From aa09311c40f6d3aa6037065e17d257ffa48b9402 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 15 Oct 2026 18:07:07 +0200 Subject: [PATCH 001/213] mm/vma: don't leak the commit charge on mmap failure Patch series "mm: various (urgent?) fixes", v1. Here are various MM fixes. The majority was reported by Josef at Anthrophic, the remainder was found while reviewing/overhauling them. Two issues/fixes reported are not included, as we are discussing (better) alternatives on the list already. I left most of the original commit messages as schloppy as they are: I don't particularly care as long as the code is in good shape and as long as they are not actively wrong. But I (and Mike) could not be stopped from overhauling at least some of them. While I don't think there is anything too critical in here, I think this is a good opportunity to see how fast we can get pre-screened/pre-reviewed fixes upstream that might be security-related. The majority are hugetlb + uffd issues, that are not actually that relevant for most installations. But all things that should just get fixed. Some uffd fixes need proper follow ups to fix some more of the root causes of leftover UFFD PTE bits (or proof that there are none remaining). This patch (of 10): When an accountable mapping replaces existing mappings (MAP_FIXED), __mmap_setup() only charges the difference between the size of the new mapping and the VM_ACCOUNT pages it replaces. It then zeroes vms->nr_accounted so that unmapping the old VMAs does not return their charge, which now backs the new mapping. If the operation fails after that point, the error path returns map->charged, i.e. only the difference. By then vms_clean_up_area() has zapped the page tables and closed the old VMAs, so vms_abort_munmap_vmas() cannot reattach them. It leaves a gap and finishes the unmap with vms_complete_munmap_vmas(), which uncharges vms->nr_accounted == 0 pages. The old VMAs are gone, the new one was never created, and the charge of the old ones is never returned, not even when the process exits. No privilege is needed to hit this, only a file whose mmap hook fails, for example a socket (sock_no_mmap() returns -ENODEV): p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); mmap(p, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_FIXED, socket_fd, 0); Every iteration adds len to Committed_AS for good. With vm.overcommit_memory=2 this lets any user make all further accountable allocations in the system fail with -ENOMEM. Once the new mapping has taken over the charge of the mappings it replaces, it owns the charge for all of its pages. Record that in map->charged, so that the error path returns all of it. A failure in __mmap_setup() itself is not affected: it happens before the charge is taken over and the old VMAs are reattached with their charge intact. Original fix generated by an LLM and sent in private by Josef. [david@kernel.org: call variable "uncharged" and make it const, adjust/simplify comments] Link: https://lore.kernel.org/20261015-urgent_fixes-v1-0-42796911043b@kernel.org Link: https://lore.kernel.org/20261015-urgent_fixes-v1-1-42796911043b@kernel.org Fixes: 4f87153e82c4 ("mm: change failure of MAP_FIXED to restoring the gap on failure") Signed-off-by: Josef Bacik Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Pedro Falcato Reviewed-by: Jose A. Perez de Azpillaga Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Muchun Song Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: --- mm/vma.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/vma.c b/mm/vma.c index f56317ee8247..5ab3fe59f06b 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -2577,14 +2577,17 @@ static int __mmap_setup(struct mmap_state *map, struct vm_area_desc *desc, /* Private writable mapping: check memory availability. */ if (accountable_mapping(map)) { - map->charged = map->pglen; - map->charged -= vms->nr_accounted; - if (map->charged) { - error = security_vm_enough_memory_mm(map->mm, map->charged); + const unsigned long uncharged = map->pglen - vms->nr_accounted; + + /* Charge only what's not already charged by mappings we replace. */ + if (uncharged) { + error = security_vm_enough_memory_mm(map->mm, uncharged); if (error) return error; } + /* Our new mapping owns the entire charge now. */ + map->charged = map->pglen; vms->nr_accounted = 0; vma_flags_set(&map->vma_flags, VMA_ACCOUNT_BIT); } From 824cc56bf43c714319992bad3558803cac6e77c0 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 15 Oct 2026 18:07:08 +0200 Subject: [PATCH 002/213] mm/hugetlb: fix use-after-free of a new anon folio For a fault in a private mapping with nothing in the page cache, hugetlb_no_page() allocates a folio, locks it and maps it. The folio has one reference, which belongs to the page table entry from then on. After dropping the page table lock the function still sets the migratable flag of the folio and unlocks it. Whoever zaps the entry in between frees the folio. The fault holds the fault mutex and the lock of the vma. MADV_DONTNEED and the first unmap of a hole punch, hugetlb_vmdelete_list(), do not take the fault mutex, and a private vma only has a lock if it owns a reservation map: not with MAP_NORESERVE, and not in the child after fork(). Then the flag is set in, and folio_unlock() is called on, a folio that is free or has a new owner. A surplus page goes back to the page allocator while it is locked: BUG: Bad page state in process init pfn:108e00 flags: 0x200000000000059(locked|uptodate|dirty|head|node=0|zone=2) page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set A page of the pool can be given to the next user, of any file, who locks it and loses the lock to the late folio_unlock(): kernel BUG at mm/filemap.c:1532! hugetlb_no_page+0x111b/0x23e0 hugetlb_fault+0x3a9/0x14a0 An unprivileged process is enough: threads that write to a MAP_PRIVATE | MAP_NORESERVE hugetlb mapping, threads that call MADV_DONTNEED on it, and one that takes the freed pages for another file. Until commit 3a47c54f09c4 ("hugetlbfs: revert use i_mmap_rwsem for more pmd sharing synchronization") the fault held i_mmap_rwsem for reading, which everybody who zaps takes for writing. The lock that commit bf4916922c60 ("hugetlbfs: extend hugetlb_vma_lock to private VMAs") gave to private vmas closed the window again, but only for those with a reservation map. Be done with the folio before the page table lock is dropped, as hugetlb_wp() is with its copy. The function already unlocks page cache folios under that lock. Original fix generated by an LLM and sent in private by Josef. [david@kernel.org: simplify+move comment] Link: https://lore.kernel.org/20261015-urgent_fixes-v1-2-42796911043b@kernel.org Fixes: 3a47c54f09c4 ("hugetlbfs: revert use i_mmap_rwsem for more pmd sharing synchronization") Signed-off-by: Josef Bacik Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: --- mm/hugetlb.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b017ae1dfd6f..16a3abb985be 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5991,8 +5991,6 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, ret = hugetlb_wp(vmf); } - spin_unlock(vmf->ptl); - /* * Only set hugetlb_migratable in newly allocated pages. Existing pages * found in the pagecache may not have hugetlb_migratable if they have @@ -6003,6 +6001,12 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, if (folio_locked) folio_unlock(folio); + + /* + * Hold the PTL until we're done with the folio: the page table owns the + * folio reference now. + */ + spin_unlock(vmf->ptl); out: hugetlb_vma_unlock_read(vma); From d8a9cd7fd2fb01211430c933932da58839b8ca65 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 15 Oct 2026 18:07:09 +0200 Subject: [PATCH 003/213] mm/hugetlb: fix use-after-free in hugetlb_mfill_atomic_pte() Let's fix another instance where we use a folio after dropping the PTL, where someone else could e.g., MADV_DONTNEED it once we drop the PTL, making us trigger a use-after-free on the folio. Link: https://lore.kernel.org/20261015-urgent_fixes-v1-3-42796911043b@kernel.org Fixes: cb6acd01e2e4 ("hugetlbfs: fix races and page leaks during migration") Fixes: 1c9e8def43a3 ("userfaultfd: hugetlbfs: add UFFDIO_COPY support for shared mappings") Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reported-by: Lorenzo Stoakes (ARM) Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: Josef Bacik Cc: --- mm/hugetlb.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 16a3abb985be..5db64fa98404 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6520,11 +6520,16 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, /* No need to invalidate - it was non-present before */ update_mmu_cache(dst_vma, dst_addr, dst_pte); - spin_unlock(ptl); if (!is_continue) folio_set_hugetlb_migratable(folio); if (vm_shared || is_continue) folio_unlock(folio); + + /* + * Hold the PTL until we're done with the folio: the page table owns the + * folio reference now. + */ + spin_unlock(ptl); ret = 0; out: return ret; From ae99c503eb362b42cd5d24d80bc6cba669acc4c2 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 15 Oct 2026 18:07:10 +0200 Subject: [PATCH 004/213] mm/hugetlb: unmap the page where the other VMA maps it If the owner of a MAP_PRIVATE hugetlb mapping fails to allocate a page for CoW, unmap_ref_private() unmaps the old page from all other private VMAs that have no reservation of their own. It finds them by the offset in the file, but then zaps each of them at the address of the faulting VMA. Nothing says that another VMA maps the offset at that address: a child can have moved its copy of the VMA with mremap(), and a second mapping of the file is wherever mmap() has put it. So __unmap_hugepage_range() walks the page tables of the other mm at an address outside of iter_vma, where i_mmap_rwsem of this file protects nothing and where anything can be mapped, and the page stays mapped where it really is. All of the following was seen on v7.3-rc6+ (g6c377d19d4a5) in KVM guests, as a user without any privilege. It takes a pool of 2 MiB pages with no free page that is not reserved, so that the allocation fails: the user can take or reserve what is free itself. - The child moves its VMA away. The page is never unmapped, the mapcount stays 2, and the owner goes round retry_avoidcopy in hugetlb_wp() until the child is gone or a page becomes free. SIGKILL to the owner does not end it. 5 of 5 runs. - The owner's mm has a second private mapping of the offset, made with MAP_NORESERVE. The owner zaps its own entry and reads zeroes where its data was. 5 of 5 runs. - The child moves its VMA away and maps a 1 GiB hugetlb page at the old address, so this one takes a pool with a 1 GiB page as well. huge_pte_offset() for the 2 MiB size steps through the PUD leaf, so the "PMD table" is the 1 GiB page itself and its "page table lock" is folio->private of that page, which holds 2. Kernels with lock debugging oops in __unmap_hugepage_range(), 7 of 7 runs. Others spin on that word for ever with preemption disabled, 5 of 5 runs: watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [u_det3:130] RIP: 0010:queued_spin_lock_slowpath+0x8a/0x290 RAX: 0000000000000002 RBX: ffffea0001000028 Call Trace: __unmap_hugepage_range+0x154/0x6d0 unmap_hugepage_range+0xc1/0x140 hugetlb_wp+0x55c/0x700 hugetlb_fault+0x5f3/0x8f0 RCU stalls follow, and killing all tasks of the user changes nothing. Four such owners, whose children share a single 1 GiB page, took down the whole guest with its four CPUs, 2 of 2 runs: one reported soft lockups on all four, the other gave no sign of life any more. i_mmap_rwsem stays write locked. With a MAP_PRIVATE mapping of a hugetlbfs file that belongs to another user, and that the attacker can only read, the next mmap() of that file by its owner sleeps uninterruptibly in __mmap_region(), 4 of 4 runs. Compute the address from the offset for each VMA. The tree walk returns only VMAs that cover pgoff, and vm_start, vm_end and vm_pgoff of a hugetlb VMA are aligned to the huge page size, as pgoff is, so the whole range lies inside iter_vma. Its page tables can not go away there, since the VMA is unlinked from the file under i_mmap_rwsem first. With the right address the page is unmapped from all VMAs but the owner's, so the retry in hugetlb_wp() finds a mapcount of 1 and reuses the page. A child that has moved its copy of the VMA now loses the page, and gets SIGBUS when it faults on a missing page of that VMA. That is what the comment above the function has always promised to those who do not own the reservation, and what already happens to a child that has left the VMA in place. The address has been wrong from the start, and the second case needs only mmap() and fork(). mremap() of hugetlb VMAs, which the other cases use, came with commit 550a7d60bd5e ("mm, hugepages: add mremap() support for hugepage backed vma") in v5.16. Original fix generated by an LLM and sent in private by Josef. [david@kernel.org: switch to vma_filebacked_address, add VM_WARN_ON_ONCE, refine comments] Link: https://lore.kernel.org/20261015-urgent_fixes-v1-4-42796911043b@kernel.org Fixes: 04f2cbe35699 ("hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed") Signed-off-by: Josef Bacik Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reported-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: --- mm/hugetlb.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5db64fa98404..4fca859bbd7e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5482,6 +5482,8 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping; pgoff_t pgoff; + VM_WARN_ON_ONCE(!folio_test_anon(folio)); + /* * vm_pgoff is in PAGE_SIZE units, hence the different calculation * from page cache lookup which is in HPAGE_SIZE units. @@ -5516,10 +5518,17 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, * could insert a zeroed page instead of the data existing * from the time of fork. This would look like data corruption */ - if (!is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) - unmap_hugepage_range(iter_vma, address, - address + huge_page_size(h), - folio, 0); + if (is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) + continue; + + /* + * The address in other VMAs can differ. Even though our old + * folio is anon, we always operate on the underlying hugetlb + * file. + */ + address = vma_filebacked_address(iter_vma, pgoff, 1); + unmap_hugepage_range(iter_vma, address, + address + huge_page_size(h), folio, 0); } i_mmap_unlock_write(mapping); } From d42032c0fe3ca007986c1229128775af0340fbec Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 15 Oct 2026 18:07:11 +0200 Subject: [PATCH 005/213] mm/hugetlb: don't clear the resv_map of a VMA whose lock is held Since commit bf4916922c60 ("hugetlbfs: extend hugetlb_vma_lock to private VMAs") the lock of a private hugetlb VMA that owns a reservation is the rw_sema of its resv_map. The hugetlb_vma_*lock*() helpers find it through vma->vm_private_data, anew for every lock and every unlock. At the end of mremap() the old VMA has to let go of the resv_map, which now belongs to the new VMA: clear_vma_resv_huge_pages() stores NULL into vm_private_data. mremap() holds mmap_lock for writing and has write locked the VMA, which keeps away everybody who comes through the mm. It does not keep away those who find the VMA through the file. For hole punch and truncation hugetlb_vmdelete_list() does hugetlb_vma_trylock_write(vma); locks resv_map->rw_sema unmap_hugepage_range(vma, ...); vm_private_data = NULL hugetlb_vma_unlock_write(vma); no resv_map: does nothing and the rw_sema stays write locked for ever: WARNING: lock held when returning to user space! mremap_punch/116 is leaving the kernel with locks still held! locks held by mremap_punch/116: 1, last CPU#1: #0: (&resv_map->rw_sema){+.+.}-{4:4}, at: hugetlb_vmdelete_list+0x46/0x160 The next fault in the new VMA sleeps in hugetlb_vma_lock_read(), uninterruptibly, and it holds its entry of hugetlb_fault_mutex_table[]. That table is global, so a fault of any other process on a page that hashes to the same entry sleeps uninterruptibly as well. All those who take the lock of a private VMA without taking the mmap/vma lock are affected: rmap walks. So hold i_mmap_rwsem around the clearing of the resv_map. No change of behaviour, other than that mremap() of a hugetlb VMA takes i_mmap_rwsem once more. Original fix generated by an LLM and sent in private by Josef. [david: avoid local variable, move and refine comment, description cleanups+simplifications] Link: https://lore.kernel.org/20261015-urgent_fixes-v1-5-42796911043b@kernel.org Fixes: bf4916922c60 ("hugetlbfs: extend hugetlb_vma_lock to private VMAs") Signed-off-by: Josef Bacik Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reported-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: --- mm/hugetlb.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4fca859bbd7e..4a7c52802cde 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7509,6 +7509,15 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) */ void fixup_hugetlb_reservations(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) - clear_vma_resv_huge_pages(vma); + if (!is_vm_hugetlb_page(vma)) + return; + + /* + * Rmap walks might still find this VMA and take the lock + * inside the reservation map for __vma_private_lock(). Don't clear + * the reservation map while that is possible. + */ + i_mmap_lock_write(vma->vm_file->f_mapping); + clear_vma_resv_huge_pages(vma); + i_mmap_unlock_write(vma->vm_file->f_mapping); } From f61e1cf15fb44f9cf89d43d8ef01f926322bb1c4 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 15 Oct 2026 18:07:12 +0200 Subject: [PATCH 006/213] mm/hugetlb: fix endless write fault on a leftover uffd bit hugetlb_fault() resolves the uffd bit on a write fault only if the VMA is registered for userfaultfd write protection, and hugetlb_wp() bails out for any entry with the uffd bit set assuming hugetlb_fault() has already resolved it. When a read-only hugetlb entry with the uffd bit set is mapped in a VMA that is not registered for uffd-wp, a write fault to it never makes progress. If the access comes from the kernel, e.g. read(2) from /dev/zero into such page, the task spins in the page fault handler, ignores SIGKILL and keeps holding the locks taken by the system call. Such entries should not exist, but an unprivileged user can create them, for example with MADV_DONTNEED of a page protected with UFFDIO_RWPROTECT or with UFFDIO_COPY in MODE_WP racing with UFFDIO_UNREGISTER. Clear the uffd bit when the VMA is not registered for uffd-wp and let the fault proceed to hugetlb_wp(), like do_wp_page() does for regular pages. Note that a FOLL_FORCE write to a page protected with UFFDIO_RWPROTECT in a private VMA that was made PROT_NONE now drops the RWP protection of that page. Original fix generated by an LLM and sent in private by Josef. [rppt@kernel.org: massage commit message] [david@kernel.org: convert comment into a TODO] Link: https://lore.kernel.org/20261015-urgent_fixes-v1-6-42796911043b@kernel.org Fixes: 60d5b473d61b ("mm/hugetlb: fix uffd wr-protection for CoW optimization path") Signed-off-by: Josef Bacik Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reported-by: Chris Mason Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Muchun Song Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: --- mm/hugetlb.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4a7c52802cde..d3adafbea160 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6226,15 +6226,19 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, goto out_ptl; /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd(huge_ptep_get(mm, vmf.address, vmf.pte)) && + if (huge_pte_uffd(huge_ptep_get(mm, vmf.address, vmf.pte)) && (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { - if (!userfaultfd_wp_async(vma)) { + if (userfaultfd_wp(vma) && !userfaultfd_wp_async(vma)) { spin_unlock(vmf.ptl); hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); return handle_userfault(&vmf, VM_UFFD_WP); } + /* + * TODO: fix all cases where uffd PTE bits remain in VMAs that + * are no longer registered with uffd. + */ vmf.orig_pte = huge_pte_clear_uffd(vmf.orig_pte); set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte, huge_page_size(hstate_vma(vma))); From 811777d82038d4fd3f5d3bf986f682df463b6f21 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 15 Oct 2026 18:07:13 +0200 Subject: [PATCH 007/213] mm/hugetlb: tolerate uffd bits set without uffd for PROT_NONE PTEs The whole uffd handling when a VMA is unregistered from UFFD is shaky: there are still some scenarios where we might have stale uffd PTE bits in VMAs no longer registered for UFFD, which confuses other code. Let's just tolerate that for now and simply clear the PTE bit. Add a TODO to really clean up any uffd leftovers when unregistering a VMA from userfaultfd. The handling is now more similar to what we have in handle_pte_fault(). Link: https://lore.kernel.org/20261015-urgent_fixes-v1-7-42796911043b@kernel.org Fixes: 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP") Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Acked-by: Kiryl Shutsemau (Meta) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Muchun Song Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Mike Rapoport Cc: Josef Bacik Cc: --- mm/hugetlb.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d3adafbea160..a4fede4be663 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6169,16 +6169,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * NUMA hinting), so let normal hugetlb fault handling proceed. */ if (pte_protnone(vmf.orig_pte) && vma_is_accessible(vma) && - userfaultfd_rwp(vma) && huge_pte_uffd(vmf.orig_pte)) { + huge_pte_uffd(vmf.orig_pte)) { spinlock_t *ptl; pte_t pte; /* Sync: drop hugetlb locks before blocking in handle_userfault() */ - if (!userfaultfd_rwp_async(vma)) + if (userfaultfd_rwp(vma) && !userfaultfd_rwp_async(vma)) return hugetlb_handle_userfault(&vmf, mapping, VM_UFFD_RWP); ptl = huge_pte_lock(h, mm, vmf.pte); pte = huge_ptep_get(mm, vmf.address, vmf.pte); + /* + * TODO: fix all cases where uffd PTE bits remain in VMAs that + * are no longer registered with uffd. + */ if (pte_protnone(pte) && huge_pte_uffd(pte)) { unsigned int shift = huge_page_shift(h); From c37e9d9411a12240a60ca3796d3cbb496e6258a2 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 15 Oct 2026 18:07:14 +0200 Subject: [PATCH 008/213] mm/hugetlb: tolerate UFFD_WP markers outside uffd-wp VMAs UFFD PTE bit handling is a bit like swiss cheese, in particular with recently added RWP mode. Having uffd-wp markers outside uffd-wp VMAs already confused this code in the past and resulted in uffd-pte bits confusing other code later. Let's play safe and just ignore these markers in hugetlb_no_page(), not silently propagating the uffd-wp PTE bit. Link: https://lore.kernel.org/20261015-urgent_fixes-v1-8-42796911043b@kernel.org Fixes: 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP") Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Muchun Song Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: Josef Bacik Cc: --- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a4fede4be663..c6c45de147d2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5981,8 +5981,11 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, /* * If this pte was previously wr-protected, keep it wr-protected even * if populated. + * + * TODO: fix all cases where uffd-wp markers remain in VMAs that are no + * longer registered with uffd. */ - if (unlikely(pte_is_uffd_wp_marker(vmf->orig_pte))) + if (unlikely(userfaultfd_wp(vma) && pte_is_uffd_wp_marker(vmf->orig_pte))) new_pte = huge_pte_mkuffd(new_pte); set_huge_pte_at(mm, vmf->address, vmf->pte, new_pte, huge_page_size(h)); From f6605fc6b14bd5fc0fea90fc8226573adcec352b Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 15 Oct 2026 18:07:15 +0200 Subject: [PATCH 009/213] mm/hugetlb: don't leave a uffd-wp marker when zapping an RWP entry An entry that is protected with UFFDIO_RWPROTECT is PAGE_NONE with the uffd bit set, the bit that userfaultfd write protection uses too. When __unmap_hugepage_range() zaps an entry with that bit it leaves a PTE_MARKER_UFFD_WP behind, whatever the VMA is registered for. In a VMA registered with UFFDIO_REGISTER_MODE_RWP that marker means nothing. The next fault has hugetlb_no_page() turn it into a present, read-only entry with the uffd bit. That is not an RWP entry, since it is not PAGE_NONE, and the VMA has no VM_UFFD_WP (the two modes can not be registered together). An earlier fix solved some side effects, now just stop the zap from wrongly installing a marker. Leave a marker only if the VMA is registered for write protection, both for present and for non-present entries. The zap of ordinary page tables has the same test, in cond_install_uffd_wp_ptes(), and hugetlb_change_protection() says that "uffd_rwp does not install markers". After a zap the range of an RWP VMA is simply empty, as it is for anonymous memory and shmem. Nothing changes for VMAs with VM_UFFD_WP. In a VMA that is not registered at all a leftover uffd bit or marker is now dropped by the zap and no longer kept alive. Original fix generated by an LLM and sent in private by Josef. [david@kernel.org: a bit of description overhaul+simplification] Link: https://lore.kernel.org/20261015-urgent_fixes-v1-9-42796911043b@kernel.org Fixes: 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP") Signed-off-by: Josef Bacik Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reported-by: Chris Mason Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Acked-by: Kiryl Shutsemau (Meta) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Muchun Song Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Mike Rapoport Cc: --- mm/hugetlb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c6c45de147d2..a949b446cbea 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5309,7 +5309,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, * drop the uffd-wp bit in this zap, then replace the * pte with a marker. */ - if (pte_swp_uffd_any(pte) && + if (userfaultfd_wp(vma) && pte_swp_uffd_any(pte) && !(zap_flags & ZAP_FLAG_DROP_MARKER)) set_huge_pte_at(mm, address, ptep, make_pte_marker(PTE_MARKER_UFFD_WP), @@ -5345,7 +5345,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, if (huge_pte_dirty(pte)) folio_mark_dirty(folio); /* Leave a uffd-wp pte marker if needed */ - if (huge_pte_uffd(pte) && + if (userfaultfd_wp(vma) && huge_pte_uffd(pte) && !(zap_flags & ZAP_FLAG_DROP_MARKER)) set_huge_pte_at(mm, address, ptep, make_pte_marker(PTE_MARKER_UFFD_WP), From cc078ca0b5b79b4a9e514b7f1b489f2afc83b195 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 15 Oct 2026 18:07:16 +0200 Subject: [PATCH 010/213] mm/mremap: fix calling fixup_hugetlb_reservations() on the wrong VMA In case we fail to move the page tables, for example, because we fail to allocate a page table, we actually keep the old VMA. Calling fixup_hugetlb_reservations() on the old VMA is wrong. Fix it by just calling it on the VMA that will be removed. The user-visible impact is that the VMA loses its reservations, meaning that the existing reservations get stranded (leaked) and new hugetlb folios that are getting allocated in the VMA cannot make use of the reservations, failing these allocations. Link: https://lore.kernel.org/20261015-urgent_fixes-v1-10-42796911043b@kernel.org Fixes: b714ccb02a76 ("mm/mremap: complete refactor of move_vma()") Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Reviewed-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Oscar Salvador Cc: Miaohe Lin Cc: Naoya Horiguchi Cc: Adam Litke Cc: Mel Gorman Cc: Rik van Riel Cc: Peter Xu Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: Josef Bacik Cc: --- mm/mremap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 444f37d7cd28..ace0c6a343b1 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -1345,7 +1345,7 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, mremap_userfaultfd_prep(new_vma, vrm->uf); } - fixup_hugetlb_reservations(vma); + fixup_hugetlb_reservations(vrm->vma); *new_vma_ptr = new_vma; return err; From 92635a3ffe8bb48304f50625dd591a27d7505698 Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Wed, 23 Sep 2026 10:05:15 +0800 Subject: [PATCH 011/213] mm/hugetlb: account migration target folio in per-node NR_HUGETLB vmstat Patch series "mm: fix hugetlb NR_HUGETLB accounting on folio migration", v2. The hugeTLB counters added by 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg") are maintained in two per-node places: - the per-node vmstat counter NR_HUGETLB, exposed as nr_hugetlb in /proc/vmstat; - the per-node memcg lruvec stat, exposed via memory.numa_stat. Both are accounted against the folio's node, and both drift when a hugetlb folio is migrated, though in different ways. A migration target folio is allocated by alloc_hugetlb_folio_nodemask() and inherits the old folio's state without ever being accounted, while the old folio is freed right after and its free is accounted. That alone loses vmstat accounting: the target node has no matching increment for the decrement on the old node, so /proc/vmstat's nr_hugetlb shrinks by nr_pages per migration. Patch 1 accounts the folio where it is obtained, so the increment pairs with the free in free_huge_folio() on the successful as well as the failed migration path. The memfd page cache preallocation helper has the same asymmetry and is fixed in the same patch. The per-node lruvec stat breaks differently. mem_cgroup_migrate() moves the charge to the new folio and drops the old folio's memcg data, so the old folio's free right after migration skips the memcg per-node decrement; the count stays attributed to the old node for the rest of the charge's life, and the target folio never gets an increment on its new node. Patch 2 moves that per-node accounting along with the charge, in move_hugetlb_state(). This patch (of 2): The NR_HUGETLB vmstat counter is maintained per folio's node: incremented when a huge page is handed to a user via hugetlb_alloc_folio() and decremented when it is returned to the pool via free_huge_folio(). A folio obtained by alloc_hugetlb_folio_nodemask() never goes through hugetlb_alloc_folio(), so it is never accounted, while its free always is. For a migration target this means the target node gets no matching increment for the decrement on the old node, so the global nr_hugetlb in /proc/vmstat drops by nr_pages for each migration. The same asymmetry affects the failed migration path, which frees the target again right away, and the temporary folio hugetlb_mfill_atomic_pte() takes from the same helper. alloc_hugetlb_folio_reserve(), used to preallocate the memfd page cache folios, has the same asymmetry: the folio is handed to a user without being accounted, while its free is accounted through free_huge_folio(). Account the folio where it is obtained, in alloc_hugetlb_folio_nodemask() and alloc_hugetlb_folio_reserve(), so that the increment pairs with the decrement in free_huge_folio(): a successful migration hands the folio to a user, a failed one frees it again. Link: https://lore.kernel.org/20260923-for-hugetlb_state3-v2-0-e8a36245bfab@kylinos.cn Link: https://lore.kernel.org/20260923-for-hugetlb_state3-v2-1-e8a36245bfab@kylinos.cn Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg") Signed-off-by: Hongfu Li Signed-off-by: Andrew Morton Tested-by: Joshua Hahn Reviewed-by: Joshua Hahn Acked-by: Muchun Song Acked-by: Oscar Salvador Cc: David Hildenbrand Cc: Shakeel Butt Cc: Michal Hocko Cc: Johannes Weiner Cc: Nhat Pham Cc: Michal Hocko Cc: Roman Gushchin Cc: Chris Down Cc: --- mm/hugetlb.c | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a949b446cbea..4a07eada7779 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2206,6 +2206,11 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, } spin_unlock_irq(&hugetlb_lock); + + if (folio) + lruvec_stat_mod_folio(folio, NR_HUGETLB, + folio_nr_pages(folio)); + return folio; } @@ -2213,24 +2218,30 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask, bool allow_alloc_fallback) { - spin_lock_irq(&hugetlb_lock); - if (available_huge_pages(h)) { - struct folio *folio; + struct folio *folio = NULL; + spin_lock_irq(&hugetlb_lock); + if (available_huge_pages(h)) folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, preferred_nid, nmask); - if (folio) { - spin_unlock_irq(&hugetlb_lock); - return folio; - } - } spin_unlock_irq(&hugetlb_lock); - /* We cannot fallback to other nodes, as we could break the per-node pool. */ - if (!allow_alloc_fallback) - gfp_mask |= __GFP_THISNODE; + if (!folio) { + /* + * We cannot fallback to other nodes, as we could break the + * per-node pool. + */ + if (!allow_alloc_fallback) + gfp_mask |= __GFP_THISNODE; - return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask); + folio = alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, + nmask); + } + + if (folio) + lruvec_stat_mod_folio(folio, NR_HUGETLB, folio_nr_pages(folio)); + + return folio; } static nodemask_t *policy_mbind_nodemask(gfp_t gfp) From 02e0c8cc2815f0082b5b5aabafe3b46688082adc Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Wed, 23 Sep 2026 10:05:16 +0800 Subject: [PATCH 012/213] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio memory.numa_stat exposes per-node hugetlb counters from per-node lruvec stats. These stats are accounted against folio_nid(): incremented on the folio's node when handed to a user, decremented when the folio is returned to the pool. During hugetlb folio migration, mem_cgroup_migrate() moves the charge to the new folio and drops the memcg data of the old one, so the free of the old folio right after the migration skips the memcg per-node lruvec decrement. The hugetlb count stays attributed to the old node for the rest of the life of the charge, while the target folio gets no increment on the new node; its later free decrements a counter that was never incremented. Migrate the per-node lruvec accounting alongside migration. Global memcg totals remain balanced because they track resource consumption, not node placement. Link: https://lore.kernel.org/20260923-for-hugetlb_state3-v2-2-e8a36245bfab@kylinos.cn Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg") Signed-off-by: Hongfu Li Signed-off-by: Andrew Morton Tested-by: Joshua Hahn Reviewed-by: Joshua Hahn Reviewed-by: Oscar Salvador Acked-by: Muchun Song Cc: David Hildenbrand Cc: Shakeel Butt Cc: Michal Hocko Cc: Johannes Weiner Cc: Nhat Pham Cc: Michal Hocko Cc: Roman Gushchin Cc: Chris Down Cc: --- include/linux/memcontrol.h | 8 ++++++++ mm/hugetlb.c | 25 +++++++++++++++++++++++++ mm/memcontrol.c | 5 ++--- 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 64c183be8cbf..cd223f60b3a2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -983,6 +983,9 @@ unsigned long lruvec_page_state_monotonic(const struct lruvec *lruvec, unsigned long lruvec_page_state_local(const struct lruvec *lruvec, enum node_stat_item idx); +void mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val); + void mem_cgroup_flush_stats(struct mem_cgroup *memcg); void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg); @@ -1451,6 +1454,11 @@ static inline unsigned long lruvec_page_state_local(const struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } +static inline void mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ +} + static inline void mem_cgroup_flush_stats(struct mem_cgroup *memcg) { } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4a07eada7779..68f782f002d3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -7396,12 +7397,36 @@ void folio_putback_hugetlb(struct folio *folio) folio_put(folio); } +static void move_hugetlb_lruvec_stat(struct folio *old_folio, + struct folio *new_folio) +{ + struct mem_cgroup *memcg; + long nr_pages = folio_nr_pages(old_folio); + int old_nid = folio_nid(old_folio); + int new_nid = folio_nid(new_folio); + + if (old_nid == new_nid) + return; + + guard(rcu)(); + + memcg = folio_memcg(new_folio); + if (!memcg) + return; + + mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)), + NR_HUGETLB, -nr_pages); + mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)), + NR_HUGETLB, nr_pages); +} + void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, enum migrate_reason reason) { struct hstate *h = folio_hstate(old_folio); hugetlb_cgroup_migrate(old_folio, new_folio); + move_hugetlb_lruvec_stat(old_folio, new_folio); folio_set_owner_migrate_reason(new_folio, reason); /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4d00748c8a5b..f2ad4de8d805 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1015,9 +1015,8 @@ static void __mod_memcg_lruvec_state(struct mem_cgroup_per_node *pn, put_cpu(); } -static void mod_memcg_lruvec_state(struct lruvec *lruvec, - enum node_stat_item idx, - int val) +void mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) { struct pglist_data *pgdat = lruvec_pgdat(lruvec); struct mem_cgroup_per_node *pn; From bd212a7dbdb151e9bacb02857a222e25b28c229d Mon Sep 17 00:00:00 2001 From: Yeoreum Yun Date: Tue, 29 Sep 2026 11:33:18 +0100 Subject: [PATCH 013/213] kselftest: mm: return fail when child test result is fail in khugepaged Patch series "kselftest: mm: fix intermittent failure khugepaged test", v4. There are intermittent failures in collapse_max_ptes_swap() and collapse_max_ptes_shared() when using the khugepaged_context: # Run test: collapse_max_ptes_shared (khugepaged:anon) # Allocate huge page... OK # Share huge page over fork()... OK # Trigger CoW on page 1023 of 2048... OK # Maybe collapse with max_ptes_shared exceeded.... OK # Trigger CoW on page 1024 of 2048... Fail Bail out! Unexpected huge page # Planned tests != run tests (26 != 23) # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0 # Run test: collapse_max_ptes_swap (khugepaged:anon) # Swapout 257 of 2048 pages... OK # Maybe collapse with max_ptes_swap exceeded.... OK # Swapout 256 of 2048 pages... OK Bail out! Unexpected huge page # Planned tests != run tests (26 != 17) # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0 This happens because khugepaged may collapse the pages before wait_for_scan() is called, causing a sanity check that expects uncollapsed pages to fail. For example, in collapse_max_ptes_swap(), after faulting the pages back in and paging out up to max_ptes_swap pages, khugepaged may collapse them again before c->collapse() is called. To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been collapsed by wait_for_scan() for anon. This prevents khugepaged from collapsing it again before c->collapse() is called. Also, fix false-positive results when a child process fails in tests such as collapse_fork*() or collapse_max_ptes_shared(): # ------------------------- # running ./khugepaged -s 2 # ------------------------- # # Run test: collapse_max_ptes_shared (khugepaged:anon) # Allocate huge page... OK # Share huge page over fork()... OK # Trigger CoW on page 1023 of 2048... OK # Maybe collapse with max_ptes_shared exceeded.... OK # Trigger CoW on page 1024 of 2048... Fail Bail out! Unexpected huge page # Planned tests != run tests (26 != 23) # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0 // child failed. # Check if parent still has huge page... OK // parent hpage success ok 24 collapse_max_ptes_shared // considered as success ... # Totals: pass:26 fail:0 xfail:0 xpass:0 skip:0 error:0 This failure was observed on NVIDIA Spark with 16KB page. This patch (of 2): Although the child process in collapse_fork*() or collapse_max_ptes_shared() reports `KSFT_FAIL`, the result is ignored because the test only checks whether the parent's page was collapsed into a huge page. As a result, the test is considered successful whenever the parent's page is a huge page, even if the child test fails, as shown below: # # Run test: collapse_max_ptes_shared (khugepaged:anon) # Allocate huge page... OK # Share huge page over fork()... OK # Trigger CoW on page 1023 of 2048... OK # Maybe collapse with max_ptes_shared exceeded.... OK # Trigger CoW on page 1024 of 2048... Fail Bail out! Unexpected huge page # Planned tests != run tests (26 != 23) # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0 // child failed. # Check if parent still has huge page... OK // parent hpage success ok 24 collapse_max_ptes_shared // considered as success ... # Totals: pass:26 fail:0 xfail:0 xpass:0 skip:0 error:0 To address this, propagate the child's failure and skip the subsequent check in the parent. Link: https://lore.kernel.org/20260929-fix_khugepagd_fail-v4-0-2169c18f2576@arm.com Link: https://lore.kernel.org/20260929-fix_khugepagd_fail-v4-1-2169c18f2576@arm.com Signed-off-by: Yeoreum Yun Signed-off-by: Andrew Morton Reviewed-by: Baolin Wang Reviewed-by: Gregory Price (Meta) Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Acked-by: Zi Yan Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan --- tools/testing/selftests/mm/khugepaged.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index a2ac3b3ca5de..2aa7c9197158 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -1053,6 +1053,8 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) wait(&wstatus); exit_status = WEXITSTATUS(wstatus); + if (exit_status == KSFT_FAIL) + goto out; ksft_print_msg("Check if parent still has small page..."); if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) @@ -1060,6 +1062,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) else fail("Fail"); validate_memory(p, 0, page_size); +out: ops->cleanup_area(p, hpage_pmd_size); ksft_test_result_report(exit_status, "%s\n", __func__); } @@ -1100,6 +1103,8 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o wait(&wstatus); exit_status = WEXITSTATUS(wstatus); + if (exit_status == KSFT_FAIL) + goto out; ksft_print_msg("Check if parent still has huge page..."); if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) @@ -1107,6 +1112,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o else fail("Fail"); validate_memory(p, 0, hpage_pmd_size); +out: ops->cleanup_area(p, hpage_pmd_size); ksft_test_result_report(exit_status, "%s\n", __func__); } @@ -1158,6 +1164,8 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops wait(&wstatus); exit_status = WEXITSTATUS(wstatus); + if (exit_status == KSFT_FAIL) + goto out; ksft_print_msg("Check if parent still has huge page..."); if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) @@ -1165,6 +1173,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops else fail("Fail"); validate_memory(p, 0, hpage_pmd_size); +out: ops->cleanup_area(p, hpage_pmd_size); ksft_test_result_report(exit_status, "%s\n", __func__); } From f6c9f1aab305e51ef5fed624a4cfd5a9dfd9f72a Mon Sep 17 00:00:00 2001 From: Yeoreum Yun Date: Tue, 29 Sep 2026 11:33:19 +0100 Subject: [PATCH 014/213] kselftest: mm: fix intermittent failure khugepaged test There are intermittent failures in collapse_max_ptes_swap() and collapse_max_ptes_shared() when using the khugepaged_context: // while running ./khugepaged -s 2 # Run test: collapse_max_ptes_shared (khugepaged:anon) # Allocate huge page... OK # Share huge page over fork()... OK # Trigger CoW on page 1023 of 2048... OK # Maybe collapse with max_ptes_shared exceeded.... OK # Trigger CoW on page 1024 of 2048... Fail Bail out! Unexpected huge page # Planned tests != run tests (26 != 23) # Totals: pass:23 fail:0 xfail:0 xpass:0 skip:0 error:0 # Run test: collapse_max_ptes_swap (khugepaged:anon) # Swapout 257 of 2048 pages... OK # Maybe collapse with max_ptes_swap exceeded.... OK # Swapout 256 of 2048 pages... OK Bail out! Unexpected huge page # Planned tests != run tests (26 != 17) # Totals: pass:17 fail:0 xfail:0 xpass:0 skip:0 error:0 This happens because khugepaged may collapse the pages before wait_for_scan() is called, causing a sanity check that expects uncollapsed pages to fail. For example, in collapse_max_ptes_swap(), after faulting the pages back in and paging out up to max_ptes_swap pages, khugepaged may collapse them again before c->collapse() is called. To prevent this, mark the VMA with MADV_NOHUGEPAGE after it has been collapsed by wait_for_scan() for anon. This prevents khugepaged from collapsing it again before c->collapse() is called. This failure was observed on NVIDIA Spark with 16KB page. Link: https://lore.kernel.org/20260929-fix_khugepagd_fail-v4-2-2169c18f2576@arm.com Signed-off-by: Yeoreum Yun Signed-off-by: Andrew Morton Reviewed-by: Gregory Price (Meta) Reviewed-by: Baolin Wang Tested-by: Baolin Wang Acked-by: Lorenzo Stoakes (ARM) Cc: Zi Yan Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: David Hildenbrand Cc: Shuah Khan --- tools/testing/selftests/mm/khugepaged.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index 2aa7c9197158..4e888b7bf310 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -618,6 +618,16 @@ static bool wait_for_scan(const char *msg, char *p, size_t len, usleep(TICK); } + /* + * The file and shmem tests rely on refaults to install PMD mappings + * after collapse. MADV_NOHUGEPAGE would prevent those mappings. + * + * Apply MADV_NOHUGEPAGE only to anonymous VMAs to prevent khugepaged + * from unexpectedly collapsing pages during the test. + */ + if (is_anon(ops)) + madvise(p, len, MADV_NOHUGEPAGE); + return timeout == -1; } From 127293b445efb647119bf1a20e6dc943a92b0f12 Mon Sep 17 00:00:00 2001 From: Bingfang Guo Date: Mon, 21 Sep 2026 14:16:24 +0800 Subject: [PATCH 015/213] memcg: keep swap charging under RCU protection Patch series "memcg: move memcgid refcount to objcg to unpin dying memcgs", v2. Although the dying memcg problem caused by LRU pages is fixed, I can still see many dying memcgs on some workloads that use shmem and those pages are swapped out. For example, programs populating logs to tmpfs or containers sharing data using shmem. This series binds the memcgid refcount to objcgs so dying memcgs can be freed normally in this case. The memcg private ID identifies memcgs for objects that can outlive the cgroup itself: swap entries and workingset shadows. Today the ID's refcount is embedded in the css, and every outstanding ID reference (mostly swap entries) pins the css, keeping the entire memcg alive. This causes a problem: A swapped-out page holds a memcgid reference that pins the css, so the memcg cannot be freed until the page is swapped back in and charged back to its online parent. The work done by Muchun Song and Qi Zheng already charges folios to the objcg, which is reparented to its parent when the memcg offlines. This series applies similar idea to the memcg private ID: the ID's refcount moves from the css into the objcg, and the memcgid xarray holds a reference to an objcg instead of pinning the css. When the memcg offlines, the objcg is reparented and any remaining memcgid references resolve to the ancestor, so swapped-out pages no longer pin the dying memcg and get the online parent naturally on swapin. Unbinding the ID from the memcg has three consequences the series has to deal with: 1. The ID stops pinning the memcg, so the paths that relied on the ID reference to keep the memcg alive have to hold the RCU read lock instead. (Patch 1) 2. Charge and uncharge no longer necessarily happen on the same memcg: swapout charges the folio's memcg, while the slot free resolves the nearest live ancestor. The counters are hierarchical, and the MEMCG_SWAP stat is either reparented at offline (v1) or not visible (v2), so nothing leaks. But "does this entry carry a counter charge at all" can no longer be answered from the resolved memcg: root is skipped only because root's swap is not accounted, and a non-root ID whose memcg was reparented into root still carries a charge that must be released. Patch 2 adds mem_cgroup_private_id_is_root() and makes all three swap paths decide on the ID's root status. 3. An ID can now outlive the memcg it was allocated to, so the memcg resolved from an ID is not necessarily the memcg the ID was handed out for. Callers that need exactly that memcg (list_lru, the workingset and MGLRU shadow tests) now get NULL and skip the entry, while the swap paths, which only need something to account to, get the nearest live ancestor. (Patch 4) The series is four patches. The first three are preparation that keeps today's semantics while the ID is still bound to the css. Only the last patch changes behavior. This patch (of 4): This is a preparatory work for unbinding memcgid from memcg. No functional change. The swap charging path currently drops its RCU read lock after acquiring a private ID reference. This is safe because the ID reference pins the memcg's CSS. After ID references are moved to pin objcgs, that lifetime guarantee will no longer hold. Keep the RCU read lock held while accessing the memcg for counter charging, statistics and failure handling. (This matches what __memcg1_swapout() already does). Save the private ID for swap memcg recording before dropping the RCU read lock. So the swap cluster locking remains outside the RCU read-side critical section. Link: https://lore.kernel.org/20260921-bingfangguo-memcgid-rework-v2-0-6c0637dc0edb@tencent.com Link: https://lore.kernel.org/20260921-bingfangguo-memcgid-rework-v2-1-6c0637dc0edb@tencent.com Signed-off-by: Bingfang Guo Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin Cc: Shakeel Butt Cc: Dave Chinner Cc: Qi Zheng Cc: Kairui Song Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Bingfang Guo --- mm/memcontrol.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f2ad4de8d805..059eaca173bb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5954,6 +5954,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) struct page_counter *counter; struct mem_cgroup *memcg; struct obj_cgroup *objcg; + unsigned short private_id; if (do_memsw_account()) return 0; @@ -5963,30 +5964,29 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) if (!objcg) return 0; - rcu_read_lock(); - memcg = obj_cgroup_memcg(objcg); - if (!folio_test_swapcache(folio)) { - memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - rcu_read_unlock(); - return 0; - } + scoped_guard(rcu) { + memcg = obj_cgroup_memcg(objcg); + if (!folio_test_swapcache(folio)) { + memcg_memory_event(memcg, MEMCG_SWAP_FAIL); + return 0; + } - memcg = mem_cgroup_private_id_get_online(memcg, nr_pages); - /* memcg is pined by memcg ID. */ - rcu_read_unlock(); + memcg = mem_cgroup_private_id_get_online(memcg, nr_pages); + /* memcg is pined by memcg ID. */ + private_id = mem_cgroup_private_id(memcg); - if (!mem_cgroup_is_root(memcg) && - !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { - memcg_memory_event(memcg, MEMCG_SWAP_MAX); - memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - mem_cgroup_private_id_put(memcg, nr_pages); - return -ENOMEM; + if (!mem_cgroup_is_root(memcg) && + !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { + memcg_memory_event(memcg, MEMCG_SWAP_MAX); + memcg_memory_event(memcg, MEMCG_SWAP_FAIL); + mem_cgroup_private_id_put(memcg, nr_pages); + return -ENOMEM; + } + mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); } - mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); ci = swap_cluster_get_and_lock(folio); - __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages, - mem_cgroup_private_id(memcg)); + __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages, private_id); swap_cluster_unlock(ci); return 0; From ea74172be27d19d72a72647d7b1ef131662d1ded Mon Sep 17 00:00:00 2001 From: Bingfang Guo Date: Mon, 21 Sep 2026 14:16:25 +0800 Subject: [PATCH 016/213] memcg: base swap charge accounting on memcgid root status A private ID currently pins its original memcg, so testing whether the ID belongs to root is equivalent to testing whether the resolved memcg is root. That equivalence will no longer hold when IDs refer to objcgs. A non-root ID may resolve to root after reparenting, but its swap entries still carry counter charges inherited by root. Skipping their uncharge based on the resolved memcg would leave those charges behind. Use the private ID's root status to decide whether a swap entry carries a counter charge. A root-ID entry carries none, while a non-root-ID entry must release its charge even if its current accounting memcg has become root. This patch adds a new helper to check if the memcgid equals to that of the root memcg. For swap uncharging and v2 swap charging, simply decide whether to charge/uncharge memsw or swap counter based on the swap memcgid is root or not. For v1 swapout, don't recharge the memsw counter, just cancel the charge if the swap memcg ID refers to the root memcg. This makes memory and swap charging have similar semantics: one relys on the objcg's root status, and the other relys on the memcgid's root status. Link: https://lore.kernel.org/20260921-bingfangguo-memcgid-rework-v2-2-6c0637dc0edb@tencent.com Signed-off-by: Bingfang Guo Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin Cc: Shakeel Butt Cc: Dave Chinner Cc: Qi Zheng Cc: Kairui Song Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Bingfang Guo --- mm/memcontrol-v1.c | 16 ++++++++-------- mm/memcontrol-v1.h | 5 +++++ mm/memcontrol.c | 4 ++-- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index bf2c7d53b01b..ed015fdd9512 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -271,6 +271,7 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci) struct mem_cgroup *memcg, *swap_memcg; struct obj_cgroup *objcg; unsigned int nr_entries; + unsigned short private_id; VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio); VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); @@ -293,25 +294,24 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci) /* * In case the memcg owning these pages has been offlined and doesn't * have an ID allocated to it anymore, charge the closest online - * ancestor for the swap instead and transfer the memory+swap charge. + * ancestor for the swap instead and cancel the memory+swap charge + * if the ID refers to the root memcg. */ nr_entries = folio_nr_pages(folio); swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries); + private_id = mem_cgroup_private_id(swap_memcg); mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries); - __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries, - mem_cgroup_private_id(swap_memcg)); + __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries, private_id); folio_unqueue_deferred_split(folio); folio->memcg_data = 0; - if (!obj_cgroup_is_root(objcg)) + if (!obj_cgroup_is_root(objcg)) { page_counter_uncharge(&memcg->memory, nr_entries); - if (memcg != swap_memcg) { - if (!mem_cgroup_is_root(swap_memcg)) - page_counter_charge(&swap_memcg->memsw, nr_entries); - page_counter_uncharge(&memcg->memsw, nr_entries); + if (mem_cgroup_private_id_is_root(private_id)) + page_counter_uncharge(&memcg->memsw, nr_entries); } /* diff --git a/mm/memcontrol-v1.h b/mm/memcontrol-v1.h index 0952b2a783e5..7286456125db 100644 --- a/mm/memcontrol-v1.h +++ b/mm/memcontrol-v1.h @@ -22,6 +22,11 @@ void drain_all_stock(struct mem_cgroup *root_memcg); int memory_stat_show(struct seq_file *m, void *v); +static inline bool mem_cgroup_private_id_is_root(unsigned short id) +{ + return id == mem_cgroup_private_id(root_mem_cgroup); +} + struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, unsigned int n); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 059eaca173bb..c43b09178512 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5975,7 +5975,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) /* memcg is pined by memcg ID. */ private_id = mem_cgroup_private_id(memcg); - if (!mem_cgroup_is_root(memcg) && + if (!mem_cgroup_private_id_is_root(private_id) && !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); @@ -6004,7 +6004,7 @@ void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages) rcu_read_lock(); memcg = mem_cgroup_from_private_id(id); if (memcg) { - if (!mem_cgroup_is_root(memcg)) { + if (!mem_cgroup_private_id_is_root(id)) { if (do_memsw_account()) page_counter_uncharge(&memcg->memsw, nr_pages); else From 0262a5ee845c9c6b0a68ee587fffd6df01da19b5 Mon Sep 17 00:00:00 2001 From: Bingfang Guo Date: Mon, 21 Sep 2026 14:16:26 +0800 Subject: [PATCH 017/213] memcg: manipulate memcg private ID references by ID This is a preparatory work for moving memcgid from memcg to objcg. Swap entries retain a private ID rather than a memcg pointer. Once private ID references are moved to objcgs, the ID can also outlive the memcg to which it was originally assigned. So it's better to make the get and put functions accept the ID itself instead of the memcg. Rename mem_cgroup_private_id_get_online() to mem_cgroup_private_id_get(), and make it return the ID only. If the memcg is already dying, the dying memcg will still be used for charging and stats accounting in v2 swap charging path. But they are hierarchical and will be reparented after offlining so it doesn't matter. Make mem_cgroup_private_id_put() take the ID and resolve the reference holder internally. Convert swap uncharge and charge rollback to release the reference using that ID. This introduces an extra xarray lookup for now, which will be removed in the final patch. Separate the online-state reference release into mem_cgroup_private_id_kill(). The offline path already has the memcg pointer and can call the underlying put helper directly. Link: https://lore.kernel.org/20260921-bingfangguo-memcgid-rework-v2-3-6c0637dc0edb@tencent.com Signed-off-by: Bingfang Guo Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin Cc: Shakeel Butt Cc: Dave Chinner Cc: Qi Zheng Cc: Kairui Song Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Bingfang Guo --- mm/memcontrol-v1.c | 7 +++---- mm/memcontrol-v1.h | 3 +-- mm/memcontrol.c | 32 +++++++++++++++++++++++--------- 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index ed015fdd9512..b7f286888507 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -268,7 +268,7 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg) */ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci) { - struct mem_cgroup *memcg, *swap_memcg; + struct mem_cgroup *memcg; struct obj_cgroup *objcg; unsigned int nr_entries; unsigned short private_id; @@ -298,9 +298,8 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci) * if the ID refers to the root memcg. */ nr_entries = folio_nr_pages(folio); - swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries); - private_id = mem_cgroup_private_id(swap_memcg); - mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries); + private_id = mem_cgroup_private_id_get(memcg, nr_entries); + mod_memcg_state(memcg, MEMCG_SWAP, nr_entries); __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries, private_id); diff --git a/mm/memcontrol-v1.h b/mm/memcontrol-v1.h index 7286456125db..820139657184 100644 --- a/mm/memcontrol-v1.h +++ b/mm/memcontrol-v1.h @@ -27,8 +27,7 @@ static inline bool mem_cgroup_private_id_is_root(unsigned short id) return id == mem_cgroup_private_id(root_mem_cgroup); } -struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, - unsigned int n); +unsigned short mem_cgroup_private_id_get(struct mem_cgroup *memcg, unsigned int n); void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent, int idx); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c43b09178512..53c09a0e500e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4082,7 +4082,7 @@ static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg) } } -static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg, unsigned int n) +static void __mem_cgroup_private_id_put(struct mem_cgroup *memcg, unsigned int n) { if (refcount_sub_and_test(n, &memcg->private_id_ref)) { mem_cgroup_private_id_remove(memcg); @@ -4092,7 +4092,22 @@ static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg, unsigned } } -struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, unsigned int n) +static inline void mem_cgroup_private_id_put(unsigned short id, unsigned int n) +{ + struct mem_cgroup *memcg; + + lockdep_assert_in_rcu_read_lock(); + + memcg = mem_cgroup_from_private_id(id); + __mem_cgroup_private_id_put(memcg, n); +} + +static void mem_cgroup_private_id_kill(struct mem_cgroup *memcg) +{ + __mem_cgroup_private_id_put(memcg, 1); +} + +unsigned short mem_cgroup_private_id_get(struct mem_cgroup *memcg, unsigned int n) { while (!refcount_add_not_zero(n, &memcg->private_id_ref)) { /* @@ -4105,7 +4120,8 @@ struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, un } memcg = parent_mem_cgroup(memcg); } - return memcg; + + return mem_cgroup_private_id(memcg); } /** @@ -4430,7 +4446,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); - mem_cgroup_private_id_put(memcg, 1); + mem_cgroup_private_id_kill(memcg); } static void mem_cgroup_css_released(struct cgroup_subsys_state *css) @@ -5971,15 +5987,13 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) return 0; } - memcg = mem_cgroup_private_id_get_online(memcg, nr_pages); - /* memcg is pined by memcg ID. */ - private_id = mem_cgroup_private_id(memcg); + private_id = mem_cgroup_private_id_get(memcg, nr_pages); if (!mem_cgroup_private_id_is_root(private_id) && !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - mem_cgroup_private_id_put(memcg, nr_pages); + mem_cgroup_private_id_put(private_id, nr_pages); return -ENOMEM; } mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); @@ -6011,7 +6025,7 @@ void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages) page_counter_uncharge(&memcg->swap, nr_pages); } mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages); - mem_cgroup_private_id_put(memcg, nr_pages); + mem_cgroup_private_id_put(id, nr_pages); } rcu_read_unlock(); } From 8290f45995f5700b87a47b1ee8c97300dfb66dda Mon Sep 17 00:00:00 2001 From: Bingfang Guo Date: Mon, 21 Sep 2026 14:16:27 +0800 Subject: [PATCH 018/213] memcg: move memcg private ID refcount to objcg The memcg private ID is used by objects that can't afford storing a whole pointer and can outlive memcgs to track the memcg (notably swap entries). The current design holds a refcount to the css, preventing the memcg from being freed. This patch unbinds the lifetime of memcgid from the memcg so it can be freed. The idea is to move the refcount of memcgid to one of the memcg's objcg. The objcg is stored in the global memcgid xarray instead and used for retrieving the online memcg from it. So swapped out pages no longer pin the dying memcg. After the change, a memcgid can refer to a non present memcg. To handle this situation, when trying to get the original memcg from the id, compare the memcgid passed in with that of the memcg, and return NULL to indicate its death if they differ. NULL checks are added for list_lru_walk_node(), workingset_test_recent() to skip dead memcgs. For MGLRU recency test, mem_cgroup_lruvec() will substitute NULL with root_mem_cgroup. In the earlier patch, an extra xarray lookup was introduced in swap uncharging path. Now that we have the objcg pointer in the function, the extra overhead can be removed by using it for putting directly. Link: https://lore.kernel.org/20260921-bingfangguo-memcgid-rework-v2-4-6c0637dc0edb@tencent.com Signed-off-by: Bingfang Guo Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin Cc: Shakeel Butt Cc: Dave Chinner Cc: Qi Zheng Cc: Kairui Song Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Bingfang Guo --- include/linux/memcontrol.h | 7 ++-- mm/list_lru.c | 2 +- mm/memcontrol.c | 73 ++++++++++++++++++++++++++++---------- mm/workingset.c | 2 +- 4 files changed, 60 insertions(+), 24 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index cd223f60b3a2..74110a324f9e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -180,6 +180,7 @@ struct obj_cgroup { struct percpu_ref refcnt; struct mem_cgroup *memcg; atomic_t nr_charged_bytes; + refcount_t private_id_ref; union { struct list_head list; /* protected by objcg_lock */ struct rcu_head rcu; @@ -225,9 +226,6 @@ struct mem_cgroup { /* vmpressure notifications. Written on every reclaim iteration. */ struct vmpressure vmpressure; - /* Written on every swap charge and uncharge. */ - refcount_t private_id_ref; - #ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC /* MEMCG_KMEM for nmi context */ atomic_t kmem_stat; @@ -324,6 +322,9 @@ struct mem_cgroup { unsigned long zswap_max; #endif + /* The objcg holding private memcg ID. */ + struct obj_cgroup *private_id_objcg; + /* Private memcg ID. Used to ID objects that outlive the cgroup */ int private_id; diff --git a/mm/list_lru.c b/mm/list_lru.c index 8a6dd0a489e1..7edd79113cc5 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -428,7 +428,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, xa_for_each(&lru->xa, index, mlru) { rcu_read_lock(); memcg = mem_cgroup_from_private_id(index); - if (!mem_cgroup_tryget(memcg)) { + if (!memcg || !mem_cgroup_tryget(memcg)) { rcu_read_unlock(); continue; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 53c09a0e500e..a5335da5d425 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4074,6 +4074,13 @@ static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg) #define MEM_CGROUP_ID_MAX ((1UL << MEM_CGROUP_ID_SHIFT) - 1) static DEFINE_XARRAY_ALLOC1(mem_cgroup_private_ids); +static inline struct obj_cgroup *obj_cgroup_from_private_id(unsigned short id) +{ + lockdep_assert_once(rcu_read_lock_held()); + + return xa_load(&mem_cgroup_private_ids, id); +} + static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg) { if (memcg->private_id > 0) { @@ -4082,34 +4089,44 @@ static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg) } } -static void __mem_cgroup_private_id_put(struct mem_cgroup *memcg, unsigned int n) +static void __mem_cgroup_private_id_put(struct obj_cgroup *objcg, + unsigned short id, unsigned int n) { - if (refcount_sub_and_test(n, &memcg->private_id_ref)) { - mem_cgroup_private_id_remove(memcg); + struct obj_cgroup *objcg_free; - /* Memcg ID pins CSS */ - css_put(&memcg->css); + if (refcount_sub_and_test(n, &objcg->private_id_ref)) { + objcg_free = xa_erase(&mem_cgroup_private_ids, id); + VM_WARN_ON(objcg_free != objcg); + + /* Memcg ID pins the objcg */ + obj_cgroup_put(objcg); } } static inline void mem_cgroup_private_id_put(unsigned short id, unsigned int n) { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; lockdep_assert_in_rcu_read_lock(); - memcg = mem_cgroup_from_private_id(id); - __mem_cgroup_private_id_put(memcg, n); + objcg = obj_cgroup_from_private_id(id); + __mem_cgroup_private_id_put(objcg, id, n); } static void mem_cgroup_private_id_kill(struct mem_cgroup *memcg) { - __mem_cgroup_private_id_put(memcg, 1); + __mem_cgroup_private_id_put(memcg->private_id_objcg, memcg->private_id, 1); } unsigned short mem_cgroup_private_id_get(struct mem_cgroup *memcg, unsigned int n) { - while (!refcount_add_not_zero(n, &memcg->private_id_ref)) { + struct obj_cgroup *objcg; + + lockdep_assert_once(rcu_read_lock_held()); + + objcg = memcg->private_id_objcg; + + while (!refcount_add_not_zero(n, &objcg->private_id_ref)) { /* * The root cgroup cannot be destroyed, so it's refcount must * always be >= 1. @@ -4119,6 +4136,7 @@ unsigned short mem_cgroup_private_id_get(struct mem_cgroup *memcg, unsigned int break; } memcg = parent_mem_cgroup(memcg); + objcg = memcg->private_id_objcg; } return mem_cgroup_private_id(memcg); @@ -4129,11 +4147,25 @@ unsigned short mem_cgroup_private_id_get(struct mem_cgroup *memcg, unsigned int * @id: the memcg id to look up * * Caller must hold rcu_read_lock(). + * + * @return: the memcg, or NULL if the memcg referred to is already dead. */ struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id) { + struct obj_cgroup *objcg; + struct mem_cgroup *memcg; + WARN_ON_ONCE(!rcu_read_lock_held()); - return xa_load(&mem_cgroup_private_ids, id); + + objcg = obj_cgroup_from_private_id(id); + if (!objcg) + return NULL; + + memcg = obj_cgroup_memcg(objcg); + if (mem_cgroup_private_id(memcg) != id) + return NULL; + + return memcg; } struct mem_cgroup *mem_cgroup_get_from_id(u64 id) @@ -4380,9 +4412,10 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) FLUSH_TIME); lru_gen_online_memcg(memcg); - /* Online state pins memcg ID, memcg ID pins CSS */ - refcount_set(&memcg->private_id_ref, 1); - css_get(css); + /* CSS pins memcg ID, memcg ID pins obj cgroup */ + memcg->private_id_objcg = objcg; + refcount_set(&memcg->private_id_objcg->private_id_ref, 1); + obj_cgroup_get(memcg->private_id_objcg); /* * Ensure mem_cgroup_from_private_id() works once we're fully online. @@ -4394,7 +4427,7 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) * publish it here at the end of onlining. This matches the * regular ID destruction during offlining. */ - xa_store(&mem_cgroup_private_ids, memcg->private_id, memcg, GFP_KERNEL); + xa_store(&mem_cgroup_private_ids, memcg->private_id, memcg->private_id_objcg, GFP_KERNEL); return 0; free_objcg: @@ -5832,8 +5865,6 @@ static void __init memcg_struct_check(void) memory_events_local); CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_write_hot, vmpressure); - CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_write_hot, - private_id_ref); #ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_write_hot, kmem_stat); @@ -5878,6 +5909,8 @@ static void __init memcg_struct_check(void) CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly, zswap_writeback); #endif + CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly, + private_id_objcg); CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly, private_id); CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly, @@ -6013,10 +6046,12 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) */ void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages) { + struct obj_cgroup *objcg; struct mem_cgroup *memcg; rcu_read_lock(); - memcg = mem_cgroup_from_private_id(id); + objcg = obj_cgroup_from_private_id(id); + memcg = obj_cgroup_memcg(objcg); if (memcg) { if (!mem_cgroup_private_id_is_root(id)) { if (do_memsw_account()) @@ -6025,7 +6060,7 @@ void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages) page_counter_uncharge(&memcg->swap, nr_pages); } mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages); - mem_cgroup_private_id_put(id, nr_pages); + __mem_cgroup_private_id_put(objcg, id, nr_pages); } rcu_read_unlock(); } diff --git a/mm/workingset.c b/mm/workingset.c index 8412f4840ae3..1504f91cdca5 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -470,7 +470,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, * configurations instead. */ eviction_memcg = mem_cgroup_from_private_id(memcgid); - if (!mem_cgroup_tryget(eviction_memcg)) + if (eviction_memcg && !mem_cgroup_tryget(eviction_memcg)) eviction_memcg = NULL; rcu_read_unlock(); From c2d4ccfbba3fab536d7fccd8d96675d325bb4bc5 Mon Sep 17 00:00:00 2001 From: Zhijian Han Date: Tue, 22 Sep 2026 11:18:43 +0800 Subject: [PATCH 019/213] mm: fix typos in various comments Fix spelling errors found by codespell in several mm source files: - "aray" -> "array" in include/linux/mm.h - "indiciate" -> "indicate" in include/linux/mm_types.h - "aggresive" -> "aggressive" in include/linux/mm_types.h - "progated" -> "propagated" in mm/huge_memory.c - "Pressumably" -> "Presumably" in mm/hugetlb.c - "fime" -> "time" in mm/hugetlb.c - "farest" -> "farthest" in mm/list_lru.c - "trivally" -> "trivially" in mm/madvise.c - "atmost" -> "at most" in mm/memcontrol.c - "splited" -> "split" in mm/memory-failure.c - "appliable" -> "applicable" in mm/memory.c - "incase" -> "in case" in mm/page_io.c - "contigous" -> "contiguous" in mm/vmalloc.c - "probablity" -> "probability" in mm/kfence/core.c - "possesss" -> "possesses" in mm/util.c No functional changes. Link: https://lore.kernel.org/20260922031843.2857104-1-hanzhijian1991@gmail.com Signed-off-by: Zhijian Han Signed-off-by: Andrew Morton --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 4 ++-- mm/huge_memory.c | 2 +- mm/hugetlb.c | 4 ++-- mm/kfence/core.c | 2 +- mm/list_lru.c | 2 +- mm/madvise.c | 2 +- mm/memcontrol.c | 2 +- mm/memory-failure.c | 2 +- mm/memory.c | 2 +- mm/page_io.c | 2 +- mm/util.c | 2 +- mm/vmalloc.c | 2 +- 13 files changed, 15 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0a2a7fc4a442..46a380aa1c84 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4601,7 +4601,7 @@ static inline void mmap_action_simple_ioremap(struct vm_area_desc *desc, * @desc: The VMA descriptor for the VMA requiring kernel pags to be mapped. * @start: The virtual address from which to map them. * @pages: An array of struct page pointers describing the memory to map. - * @nr_pages: The number of entries in the @pages aray. + * @nr_pages: The number of entries in the @pages array. */ static inline void mmap_action_map_kernel_pages(struct vm_area_desc *desc, unsigned long start, struct page **pages, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0720a4e98286..4b631fb4ff7a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -759,7 +759,7 @@ static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) /* * While __vma_enter_locked() is working to ensure are no read-locks held on a * VMA (either while acquiring a VMA write lock or marking a VMA detached) we - * set the VM_REFCNT_EXCLUDE_READERS_FLAG in vma->vm_refcnt to indiciate to + * set the VM_REFCNT_EXCLUDE_READERS_FLAG in vma->vm_refcnt to indicate to * vma_start_read() that the reference count should be left alone. * * See the comment describing vm_refcnt in vm_area_struct for details as to @@ -1971,7 +1971,7 @@ enum { /* * MMF_HAS_PINNED: Whether this mm has pinned any pages. This can be either * replaced in the future by mm.pinned_vm when it becomes stable, or grow into - * a counter on its own. We're aggresive on this bit for now: even if the + * a counter on its own. We're aggressive on this bit for now: even if the * pinned pages were unpinned later on, we'll still keep this bit set for the * lifecycle of this mm, just for simplicity. */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d7901b8171b2..67cf8ee898ba 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3420,7 +3420,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, swp_entry = make_readable_device_private_entry( page_to_pfn(page + i)); /* - * Young and dirty bits are not progated via swp_entry + * Young and dirty bits are not propagated via swp_entry */ entry = swp_entry_to_pte(swp_entry); if (soft_dirty) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 68f782f002d3..5ccef3ce67c7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5397,7 +5397,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, int rc = vma_needs_reservation(h, vma, address); if (rc < 0) - /* Pressumably allocate_file_region_entries failed + /* Presumably allocate_file_region_entries failed * to allocate a file_region struct. Clear * hugetlb_restore_reserve so that global reserve * count will not be incremented by free_huge_folio. @@ -5619,7 +5619,7 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf) * In order to determine where this is a COW on a MAP_PRIVATE mapping it * is enough to check whether the old_folio is anonymous. This means that * the reserve for this address was consumed. If reserves were used, a - * partial faulted mapping at the fime of fork() could consume its reserves + * partial faulted mapping at the time of fork() could consume its reserves * on COW instead of the full address range. */ if (is_vma_resv_set(vma, HPAGE_RESV_OWNER) && diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 90925c646c4c..55d5c114eadf 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -156,7 +156,7 @@ atomic_t kfence_allocation_gate = ATOMIC_INIT(1); * allocations of the same source filling up the pool. * * Assuming a range of 15%-85% unique allocations in the pool at any point in - * time, the below parameters provide a probablity of 0.02-0.33 for false + * time, the below parameters provide a probability of 0.02-0.33 for false * positive hits respectively: * * P(alloc_traces) = (1 - e^(-HNUM * (alloc_traces / SIZE)) ^ HNUM diff --git a/mm/list_lru.c b/mm/list_lru.c index 7edd79113cc5..d1d256822ccc 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -587,7 +587,7 @@ static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, */ do { /* - * Keep finding the farest parent that wasn't populated + * Keep finding the farthest parent that wasn't populated * until found memcg itself. */ pos = memcg; diff --git a/mm/madvise.c b/mm/madvise.c index bef4de4425f2..7a7bfcfb8f8f 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1298,7 +1298,7 @@ static bool is_discard(int behavior) * We are restricted from madvise()'ing mseal()'d VMAs only in very particular * circumstances - discarding of data from read-only anonymous SEALED mappings. * - * This is because users cannot trivally discard data from these VMAs, and may + * This is because users cannot trivially discard data from these VMAs, and may * only do so via an appropriate madvise() call. */ static bool can_madvise_modify(struct madvise_behavior *madv_behavior) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a5335da5d425..aad0498a7bd6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -741,7 +741,7 @@ static unsigned long *memcg_events_local_array(struct mem_cgroup *memcg) * * 2) Flush the stats synchronously on reader side only when there are more than * (MEMCG_CHARGE_BATCH * nr_cpus) update events. Though this optimization - * will let stats be out of sync by atmost (MEMCG_CHARGE_BATCH * nr_cpus) but + * will let stats be out of sync by at most (MEMCG_CHARGE_BATCH * nr_cpus) but * only for 2 seconds due to (1). */ static void flush_memcg_stats_dwork(struct work_struct *w); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index d237f556b3a0..0c96eb511997 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2560,7 +2560,7 @@ int memory_failure(unsigned long pfn, int flags) /* * We're only intended to deal with the non-Compound page here. * The page cannot become compound pages again as folio has been - * splited and extra refcnt is held. + * split and extra refcnt is held. */ WARN_ON(folio_test_large(folio)); diff --git a/mm/memory.c b/mm/memory.c index 7d821b110df0..ed64455c73bd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5069,7 +5069,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * have changed, so sub pages might got charged to the wrong cgroup, * or even should be shmem. So we have to free it and fallback. * Nothing should have touched it, both anon and shmem checks if a - * large folio is fully appliable before use. + * large folio is fully applicable before use. * * This will be removed once we unify folio allocation in the swap cache * layer, where allocation of a folio stabilizes the swap entries. diff --git a/mm/page_io.c b/mm/page_io.c index 0808808f0309..c6824fcd483e 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -135,7 +135,7 @@ static bool is_folio_zero_filled(struct folio *folio) for (i = 0; i < folio_nr_pages(folio); i++) { data = kmap_local_folio(folio, i * PAGE_SIZE); /* - * Check last word first, incase the page is zero-filled at + * Check last word first, in case the page is zero-filled at * the start and has non-zero data at the end, which is common * in real-world workloads. */ diff --git a/mm/util.c b/mm/util.c index bf0513d1d3d0..cddd55cbb61c 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1240,7 +1240,7 @@ EXPORT_SYMBOL(__compat_vma_mmap); /** * compat_vma_mmap() - Apply the file's .mmap_prepare() hook to an * existing VMA and execute any requested actions. - * @file: The file which possesss an f_op->mmap_prepare() hook. + * @file: The file which possesses an f_op->mmap_prepare() hook. * @vma: The VMA to apply the .mmap_prepare() hook to. * * Ordinarily, .mmap_prepare() is invoked directly upon mmap(). However, certain diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a009cc73f1b7..22ff0c32137f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3685,7 +3685,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, /* * Initially, attempt to have the page allocator give us large order * pages. Do not attempt allocating smaller than order chunks since - * __vmap_pages_range() expects physically contigous pages of exactly + * __vmap_pages_range() expects physically contiguous pages of exactly * order long chunks. */ while (large_order > order && nr_remaining) { From bdf6fbf21268591a7a7c8f4101c37f936bc57b8b Mon Sep 17 00:00:00 2001 From: Li Zhe Date: Mon, 28 Sep 2026 10:47:23 +0800 Subject: [PATCH 020/213] mm/hugetlb: fix overbroad MMU notifiers for unshared PMDs Hugetlb currently expands MMU notifier ranges to PUD boundaries whenever PMD sharing is possible. That is only needed when huge_pmd_unshare() actually detaches a shared PMD page table, because clearing the PUD invalidates the whole PUD-sized virtual address range. For hugetlbfs hole punch, and similarly for other hugetlb unmap paths, a shared mapping can pass the "PMD sharing is possible" range test in adjust_range_if_pmd_sharing_possible() even when the hugetlbfs file does not currently have any shared PMD page tables. KVM then receives a 1G invalidation for a 2M operation and zaps unrelated secondary mappings, so the guest has to fault them back in. Avoid this by tracking active PMD-sharing attachments per hugetlbfs inode. The count is incremented only after huge_pmd_share() successfully installs a shared PMD table, and decremented when __huge_pmd_unshare() actually detaches one. Since huge_pmd_share() can run concurrently under i_mmap_lock_read(), use a 64-bit atomic counter. A zero count is used to skip the conservative notifier range expansion only after excluding concurrent PMD sharing with the mapping write lock. On a Redis-in-VM workload that punches cold 2M hugetlb pages, this patch improves P99 QPS stability while punching pages, reducing the QPS degradation ratio from 7.09% to 1.45%. Link: https://lore.kernel.org/20260928024723.87708-1-lizhe.67@bytedance.com Signed-off-by: Li Zhe Signed-off-by: Andrew Morton Reported-by: aiqi.i7 Acked-by: Muchun Song Cc: David Hildenbrand Cc: Oscar Salvador --- fs/hugetlbfs/inode.c | 1 + include/linux/hugetlb.h | 37 +++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 13 ++++++++++--- 3 files changed, 48 insertions(+), 3 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 7611a8470ea2..78e27ce0a6f6 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -921,6 +921,7 @@ static struct inode *hugetlbfs_get_inode(struct super_block *sb, simple_inode_init_ts(inode); info->resv_map = resv_map; info->seals = F_SEAL_SEAL; + hugetlbfs_pmd_sharing_init(inode); switch (mode & S_IFMT) { default: init_special_inode(inode, mode, dev); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 80a5a03e9cee..893ef4fe3ba5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -508,6 +509,9 @@ struct hugetlbfs_inode_info { struct inode vfs_inode; struct resv_map *resv_map; unsigned int seals; +#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING + atomic64_t pmd_sharing_count; +#endif }; static inline struct hugetlbfs_inode_info *HUGETLBFS_I(struct inode *inode) @@ -515,6 +519,39 @@ static inline struct hugetlbfs_inode_info *HUGETLBFS_I(struct inode *inode) return container_of(inode, struct hugetlbfs_inode_info, vfs_inode); } +#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING +static inline void hugetlbfs_pmd_sharing_init(struct inode *inode) +{ + atomic64_set(&HUGETLBFS_I(inode)->pmd_sharing_count, 0); +} + +static inline void hugetlbfs_pmd_sharing_inc(struct inode *inode) +{ + atomic64_inc(&HUGETLBFS_I(inode)->pmd_sharing_count); +} + +static inline void hugetlbfs_pmd_sharing_dec(struct inode *inode) +{ + atomic64_dec(&HUGETLBFS_I(inode)->pmd_sharing_count); +} + +static inline bool hugetlbfs_pmd_sharing_active(struct inode *inode) +{ + return atomic64_read(&HUGETLBFS_I(inode)->pmd_sharing_count) != 0; +} +#else +static inline void hugetlbfs_pmd_sharing_init(struct inode *inode) {} + +static inline void hugetlbfs_pmd_sharing_inc(struct inode *inode) {} + +static inline void hugetlbfs_pmd_sharing_dec(struct inode *inode) {} + +static inline bool hugetlbfs_pmd_sharing_active(struct inode *inode) +{ + return false; +} +#endif + extern const struct vm_operations_struct hugetlb_vm_ops; struct file *hugetlb_file_setup(const char *name, size_t size, vma_flags_t acct, int creat_flags, int page_size_log); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5ccef3ce67c7..fd41ff7432a6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5427,10 +5427,12 @@ void __hugetlb_zap_begin(struct vm_area_struct *vma, if (!vma->vm_file) /* hugetlbfs_file_mmap error */ return; - adjust_range_if_pmd_sharing_possible(vma, start, end); hugetlb_vma_lock_write(vma); - if (vma->vm_file) + if (vma->vm_file) { i_mmap_lock_write(vma->vm_file->f_mapping); + if (hugetlbfs_pmd_sharing_active(file_inode(vma->vm_file))) + adjust_range_if_pmd_sharing_possible(vma, start, end); + } } void __hugetlb_zap_end(struct vm_area_struct *vma, @@ -5469,7 +5471,10 @@ void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, start, end); - adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); + i_mmap_assert_write_locked(vma->vm_file->f_mapping); + if (hugetlbfs_pmd_sharing_active(file_inode(vma->vm_file))) + adjust_range_if_pmd_sharing_possible(vma, &range.start, + &range.end); mmu_notifier_invalidate_range_start(&range); tlb_gather_mmu(&tlb, vma->vm_mm); @@ -7101,6 +7106,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, if (pud_none(*pud)) { pud_populate(mm, pud, (pmd_t *)((unsigned long)spte & PAGE_MASK)); + hugetlbfs_pmd_sharing_inc(file_inode(vma->vm_file)); mm_inc_nr_pmds(mm); } else { ptdesc_pmd_pts_dec(virt_to_ptdesc(spte)); @@ -7132,6 +7138,7 @@ static int __huge_pmd_unshare(struct mmu_gather *tlb, pud_clear(pud); tlb_unshare_pmd_ptdesc(tlb, virt_to_ptdesc(ptep), addr); + hugetlbfs_pmd_sharing_dec(file_inode(vma->vm_file)); mm_dec_nr_pmds(mm); return 1; From 52b64a9a7eedd663a88e519384e72204eac99a16 Mon Sep 17 00:00:00 2001 From: Park Tae-sun Date: Wed, 23 Sep 2026 18:26:48 +0900 Subject: [PATCH 021/213] selftests/mm: fix mlock2 errno handling and false PASS on ENOSYS While inspecting selftests/mm syscall wrappers, I noticed that mlock2_() in mlock2.h handles the syscall return value differently from other wrappers: int ret = syscall(__NR_mlock2, start, len, flags); if (ret) { errno = ret; return -1; } Commit 1ddae9d67ee1 ("selftests/mm/mlock: print error on failure") introduced this intending to make mlock2_() behave like libc by setting errno and returning -1. However, glibc syscall(2) already returns -1 on failure and sets positive errno. Assigning "errno = ret;" overwrites errno with -1. To verify this, mlock2 was disabled in the kernel (via sys_ni_syscall) to return -ENOSYS. Testing revealed two interrelated defects: 1. In the unmodified test, mlock2_() clobbered errno to -1. The check "if (ret && errno == ENOSYS)" in main() was bypassed, resulting in an immediate crash in the first test: ~ # ./mlock2-tests TAP version 13 1..15 Bail out! mlock2(0): Unknown error -1 # Planned tests != run tests (15 != 0) # Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0 (exit code: 1 - FAIL) 2. After restoring mlock2_() to directly return syscall(), errno correctly retained ENOSYS (38), entering the ENOSYS check in main(). However, it then called ksft_finished(): ~ # ./mlock2-tests TAP version 13 # Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0 ~ # echo $? 0 Because ksft_set_plan() had not been called yet (ksft_plan == 0) and zero tests ran (ksft_pass == 0), ksft_finished() evaluated 0 == 0 as success and exited with KSFT_PASS (code 0) without any TAP skip header. Fix both issues by: 1. Returning the syscall() result directly in mlock2_() so that errno is preserved. 2. Calling ksft_exit_skip() on ENOSYS so unsupported kernels report a TAP skip ("1..0 # SKIP ...") and exit with KSFT_SKIP (code 4). Verification on the mlock2-disabled kernel: ~ # ./mlock2-tests TAP version 13 1..0 # SKIP mlock2() syscall is not supported ~ # echo $? 4 Re-enabling mlock2 in the kernel confirmed all 15 tests pass cleanly: ~ # ./mlock2-tests TAP version 13 1..15 ok 1 test_mlock_lock: Locked ... ok 15 test_mlockall_future_droppable: droppable memory not locked # Totals: pass:15 fail:0 xfail:0 xpass:0 skip:0 error:0 ~ # echo $? 0 Link: https://lore.kernel.org/20260923-selftests-mm-mlock2-fix-v1-1-750b627854c6@dgu.ac.kr Fixes: 1ddae9d67ee1 ("selftests/mm/mlock: print error on failure") Fixes: 65c89684896d ("selftests/mm: mlock2-tests: conform test to TAP format output") Signed-off-by: Park Tae-sun Signed-off-by: Andrew Morton Reviewed-by: Gregory Price Acked-by: David Hildenbrand (Arm) Reviewed-by: Muhammad Usama Anjum Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan Cc: Brendan Jackman --- tools/testing/selftests/mm/mlock2-tests.c | 2 +- tools/testing/selftests/mm/mlock2.h | 8 +------- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/mm/mlock2-tests.c b/tools/testing/selftests/mm/mlock2-tests.c index e16e288cc7c1..144b550813a6 100644 --- a/tools/testing/selftests/mm/mlock2-tests.c +++ b/tools/testing/selftests/mm/mlock2-tests.c @@ -502,7 +502,7 @@ int main(int argc, char **argv) ret = mlock2_(map, size, MLOCK_ONFAULT); if (ret && errno == ENOSYS) - ksft_finished(); + ksft_exit_skip("mlock2() syscall is not supported\n"); munmap(map, size); diff --git a/tools/testing/selftests/mm/mlock2.h b/tools/testing/selftests/mm/mlock2.h index 81e77fa41901..4417eaa5cfb7 100644 --- a/tools/testing/selftests/mm/mlock2.h +++ b/tools/testing/selftests/mm/mlock2.h @@ -6,13 +6,7 @@ static int mlock2_(void *start, size_t len, int flags) { - int ret = syscall(__NR_mlock2, start, len, flags); - - if (ret) { - errno = ret; - return -1; - } - return 0; + return syscall(__NR_mlock2, start, len, flags); } static FILE *seek_to_smaps_entry(unsigned long addr) From f60ee3f51f28896137647ccb51ede21f2e15b535 Mon Sep 17 00:00:00 2001 From: Yeoreum Yun Date: Thu, 1 Oct 2026 22:17:51 +0100 Subject: [PATCH 022/213] kselftest: mm: prevent random failure of huge page split for khugepaged MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patch series "kselftest: mm: fix some failure of split_huge_page_test", v9. split_huge_page_test can fail for the following reasons: 1. During the test, khugepaged may collapse previously split pages again, causing intermittent failures. 2. Since glibc commit 321e1fc73f (“malloc: Enable 2MB THP by default on AArch64”), glibc may call madvise(MADV_HUGEPAGE) for sufficiently large allocations made by memalign(). The underlying VMA may start at a different address from the aligned address returned by memalign(). Moreover, a subsequent madvise(MADV_HUGEPAGE) call does not split the VMA because it already has the same advice. This causes the test to fail because the check_huge_xxx() helpers incorrectly require the address returned by memalign() to match the VMA start address reported in /proc/self/smaps. Address these issues by applying MADV_NOHUGEPAGE after faulting in the huge page, preventing khugepaged from collapsing it again, and instead of relying on /proc/self/smaps, use /proc/self/pagemap and /proc/kpageflags to detect huge-page mappings and large folios: 1. If hpage_size == pmd_pagesize, check PAGE_IS_HUGE instead of using check_large_folios(), since only the mapping type matters. This identifies PMD-mapped huge pages. 2. Otherwise, use check_large_folios() to detect large folios. This covers mTHP cases. 3. Check the folio flags according to the type of huge page. Since check_huge_shmem() was required to distinguish shmem huge pages because /proc/self/smaps reports them using a dedicated “ShmemPmdMapped” entry, as opposed to “FilePmdMapped” for file-backed huge pages. Now that /proc/self/smaps is no longer used to detect huge pages and /proc/kpageflags is used instead, it is sufficient to distinguish between file-backed and anonymous pages since the ShmemPmdMapped is also kind of file-backed. Therefore, remove check_huge_shmem() and use check_huge_file() instead. This patch (of 4): There're some random failure for split_huge_page_test when khugepaged collapses pages into pmd again which had split by the test. Prevent the khugepaged's collapses for split page by setting the mapped pmd-huge-page with MADV_NOHUGEPAGE before split. Link: https://lore.kernel.org/20261001-fix_split-v9-0-0f4ba8bbdbdf@arm.com Link: https://lore.kernel.org/20261001-fix_split-v9-1-0f4ba8bbdbdf@arm.com Signed-off-by: Yeoreum Yun Signed-off-by: Andrew Morton Suggested-by: Kevin Brodsky Suggested-by: Lorenzo Stoakes (ARM) Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Zi Yan Reviewed-by: Sarthak Sharma Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan --- .../testing/selftests/mm/split_huge_page_test.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index c5d96a4b1db3..36a6719ff09a 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -108,6 +108,15 @@ static char *allocate_zero_filled_hugepage(size_t len) return result; } +static void disable_khugepaged(void *addr, size_t len) +{ + /* Disables khugepaged from collapsing pages in range into THPs */ + if (!madvise(addr, len, MADV_NOHUGEPAGE)) + return; + + ksft_exit_fail_perror("MADV_NOHUGEPAGE failed"); +} + static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hpages, size_t len) { unsigned long rss_anon_before, rss_anon_after; @@ -120,6 +129,8 @@ static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hp if (!rss_anon_before) ksft_exit_fail_msg("No RssAnon is allocated before split\n"); + disable_khugepaged(one_page, len); + /* split all THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)one_page, (uint64_t)one_page + len, 0); @@ -167,6 +178,8 @@ static void split_pmd_thp_to_order(int order) if (!check_huge_anon(one_page, 4 * pmd_pagesize, 4, pmd_pagesize)) ksft_exit_fail_msg("No THP is allocated\n"); + disable_khugepaged(one_page, len); + /* split all THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)one_page, (uint64_t)one_page + len, order); @@ -215,6 +228,8 @@ static void split_pte_mapped_thp(void) goto out; } + disable_khugepaged(thp_area, thp_area_size); + /* * To challenge spitting code, we will mremap a single page of each * THP (page[i] of thp[i]) in the thp_area into page_area. This will @@ -482,6 +497,7 @@ static int create_pagecache_thp_and_fd(const char *testfile, size_t fd_size, ksft_test_result_skip("Pagecache folio split skipped\n"); return -2; } + disable_khugepaged(*addr, fd_size); return 0; err_out_close: close(*fd); From 6d6404c897a2c609cb88bd08d9fa9b2e8e9cd384 Mon Sep 17 00:00:00 2001 From: Yeoreum Yun Date: Thu, 1 Oct 2026 22:17:52 +0100 Subject: [PATCH 023/213] kselftest: mm: replace usage of /proc/self/smaps for __check_pmd_huge() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since glibc commit 321e1fc73f (“malloc: Enable 2MB THP by default on AArch64”), glibc may call madvise(MADV_HUGEPAGE) for sufficiently large allocations made by memalign(). The underlying VMA may start at a different address from the aligned address returned by memalign(). Furthermore, a subsequent madvise(MADV_HUGEPAGE) call does not split the VMA because the flag is already set. This causes split_huge_page_test to fail because the check_pmd_huge() helpers incorrectly require the address returned by memalign() to match the VMA start address reported in /proc/self/smaps. Instead of relying on /proc/self/smaps, use /proc/self/pagemap and /proc/kpageflags to detect huge-page mappings checking PAGE_IS_HUGE and PAGE_IS_FILE according to type of huge page. Since shmem pages are also file-backed, simply check whether the page is file-backed. Link: https://lore.kernel.org/20261001-fix_split-v9-2-0f4ba8bbdbdf@arm.com Fixes: 642bc52aed9c ("selftests: vm: bring common functions to a new file") Signed-off-by: Yeoreum Yun Signed-off-by: Andrew Morton Suggested-by: David Hildenbrand (Arm) Reviewed-by: Sarthak Sharma Reviewed-by: Baolin Wang Acked-by: David Hildenbrand (Arm) Tested-by: Baolin Wang Acked-by: Zi Yan Acked-by: Lorenzo Stoakes (ARM) Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan Cc: Kevin Brodsky --- tools/testing/selftests/mm/vm_util.c | 78 +++++++++++++++++----------- 1 file changed, 49 insertions(+), 29 deletions(-) diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index a0ab78ceedb1..a3c3a2706c22 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -351,24 +351,6 @@ char *__get_smap_entry(void *addr, const char *pattern, char *buf, size_t len) return entry; } -static bool __check_pmd_huge(void *addr, char *pattern, int nr_hpages, - uint64_t hpage_size) -{ - char buffer[MAX_LINE_LENGTH]; - uint64_t thp = -1; - char *entry; - - entry = __get_smap_entry(addr, pattern, buffer, sizeof(buffer)); - if (!entry) - goto err_out; - - if (sscanf(entry, "%9" SCNu64 " kB", &thp) != 1) - ksft_exit_fail_msg("Reading smap error\n"); - -err_out: - return thp == (nr_hpages * (hpage_size >> 10)); -} - static bool check_large_folios(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { @@ -410,6 +392,50 @@ static bool check_large_folios(void *addr, size_t len, int nr_hpages, return ret; } +enum check_huge_type { + CHECK_HUGE_ANON, + CHECK_HUGE_FILE, +}; + +static bool check_huge_type(uint64_t categories, enum check_huge_type type) +{ + const bool file = categories & PAGE_IS_FILE; + + switch (type) { + case CHECK_HUGE_ANON: + return !file; + case CHECK_HUGE_FILE: + return file; + } + + return false; +} + +static bool __check_pmd_huge(void *addr, size_t len, int nr_hpages, + uint64_t hpage_size, enum check_huge_type type) +{ + int pagemap_fd; + int nr_pmd_mappings = 0; + uint64_t categories; + char *start = addr; + char *end = start + len; + + pagemap_fd = open(PAGEMAP_PATH, O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + + for (; start < end; start += hpage_size) { + categories = pagemap_scan_get_categories(pagemap_fd, start); + if (!(categories & PAGE_IS_HUGE)) + continue; + if (check_huge_type(categories, type)) + nr_pmd_mappings++; + } + close(pagemap_fd); + + return nr_hpages == nr_pmd_mappings; +} + bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { uint64_t pmd_pagesize = read_pmd_pagesize(); @@ -418,7 +444,8 @@ bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) ksft_exit_fail_msg("reading PMD pagesize failed\n"); if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, "AnonHugePages: ", nr_hpages, hpage_size); + return __check_pmd_huge(addr, len, nr_hpages, hpage_size, + CHECK_HUGE_ANON); return check_large_folios(addr, len, nr_hpages, hpage_size); } @@ -431,22 +458,15 @@ bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) ksft_exit_fail_msg("reading PMD pagesize failed\n"); if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, "FilePmdMapped:", nr_hpages, hpage_size); + return __check_pmd_huge(addr, len, nr_hpages, hpage_size, + CHECK_HUGE_FILE); return check_large_folios(addr, len, nr_hpages, hpage_size); } bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { - uint64_t pmd_pagesize = read_pmd_pagesize(); - - if (!pmd_pagesize) - ksft_exit_fail_msg("reading PMD pagesize failed\n"); - - if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, "ShmemPmdMapped:", nr_hpages, hpage_size); - - return check_large_folios(addr, len, nr_hpages, hpage_size); + return check_huge_file(addr, len, nr_hpages, hpage_size); } int64_t allocate_transhuge(void *ptr, int pagemap_fd) From f47d99c9beb3dbc05b1d5ff199c6da95acaef9b4 Mon Sep 17 00:00:00 2001 From: Yeoreum Yun Date: Thu, 1 Oct 2026 22:17:53 +0100 Subject: [PATCH 024/213] kselftest: mm: integrate huge page checks check_large_folios() only checks for large folios without distinguishing between anonymous and file-backed huge pages. To add huge page type checking, integrate the huge page checks into __check_huge() and __check_type(): 1. If hpage_size == pmd_pagesize, check PAGE_IS_HUGE instead of using check_large_folios(), since only the mapping type matters. This identifies PMD-mapped huge pages. 2. Otherwise, use check_large_folios() to detect large folios. This covers mTHP cases. 3. Check the folio flags according to the huge page type via __check_type() Link: https://lore.kernel.org/20261001-fix_split-v9-3-0f4ba8bbdbdf@arm.com Signed-off-by: Yeoreum Yun Signed-off-by: Andrew Morton Suggested-by: David Hildenbrand (Arm) Suggested-by: Zi Yan Acked-by: Zi Yan Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan Cc: Kevin Brodsky --- tools/testing/selftests/mm/vm_util.c | 106 ++++++++++++++++----------- 1 file changed, 64 insertions(+), 42 deletions(-) diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index a3c3a2706c22..65c06e17c326 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -392,76 +392,98 @@ static bool check_large_folios(void *addr, size_t len, int nr_hpages, return ret; } -enum check_huge_type { - CHECK_HUGE_ANON, - CHECK_HUGE_FILE, +enum check_type { + CHECK_TYPE_ANON, + CHECK_TYPE_FILE, }; -static bool check_huge_type(uint64_t categories, enum check_huge_type type) -{ - const bool file = categories & PAGE_IS_FILE; - - switch (type) { - case CHECK_HUGE_ANON: - return !file; - case CHECK_HUGE_FILE: - return file; - } - - return false; -} - -static bool __check_pmd_huge(void *addr, size_t len, int nr_hpages, - uint64_t hpage_size, enum check_huge_type type) +static bool __check_type(void *addr, size_t len, uint64_t page_size, + enum check_type type) { + bool ret = false; int pagemap_fd; - int nr_pmd_mappings = 0; - uint64_t categories; char *start = addr; char *end = start + len; + uint64_t categories; pagemap_fd = open(PAGEMAP_PATH, O_RDONLY); if (pagemap_fd < 0) ksft_exit_fail_perror("open pagemap"); + for (; start < end; start += page_size) { + categories = pagemap_scan_get_categories(pagemap_fd, start); + if ((categories & PAGE_IS_PRESENT) != PAGE_IS_PRESENT) + continue; + + if ((type == CHECK_TYPE_FILE) != !!(categories & PAGE_IS_FILE)) + goto out; + } + + ret = true; + +out: + close(pagemap_fd); + return ret; +} + +static bool __check_huge(void *addr, size_t len, int nr_hpages, + uint64_t hpage_size) +{ + bool ret = false; + int pagemap_fd; + int nr_pmd_mappings = 0; + uint64_t pmd_pagesize; + uint64_t categories; + char *start = addr; + char *end = start + len; + + pmd_pagesize = read_pmd_pagesize(); + if (!pmd_pagesize) + ksft_exit_fail_msg("reading PMD pagesize failed\n"); + + pagemap_fd = open(PAGEMAP_PATH, O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + + if (hpage_size != pmd_pagesize) { + ret = check_large_folios(addr, len, nr_hpages, hpage_size); + goto out; + } + for (; start < end; start += hpage_size) { categories = pagemap_scan_get_categories(pagemap_fd, start); - if (!(categories & PAGE_IS_HUGE)) - continue; - if (check_huge_type(categories, type)) + if (categories & PAGE_IS_HUGE) nr_pmd_mappings++; } - close(pagemap_fd); - return nr_hpages == nr_pmd_mappings; + if (nr_pmd_mappings != nr_hpages) + goto out; + + ret = true; + +out: + close(pagemap_fd); + return ret; } bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { - uint64_t pmd_pagesize = read_pmd_pagesize(); + const uint64_t scan_mapping_size = (nr_hpages > 0) ? hpage_size : psize(); - if (!pmd_pagesize) - ksft_exit_fail_msg("reading PMD pagesize failed\n"); + if (!__check_huge(addr, len, nr_hpages, hpage_size)) + return false; - if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, len, nr_hpages, hpage_size, - CHECK_HUGE_ANON); - - return check_large_folios(addr, len, nr_hpages, hpage_size); + return __check_type(addr, len, scan_mapping_size, CHECK_TYPE_ANON); } bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { - uint64_t pmd_pagesize = read_pmd_pagesize(); + const uint64_t scan_mapping_size = (nr_hpages > 0) ? hpage_size : psize(); - if (!pmd_pagesize) - ksft_exit_fail_msg("reading PMD pagesize failed\n"); + if (!__check_huge(addr, len, nr_hpages, hpage_size)) + return false; - if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, len, nr_hpages, hpage_size, - CHECK_HUGE_FILE); - - return check_large_folios(addr, len, nr_hpages, hpage_size); + return __check_type(addr, len, scan_mapping_size, CHECK_TYPE_FILE); } bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) From 49c77c5de353f47c9607ba9ef9a6748a361ad975 Mon Sep 17 00:00:00 2001 From: Yeoreum Yun Date: Thu, 1 Oct 2026 22:17:54 +0100 Subject: [PATCH 025/213] kselftest: mm: remove check_huge_shmem() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit check_huge_shmem() was required to distinguish shmem huge pages because /proc/self/smaps reports them using a dedicated “ShmemPmdMapped” entry, as opposed to “FilePmdMapped” for file-backed huge pages. Now that /proc/self/smaps is no longer used to detect huge pages and /proc/kpageflags is used instead, it is sufficient to distinguish between file-backed and anonymous pages since the ShmemPmdMapped is also kind of file-backed. Therefore, remove check_huge_shmem() and use check_huge_file() instead and cleanup khugepaged's check_huge operation in mem_ops. Link: https://lore.kernel.org/20261001-fix_split-v9-4-0f4ba8bbdbdf@arm.com Signed-off-by: Yeoreum Yun Signed-off-by: Andrew Morton Suggested-by: David Hildenbrand (Arm) Reviewed-by: Baolin Wang Reviewed-by: Sarthak Sharma Acked-by: Zi Yan Acked-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan Cc: Kevin Brodsky --- .../selftests/mm/folio_split_race_test.c | 2 +- tools/testing/selftests/mm/khugepaged.c | 38 +++---------------- tools/testing/selftests/mm/uffd-common.c | 4 +- tools/testing/selftests/mm/vm_util.c | 5 --- tools/testing/selftests/mm/vm_util.h | 1 - 5 files changed, 9 insertions(+), 41 deletions(-) diff --git a/tools/testing/selftests/mm/folio_split_race_test.c b/tools/testing/selftests/mm/folio_split_race_test.c index e4660bf89b62..1a9840b73e61 100644 --- a/tools/testing/selftests/mm/folio_split_race_test.c +++ b/tools/testing/selftests/mm/folio_split_race_test.c @@ -181,7 +181,7 @@ static uint64_t run_iteration(void) for (i = 0; i < TOTAL_PAGES; i++) fill_page(mmap_base, i); - if (!check_huge_shmem(mmap_base, FILE_SIZE, NR_PMD_PAGE, pmd_pagesize)) + if (!check_huge_file(mmap_base, FILE_SIZE, NR_PMD_PAGE, pmd_pagesize)) ksft_exit_fail_msg("No shmem THP is allocated\n"); if (pthread_barrier_init(&ctl.barrier, NULL, NUM_READER_THREADS + 1) != 0) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index 4e888b7bf310..1bc7acc66a46 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -57,7 +57,7 @@ struct mem_ops { void *(*setup_area)(int nr_hpages); void (*cleanup_area)(void *p, unsigned long size); void (*fault)(void *p, unsigned long start, unsigned long end); - bool (*check_huge)(void *addr, size_t len, int nr_hpages, unsigned long hpage_size); + bool (*check_huge)(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); const char *name; }; @@ -343,12 +343,6 @@ static void anon_fault(void *p, unsigned long start, unsigned long end) fill_memory(p, start, end); } -static bool anon_check_huge(void *addr, size_t len, int nr_hpages, - unsigned long hpage_size) -{ - return check_huge_anon(addr, len, nr_hpages, hpage_size); -} - static void *file_setup_area_common(int nr_hpages, enum file_setup_ops setup) { const int open_opt = setup == FILE_SETUP_READ_ONLY_FS ? O_RDONLY : O_RDWR; @@ -444,20 +438,6 @@ static void file_fault_write(void *p, unsigned long start, unsigned long end) ksft_exit_fail_perror("madvise(MADV_POPULATE_WRITE)"); } -static bool file_check_huge(void *addr, size_t len, int nr_hpages, - unsigned long hpage_size) -{ - switch (finfo.type) { - case VMA_FILE: - return check_huge_file(addr, len, nr_hpages, hpage_size); - case VMA_SHMEM: - return check_huge_shmem(addr, len, nr_hpages, hpage_size); - default: - ksft_exit_fail_msg("Unknown VMA type\n"); - return false; - } -} - static void *shmem_setup_area(int nr_hpages) { void *p; @@ -481,17 +461,11 @@ static void shmem_cleanup_area(void *p, unsigned long size) close(finfo.fd); } -static bool shmem_check_huge(void *addr, size_t len, int nr_hpages, - unsigned long hpage_size) -{ - return check_huge_shmem(addr, len, nr_hpages, hpage_size); -} - static struct mem_ops __anon_ops = { .setup_area = &anon_setup_area, .cleanup_area = &anon_cleanup_area, .fault = &anon_fault, - .check_huge = &anon_check_huge, + .check_huge = &check_huge_anon, .name = "anon", }; @@ -499,7 +473,7 @@ static struct mem_ops __read_only_file_ops = { .setup_area = &file_setup_read_only_area, .cleanup_area = &file_cleanup_area, .fault = &file_fault_read, - .check_huge = &file_check_huge, + .check_huge = &check_huge_file, .name = "file", }; @@ -507,7 +481,7 @@ static struct mem_ops __read_write_file_read_ops = { .setup_area = &file_setup_read_write_fs_read_area, .cleanup_area = &file_cleanup_area, .fault = &file_fault_read_and_flush, - .check_huge = &file_check_huge, + .check_huge = &check_huge_file, .name = "file", }; @@ -515,7 +489,7 @@ static struct mem_ops __read_write_file_write_ops = { .setup_area = &file_setup_read_write_fs_write_area, .cleanup_area = &file_cleanup_area, .fault = &file_fault_write, - .check_huge = &file_check_huge, + .check_huge = &check_huge_file, .name = "file", }; @@ -523,7 +497,7 @@ static struct mem_ops __shmem_ops = { .setup_area = &shmem_setup_area, .cleanup_area = &shmem_cleanup_area, .fault = &anon_fault, - .check_huge = &shmem_check_huge, + .check_huge = &check_huge_file, .name = "shmem", }; diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index 1fb967ef4985..a76851aced4d 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -196,8 +196,8 @@ static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *p, int { size_t len = expect_nr_hpages * read_pmd_pagesize(); - if (!check_huge_shmem(gopts->area_dst_alias, len, expect_nr_hpages, - read_pmd_pagesize())) + if (!check_huge_file(gopts->area_dst_alias, len, expect_nr_hpages, + read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); } diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index 65c06e17c326..515031b8283d 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -486,11 +486,6 @@ bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) return __check_type(addr, len, scan_mapping_size, CHECK_TYPE_FILE); } -bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) -{ - return check_huge_file(addr, len, nr_hpages, hpage_size); -} - int64_t allocate_transhuge(void *ptr, int pagemap_fd) { uint64_t ent[2]; diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h index 072a6c756c51..b5d59729d432 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -96,7 +96,6 @@ uint64_t read_pmd_pagesize(void); unsigned long rss_anon(void); bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); -bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); int64_t allocate_transhuge(void *ptr, int pagemap_fd); int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags); int gather_folio_orders(char *vaddr_start, size_t len, From 7c7180be976c8fed695676d1e8daf311ec08e91a Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Thu, 24 Sep 2026 12:11:03 -0700 Subject: [PATCH 026/213] mm: remove the unused zone->unaccepted_cleanup Commit fefc07518227 ("mm/page_alloc: fix race condition in unaccepted memory handling") removed the zones_with_unaccepted_pages static key and the work that decremented it, but left the work_struct behind in struct zone. Nothing uses it anymore, so let's remove it. Link: https://lore.kernel.org/20260924191103.3475117-1-usama.arif@linux.dev Signed-off-by: Usama Arif Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Shakeel Butt Reviewed-by: Johannes Weiner Reviewed-by: Barry Song Reviewed-by: Kiryl Shutsemau (Meta) Cc: Axel Rasmussen Cc: Baolin Wang Cc: Baoquan He Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Qi Zheng Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie --- include/linux/mmzone.h | 3 --- 1 file changed, 3 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8e4e0bda3b58..4a5c826daec7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1077,9 +1077,6 @@ struct zone { #ifdef CONFIG_UNACCEPTED_MEMORY /* Pages to be accepted. All pages on the list are MAX_PAGE_ORDER */ struct list_head unaccepted_pages; - - /* To be called once the last page in the zone is accepted */ - struct work_struct unaccepted_cleanup; #endif /* zone flags, see below */ From b32f336987744d364444a35ec490b8ba6ec60c6b Mon Sep 17 00:00:00 2001 From: Anshuman Khandual Date: Wed, 16 Sep 2026 10:19:32 +0530 Subject: [PATCH 027/213] arm64/mm: move __check_safe_pte_update() Patch series "arm64/mm: Standardize printing for pgtable entries", v3. Standardize printing for pgtable entries using recently introduced generic helper ptval_bytes_to_hex_str() in core MM which automatically enables 128 bits entries when added later. But first move __check_safe_pte_update() outside to avoid a cyclic dependency while accessing these afore mentioned core MM helpers defined in . This replaces the original page table entry print standardisation proposal which was part of the D128 series [1]. This patch (of 2): The page table entry print helpers and related macros which are defined in will not be accessible in platform which is basically caused by cycling dependency. Move __check_safe_pte_update() inside arch/arm64/mm/mmu.c as a preparation for subsequent usage of the afore mentioned generic MM helpers. While here drop IS_ENABLED(CONFIG_DEBUG_VM), although wrap __check_safe_pte_update() inside #ifdef CONFIG_DEBUG_VM that preserves the current code optimization which is achieved via the static inline functions. This does not cause any functional change. [david@kernel.org: aarch64-linux-ld error] Link: https://lore.kernel.org/99023678-fd8a-4eef-8e43-a07b504e3e96@kernel.org Link: https://lore.kernel.org/20260916044933.2689426-2-anshuman.khandual@arm.com Link: https://lore.kernel.org/linux-mm/20260729122452.3797443-11-anshuman.khandual@arm.com/ [1] Signed-off-by: Anshuman Khandual Signed-off-by: Andrew Morton Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Catalin Marinas Cc: Will Deacon Cc: Mike Rapoport Cc: Lorenzo Stoakes --- arch/arm64/include/asm/pgtable.h | 49 ++++---------------------------- arch/arm64/mm/mmu.c | 44 ++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 44 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e89ec5f4787b..763c5a411d64 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -387,52 +387,13 @@ static inline pte_t __ptep_get(pte_t *ptep) extern void __sync_icache_dcache(pte_t pteval); bool pgattr_change_is_safe(pteval_t old, pteval_t new); -/* - * PTE bits configuration in the presence of hardware Dirty Bit Management - * (PTE_WRITE == PTE_DBM): - * - * Dirty Writable | PTE_RDONLY PTE_WRITE PTE_DIRTY (sw) - * 0 0 | 1 0 0 - * 0 1 | 1 1 0 - * 1 0 | 1 0 1 - * 1 1 | 0 1 x - * - * When hardware DBM is not present, the software PTE_DIRTY bit is updated via - * the page fault mechanism. Checking the dirty status of a pte becomes: - * - * PTE_DIRTY || (PTE_WRITE && !PTE_RDONLY) - */ - -static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, - pte_t pte) +#ifdef CONFIG_DEBUG_VM +void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte); +#else +static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte) { - pte_t old_pte; - - if (!IS_ENABLED(CONFIG_DEBUG_VM)) - return; - - old_pte = __ptep_get(ptep); - - if (!pte_valid(old_pte) || !pte_valid(pte)) - return; - if (mm != current->active_mm && atomic_read(&mm->mm_users) <= 1) - return; - - /* - * Check for potential race with hardware updates of the pte - * (__ptep_set_access_flags safely changes valid ptes without going - * through an invalid entry). - */ - VM_WARN_ONCE(!pte_young(pte), - "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); - VM_WARN_ONCE(pte_write(old_pte) && !pte_dirty(pte), - "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); - VM_WARN_ONCE(!pgattr_change_is_safe(pte_val(old_pte), pte_val(pte)), - "%s: unsafe attribute change: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); } +#endif /* CONFIG_DEBUG_VM */ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 79d90226fd5d..05b7b52d761c 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -2393,3 +2393,47 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val) return 0; } #endif + +/* + * PTE bits configuration in the presence of hardware Dirty Bit Management + * (PTE_WRITE == PTE_DBM): + * + * Dirty Writable | PTE_RDONLY PTE_WRITE PTE_DIRTY (sw) + * 0 0 | 1 0 0 + * 0 1 | 1 1 0 + * 1 0 | 1 0 1 + * 1 1 | 0 1 x + * + * When hardware DBM is not present, the software PTE_DIRTY bit is updated via + * the page fault mechanism. Checking the dirty status of a pte becomes: + * + * PTE_DIRTY || (PTE_WRITE && !PTE_RDONLY) + */ +#ifdef CONFIG_DEBUG_VM +void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte) +{ + pte_t old_pte; + + old_pte = __ptep_get(ptep); + + if (!pte_valid(old_pte) || !pte_valid(pte)) + return; + if (mm != current->active_mm && atomic_read(&mm->mm_users) <= 1) + return; + + /* + * Check for potential race with hardware updates of the pte + * (__ptep_set_access_flags safely changes valid ptes without going + * through an invalid entry). + */ + VM_WARN_ONCE(!pte_young(pte), + "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", + __func__, pte_val(old_pte), pte_val(pte)); + VM_WARN_ONCE(pte_write(old_pte) && !pte_dirty(pte), + "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", + __func__, pte_val(old_pte), pte_val(pte)); + VM_WARN_ONCE(!pgattr_change_is_safe(pte_val(old_pte), pte_val(pte)), + "%s: unsafe attribute change: 0x%016llx -> 0x%016llx", + __func__, pte_val(old_pte), pte_val(pte)); +} +#endif /* CONFIG_DEBUG_VM */ From d4113e998367cf3aec7a33589c8c378d01aed5ba Mon Sep 17 00:00:00 2001 From: Anshuman Khandual Date: Wed, 16 Sep 2026 10:19:33 +0530 Subject: [PATCH 028/213] arm64/mm: standardize printing for pgtable entries Standardize printing for pgtable entries using recently introduced generic helper ptval_bytes_to_hex_str() in core MM which automatically enables 128 bits entries when added later. Link: https://lore.kernel.org/20260916044933.2689426-3-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Signed-off-by: Andrew Morton Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Catalin Marinas Cc: Will Deacon Cc: Mike Rapoport Cc: Lorenzo Stoakes --- arch/arm64/mm/fault.c | 16 +++++++++++----- arch/arm64/mm/mmu.c | 16 ++++++++++------ 2 files changed, 21 insertions(+), 11 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 75c3e463df2e..2cecf6ba6df7 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -131,6 +131,7 @@ static inline unsigned long mm_to_pgd_phys(struct mm_struct *mm) */ static void show_pte(unsigned long addr) { + char pxd_str[PTVAL_STR_MAX]; struct mm_struct *mm; pgd_t *pgdp; pgd_t pgd; @@ -160,7 +161,8 @@ static void show_pte(unsigned long addr) pgdp = pgd_offset(mm, addr); pgd = READ_ONCE(*pgdp); - pr_alert("[%016lx] pgd=%016llx", addr, pgd_val(pgd)); + ptval_to_str(pxd_str, pgd_val(pgd)); + pr_alert("[%016lx] pgd=%s", addr, pxd_str); do { p4d_t *p4dp, p4d; @@ -173,19 +175,22 @@ static void show_pte(unsigned long addr) p4dp = p4d_offset_lockless(pgdp, pgd, addr); p4d = READ_ONCE(*p4dp); - pr_cont(", p4d=%016llx", p4d_val(p4d)); + ptval_to_str(pxd_str, p4d_val(p4d)); + pr_cont(", p4d=%s", pxd_str); if (p4d_none(p4d) || p4d_bad(p4d)) break; pudp = pud_offset_lockless(p4dp, p4d, addr); pud = READ_ONCE(*pudp); - pr_cont(", pud=%016llx", pud_val(pud)); + ptval_to_str(pxd_str, pud_val(pud)); + pr_cont(", pud=%s", pxd_str); if (pud_none(pud) || pud_bad(pud)) break; pmdp = pmd_offset_lockless(pudp, pud, addr); pmd = READ_ONCE(*pmdp); - pr_cont(", pmd=%016llx", pmd_val(pmd)); + ptval_to_str(pxd_str, pmd_val(pmd)); + pr_cont(", pmd=%s", pxd_str); if (pmd_none(pmd) || pmd_bad(pmd)) break; @@ -194,7 +199,8 @@ static void show_pte(unsigned long addr) break; pte = __ptep_get(ptep); - pr_cont(", pte=%016llx", pte_val(pte)); + ptval_to_str(pxd_str, pte_val(pte)); + pr_cont(", pte=%s", pxd_str); pte_unmap(ptep); } while(0); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 05b7b52d761c..7343ac9294f8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -2412,6 +2412,8 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val) #ifdef CONFIG_DEBUG_VM void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte) { + char pte_str_old[PTVAL_STR_MAX]; + char pte_str[PTVAL_STR_MAX]; pte_t old_pte; old_pte = __ptep_get(ptep); @@ -2426,14 +2428,16 @@ void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte) * (__ptep_set_access_flags safely changes valid ptes without going * through an invalid entry). */ + ptval_to_str(pte_str, pte_val(pte)); + ptval_to_str(pte_str_old, pte_val(old_pte)); VM_WARN_ONCE(!pte_young(pte), - "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); + "%s: racy access flag clearing: %s -> %s", + __func__, pte_str_old, pte_str); VM_WARN_ONCE(pte_write(old_pte) && !pte_dirty(pte), - "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); + "%s: racy dirty state clearing: %s -> %s", + __func__, pte_str_old, pte_str); VM_WARN_ONCE(!pgattr_change_is_safe(pte_val(old_pte), pte_val(pte)), - "%s: unsafe attribute change: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); + "%s: unsafe attribute change: %s -> %s", + __func__, pte_str_old, pte_str); } #endif /* CONFIG_DEBUG_VM */ From 2e83e2ce34ff9200b58ac92f4c94a31c9adcba52 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sat, 26 Sep 2026 12:29:29 +0300 Subject: [PATCH 029/213] arch, mm: promote DEBUG_WX to CHECK_WX Verification that the kernel does not have writable + executable mappings is about detecting security risks rather than a pure debug feature. Major distribution configurations enable it in their kernels as well as defconfigs of most architectures that have ARCH_HAS_DEBUG_WX. Rename relevant generic configuration options to use CHECK_WX and move their definitions from mm/Kconfig.debug to mm/Kconfig. Rename *debug_checkwx() funcitons and macros to *pgtable_checkwx(). For arm that does not widely enable it, only rename its variants of the config options. Enabling CHECK_WX adds a few kilobytes to the kernel binary and while the added size can be slightly reduced with churny updates of architecture implementations of ptdump, the core functionality takes most of the added size. It cannot be moved to .init.text because the verification has to happen after init sections are freed. With this, make generic CHECK_WX default to STRICT_KERNEL_RWX while still leaving users targeting small kernels the possibility to opt-out. Link: https://lore.kernel.org/20260926-direct-map-verify-wx-v2-1-efcd64a6b74a@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Suggested-by: Dave Hansen Acked-by: Lorenzo Stoakes (ARM) Acked-by: Dave Hansen Acked-by: David Hildenbrand (Arm) Reviewed-by: Anshuman Khandual Cc: Albert Ou Cc: Alexander Gordeev Cc: Alexandre Ghiti Cc: Borislav Petkov Cc: Catalin Marinas Cc: Christophe Leroy Cc: Christian Borntraeger Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Madhavan Srinivasan Cc: Mark Rutland Cc: Michael Ellerman Cc: Michal Hocko Cc: Nicholas Piggin Cc: Palmer Dabbelt Cc: Paul Walmsley Cc: H. Peter Anvin Cc: Ritesh Harjani (IBM) Cc: Russell King Cc: Shrikanth Hegde Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Gleixner Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: Will Deacon --- arch/arm/Kconfig.debug | 2 +- arch/arm/configs/aspeed_g4_defconfig | 2 +- arch/arm/configs/aspeed_g5_defconfig | 2 +- arch/arm/configs/shmobile_defconfig | 2 +- arch/arm/include/asm/ptdump.h | 6 ++-- arch/arm/mm/init.c | 2 +- arch/arm64/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/powerpc/configs/ppc64_defconfig | 2 +- arch/powerpc/mm/ptdump/ptdump.c | 2 +- arch/riscv/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/s390/configs/debug_defconfig | 2 +- arch/s390/configs/defconfig | 2 +- arch/s390/mm/dump_pagetables.c | 2 +- arch/x86/Kconfig | 2 +- arch/x86/configs/x86_64_defconfig | 2 +- arch/x86/include/asm/pgtable.h | 6 ++-- arch/x86/mm/pti.c | 2 +- include/linux/ptdump.h | 4 +-- init/main.c | 2 +- kernel/configs/debug.config | 2 +- mm/Kconfig | 41 ++++++++++++++++++++++++++++ mm/Kconfig.debug | 39 -------------------------- 24 files changed, 68 insertions(+), 66 deletions(-) diff --git a/arch/arm/Kconfig.debug b/arch/arm/Kconfig.debug index 366f162e147d..abcf14f10276 100644 --- a/arch/arm/Kconfig.debug +++ b/arch/arm/Kconfig.debug @@ -17,7 +17,7 @@ config ARM_PTDUMP_DEBUGFS kernel. If in doubt, say "N" -config ARM_DEBUG_WX +config ARM_CHECK_WX bool "Warn on W+X mappings at boot" depends on MMU select ARM_PTDUMP_CORE diff --git a/arch/arm/configs/aspeed_g4_defconfig b/arch/arm/configs/aspeed_g4_defconfig index f86dd4ce7d0d..2c9d5a644ae9 100644 --- a/arch/arm/configs/aspeed_g4_defconfig +++ b/arch/arm/configs/aspeed_g4_defconfig @@ -249,7 +249,7 @@ CONFIG_DEBUG_INFO_REDUCED=y CONFIG_GDB_SCRIPTS=y CONFIG_STRIP_ASM_SYMS=y CONFIG_DEBUG_FS=y -CONFIG_ARM_DEBUG_WX=y +CONFIG_ARM_CHECK_WX=y CONFIG_SCHED_STACK_END_CHECK=y CONFIG_PANIC_ON_OOPS=y CONFIG_PANIC_TIMEOUT=-1 diff --git a/arch/arm/configs/aspeed_g5_defconfig b/arch/arm/configs/aspeed_g5_defconfig index 45b937419dbd..1327a09e163a 100644 --- a/arch/arm/configs/aspeed_g5_defconfig +++ b/arch/arm/configs/aspeed_g5_defconfig @@ -300,7 +300,7 @@ CONFIG_DEBUG_INFO_REDUCED=y CONFIG_GDB_SCRIPTS=y CONFIG_STRIP_ASM_SYMS=y CONFIG_DEBUG_FS=y -CONFIG_ARM_DEBUG_WX=y +CONFIG_ARM_CHECK_WX=y CONFIG_SCHED_STACK_END_CHECK=y CONFIG_PANIC_ON_OOPS=y CONFIG_PANIC_TIMEOUT=-1 diff --git a/arch/arm/configs/shmobile_defconfig b/arch/arm/configs/shmobile_defconfig index 6f9696e9fe17..cc22e22b989e 100644 --- a/arch/arm/configs/shmobile_defconfig +++ b/arch/arm/configs/shmobile_defconfig @@ -225,4 +225,4 @@ CONFIG_CMA_SIZE_MBYTES=64 CONFIG_PRINTK_TIME=y CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_FS=y -CONFIG_ARM_DEBUG_WX=y +CONFIG_ARM_CHECK_WX=y diff --git a/arch/arm/include/asm/ptdump.h b/arch/arm/include/asm/ptdump.h index 46a4575146ee..3c5245220ecf 100644 --- a/arch/arm/include/asm/ptdump.h +++ b/arch/arm/include/asm/ptdump.h @@ -32,10 +32,10 @@ void ptdump_check_wx(void); #endif /* CONFIG_ARM_PTDUMP_CORE */ -#ifdef CONFIG_ARM_DEBUG_WX -#define arm_debug_checkwx() ptdump_check_wx() +#ifdef CONFIG_ARM_CHECK_WX +#define arm_pgtable_checkwx() ptdump_check_wx() #else -#define arm_debug_checkwx() do { } while (0) +#define arm_pgtable_checkwx() do { } while (0) #endif #endif /* __ASM_PTDUMP_H */ diff --git a/arch/arm/mm/init.c b/arch/arm/mm/init.c index 0cc1bf04686d..c515faf22eeb 100644 --- a/arch/arm/mm/init.c +++ b/arch/arm/mm/init.c @@ -403,7 +403,7 @@ static int __mark_rodata_ro(void *unused) void mark_rodata_ro(void) { stop_machine(__mark_rodata_ro, NULL, NULL); - arm_debug_checkwx(); + arm_pgtable_checkwx(); } #else diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b6c2dd8b2612..b51d23a62c92 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -11,7 +11,7 @@ config ARM64 select ACPI_MCFG if (ACPI && PCI) select ACPI_SPCR_TABLE if ACPI select ACPI_PPTT if ACPI - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_BINFMT_ELF_EXTRA_PHDRS select ARCH_BINFMT_ELF_STATE select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 0767cfcbaa42..877393207e3b 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -130,7 +130,7 @@ config PPC select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE - select ARCH_HAS_DEBUG_WX if STRICT_KERNEL_RWX + select ARCH_HAS_CHECK_WX if STRICT_KERNEL_RWX select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES select ARCH_HAS_DMA_OPS if PPC64 diff --git a/arch/powerpc/configs/ppc64_defconfig b/arch/powerpc/configs/ppc64_defconfig index 1eb8e3457e8b..5c33f0bba0e3 100644 --- a/arch/powerpc/configs/ppc64_defconfig +++ b/arch/powerpc/configs/ppc64_defconfig @@ -393,7 +393,7 @@ CONFIG_MAGIC_SYSRQ=y CONFIG_PAGE_OWNER=y CONFIG_PAGE_POISONING=y CONFIG_DEBUG_RODATA_TEST=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_DEBUG_STACK_USAGE=y CONFIG_DEBUG_VM=y # CONFIG_DEBUG_VM_PGTABLE is not set diff --git a/arch/powerpc/mm/ptdump/ptdump.c b/arch/powerpc/mm/ptdump/ptdump.c index 0d499aebee72..3451351b756b 100644 --- a/arch/powerpc/mm/ptdump/ptdump.c +++ b/arch/powerpc/mm/ptdump/ptdump.c @@ -191,7 +191,7 @@ static void note_prot_wx(struct pg_state *st, unsigned long addr) if (!pte_write(pte) || !pte_exec(pte)) return; - WARN_ONCE(IS_ENABLED(CONFIG_DEBUG_WX), + WARN_ONCE(IS_ENABLED(CONFIG_CHECK_WX), "powerpc/mm: Found insecure W+X mapping at address %p/%pS\n", (void *)st->start_address, (void *)st->start_address); diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0db108ea1466..f409f264d8c4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -29,7 +29,7 @@ config RISCV select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_HAS_DELAY_TIMER select ARCH_HAS_ELF_CORE_EFLAGS if BINFMT_ELF && ELF_CORE select ARCH_HAS_FAST_MULTIPLIER diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index a34376c05f6e..4bbb2c89bce0 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -92,7 +92,7 @@ config S390 select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_OPS if PCI select ARCH_HAS_ELF_RANDOMIZE diff --git a/arch/s390/configs/debug_defconfig b/arch/s390/configs/debug_defconfig index 3dae71474333..68d53c0bc8db 100644 --- a/arch/s390/configs/debug_defconfig +++ b/arch/s390/configs/debug_defconfig @@ -841,7 +841,7 @@ CONFIG_DEBUG_PAGEALLOC=y CONFIG_SLUB_DEBUG_ON=y CONFIG_PAGE_OWNER=y CONFIG_DEBUG_RODATA_TEST=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_PTDUMP_DEBUGFS=y CONFIG_DEBUG_OBJECTS=y CONFIG_DEBUG_OBJECTS_SELFTEST=y diff --git a/arch/s390/configs/defconfig b/arch/s390/configs/defconfig index 6f5722634b4d..8e5cfc695121 100644 --- a/arch/s390/configs/defconfig +++ b/arch/s390/configs/defconfig @@ -820,7 +820,7 @@ CONFIG_DEBUG_INFO_DWARF4=y CONFIG_GDB_SCRIPTS=y CONFIG_DEBUG_SECTION_MISMATCH=y CONFIG_MAGIC_SYSRQ=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_PTDUMP_DEBUGFS=y CONFIG_DEBUG_MEMORY_INIT=y CONFIG_PANIC_ON_OOPS=y diff --git a/arch/s390/mm/dump_pagetables.c b/arch/s390/mm/dump_pagetables.c index 89badbe72ae7..a23a0bd4d8a8 100644 --- a/arch/s390/mm/dump_pagetables.c +++ b/arch/s390/mm/dump_pagetables.c @@ -86,7 +86,7 @@ static void note_prot_wx(struct pg_state *st, unsigned long addr) */ if (addr == PAGE_SIZE && (nospec_uses_trampoline() || !cpu_has_bear())) return; - WARN_ONCE(IS_ENABLED(CONFIG_DEBUG_WX), + WARN_ONCE(IS_ENABLED(CONFIG_CHECK_WX), "s390/mm: Found insecure W+X mapping at address %pS\n", (void *)st->start_address); st->wx_pages += (addr - st->start_address) / PAGE_SIZE; diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 6e5e462ec059..6cb70e47d0bd 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -109,7 +109,7 @@ config X86 select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE select ARCH_HAS_SYSCALL_WRAPPER select ARCH_HAS_UBSAN - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_HAS_ZONE_DMA_SET if EXPERT select ARCH_HAVE_NMI_SAFE_CMPXCHG select ARCH_HAVE_EXTRA_ELF_NOTES diff --git a/arch/x86/configs/x86_64_defconfig b/arch/x86/configs/x86_64_defconfig index 269f7d808be4..e6896aeb77d8 100644 --- a/arch/x86/configs/x86_64_defconfig +++ b/arch/x86/configs/x86_64_defconfig @@ -263,7 +263,7 @@ CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_PRINTK_TIME=y CONFIG_DEBUG_KERNEL=y CONFIG_MAGIC_SYSRQ=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_DEBUG_STACK_USAGE=y CONFIG_SCHEDSTATS=y CONFIG_BLK_DEV_IO_TRACE=y diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index d551120a7c88..ef0252a09c28 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -41,10 +41,10 @@ void ptdump_walk_user_pgd_level_checkwx(void); #define pgprot_encrypted(prot) __pgprot(cc_mkenc(pgprot_val(prot))) #define pgprot_decrypted(prot) __pgprot(cc_mkdec(pgprot_val(prot))) -#ifdef CONFIG_DEBUG_WX -#define debug_checkwx_user() ptdump_walk_user_pgd_level_checkwx() +#ifdef CONFIG_CHECK_WX +#define pgtable_checkwx_user() ptdump_walk_user_pgd_level_checkwx() #else -#define debug_checkwx_user() do { } while (0) +#define pgtable_checkwx_user() do { } while (0) #endif extern spinlock_t pgd_lock; diff --git a/arch/x86/mm/pti.c b/arch/x86/mm/pti.c index 598f553cc871..31055ee6f1de 100644 --- a/arch/x86/mm/pti.c +++ b/arch/x86/mm/pti.c @@ -688,5 +688,5 @@ void pti_finalize(void) pti_clone_entry_text(true); pti_clone_kernel_text(); - debug_checkwx_user(); + pgtable_checkwx_user(); } diff --git a/include/linux/ptdump.h b/include/linux/ptdump.h index 240bd3bff18d..af18d1459b2f 100644 --- a/include/linux/ptdump.h +++ b/include/linux/ptdump.h @@ -31,9 +31,9 @@ bool ptdump_walk_pgd_level_core(struct seq_file *m, void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm, pgd_t *pgd); bool ptdump_check_wx(void); -static inline void debug_checkwx(void) +static inline void pgtable_checkwx(void) { - if (IS_ENABLED(CONFIG_DEBUG_WX)) + if (IS_ENABLED(CONFIG_CHECK_WX)) ptdump_check_wx(); } diff --git a/init/main.c b/init/main.c index 31f2bf54976a..a87a3d52f3e2 100644 --- a/init/main.c +++ b/init/main.c @@ -1535,7 +1535,7 @@ static void mark_readonly(void) flush_module_init_free_work(); jump_label_init_ro(); mark_rodata_ro(); - debug_checkwx(); + pgtable_checkwx(); rodata_test(); } else if (IS_ENABLED(CONFIG_STRICT_KERNEL_RWX)) { pr_info("Kernel memory protection disabled.\n"); diff --git a/kernel/configs/debug.config b/kernel/configs/debug.config index 307c97ac5fa9..ac878669c193 100644 --- a/kernel/configs/debug.config +++ b/kernel/configs/debug.config @@ -50,7 +50,7 @@ CONFIG_DEBUG_NET=y # CONFIG_DEBUG_PAGEALLOC is not set # CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is not set # CONFIG_DEBUG_RODATA_TEST is not set -# CONFIG_DEBUG_WX is not set +# CONFIG_CHECK_WX is not set # CONFIG_KFENCE is not set # CONFIG_PAGE_POISONING is not set # CONFIG_SLUB_STATS is not set diff --git a/mm/Kconfig b/mm/Kconfig index edb4a6c0a870..acefc994d9a8 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1494,6 +1494,47 @@ config LAZY_MMU_MODE_KUNIT_TEST If unsure, say N. +config ARCH_HAS_CHECK_WX + bool + +config CHECK_WX + bool "Warn on W+X mappings at boot" + default STRICT_KERNEL_RWX + depends on ARCH_HAS_CHECK_WX + depends on ARCH_HAS_PTDUMP + depends on MMU + select PTDUMP + help + Generate a warning if any W+X mappings are found at boot. + + This is useful for discovering cases where the kernel is leaving W+X + mappings after applying NX, as such mappings are a security risk. + + Look for a message in dmesg output like this: + + /mm: Checked W+X mappings: passed, no W+X pages found. + + or like this, if the check failed: + + /mm: Checked W+X mappings: failed, W+X pages found. + + Note that even if the check fails, your kernel is possibly + still fine, as W+X mappings are not a security hole in + themselves, what they do is that they make the exploitation + of other unfixed kernel bugs easier. + + There is no runtime or memory usage effect of this option + once the kernel has booted up - it's a one time check. + + If in doubt, say "Y". + +config ARCH_HAS_PTDUMP + bool + +config PTDUMP + bool + + source "mm/damon/Kconfig" endmenu diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 9eaa25d1cf23..9ae75229c76d 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -180,45 +180,6 @@ config DEBUG_RODATA_TEST help This option enables a testcase for the setting rodata read-only. -config ARCH_HAS_DEBUG_WX - bool - -config DEBUG_WX - bool "Warn on W+X mappings at boot" - depends on ARCH_HAS_DEBUG_WX - depends on ARCH_HAS_PTDUMP - depends on MMU - select PTDUMP - help - Generate a warning if any W+X mappings are found at boot. - - This is useful for discovering cases where the kernel is leaving W+X - mappings after applying NX, as such mappings are a security risk. - - Look for a message in dmesg output like this: - - /mm: Checked W+X mappings: passed, no W+X pages found. - - or like this, if the check failed: - - /mm: Checked W+X mappings: failed, W+X pages found. - - Note that even if the check fails, your kernel is possibly - still fine, as W+X mappings are not a security hole in - themselves, what they do is that they make the exploitation - of other unfixed kernel bugs easier. - - There is no runtime or memory usage effect of this option - once the kernel has booted up - it's a one time check. - - If in doubt, say "Y". - -config ARCH_HAS_PTDUMP - bool - -config PTDUMP - bool - config PTDUMP_DEBUGFS bool "Export kernel pagetable layout to userspace via debugfs" depends on DEBUG_KERNEL From 499baaff382aaccb67fbceea92edebb8fa4ba876 Mon Sep 17 00:00:00 2001 From: Palla Raghunath Date: Fri, 25 Sep 2026 21:54:49 +0100 Subject: [PATCH 030/213] mm/vmalloc: do not warn on -ENOMEM from va_clip() in pcpu_get_vm_areas() When pcpu_get_vm_areas() has to split a free vmap_area in the middle (NE_FIT_TYPE), va_clip() needs an extra vmap_area object. It takes the per-cpu ne_fit_preload_node if one is there, and otherwise falls back to kmem_cache_alloc(GFP_NOWAIT), which may fail and return -ENOMEM. pcpu_get_vm_areas() never preloads, and a single call can do more than one such split: on a NUMA system it places one area per node group, so the first split consumes the preloaded object and the next one depends on the GFP_NOWAIT allocation. That failure is expected and already handled: the recovery path returns the areas clipped so far to the free tree, purges lazily freed areas and retries. But the error is checked with WARN_ON_ONCE(), so a transient allocation failure under memory pressure or fault injection triggers a kernel warning, and a panic with panic_on_warn. syzbot hit this on a two-node VM while creating a per-cpu BPF array map. Keep the WARN_ON_ONCE() for errors other than -ENOMEM, which do indicate a bug, and take the recovery path either way. This matches what commit b9183788a2de ("mm/vmalloc: do not warn on -ENOMEM from va_alloc()") did for the other va_clip() caller. Link: https://lore.kernel.org/20260925205450.21262-1-raghunathpalla.0209@gmail.com Fixes: 1b23ff80b399 ("mm/vmalloc: invoke classify_va_fit_type() in adjust_va_to_fit_type()") Signed-off-by: Palla Raghunath Signed-off-by: Andrew Morton Reported-by: Closes: https://syzkaller.appspot.com/bug?extid=442828bb356b10813a47 Reviewed-by: Andrew Morton Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Dev Jain Cc: Shuah Khan Cc: Brigham Campbell Cc: Baoquan He --- mm/vmalloc.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 22ff0c32137f..4b0bcc998f1d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -5122,9 +5122,14 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, ret = va_clip(&free_vmap_area_root, &free_vmap_area_list, va, start, size); - if (WARN_ON_ONCE(unlikely(ret))) - /* It is a BUG(), but trigger recovery instead. */ + if (unlikely(ret)) { + /* + * -ENOMEM from the GFP_NOWAIT fallback is expected. + * Anything else is a BUG(), but trigger recovery instead. + */ + WARN_ON_ONCE(ret != -ENOMEM); goto recovery; + } /* Allocated area. */ va = vas[area]; From d1450631619d4d6dcfb641bd617d1e42a2861cca Mon Sep 17 00:00:00 2001 From: Zhang Yi Date: Mon, 28 Sep 2026 20:08:30 +0800 Subject: [PATCH 031/213] mm/truncate: align truncation boundaries to mapping minimum folio order Patch series "mm/truncate: fix data loss when truncating straddling large folios", v5. This is the fifth version fixing data loss when truncating straddling large folios caught on the upcomming ext4 + iomap buffered I/O conversion. When truncate_inode_pages_range() punches a hole or truncates a file, truncate_inode_partial_folio() splits a large folio so that the caller can drop the in-range sub-folios while keeping the out-of-range tail intact. This series fixes three distinct problems in that path that can each lose the valid out-of-range tail of a straddling folio, plus a follow-up that clarifies the return value semantics. Patch 01 aligns the truncation boundaries inwards to the mapping minimum folio order in truncate_inode_pages_range(). With a non-zero min_order, folio_split() stops at min_order instead of order 0, so a boundary computed at page granularity can land inside a min-order-aligned sub-folio and the truncate loop drops that whole chunk, valid tail included, causing data loss. Patch 02 looks the end-edge straddler up by its page index through __filemap_get_folio() in truncate_inode_partial_folio(). After the first split the straddler is unlocked and only transiently ref'd in the page cache, so the page pointer derived from the original folio can be freed and reallocated as a different folio in the same mapping, and the mapping check cannot catch it, which may cause incorrect splitting and potential data loss. Patch 03 reworks the contract between truncate_inode_partial_folio() and its callers. If the second split of the straddler fails, the function reported success unconditionally, and the leftover incorrect end position could cause the truncate loop to drop that valid tail. After rework, it tells the caller the exact page range safe to discard via new pstart/pend out-parameters, so the truncate loop never touches a straddling folio that still holds valid out-of-range data. Patch 04 clarifies the return value semantics to "at least one split succeeded", which is all the shmem caller needs to decide whether to reset its scan loop. The second patch fixes a pre-existing race issue that is reachable today, so it is Cc'd to stable. Patches 01 and 03 require a dirty large folio that carries no filesystem private data, so they are not reachable on current filesystems. They were found while developing the upcoming ext4 iomap buffered I/O path. [1] In addition, another two pre-existing issues were found during the development of this series: - On 32-bit systems with a non-zero min_order filesystem, truncating or any operation operation involving folio_next_index() on the folio containing MAX_LFS_FILESIZE can cause the index calculation to overflow, posing an unpredictable risk. [2] - In shmem, at the last call site of truncate_inode_partial_folio() in shmem_undo_range(), the restart-loop check is wrong, which leaves split sub-folios behind. [3] These two issues need to be fixed separately. This patch (of 4): When the mapping has a non-zero minimum folio order (min_order), folio_split() in truncate_inode_partial_folio() stops at min_order instead of order 0, so the sub-folio containing a split point stays aligned to 1 << min_order rather than to a single page. The original boundaries in truncate_inode_pages_range() were based on page granularity, so either boundary could land inside the min_order chunk at its edge, and the truncation loop would drop that whole chunk, valid out-of-range tail included. For example, a 64K (order-4) folio with min_order = 2 (16K) punched from offset 0 to 36K: split @p0 -> [p0-p3, p4-p7, p8-p15] # non-uniform, min_order folio2 = p8-p15 # straddles: p8 in range, p9-p15 tail valid 2nd split of folio2 -> [p8-p11, p12-p15] # success end(old) = p9 # BUG: p9 inside [p8-p11] loop truncates ... p8-p11 # p9-p11's valid tail is lost It has gone unnoticed so far for two reasons. A non-zero min_order is only used by filesystems with a block or sector size larger than the page size, and those either always write back the affected range before punching a hole or truncating, or they carry filesystem private data on dirty folios (e.g. buffer_head), which makes filemap_release_folio() fail and folio_split() abort with -EBUSY, so the folio is never split and the old start/end boundaries remain valid. The bug only becomes reachable on paths that truncate dirty large folios without prior writeback and without filesystem private data, such as the upcoming ext4 iomap buffered I/O path. Align both start (rounded up) and end (rounded down) to the mapping minimum folio order so they always fall on a folio boundary. Link: https://lore.kernel.org/20260928120833.3440834-2-yi.zhang@huaweicloud.com Link: https://lore.kernel.org/linux-fsdevel/a638a8fb-c184-4069-ae33-379ec12cd514@huaweicloud.com/ [1] Link: https://lore.kernel.org/linux-mm/5a454f2a-8ae2-491d-b903-750c945cfb9d@huaweicloud.com/ [2] Link: https://lore.kernel.org/linux-mm/5pthbyxtn7q6xi4fmkofvksmcjzfnujcw2g4fxmxjzfin5pbgf@zui3vcimb4cv/ [3] Fixes: e220917fa5077 ("mm: split a folio in minimum folio order chunks") Signed-off-by: Zhang Yi Signed-off-by: Andrew Morton Reported-by: Joanne Koong Closes: https://lore.kernel.org/linux-mm/CAJnrk1bQYUe6+1ryyJur5EEnZYrC+_5AYsy=OWzVRgD4202y1g@mail.gmail.com/ Suggested-by: Zi Yan Reviewed-by: Jan Kara Reviewed-by: Brian Foster Reviewed-by: Zi Yan Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Hugh Dickins Cc: Baolin Wang Cc: Matthew Wilcox (Oracle) Cc: Darrick J. Wong Cc: Cc: Yang Erkun Cc: Zhihao Cheng Cc: Kefeng Wang Cc: --- mm/truncate.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index b58ba940be47..8a28f4a21267 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -345,9 +345,11 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio) * @lstart: offset from which to truncate * @lend: offset to which to truncate (inclusive) * - * Truncate the page cache, removing the pages that are between - * specified offsets (and zeroing out partial pages - * if lstart or lend + 1 is not page aligned). + * Truncate the page cache, removing the folios that are between specified + * offsets (and zeroing out partial folios if lstart or lend + 1 is not + * folio aligned). For mappings with a non-zero minimum folio order, the + * boundaries are aligned inwards to 1 << min_order so the edge sub-folio + * straddling the range is kept. * * Truncate takes two passes - the first pass is nonblocking. It will not * block on page locks and it will not block on writeback. The second pass @@ -366,6 +368,7 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio) void truncate_inode_pages_range(struct address_space *mapping, loff_t lstart, uoff_t lend) { + pgoff_t min_nrpages = mapping_min_folio_nrpages(mapping); pgoff_t start; /* inclusive */ pgoff_t end; /* exclusive */ struct folio_batch fbatch; @@ -379,9 +382,8 @@ void truncate_inode_pages_range(struct address_space *mapping, return; /* - * 'start' and 'end' always covers the range of pages to be fully - * truncated. Partial pages are covered with 'partial_start' at the - * start of the range and 'partial_end' at the end of the range. + * 'start' and 'end' always covers the range of folios to be fully + * truncated, with both boundaries aligned inwards to 1 << min_order. * Note that 'end' is exclusive while 'lend' is inclusive. */ start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT; @@ -395,6 +397,10 @@ void truncate_inode_pages_range(struct address_space *mapping, else end = (lend + 1) >> PAGE_SHIFT; + start = round_up(start, min_nrpages); + if (end != (pgoff_t)-1) + end = round_down(end, min_nrpages); + folio_batch_init(&fbatch); index = start; while (index < end && find_lock_entries(mapping, &index, end - 1, From f47dee903b9f0f1e4e7181e8a398a7678cabf0a5 Mon Sep 17 00:00:00 2001 From: Zhang Yi Date: Mon, 28 Sep 2026 20:08:31 +0800 Subject: [PATCH 032/213] mm/truncate: look up the end-edge straddler by index In truncate_inode_partial_folio(), after the first split at the start edge, folio_split() unlocks and drops the refcount of the after-split sub-folios. The sub-folio straddling the end of the truncation range is therefore unlocked and only transiently ref'd in the page cache while the code still derives it from a page pointer inside the original folio. Between the first split finishing and page_folio() resolving split_at2, that tail page can be reclaimed, freed and reallocated as a new large folio in the same mapping at a different file offset. folio2 then points at a folio that does not cover the end boundary, yet folio2->mapping == folio->mapping still holds, so the stale pointer passes the mapping check and folio_split_or_unmap() splits a folio at a wrong position (or, with a transient refcount, a use-after-free window opens between try_get and the split). __folio_split()'s own folio != page_folio(split_at) check cannot catch this either since split_at2 has been reallocated as part of the new folio, so page_folio(split_at2) resolves back to folio2. Look the straddler up by its page index instead. __filemap_get_folio() returns the folio currently covering the boundary, ref'd and locked, with the mapping validated under the lock, so the split target is always the real folio at the end edge. Link: https://lore.kernel.org/20260928120833.3440834-3-yi.zhang@huaweicloud.com Link: https://lore.kernel.org/linux-mm/DLGXT0ERY79Z.3C5DYVJVX6S9Z@nvidia.com/ Fixes: 7460b470a131 ("mm/truncate: use folio_split() in truncate operation") Signed-off-by: Zhang Yi Signed-off-by: Andrew Morton Reported-by: Sashiko Closes: https://sashiko.dev/#/message/20260916094500.C30061F00893%40smtp.kernel.org Suggested-by: Jan Kara Suggested-by: Zi Yan Reviewed-by: Jan Kara Reviewed-by: Brian Foster Acked-by: Zi Yan Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Hugh Dickins Cc: Baolin Wang Cc: Matthew Wilcox (Oracle) Cc: Joanne Koong Cc: Darrick J. Wong Cc: Cc: Yang Erkun Cc: Zhihao Cheng Cc: Kefeng Wang Cc: Cc: --- mm/truncate.c | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 8a28f4a21267..f252619ae6c3 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -259,30 +259,32 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) * for shmem truncate */ struct folio *folio2; + pgoff_t end_idx; if (offset + length == size) goto no_split; - split_at2 = folio_page(folio, - PAGE_ALIGN_DOWN(offset + length) / PAGE_SIZE); - folio2 = page_folio(split_at2); - - if (!folio_try_get(folio2)) + /* + * After the first split at the start edge, the folio at the + * end edge may be freed and reused concurrently. + * __filemap_get_folio() looks up the straddler at end_idx + * and returns it locked and ref'd with the mapping + * validated. + */ + end_idx = (pos + offset + length) >> PAGE_SHIFT; + folio2 = __filemap_get_folio(folio->mapping, end_idx, + FGP_LOCK | FGP_NOWAIT, 0); + if (IS_ERR(folio2)) goto no_split; + /* make sure folio2 is large */ if (!folio_test_large(folio2)) goto out; - if (!folio_trylock(folio2)) - goto out; - - /* make sure folio2 is large and does not change its mapping */ - if (folio_test_large(folio2) && - folio2->mapping == folio->mapping) - folio_split_or_unmap(folio2, split_at2, min_order); - - folio_unlock(folio2); + split_at2 = folio_page(folio2, (end_idx - folio2->index)); + folio_split_or_unmap(folio2, split_at2, min_order); out: + folio_unlock(folio2); folio_put(folio2); no_split: return true; From ad2747979368278ec0193d71574ba5f1a225d409 Mon Sep 17 00:00:00 2001 From: Zhang Yi Date: Mon, 28 Sep 2026 20:08:32 +0800 Subject: [PATCH 033/213] mm/truncate: fix data loss when splitting straddling large folios fails truncate_inode_partial_folio() splits a large folio so that the caller's truncate loop can drop the in-range sub-folios while keeping the out-of-range tail. The first split at the punch start edge is non-uniform, which leaves the sub-folio at the truncation end edge as large as possible, this means it may still straddle the range, holding both zeroed in-range and valid out-of-range data. The function then attempts a second split at offset + length to isolate that tail. If the second split fails the straddling sub-folio stays merged. The function returned true unconditionally on all exit paths of the success block, telling the caller it was fully handled. The caller kept its default end and the truncate loop truncated every sub-folio below it, including the merged straddler, discarding the valid out-of-range tail. For example, a 4-page order-2 folio punched from offset 0 to the middle of the last page: truncate_inode_pages_range() truncate_inode_partial_folio() # same_folio == true 1st split at page0 -> [p0, p1, p2-3] # non-uniform, success folio2 = p2-3 # straddles: p2 zeroed, p3 tail valid 2nd split of folio2 fails / cannot lock return true # BUG: caller keeps default end end = 3 loop truncates p0, p1, p2-3 # p3's valid tail is lost This became reachable after commit 7460b470a131 ("mm/truncate: use folio_split() in truncate operation") replaced the atomic split_folio() with folio_split(), whose non-uniform split can partially split a folio and leave the end edge merged. It has gone unnoticed because a dirty large folio normally carries the filesystem's private data, for example buffer_head, so filemap_release_folio() fails on a dirty folio and folio_split() aborts with -EBUSY before any split, leaving the straddler safely unsplit. The bug is only reachable on paths that produce dirty large folios without filesystem private data, and it was caught on the upcoming ext4 iomap buffered I/O path when no ifs is attached. Rework the contract so the caller is told the folio range to discard: - Add pgoff_t *pstart and *pend out-parameters that receive the folio range fully covered by [lstart, lend] after any split (or none), aligned inwards to min_order, i.e. the folios wholly within the range and safe to discard. - Report a reliable end position to the caller. The straddler is looked up at an index aligned inwards to the mapping minimum folio order, and *pend is set to that boundary on success. If nothing covers the boundary, discarding up to it stays safe. If the straddler is locked by someone else, fall back to folio->index. This best-effort fallback may leave the in-range sub-folios to a later pass but never discards the out-of-range tail. If the straddler cannot be split, fall back to folio2->index so the caller keeps the out-of-range tail. - Rename the byte-range parameters start/end to lstart/lend to better express their semantics. Callers in truncate_inode_pages_range() and shmem_undo_range() pass &pstart for the folio at the start edge and &pend for the folio at the end edge, so the truncate loop drops exactly the fully covered pages and never touches a straddling folio that still holds valid out-of-range data. Link: https://lore.kernel.org/20260928120833.3440834-4-yi.zhang@huaweicloud.com Fixes: 7460b470a131 ("mm/truncate: use folio_split() in truncate operation") Signed-off-by: Zhang Yi Signed-off-by: Andrew Morton Suggested-by: Brian Foster Link: https://lore.kernel.org/linux-fsdevel/anH-WKA1coW6wtfG@bfoster/ Reviewed-by: Jan Kara Reviewed-by: Brian Foster Acked-by: Zi Yan Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Hugh Dickins Cc: Baolin Wang Cc: Matthew Wilcox (Oracle) Cc: Joanne Koong Cc: Darrick J. Wong Cc: Cc: Yang Erkun Cc: Zhihao Cheng Cc: Kefeng Wang Cc: --- mm/internal.h | 4 +-- mm/shmem.c | 13 +++---- mm/truncate.c | 93 +++++++++++++++++++++++++++++++++++---------------- 3 files changed, 71 insertions(+), 39 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 05179c4b2090..704f3e3ef2ae 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -611,8 +611,8 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start, unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); int truncate_inode_folio(struct address_space *mapping, struct folio *folio); -bool truncate_inode_partial_folio(struct folio *folio, loff_t start, - loff_t end); +bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, + loff_t lend, pgoff_t *pstart, pgoff_t *pend); long mapping_evict_folio(struct address_space *mapping, struct folio *folio); unsigned long mapping_try_invalidate(struct address_space *mapping, pgoff_t start, pgoff_t end, unsigned long *nr_failed); diff --git a/mm/shmem.c b/mm/shmem.c index 9f0965b8b59c..fadf23f9e21e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1386,11 +1386,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (folio) { same_folio = lend < folio_next_pos(folio); folio_mark_dirty(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) { - start = folio_next_index(folio); - if (same_folio) - end = folio->index; - } + truncate_inode_partial_folio(folio, lstart, lend, &start, + same_folio ? &end : NULL); folio_unlock(folio); folio_put(folio); folio = NULL; @@ -1400,8 +1397,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, folio = shmem_get_partial_folio(inode, lend >> PAGE_SHIFT); if (folio) { folio_mark_dirty(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) - end = folio->index; + truncate_inode_partial_folio(folio, lstart, lend, NULL, &end); folio_unlock(folio); folio_put(folio); } @@ -1469,7 +1465,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (!folio_test_large(folio)) { truncate_inode_folio(mapping, folio); - } else if (truncate_inode_partial_folio(folio, lstart, lend)) { + } else if (truncate_inode_partial_folio(folio, + lstart, lend, NULL, NULL)) { /* * If we split a page, reset the loop so * that we pick up the new sub pages. diff --git a/mm/truncate.c b/mm/truncate.c index f252619ae6c3..9eb108769149 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -206,30 +206,43 @@ static int folio_split_or_unmap(struct folio *folio, struct page *split_at, /* * Handle partial folios. The folio may be entirely within the * range if a split has raced with us. If not, we zero the part of the - * folio that's within the [start, end] range, and then split the folio if + * folio that's within the [lstart, lend] range, and then split the folio if * it's large. split_page_range() will discard pages which now lie beyond * i_size, and we rely on the caller to discard pages which lie within a * newly created hole. * + * When @pstart and/or @pend are non-NULL they receive the indexes of the + * folio range fully covered by [lstart, lend] after any split (or none), + * aligned inwards to min_order, i.e. the range of folios wholly within + * [lstart, lend] and so safe to discard. + * * Returns false if splitting failed so the caller can avoid * discarding the entire folio which is stubbornly unsplit. */ -bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) +bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, + loff_t lend, pgoff_t *pstart, pgoff_t *pend) { loff_t pos = folio_pos(folio); size_t size = folio_size(folio); unsigned int offset, length; struct page *split_at, *split_at2; + unsigned long min_nrbytes; unsigned int min_order; - if (pos < start) - offset = start - pos; + if (pos < lstart) + offset = lstart - pos; else offset = 0; - if (pos + size <= (u64)end) + if (pos + size <= (u64)lend) length = size - offset; else - length = end + 1 - pos - offset; + length = lend + 1 - pos - offset; + + if (pstart) + *pstart = offset ? folio_next_index(folio) : folio->index; + if (pend) + *pend = (pos + size > (u64)lend) ? folio->index : + folio_next_index(folio); folio_wait_writeback(folio); if (length == size) { @@ -247,10 +260,12 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) if (folio_needs_release(folio)) folio_invalidate(folio, offset, length); - if (!folio_test_large(folio)) - return true; min_order = mapping_min_folio_order(folio->mapping); + min_nrbytes = mapping_min_folio_nrbytes(folio->mapping); + if (folio_order(folio) == min_order) + return true; + split_at = folio_page(folio, PAGE_ALIGN_DOWN(offset) / PAGE_SIZE); if (!folio_split_or_unmap(folio, split_at, min_order)) { /* @@ -259,34 +274,57 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) * for shmem truncate */ struct folio *folio2; - pgoff_t end_idx; + pgoff_t end, aligned_end = round_down(pos + offset + length, + min_nrbytes) >> PAGE_SHIFT; + if (pstart) + *pstart = round_up(pos + offset, + min_nrbytes) >> PAGE_SHIFT; + + end = aligned_end; if (offset + length == size) - goto no_split; + goto out; /* * After the first split at the start edge, the folio at the * end edge may be freed and reused concurrently. - * __filemap_get_folio() looks up the straddler at end_idx + * __filemap_get_folio() looks up the straddler at aligned_end * and returns it locked and ref'd with the mapping * validated. */ - end_idx = (pos + offset + length) >> PAGE_SHIFT; - folio2 = __filemap_get_folio(folio->mapping, end_idx, + folio2 = __filemap_get_folio(folio->mapping, aligned_end, FGP_LOCK | FGP_NOWAIT, 0); - if (IS_ERR(folio2)) - goto no_split; - - /* make sure folio2 is large */ - if (!folio_test_large(folio2)) + if (IS_ERR(folio2)) { + /* + * No sub-folio straddles the boundary when + * aligned_end is empty, so discarding up to it is + * safe. Otherwise the straddler is locked by + * someone else and we cannot obtain a reliable end + * position, so we fall back to folio->index, which + * is safe but leaves the sub-folios split off at + * the offset edge in the page cache. + */ + if (PTR_ERR(folio2) != -ENOENT) + end = folio->index; goto out; + } - split_at2 = folio_page(folio2, (end_idx - folio2->index)); - folio_split_or_unmap(folio2, split_at2, min_order); -out: + /* Already at the minimum order, nothing to split */ + if (folio_order(folio2) == min_order) + goto out_put; + + split_at2 = folio_page(folio2, (aligned_end - folio2->index)); + + /* Split failed, keep the straddler intact */ + if (folio_split_or_unmap(folio2, split_at2, min_order)) + end = folio2->index; + +out_put: folio_unlock(folio2); folio_put(folio2); -no_split: +out: + if (pend) + *pend = end; return true; } if (folio_test_dirty(folio)) @@ -421,11 +459,8 @@ void truncate_inode_pages_range(struct address_space *mapping, folio = __filemap_get_folio(mapping, lstart >> PAGE_SHIFT, FGP_LOCK, 0); if (!IS_ERR(folio)) { same_folio = lend < folio_next_pos(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) { - start = folio_next_index(folio); - if (same_folio) - end = folio->index; - } + truncate_inode_partial_folio(folio, lstart, lend, &start, + same_folio ? &end : NULL); folio_unlock(folio); folio_put(folio); folio = NULL; @@ -435,8 +470,8 @@ void truncate_inode_pages_range(struct address_space *mapping, folio = __filemap_get_folio(mapping, lend >> PAGE_SHIFT, FGP_LOCK, 0); if (!IS_ERR(folio)) { - if (!truncate_inode_partial_folio(folio, lstart, lend)) - end = folio->index; + truncate_inode_partial_folio(folio, lstart, lend, + NULL, &end); folio_unlock(folio); folio_put(folio); } From f9be69fcacf01e945df4c68b488474cf4b6e7181 Mon Sep 17 00:00:00 2001 From: Zhang Yi Date: Mon, 28 Sep 2026 20:08:33 +0800 Subject: [PATCH 034/213] mm/truncate: clarify return value of truncate_inode_partial_folio() With the earlier rework the callers no longer rely on the return value of truncate_inode_partial_folio() to decide whether to adjust the truncation range. The pstart/pend out-parameters carry that information instead. The callers now only use the return value as a flag indicating whether the loop should be reset to pick up newly split sub-folios on the shmem path. Return true if at least one split succeeded, and false otherwise. This clarifies the existing confusing return value semantics. Link: https://lore.kernel.org/20260928120833.3440834-5-yi.zhang@huaweicloud.com Signed-off-by: Zhang Yi Signed-off-by: Andrew Morton Reviewed-by: Jan Kara Reviewed-by: Brian Foster Acked-by: Zi Yan Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Hugh Dickins Cc: Baolin Wang Cc: Matthew Wilcox (Oracle) Cc: Joanne Koong Cc: Darrick J. Wong Cc: Cc: Yang Erkun Cc: Zhihao Cheng Cc: Kefeng Wang Cc: --- mm/truncate.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 9eb108769149..5b1b13cf8ff0 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -216,8 +216,7 @@ static int folio_split_or_unmap(struct folio *folio, struct page *split_at, * aligned inwards to min_order, i.e. the range of folios wholly within * [lstart, lend] and so safe to discard. * - * Returns false if splitting failed so the caller can avoid - * discarding the entire folio which is stubbornly unsplit. + * Return %true if at least one split succeeded, %false otherwise. */ bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, loff_t lend, pgoff_t *pstart, pgoff_t *pend) @@ -247,7 +246,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, folio_wait_writeback(folio); if (length == size) { truncate_inode_folio(folio->mapping, folio); - return true; + return false; } /* @@ -264,7 +263,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, min_order = mapping_min_folio_order(folio->mapping); min_nrbytes = mapping_min_folio_nrbytes(folio->mapping); if (folio_order(folio) == min_order) - return true; + return false; split_at = folio_page(folio, PAGE_ALIGN_DOWN(offset) / PAGE_SIZE); if (!folio_split_or_unmap(folio, split_at, min_order)) { @@ -327,10 +326,9 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, *pend = end; return true; } - if (folio_test_dirty(folio)) - return false; - truncate_inode_folio(folio->mapping, folio); - return true; + if (!folio_test_dirty(folio)) + truncate_inode_folio(folio->mapping, folio); + return false; } /* From f73ecbc0589c8df8f18e444f5f633de01ebb0d98 Mon Sep 17 00:00:00 2001 From: Karl Mehltretter Date: Mon, 28 Sep 2026 01:58:33 -0700 Subject: [PATCH 035/213] mm/damon/core: preserve the quota passed to damon_new_scheme() Patch series "mm/damon/core: preserve quota state when constructing schemes", v3. damon_commit_ctx() first commits the running context's parameters to a temporary context for validating proposed updates. Constructing the temporary schemes clears the running schemes' quota state because damon_new_scheme() initializes the quota passed as a parameter before copying it to the new scheme. Even an update later rejected with -EINVAL loses the running quota state. Initialize the new scheme's copy instead, and add KUnit tests for the constructor and for accepted and rejected context updates. Note: below are test results and changelogs that could be removed from the final commit log. In the v1 live test with damo, a scheme with a plain 64 KiB size quota and a 60-second reset interval uses its quota, and a full "damo tune" with unchanged parameters then lets it try another 64 KiB within the same window. With the fix, sz_tried stays at 64 KiB. KUnit was rerun after the rebase on x86-64 and i386. With only patch 2 applied, the two new tests fail. With the fix, all 46 DAMON KUnit tests pass on both architectures. The v1 DAMON selftests showed no new failures (QEMU TCG guest; the wss_estimation test missed its accuracy bounds with and without the fix). This patch (of 2): damon_commit_ctx() first commits the running context's parameters to a temporary context for validating proposed updates. When damon_commit_schemes() creates the temporary schemes, it passes the running scheme's quota as the quota parameter of damon_new_scheme(). damon_new_scheme() calls damos_quota_init() on that quota before copying it to the new scheme. This clears the running scheme's effective quota, feedback input and charging state. Even an update rejected with -EINVAL loses the running quota state. For a size quota, this discards the bytes already charged and allows the scheme to use a fresh quota before the reset interval has elapsed. For a goal-driven quota, the consist tuner loses its accumulated input and restarts from its minimum input. A time quota loses its throughput estimate and falls back to the initial estimate. The end users will show DAMOS works more or less aggressively than expected for online-commit updates of quotas. DAMON provides best efforts by default. DAMON parameters online commit is supposed to be executed only occasionally. Hence, the issue wouldn't be critical on sane setups. For user_input type quota goals, online commit of the user input score is expected to be frequent. But, for the case commit_schemes_quota_goals command is recommended for optimal execution, and it doesn't have this bug. Commit 60bd24f272d0 ("mm/damon/sysfs: test commit input against realistic destination") introduced this problem in v6.19 when sysfs validation began committing the running context's parameters to a temporary context. Commit b90408ef1163 ("mm/damon/core: safely validate src on damon_commit_ctx()") later moved that validation into the core API, exposing other callers including DAMON_RECLAIM and DAMON_LRU_SORT. Sashiko reported the same side effect [1] on the RFC of the core API change. Copy the quota to the new scheme first, then initialize that copy. Make damos_quota_init() return void, since its return value is no longer needed. Link: https://lore.kernel.org/20260928085835.7675-1-sj@kernel.org Link: https://lore.kernel.org/20260928085835.7675-2-sj@kernel.org Link: https://lore.kernel.org/r/20260702212143.0CB6D1F00A3D@smtp.kernel.org/ [1] Fixes: 60bd24f272d0 ("mm/damon/sysfs: test commit input against realistic destination") Signed-off-by: Karl Mehltretter Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Assisted-by: LLM Cc: Bijan Tabatabai Cc: --- mm/damon/core.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/damon/core.c b/mm/damon/core.c index 733025b36745..3a61e5fdfb62 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -734,7 +734,7 @@ static bool damos_quota_goals_empty(struct damos_quota *q) } /* initialize fields of @quota that normally API users wouldn't set */ -static struct damos_quota *damos_quota_init(struct damos_quota *quota) +static void damos_quota_init(struct damos_quota *quota) { quota->esz = 0; quota->total_charged_sz = 0; @@ -744,7 +744,6 @@ static struct damos_quota *damos_quota_init(struct damos_quota *quota) quota->charge_target_from = NULL; quota->charge_addr_from = 0; quota->esz_bp = 0; - return quota; } struct damos *damon_new_scheme(struct damos_access_pattern *pattern, @@ -776,7 +775,8 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, scheme->last_applied = NULL; INIT_LIST_HEAD(&scheme->list); - scheme->quota = *(damos_quota_init(quota)); + scheme->quota = *quota; + damos_quota_init(&scheme->quota); /* quota.goals should be separately set by caller */ INIT_LIST_HEAD(&scheme->quota.goals); From f262c5b460de447511a4725e1ee433a74f772730 Mon Sep 17 00:00:00 2001 From: Karl Mehltretter Date: Mon, 28 Sep 2026 01:58:34 -0700 Subject: [PATCH 036/213] mm/damon/tests/core-kunit: test preservation of quota state Check that damon_new_scheme() initializes the new scheme's quota without changing the quota passed as a parameter. Cover all eight fields initialized by damos_quota_init(). Also check that damon_commit_ctx() preserves those fields in the destination scheme for both accepted and rejected parameter updates. Use an invalid min_region_sz for the rejected update and confirm that returning -EINVAL leaves the running quota state unchanged. Without the preceding fix, all eight fields are cleared in the constructor test and in both context update cases. Link: https://lore.kernel.org/20260928085835.7675-3-sj@kernel.org Signed-off-by: Karl Mehltretter Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Assisted-by: LLM Cc: Bijan Tabatabai Cc: Brendan Higgins Cc: David Gow --- mm/damon/tests/core-kunit.h | 107 ++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index df84d9cc7d20..ce17f4458308 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -812,6 +812,48 @@ static void damos_test_new_filter(struct kunit *test) damos_destroy_filter(filter); } +static void damos_test_new_scheme_keeps_src_quota(struct kunit *test) +{ + struct damos_access_pattern pattern = {}; + struct damon_target target = {}; + struct damos_quota quota = { + .sz = SZ_64K, + .esz = 123, + .esz_bp = 456, + .total_charged_sz = 789, + .total_charged_ns = 1011, + .charged_sz = 12, + .charged_from = 13, + .charge_target_from = &target, + .charge_addr_from = 14, + }; + struct damos_watermarks wmarks = {}; + struct damos *s; + + s = damon_new_scheme(&pattern, DAMOS_STAT, 0, "a, &wmarks, + NUMA_NO_NODE); + if (!s) + kunit_skip(test, "scheme alloc fail"); + KUNIT_EXPECT_EQ(test, s->quota.sz, (unsigned long)SZ_64K); + KUNIT_EXPECT_EQ(test, s->quota.esz, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.esz_bp, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_sz, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_ns, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_sz, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_from, 0ul); + KUNIT_EXPECT_PTR_EQ(test, s->quota.charge_target_from, NULL); + KUNIT_EXPECT_EQ(test, s->quota.charge_addr_from, 0ul); + KUNIT_EXPECT_EQ(test, quota.esz, 123ul); + KUNIT_EXPECT_EQ(test, quota.esz_bp, 456ul); + KUNIT_EXPECT_EQ(test, quota.total_charged_sz, 789ul); + KUNIT_EXPECT_EQ(test, quota.total_charged_ns, 1011ul); + KUNIT_EXPECT_EQ(test, quota.charged_sz, 12ul); + KUNIT_EXPECT_EQ(test, quota.charged_from, 13ul); + KUNIT_EXPECT_PTR_EQ(test, quota.charge_target_from, &target); + KUNIT_EXPECT_EQ(test, quota.charge_addr_from, 14ul); + damon_destroy_scheme(s); +} + static void damos_test_commit_quota_goal_for(struct kunit *test, struct damos_quota_goal *dst, struct damos_quota_goal *src) @@ -1572,6 +1614,69 @@ static void damon_test_commit_ctx(struct kunit *test) damon_destroy_ctx(dst); } +static void damon_test_commit_ctx_keeps_quota_for(struct kunit *test, + unsigned long min_region_sz, int expected_err) +{ + struct damos_access_pattern pattern = {}; + struct damos_quota quota = {.sz = SZ_64K}; + struct damos_watermarks wmarks = {}; + struct damon_ctx *src, *dst; + struct damon_target *target; + struct damos *s; + + dst = damon_new_ctx(); + if (!dst) + kunit_skip(test, "dst alloc fail"); + target = damon_new_target(); + if (!target) { + damon_destroy_ctx(dst); + kunit_skip(test, "target alloc fail"); + } + damon_add_target(dst, target); + s = damon_new_scheme(&pattern, DAMOS_STAT, 0, "a, &wmarks, + NUMA_NO_NODE); + if (!s) { + damon_destroy_ctx(dst); + kunit_skip(test, "scheme alloc fail"); + } + damon_add_scheme(dst, s); + + /* Copy the parameters before populating dst's runtime quota state. */ + src = damon_new_test_ctx(dst); + if (!src) { + damon_destroy_ctx(dst); + kunit_skip(test, "src alloc fail"); + } + src->min_region_sz = min_region_sz; + s->quota.esz = 123; + s->quota.esz_bp = 456; + s->quota.total_charged_sz = 789; + s->quota.total_charged_ns = 1011; + s->quota.charged_sz = 12; + s->quota.charged_from = 13; + s->quota.charge_target_from = target; + s->quota.charge_addr_from = 14; + + KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), expected_err); + KUNIT_EXPECT_EQ(test, s->quota.esz, 123ul); + KUNIT_EXPECT_EQ(test, s->quota.esz_bp, 456ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_sz, 789ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_ns, 1011ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_sz, 12ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_from, 13ul); + KUNIT_EXPECT_PTR_EQ(test, s->quota.charge_target_from, target); + KUNIT_EXPECT_EQ(test, s->quota.charge_addr_from, 14ul); + damon_destroy_ctx(src); + damon_destroy_ctx(dst); +} + +static void damon_test_commit_ctx_keeps_quota(struct kunit *test) +{ + /* Only power of two min_region_sz is allowed. */ + damon_test_commit_ctx_keeps_quota_for(test, 4096, 0); + damon_test_commit_ctx_keeps_quota_for(test, 4095, -EINVAL); +} + static void damon_test_valid_probe_params(struct kunit *test) { struct damon_ctx *ctx; @@ -2259,6 +2364,7 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_mvsum), KUNIT_CASE(damon_test_nr_accesses_mvsum), KUNIT_CASE(damos_test_new_filter), + KUNIT_CASE(damos_test_new_scheme_keeps_src_quota), KUNIT_CASE(damos_test_commit_quota_goal), KUNIT_CASE(damos_test_commit_quota_goals), KUNIT_CASE(damos_test_commit_quota), @@ -2270,6 +2376,7 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_commit_filter), KUNIT_CASE(damon_test_commit_probes), KUNIT_CASE(damon_test_commit_ctx), + KUNIT_CASE(damon_test_commit_ctx_keeps_quota), KUNIT_CASE(damon_test_valid_probe_params), KUNIT_CASE(damos_test_filter_out), KUNIT_CASE(damos_test_apply_scheme_filtered_sz), From 4293c54b72e75a9d8d2e79b1498cbc782267a4db Mon Sep 17 00:00:00 2001 From: Donggeun Yoo Date: Mon, 28 Sep 2026 01:48:14 -0700 Subject: [PATCH 037/213] mm/damon/core: prevent size quota overflow in the temporal goal tuner Patch series "mm/damon: fix the temporal goal tuner's size quota conversion", v5. damos_goal_tune_esz_bp_temporal() hands the size quota to damos_set_effective_quota() through quota->esz_bp in basis points, and the multiply that gets it there is unchecked. On 32-bit it wraps above 429496 bytes, and a wrapped product below 10000 divides to a zero effective quota. damos_quota_is_full() is then true on the first test of every charge window, so the scheme makes no progress for as long as the goal is unachieved. Patch 1 bounds the conversion. Patch 2 pins the boundary in the core kunit suite, where the new test would fail without patch 1 on any word size. This patch (of 2): damos_goal_tune_esz_bp_temporal() converts the scheme's size quota into basis points with "quota->esz_bp = quota->sz * 10000", both unsigned long, and damos_set_effective_quota() divides the result back by 10000. quotas/bytes is unbounded; bytes_store() hands it to kstrtoul() as is. On 32-bit the product wraps for any size quota above ULONG_MAX / 10000, that is 429496 bytes. A wrapped product below 10000 divides to a zero effective quota: 429497 gives 0. damos_quota_is_full() is then true on the first test of every charge window. Other wrapped values are wrong without being zero: 500000 gives 70503. Triggering this needs a scheme with a quota goal, the temporal goal tuner, and a size quota above ULONG_MAX / 10000 -- 429496 bytes on 32-bit, 1844674407370955 on 64-bit. The scheme then makes no progress for as long as the goal is unachieved, which is easy to notice, and writing a smaller size quota restores it. Nothing is corrupted and nothing leaks. This is unlikely to be hit on a tested setup. addr_unit does not cover this. It only scales the numbers a paddr context writes to quotas/bytes, so a large enough scaled value wraps just the same, and vaddr and fvaddr contexts take raw byte values. Bound the multiply. Link: https://lore.kernel.org/20260928084816.5575-1-sj@kernel.org Link: https://lore.kernel.org/20260928084816.5575-2-sj@kernel.org Fixes: af738a6a00c1 ("mm/damon/core: introduce DAMOS_QUOTA_GOAL_TUNER_TEMPORAL") Signed-off-by: Donggeun Yoo Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Cc: # 7.1.x --- mm/damon/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/damon/core.c b/mm/damon/core.c index 3a61e5fdfb62..46ef570a4766 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -3291,7 +3291,7 @@ static void damos_goal_tune_esz_bp_temporal(struct damon_ctx *c, if (score >= 10000) quota->esz_bp = 0; - else if (quota->sz) + else if (quota->sz && quota->sz <= ULONG_MAX / 10000) quota->esz_bp = quota->sz * 10000; else quota->esz_bp = ULONG_MAX; From 59a484182018cb0fce06a90c8b712af001b0f9ff Mon Sep 17 00:00:00 2001 From: Donggeun Yoo Date: Mon, 28 Sep 2026 01:48:15 -0700 Subject: [PATCH 038/213] mm/damon/tests/core-kunit: test the temporal tuner's size quota conversion damos_goal_tune_esz_bp_temporal() encodes the size quota in basis points, so the conversion is exact only up to ULONG_MAX / 10000. Pin the three sizes around that boundary: the largest one that fits, the first one that does not, and ULONG_MAX. Link: https://lore.kernel.org/20260928084816.5575-3-sj@kernel.org Signed-off-by: Donggeun Yoo Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Cc: Brendan Higgins Cc: David Gow --- mm/damon/tests/core-kunit.h | 48 +++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index ce17f4458308..a8e22246facf 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -2347,6 +2347,53 @@ static void damon_test_rand(struct kunit *test) } } +static void damos_test_esz_goal_temporal(struct kunit *test) +{ + struct damos_access_pattern pattern = {}; + struct damos_watermarks wmarks = {}; + struct damos_quota quota = { + .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL, + }; + struct damos_quota_goal *goal; + struct damon_ctx *ctx; + struct damos *s; + + ctx = damon_new_ctx(); + KUNIT_ASSERT_NOT_NULL(test, ctx); + + s = damon_new_scheme(&pattern, DAMOS_STAT, 0, "a, &wmarks, + NUMA_NO_NODE); + if (!s) { + damon_destroy_ctx(ctx); + kunit_skip(test, "scheme alloc fail"); + } + damon_add_scheme(ctx, s); + + goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, 10000); + if (!goal) { + damon_destroy_ctx(ctx); + kunit_skip(test, "quota goal alloc fail"); + } + goal->current_value = 0; + damos_add_quota_goal(&s->quota, goal); + + /* The largest size quota the basis-point conversion can hold. */ + s->quota.sz = ULONG_MAX / 10000; + damos_set_effective_quota(ctx, s); + KUNIT_EXPECT_EQ(test, s->quota.esz, ULONG_MAX / 10000); + + /* Any larger one saturates instead of wrapping. */ + s->quota.sz = ULONG_MAX / 10000 + 1; + damos_set_effective_quota(ctx, s); + KUNIT_EXPECT_EQ(test, s->quota.esz, ULONG_MAX / 10000); + + s->quota.sz = ULONG_MAX; + damos_set_effective_quota(ctx, s); + KUNIT_EXPECT_EQ(test, s->quota.esz, ULONG_MAX / 10000); + + damon_destroy_ctx(ctx); +} + static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_target), KUNIT_CASE(damon_test_regions), @@ -2388,6 +2435,7 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_is_last_region), KUNIT_CASE(damon_test_walk_control_obsolete), KUNIT_CASE(damon_test_rand), + KUNIT_CASE(damos_test_esz_goal_temporal), {}, }; From 9bdad0994d24f5a0bf106f1e931039c52b85b0e7 Mon Sep 17 00:00:00 2001 From: "Zenghui Yu (Huawei)" Date: Mon, 28 Sep 2026 01:39:55 -0700 Subject: [PATCH 039/213] mm/damon/api: remove unused NR_DAMOS_* enumerators Patch series "mm/damon/core: cleanup code, reduce stack usage, and add kunit". Three various improvements. Patch 1 from Zenghui Yu (Huawei) cleans up unused enums. Patch 2 from Arnd Bergmann reduces kdamond's stack usage. Patch 3 from Karl Mehltretter adds a kunit test for uninitialized PSI quota goal stage logic. This patch (of 3): NR_DAMOS_ACTIONS, NR_DAMOS_QUOTA_GOAL_METRICS, and NR_DAMOS_WMARK_METRICS are not referenced by any code. Remove them. They can be reintroduced if a real user comes up. Link: https://lore.kernel.org/20260928083959.4030-1-sj@kernel.org Link: https://lore.kernel.org/20260928083959.4030-2-sj@kernel.org Signed-off-by: Zenghui Yu (Huawei) Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park --- include/linux/damon.h | 8 -------- 1 file changed, 8 deletions(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 844b175120f0..6a29dc2ac8db 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -117,7 +117,6 @@ struct damon_target { * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions. * @DAMOS_MIGRATE_COLD: Migrate the regions prioritizing colder regions. * @DAMOS_STAT: Do nothing but count the stat. - * @NR_DAMOS_ACTIONS: Total number of DAMOS actions * * The support of each action is up to running &struct damon_operations. * Refer to 'Operation Action' section of Documentation/mm/damon/design.rst for @@ -137,7 +136,6 @@ enum damos_action { DAMOS_MIGRATE_HOT, DAMOS_MIGRATE_COLD, DAMOS_STAT, /* Do nothing but only record the stat */ - NR_DAMOS_ACTIONS, }; /** @@ -154,9 +152,6 @@ enum damos_action { * @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP: Scheme-eligible memory ratio of a * node in basis points (0-10000). * @DAMOS_QUOTA_HUGEPAGE_MEM_BP: Huge page to total used memory ratio. - * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics. - * - * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported. */ enum damos_quota_goal_metric { DAMOS_QUOTA_USER_INPUT, @@ -169,7 +164,6 @@ enum damos_quota_goal_metric { DAMOS_QUOTA_INACTIVE_MEM_BP, DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP, DAMOS_QUOTA_HUGEPAGE_MEM_BP, - NR_DAMOS_QUOTA_GOAL_METRICS, }; /** @@ -313,12 +307,10 @@ struct damos_quota { * * @DAMOS_WMARK_NONE: Ignore the watermarks of the given scheme. * @DAMOS_WMARK_FREE_MEM_RATE: Free memory rate of the system in [0,1000]. - * @NR_DAMOS_WMARK_METRICS: Total number of DAMOS watermark metrics */ enum damos_wmark_metric { DAMOS_WMARK_NONE, DAMOS_WMARK_FREE_MEM_RATE, - NR_DAMOS_WMARK_METRICS, }; /** From d4aa83a7b62be607833c74a6676e6400f3085636 Mon Sep 17 00:00:00 2001 From: Arnd Bergmann Date: Mon, 28 Sep 2026 01:39:56 -0700 Subject: [PATCH 040/213] mm/damon/core: reduce stack usage further In a previous patch, I had annotated kdamond_tune_intervals() as noinline_for_stack in order to not exceed the stack frame warning limit. Unfortunately, my current linux-next randconfig builds show a similar problem again with clang-21: mm/damon/core.c:3953:12: error: stack frame size (1288) exceeds limit (1280) in 'kdamond_fn' [-Werror,-Wframe-larger-than] 3953 | static int kdamond_fn(void *data) Do the same thing with kdamond_apply_schemes(), kdamond_merge_regions(), and kdamond_split_regions(), which also have individually large stacks. This should work to keep the deepest total stack depth down much more as well as avoid the warning. I also tried to reduce the complexity of kdamond_fn() itself by splitting out the while() loop into a separate function. While this arguably led to slightly more readable code, it had no effect on the total stack usage and just made the new function the largest stack user and had a nonzero risk of me getting the conversion wrong. Link: https://lore.kernel.org/20260928083959.4030-3-sj@kernel.org Fixes: 5a00cae64de1 ("mm/damon/core: reduce kernel stack usage") Signed-off-by: Arnd Bergmann Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Cc: Nick Desaulniers Cc: Bill Wendling Cc: Justin Stitt Cc: Ravi Jonnalagadda Cc: Nathan Chancellor --- mm/damon/core.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/damon/core.c b/mm/damon/core.c index 46ef570a4766..5ecbea5d71e1 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -3433,7 +3433,7 @@ static void damos_trace_stat(struct damon_ctx *c, struct damos *s) trace_call__damos_stat_after_apply_interval(cidx, sidx, &s->stat); } -static void kdamond_apply_schemes(struct damon_ctx *c) +static noinline_for_stack void kdamond_apply_schemes(struct damon_ctx *c) { struct damon_target *t; struct damos *s; @@ -3589,8 +3589,9 @@ static void damon_merge_regions_of(struct damon_target *t, unsigned int thres, * while DAMON is running. For such a case, repeat merging until the limit is * met while increasing @threshold up to possible maximum level. */ -static void kdamond_merge_regions(struct damon_ctx *c, unsigned int threshold, - unsigned long sz_limit) +static noinline_for_stack void kdamond_merge_regions(struct damon_ctx *c, + unsigned int threshold, + unsigned long sz_limit) { struct damon_target *t; unsigned int nr_regions; @@ -3734,7 +3735,7 @@ static void damon_split_some_regions(struct damon_ctx *ctx, * split was unnecessarily made, later 'kdamond_merge_regions()' will revert * it. */ -static void kdamond_split_regions(struct damon_ctx *ctx) +static noinline_for_stack void kdamond_split_regions(struct damon_ctx *ctx) { struct damon_target *t; unsigned long nr_regions = 0; From e444edf7dbe640fab592ba042e1a77f93e976ebb Mon Sep 17 00:00:00 2001 From: Karl Mehltretter Date: Mon, 28 Sep 2026 01:39:57 -0700 Subject: [PATCH 041/213] mm/damon/tests/core-kunit: test PSI goal values with explicit samples Test the PSI current-value helper with explicit totals so the result does not depend on the test system's memory pressure. Cover an unmeasured consist goal, unmeasured temporal goals with zero and nonzero effective quotas, and measured rounds for both tuners. Check last_psi_total after each call. Link: https://lore.kernel.org/20260928083959.4030-4-sj@kernel.org Signed-off-by: Karl Mehltretter Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Assisted-by: LLM Cc: Lian Wang Cc: Kunwu Chan Cc: Brendan Higgins Cc: David Gow --- mm/damon/tests/core-kunit.h | 43 +++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index a8e22246facf..2111faa58153 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -952,6 +952,48 @@ static void damos_test_commit_quota_goal(struct kunit *test) }); } +static void damos_test_set_psi_current_val(struct kunit *test) +{ + struct damos s = { + .quota.goal_tuner = DAMOS_QUOTA_GOAL_TUNER_CONSIST, + }; + struct damos_quota_goal goal = { + .metric = DAMOS_QUOTA_SOME_MEM_PSI_US, + .target_value = 100, + .last_psi_total = U64_MAX, + }; + + /* uninitialized last_psi_total keeps the consist tuner quota */ + damos_set_psi_current_val(1000, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 100ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 1000ull); + + /* initialized last_psi_total gives the delta */ + damos_set_psi_current_val(1030, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 30ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 1030ull); + + /* temporal tuner keeps a zero quota */ + s.quota.goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL; + s.quota.esz = 0; + goal.last_psi_total = U64_MAX; + damos_set_psi_current_val(2000, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 100ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 2000ull); + + /* temporal tuner keeps a non-zero quota */ + s.quota.esz = SZ_64K; + goal.last_psi_total = U64_MAX; + damos_set_psi_current_val(3000, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 0ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 3000ull); + + /* temporal tuner uses the measured PSI delta */ + damos_set_psi_current_val(3250, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 250ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 3250ull); +} + static void damos_test_commit_quota_goals_for(struct kunit *test, struct damos_quota_goal *dst_goals, int nr_dst_goals, struct damos_quota_goal *src_goals, int nr_src_goals) @@ -2413,6 +2455,7 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damos_test_new_filter), KUNIT_CASE(damos_test_new_scheme_keeps_src_quota), KUNIT_CASE(damos_test_commit_quota_goal), + KUNIT_CASE(damos_test_set_psi_current_val), KUNIT_CASE(damos_test_commit_quota_goals), KUNIT_CASE(damos_test_commit_quota), KUNIT_CASE(damos_test_commit_dests), From f4399c9927c46bd637b2b9a7677e3aa281c1ade4 Mon Sep 17 00:00:00 2001 From: Ye Liu Date: Mon, 28 Sep 2026 16:15:54 +0800 Subject: [PATCH 042/213] mm/vmalloc: fix vmalloc_dump_obj address alignment for last-page lookups Patch series "mm/vmalloc: fix vmalloc_dump_obj VA lookup", v4. vmalloc_dump_obj() has two bugs that cause it to miss vmalloc allocations. 1. PAGE_ALIGN() rounds up, pushing last-page addresses to va_end and outside the VA lookup range. 2. The function searches only one vmap node, but allocations larger than 64 KiB may span multiple vmap zones whose VA is stored in a different node's rb-tree. Patch 1 fixes the alignment, patch 2 fixes the cross-zone search. This patch (of 2): vmalloc_dump_obj() uses PAGE_ALIGN() to normalize the input address before looking it up in the per-node busy tree. PAGE_ALIGN() rounds up, which can push an address in the last page of a vmalloc allocation to va_end -- outside the [va_start, va_end) range that __find_vmap_area() searches. This causes the lookup to miss the VA and return false, degrading diagnostic output in OOM dumps and KASAN reports to the less informative "vmalloc memory" fallback. The upward alignment can also change the addr_to_node() mapping when the page boundary crosses a vmap zone boundary, causing the search to hit the wrong node entirely. Use PAGE_ALIGN_DOWN() instead, which rounds down to the page containing the address. This keeps the address within the VA range and preserves the correct node mapping. Link: https://lore.kernel.org/20260928-vmalloc_dump_obj-v4-0-6f288a431edc@linux.dev Link: https://lore.kernel.org/20260928-vmalloc_dump_obj-v4-1-6f288a431edc@linux.dev Signed-off-by: Ye Liu Signed-off-by: Andrew Morton Reviewed-by: Uladzislau Rezki (Sony) Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Alexandre Ghiti --- mm/vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4b0bcc998f1d..e3c412430a6f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -5268,7 +5268,7 @@ bool vmalloc_dump_obj(void *object) unsigned long addr; unsigned long nr_pages; - addr = PAGE_ALIGN((unsigned long) object); + addr = PAGE_ALIGN_DOWN((unsigned long) object); vn = addr_to_node(addr); if (!spin_trylock(&vn->busy.lock)) From 4b52da012040e1bdcbe93b7bb8b407568b559249 Mon Sep 17 00:00:00 2001 From: Ye Liu Date: Mon, 28 Sep 2026 16:15:55 +0800 Subject: [PATCH 043/213] mm/vmalloc: fix vmalloc_dump_obj cross-zone VA lookup vmalloc_dump_obj() searches only one vmap node (addr_to_node(addr)), but a vmalloc allocation may span multiple vmap zones. The VA is stored in only one node's rb-tree (addr_to_node(va_start)), so an object pointer in a different zone than va_start maps to a different node and the search misses. This affects any allocation larger than vmap_zone_size (64 KiB) on multi-CPU systems. Extract find_vmap_area_lock() from find_vmap_area() to share the cross-node iteration logic. The helper supports both spin_lock and spin_trylock, the latter for atomic dump contexts (OOM, KASAN, RCU). Link: https://lore.kernel.org/20260928-vmalloc_dump_obj-v4-2-6f288a431edc@linux.dev Signed-off-by: Ye Liu Signed-off-by: Andrew Morton Reviewed-by: Uladzislau Rezki (Sony) Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Alexandre Ghiti --- mm/vmalloc.c | 108 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 66 insertions(+), 42 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e3c412430a6f..81cad5095889 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2503,67 +2503,94 @@ static void free_unmap_vmap_area(struct vmap_area *va) free_vmap_area_noflush(va); } -struct vmap_area *find_vmap_area(unsigned long addr) +static inline int next_vmap_node_id(int i) +{ + return (i + nr_vmap_nodes - 1) % nr_vmap_nodes; +} + +enum vmap_lock_mode { + VMAP_LOCK, + VMAP_TRYLOCK, +}; + +/* + * Search for a vmap_area at @addr across all vmap nodes. An + * addr_to_node_id(addr) converts an address to a node index where + * a VA is located. If VA spans several zones and passed addr is not + * the same as va->va_start, what is not common, we may need to scan + * extra nodes. See an example: + * + * <----va----> + * -|-----|-----|-----|-----|- + * 1 2 0 1 + * + * VA resides in node 1 whereas it spans 1, 2 an 0. If passed addr + * is within 2 or 0 nodes we should do extra work. + * + * Returns the VA with @locked_vn->busy.lock held; the caller must + * release it. If @mode is VMAP_TRYLOCK, nodes that cannot be locked + * are skipped. + */ +static struct vmap_area * +find_vmap_area_lock(unsigned long addr, struct vmap_node **locked_vn, + enum vmap_lock_mode mode) { struct vmap_node *vn; struct vmap_area *va; int i, j; + *locked_vn = NULL; + if (unlikely(!vmap_initialized)) return NULL; - /* - * An addr_to_node_id(addr) converts an address to a node index - * where a VA is located. If VA spans several zones and passed - * addr is not the same as va->va_start, what is not common, we - * may need to scan extra nodes. See an example: - * - * <----va----> - * -|-----|-----|-----|-----|- - * 1 2 0 1 - * - * VA resides in node 1 whereas it spans 1, 2 an 0. If passed - * addr is within 2 or 0 nodes we should do extra work. - */ i = j = addr_to_node_id(addr); do { vn = &vmap_nodes[i]; - spin_lock(&vn->busy.lock); - va = __find_vmap_area(addr, &vn->busy.root); - spin_unlock(&vn->busy.lock); + if (mode == VMAP_LOCK) { + spin_lock(&vn->busy.lock); + } else { + if (!spin_trylock(&vn->busy.lock)) + continue; + } - if (va) + va = __find_vmap_area(addr, &vn->busy.root); + if (va) { + *locked_vn = vn; return va; - } while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j); + } + + spin_unlock(&vn->busy.lock); + } while ((i = next_vmap_node_id(i)) != j); return NULL; } +struct vmap_area *find_vmap_area(unsigned long addr) +{ + struct vmap_node *vn; + struct vmap_area *va; + + va = find_vmap_area_lock(addr, &vn, VMAP_LOCK); + if (va) + spin_unlock(&vn->busy.lock); + + return va; +} + static struct vmap_area *find_unlink_vmap_area(unsigned long addr) { struct vmap_node *vn; struct vmap_area *va; - int i, j; - /* - * Check the comment in the find_vmap_area() about the loop. - */ - i = j = addr_to_node_id(addr); - do { - vn = &vmap_nodes[i]; - - spin_lock(&vn->busy.lock); - va = __find_vmap_area(addr, &vn->busy.root); - if (va) - unlink_va(va, &vn->busy.root); + va = find_vmap_area_lock(addr, &vn, VMAP_LOCK); + if (va) { + unlink_va(va, &vn->busy.root); spin_unlock(&vn->busy.lock); + } - if (va) - return va; - } while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j); - - return NULL; + return va; } /*** Per cpu kva allocator ***/ @@ -5269,14 +5296,11 @@ bool vmalloc_dump_obj(void *object) unsigned long nr_pages; addr = PAGE_ALIGN_DOWN((unsigned long) object); - vn = addr_to_node(addr); - if (!spin_trylock(&vn->busy.lock)) - return false; - - va = __find_vmap_area(addr, &vn->busy.root); + va = find_vmap_area_lock(addr, &vn, VMAP_TRYLOCK); if (!va || !va->vm) { - spin_unlock(&vn->busy.lock); + if (va) + spin_unlock(&vn->busy.lock); return false; } From 89073077600b7cb3734d53f0fe535b302b01e859 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:24 +0800 Subject: [PATCH 044/213] gpu: ipu-v3: don't use GFP_DMA when calling dma_alloc_coherent() Patch series "Don't use GFP_DMA when calling dma_alloc_coherent". This series picks up the first part of an earlier cleanup series [1] that was prepared back in the year of 2022, but for various reasons never made it merged and has been sitting in a local tree since then. This subset only touches the call sites where GFP_DMA is passed to dma_alloc_coherent() (and its dma_alloc_wc()/dmam_alloc_coherent() variants), which is the most self-contained and least risky slice of that work. That GFP_DMA is simply redundant here: the DMA API derives the allocation zone from the device's coherent_dma_mask (together with bus_dma_limit) and ignores the GFP_DMA flag passed by the caller. Removing the redundant GFP_DMA won't harm anything, while keeps it from being blindly copied into new code. This patch (of 12): dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-1-hebaoquan@kylinos.cn Link: https://lore.kernel.org/20260903111836.1777265-2-hebaoquan@kylinos.cn Link: https://lore.kernel.org/all/20220219005221.634-1-bhe@redhat.com/T/#u [1] Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/gpu/ipu-v3/ipu-image-convert.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/ipu-v3/ipu-image-convert.c b/drivers/gpu/ipu-v3/ipu-image-convert.c index 29b6d36c9bb6..493be4d4b091 100644 --- a/drivers/gpu/ipu-v3/ipu-image-convert.c +++ b/drivers/gpu/ipu-v3/ipu-image-convert.c @@ -371,7 +371,7 @@ static int alloc_dma_buf(struct ipu_image_convert_priv *priv, { buf->len = PAGE_ALIGN(size); buf->virt = dma_alloc_coherent(priv->ipu->dev, buf->len, &buf->phys, - GFP_DMA | GFP_KERNEL); + GFP_KERNEL); if (!buf->virt) { dev_err(priv->ipu->dev, "failed to alloc dma buffer\n"); return -ENOMEM; From a856d82102a52d3250cf08e09c68d39a4fdbfefb Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:25 +0800 Subject: [PATCH 045/213] drm/sti: don't use GFP_DMA when calling dma_alloc_wc() dma_alloc_wc() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-3-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/gpu/drm/sti/sti_cursor.c | 4 ++-- drivers/gpu/drm/sti/sti_hqvdp.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/sti/sti_cursor.c b/drivers/gpu/drm/sti/sti_cursor.c index d0b89b28e50c..a0ec671477fd 100644 --- a/drivers/gpu/drm/sti/sti_cursor.c +++ b/drivers/gpu/drm/sti/sti_cursor.c @@ -240,7 +240,7 @@ static int sti_cursor_atomic_check(struct drm_plane *drm_plane, cursor->pixmap.base = dma_alloc_wc(cursor->dev, cursor->pixmap.size, &cursor->pixmap.paddr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!cursor->pixmap.base) { DRM_ERROR("Failed to allocate memory for pixmap\n"); return -EINVAL; @@ -380,7 +380,7 @@ struct drm_plane *sti_cursor_create(struct drm_device *drm_dev, /* Allocate clut buffer */ size = 0x100 * sizeof(unsigned short); cursor->clut = dma_alloc_wc(dev, size, &cursor->clut_paddr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!cursor->clut) { DRM_ERROR("Failed to allocate memory for cursor clut\n"); diff --git a/drivers/gpu/drm/sti/sti_hqvdp.c b/drivers/gpu/drm/sti/sti_hqvdp.c index cf1ed8a33b8e..b0d66834bcbe 100644 --- a/drivers/gpu/drm/sti/sti_hqvdp.c +++ b/drivers/gpu/drm/sti/sti_hqvdp.c @@ -860,7 +860,7 @@ static void sti_hqvdp_init(struct sti_hqvdp *hqvdp) size = NB_VDP_CMD * sizeof(struct sti_hqvdp_cmd); hqvdp->hqvdp_cmd = dma_alloc_wc(hqvdp->dev, size, &dma_addr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!hqvdp->hqvdp_cmd) { DRM_ERROR("Failed to allocate memory for VDP cmd\n"); return; From eae6eecd67c9bca9901d762c2d15c671d90e609f Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:27 +0800 Subject: [PATCH 046/213] spi: spi-ti-qspi: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-5-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/spi/spi-ti-qspi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/spi/spi-ti-qspi.c b/drivers/spi/spi-ti-qspi.c index 34b154922ff2..fb0d940883fb 100644 --- a/drivers/spi/spi-ti-qspi.c +++ b/drivers/spi/spi-ti-qspi.c @@ -855,7 +855,7 @@ static int ti_qspi_probe(struct platform_device *pdev) qspi->rx_bb_addr = dma_alloc_coherent(qspi->dev, QSPI_DMA_BUFFER_SIZE, &qspi->rx_bb_dma_addr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!qspi->rx_bb_addr) { dev_err(qspi->dev, "dma_alloc_coherent failed, using PIO mode\n"); From 5f0f758d36730dfbd75be3c722e40d4f98fa7d03 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:28 +0800 Subject: [PATCH 047/213] fbdev: fsl-diu-fb: don't use GFP_DMA when calling dmam_alloc_coherent() dmam_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-6-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/video/fbdev/fsl-diu-fb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/video/fbdev/fsl-diu-fb.c b/drivers/video/fbdev/fsl-diu-fb.c index b71d15794ce8..d7ed007915c1 100644 --- a/drivers/video/fbdev/fsl-diu-fb.c +++ b/drivers/video/fbdev/fsl-diu-fb.c @@ -1690,7 +1690,7 @@ static int fsl_diu_probe(struct platform_device *pdev) int ret; data = dmam_alloc_coherent(&pdev->dev, sizeof(struct fsl_diu_data), - &dma_addr, GFP_DMA | __GFP_ZERO); + &dma_addr, __GFP_ZERO); if (!data) return -ENOMEM; data->dma_addr = dma_addr; From fd3d2126e737c9c95d78d30904916ff84ebd2940 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:29 +0800 Subject: [PATCH 048/213] usb: gadget: lpc32xx_udc: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-7-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/usb/gadget/udc/lpc32xx_udc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/usb/gadget/udc/lpc32xx_udc.c b/drivers/usb/gadget/udc/lpc32xx_udc.c index 044c31869cfb..90d1eef52877 100644 --- a/drivers/usb/gadget/udc/lpc32xx_udc.c +++ b/drivers/usb/gadget/udc/lpc32xx_udc.c @@ -3080,7 +3080,7 @@ static int lpc32xx_udc_probe(struct platform_device *pdev) /* Allocate memory for the UDCA */ udc->udca_v_base = dma_alloc_coherent(&pdev->dev, UDCA_BUFF_SIZE, &dma_handle, - (GFP_KERNEL | GFP_DMA)); + GFP_KERNEL); if (!udc->udca_v_base) { dev_err(udc->dev, "error getting UDCA region\n"); retval = -ENOMEM; From 4facd31d97d9fcdbf3a71f97463ca3894fb035a2 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:30 +0800 Subject: [PATCH 049/213] usb: cdns3: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Use GFP_KERNEL instead so the allocation may reclaim as usual for probe-time allocations. Link: https://lore.kernel.org/20260903111836.1777265-8-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/usb/cdns3/cdns3-gadget.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/usb/cdns3/cdns3-gadget.c b/drivers/usb/cdns3/cdns3-gadget.c index 42311c1bfada..484d3128cf08 100644 --- a/drivers/usb/cdns3/cdns3-gadget.c +++ b/drivers/usb/cdns3/cdns3-gadget.c @@ -3376,7 +3376,7 @@ static int cdns3_gadget_start(struct cdns *cdns) /* allocate memory for setup packet buffer */ priv_dev->setup_buf = dma_alloc_coherent(priv_dev->sysdev, 8, - &priv_dev->setup_dma, GFP_DMA); + &priv_dev->setup_dma, GFP_KERNEL); if (!priv_dev->setup_buf) { ret = -ENOMEM; goto err2; From a21eff6fea4fe25f965e068cc15252b916a07fc4 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:31 +0800 Subject: [PATCH 050/213] media: staging: imx: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-9-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/staging/media/imx/imx-media-utils.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c index f119477cac6b..7bdcb88a686b 100644 --- a/drivers/staging/media/imx/imx-media-utils.c +++ b/drivers/staging/media/imx/imx-media-utils.c @@ -588,7 +588,7 @@ int imx_media_alloc_dma_buf(struct device *dev, buf->len = PAGE_ALIGN(size); buf->virt = dma_alloc_coherent(dev, buf->len, &buf->phys, - GFP_DMA | GFP_KERNEL); + GFP_KERNEL); if (!buf->virt) return -ENOMEM; From 9db28c5f1e776a45491b8794490bf11e7c592c40 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:32 +0800 Subject: [PATCH 051/213] spi: atmel: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-10-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/spi/spi-atmel.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/spi/spi-atmel.c b/drivers/spi/spi-atmel.c index c8012c82c3a7..e91e84fdbe8f 100644 --- a/drivers/spi/spi-atmel.c +++ b/drivers/spi/spi-atmel.c @@ -624,7 +624,7 @@ static int atmel_spi_configure_dma(struct spi_controller *host, if (IS_ENABLED(CONFIG_SOC_SAM_V4_V5)) { as->addr_tx_bbuf = dma_alloc_coherent(dev, SPI_MAX_DMA_XFER, &as->dma_addr_tx_bbuf, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!as->addr_tx_bbuf) { err = -ENOMEM; goto err_release_dma; @@ -632,7 +632,7 @@ static int atmel_spi_configure_dma(struct spi_controller *host, as->addr_rx_bbuf = dma_alloc_coherent(dev, SPI_MAX_DMA_XFER, &as->dma_addr_rx_bbuf, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!as->addr_rx_bbuf) { err = -ENOMEM; goto err_release_dma; From 366830e788a25fd8786d07ec1af48fdb1b64ae95 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:33 +0800 Subject: [PATCH 052/213] media: imx7-media-csi: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-11-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Reviewed-by: Frank Li Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/media/platform/nxp/imx7-media-csi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/platform/nxp/imx7-media-csi.c b/drivers/media/platform/nxp/imx7-media-csi.c index 7ddc7ba06e3d..22c0cbdc92bf 100644 --- a/drivers/media/platform/nxp/imx7-media-csi.c +++ b/drivers/media/platform/nxp/imx7-media-csi.c @@ -466,7 +466,7 @@ static int imx7_csi_alloc_dma_buf(struct imx7_csi *csi, buf->len = PAGE_ALIGN(size); buf->virt = dma_alloc_coherent(csi->dev, buf->len, &buf->dma_addr, - GFP_DMA | GFP_KERNEL); + GFP_KERNEL); if (!buf->virt) return -ENOMEM; From c9f045135314fd3f54ed8b7ccb0eba6a8e3ad45a Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:34 +0800 Subject: [PATCH 053/213] media: nxp: imx8-isi: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Remove the redundant GFP_DMA flag. Link: https://lore.kernel.org/20260903111836.1777265-12-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Reviewed-by: Frank Li Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c b/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c index f45c2aae59ce..fc907d357149 100644 --- a/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +++ b/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c @@ -773,7 +773,7 @@ static int mxc_isi_video_alloc_discard_buffers(struct mxc_isi_video *video) buf->size = PAGE_ALIGN(video->pix.plane_fmt[i].sizeimage); buf->addr = dma_alloc_coherent(video->pipe->isi->dev, buf->size, - &buf->dma, GFP_DMA | GFP_KERNEL); + &buf->dma, GFP_KERNEL); if (!buf->addr) { mxc_isi_video_free_discard_buffers(video); return -ENOMEM; From 4d7507b166f84c0c1d76b0aba19ebfd585465772 Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:35 +0800 Subject: [PATCH 054/213] mtd: rawnand: gpmi: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Use GFP_KERNEL instead so the allocation may reclaim as usual for probe-time allocations. Link: https://lore.kernel.org/20260903111836.1777265-13-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c b/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c index 527165ccc839..0d91950ae9fd 100644 --- a/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c +++ b/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c @@ -1387,7 +1387,7 @@ static int gpmi_alloc_dma_buffer(struct gpmi_nand_data *this) goto error_alloc; this->auxiliary_virt = dma_alloc_coherent(dev, geo->auxiliary_size, - &this->auxiliary_phys, GFP_DMA); + &this->auxiliary_phys, GFP_KERNEL); if (!this->auxiliary_virt) goto error_alloc; From 1528ed9f962f328280b918b066188a0fe3a5ce0e Mon Sep 17 00:00:00 2001 From: Baoquan He Date: Thu, 3 Sep 2026 19:18:36 +0800 Subject: [PATCH 055/213] usb: cdns2: don't use GFP_DMA when calling dma_alloc_coherent() dma_alloc_coherent() allocates the DMA buffer with the device's addressing limitation in mind; the DMA core picks the zone from the device's coherent DMA mask and ignores GFP_DMA passed by the caller. Use GFP_KERNEL instead so the allocation may reclaim as usual for probe-time allocations. Link: https://lore.kernel.org/20260903111836.1777265-14-hebaoquan@kylinos.cn Signed-off-by: Baoquan He Signed-off-by: Andrew Morton Cc: Christoph Hellwig Cc: Harry Yoo --- drivers/usb/gadget/udc/cdns2/cdns2-gadget.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c b/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c index 308d3c468ab1..8719f1f86e61 100644 --- a/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c +++ b/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c @@ -2341,7 +2341,7 @@ static int cdns2_gadget_start(struct cdns2_device *pdev) /* Allocate memory for setup packet buffer. */ buf = dma_alloc_coherent(pdev->dev, 8, &pdev->ep0_preq.request.dma, - GFP_DMA); + GFP_KERNEL); pdev->ep0_preq.request.buf = buf; if (!pdev->ep0_preq.request.buf) { From de6632aa7851b987d8f54129e43d75bc681f72b9 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:04 -0700 Subject: [PATCH 056/213] mm/damon/core: introduce damos_quota_goal->complement Patch series "mm/damon: introduce damos quota goal target metric complement flag". Aim-oriented DAMOS quota auto-tuning assumes the aggressiveness of the scheme (quota) and the quota goal metric are directly proportional. Depending on the scheme setup and usage, keeping the relationship can be challenging. For example, let's suppose a memory tiering approach for higher upper tier utilization. One common idea for that (TPP) is utilizing two schemes, one for promotion and the other one for demotion. The promotion scheme migrates hot pages from lower tier to upper tier, aiming for high utilization of the upper tier. The demotion scheme migrates cold pages from upper tier to lower tier, aiming for head room free memory of the upper tier. Both schemes and their goals are in direct proportion. However, for this kind of use case, we need to implement two different goal metrics (per-node memory utilization and free memory ratio) while essentially the free memory ratio is just a complemented value of the utilization. To avoid adding too many new metrics, we are adding metrics that turn out to be really needed for each found use case. For example, some_mem_psi_us, node_eligible_mem_bp and hugepage_mem_bp don't have their complemented value version. But it is not really difficult to expect use cases that their complemented version can be useful. For example, hugepage_mem_bp use case may need a way to reduce the hugepage ratio. That would require a complemented version of hugepage_mem_bp. Adding a new metric for each of such use cases could make the number of metrics unnecessarily high, and discourage flexible usages of DAMOS. Add a new flag, quota goal complement, to allow flexible tuning goal setup without unnecessarily increasing the number of metrics. The flag specifies whether to use the complemented value of the given goal metric for the tuning. For example, if the complement flag is set, the upper tier memory utilization ratio metric works the same as the free memory ratio metric for the tier. This patch (of 8): Introduce damos_quota_goal->complement for specifying whether to use a complemented value of the given goal target metric. Add the field to the data structure and implement essential core support. Handle the flag in the quota goal commit and current quota goal metric value retrieval. [akpm@linux-foundation.org: fix comment tpyo, per Gutierrez] Link: https://lore.kernel.org/20260929080113.41708-1-sj@kernel.org Link: https://lore.kernel.org/20260929080113.41708-2-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton Cc: Gutierrez Asier --- include/linux/damon.h | 2 ++ mm/damon/core.c | 26 +++++++++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 6a29dc2ac8db..42234839ce29 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -169,6 +169,7 @@ enum damos_quota_goal_metric { /** * struct damos_quota_goal - DAMOS scheme quota auto-tuning goal. * @metric: Metric to be used for representing the goal. + * @complement: Use the complement of the metric. * @target_value: Target value of @metric to achieve with the tuning. * @current_value: Current value of @metric. * @nid: Node id. @@ -191,6 +192,7 @@ enum damos_quota_goal_metric { */ struct damos_quota_goal { enum damos_quota_goal_metric metric; + bool complement; unsigned long target_value; unsigned long current_value; /* metric-dependent fields */ diff --git a/mm/damon/core.c b/mm/damon/core.c index 5ecbea5d71e1..0d27f08b350e 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -1224,6 +1224,7 @@ static int damos_commit_quota_goal( if (!src->target_value) return -EINVAL; dst->metric = src->metric; + dst->complement = src->complement; dst->target_value = src->target_value; if (dst->metric == DAMOS_QUOTA_USER_INPUT) dst->current_value = src->current_value; @@ -2960,10 +2961,18 @@ static void damos_set_psi_current_val(u64 now_psi_total, struct damos_quota_goal *goal, struct damos *s) { u64 last_psi_total = goal->last_psi_total; + unsigned long val; goal->last_psi_total = now_psi_total; if (last_psi_total != U64_MAX) { - goal->current_value = now_psi_total - last_psi_total; + val = now_psi_total - last_psi_total; + if (goal->complement) { + if (val < s->quota.reset_interval * 1000) + val = s->quota.reset_interval * 1000 - val; + else + val = 0; + } + goal->current_value = val; return; } /* uninitialized last_psi_total; make no effect this round */ @@ -3255,6 +3264,21 @@ static void damos_set_quota_goal_current_value(struct damon_ctx *c, default: break; } + if (!goal->complement) + return; + + /* update current_value to complemented value */ + + /* for user_input, users set complemented value on their own */ + if (goal->metric == DAMOS_QUOTA_USER_INPUT) + return; + /* damos_set_psi_current_val() handles complement flag itself */ + if (goal->metric == DAMOS_QUOTA_SOME_MEM_PSI_US) + return; + if (goal->current_value < 10000) + goal->current_value = 10000 - goal->current_value; + else + goal->current_value = 0; } /* Return the highest score since it makes schemes least aggressive */ From 89d16e8622e0209cf6b5cd96a939574cd4850485 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:05 -0700 Subject: [PATCH 057/213] mm/damon/core: add complement argument to damos_new_quota_goal() damos_quota_goal->complement needs to be manually set by each API callers. It is easy to make mistakes. Extend the quota goal constructor, damos_new_quota_goal() to receive and set the complement flag value. Also update all callers to use the new signature. Link: https://lore.kernel.org/20260929080113.41708-3-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: Asier Gutierrez --- include/linux/damon.h | 2 +- mm/damon/core.c | 7 ++++--- mm/damon/lru_sort.c | 5 +++-- mm/damon/reclaim.c | 5 +++-- mm/damon/sysfs-schemes.c | 2 +- mm/damon/tests/core-kunit.h | 3 ++- samples/damon/mtier.c | 2 +- 7 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 42234839ce29..63050eb2206a 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -1090,7 +1090,7 @@ bool damos_filter_for_ops(enum damos_filter_type type); void damos_destroy_filter(struct damos_filter *f); struct damos_quota_goal *damos_new_quota_goal( - enum damos_quota_goal_metric metric, + enum damos_quota_goal_metric metric, bool complement, unsigned long target_value); void damos_add_quota_goal(struct damos_quota *q, struct damos_quota_goal *g); void damos_destroy_quota_goal(struct damos_quota_goal *goal); diff --git a/mm/damon/core.c b/mm/damon/core.c index 0d27f08b350e..b63e60ef8990 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -691,7 +691,7 @@ void damos_destroy_filter(struct damos_filter *f) } struct damos_quota_goal *damos_new_quota_goal( - enum damos_quota_goal_metric metric, + enum damos_quota_goal_metric metric, bool complement, unsigned long target_value) { struct damos_quota_goal *goal; @@ -700,6 +700,7 @@ struct damos_quota_goal *damos_new_quota_goal( if (!goal) return NULL; goal->metric = metric; + goal->complement = complement; goal->target_value = target_value; if (metric == DAMOS_QUOTA_SOME_MEM_PSI_US) goal->last_psi_total = U64_MAX; @@ -1262,8 +1263,8 @@ int damos_commit_quota_goals(struct damos_quota *dst, struct damos_quota *src) damos_for_each_quota_goal_safe(src_goal, next, src) { if (j++ < i) continue; - new_goal = damos_new_quota_goal( - src_goal->metric, src_goal->target_value); + new_goal = damos_new_quota_goal(src_goal->metric, + src_goal->complement, src_goal->target_value); if (!new_goal) return -ENOMEM; err = damos_commit_quota_goal(new_goal, src_goal); diff --git a/mm/damon/lru_sort.c b/mm/damon/lru_sort.c index 273efa3c913e..64e086985eb5 100644 --- a/mm/damon/lru_sort.c +++ b/mm/damon/lru_sort.c @@ -233,12 +233,13 @@ static int damon_lru_sort_add_quota_goals(struct damos *hot_scheme, if (!active_mem_bp) return 0; - goal = damos_new_quota_goal(DAMOS_QUOTA_ACTIVE_MEM_BP, active_mem_bp); + goal = damos_new_quota_goal(DAMOS_QUOTA_ACTIVE_MEM_BP, false, + active_mem_bp); if (!goal) return -ENOMEM; damos_add_quota_goal(&hot_scheme->quota, goal); /* aim 0.2 % goal conflict, to keep little ping pong */ - goal = damos_new_quota_goal(DAMOS_QUOTA_INACTIVE_MEM_BP, + goal = damos_new_quota_goal(DAMOS_QUOTA_INACTIVE_MEM_BP, false, 10000 - active_mem_bp + 2); if (!goal) return -ENOMEM; diff --git a/mm/damon/reclaim.c b/mm/damon/reclaim.c index 42a2c9cb1343..014b0779ea6d 100644 --- a/mm/damon/reclaim.c +++ b/mm/damon/reclaim.c @@ -233,7 +233,7 @@ static int damon_reclaim_apply_parameters(void) damon_set_schemes(param_ctx, &scheme, 1); if (quota_mem_pressure_us) { - goal = damos_new_quota_goal(DAMOS_QUOTA_SOME_MEM_PSI_US, + goal = damos_new_quota_goal(DAMOS_QUOTA_SOME_MEM_PSI_US, false, quota_mem_pressure_us); if (!goal) goto out; @@ -241,7 +241,8 @@ static int damon_reclaim_apply_parameters(void) } if (quota_autotune_feedback) { - goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, 10000); + goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, false, + 10000); if (!goal) goto out; goal->current_value = quota_autotune_feedback; diff --git a/mm/damon/sysfs-schemes.c b/mm/damon/sysfs-schemes.c index bfb6f0bc3f21..06af417bc9a2 100644 --- a/mm/damon/sysfs-schemes.c +++ b/mm/damon/sysfs-schemes.c @@ -2869,7 +2869,7 @@ static int damos_sysfs_add_quota_score( if (!sysfs_goal->target_value) continue; - goal = damos_new_quota_goal(sysfs_goal->metric, + goal = damos_new_quota_goal(sysfs_goal->metric, false, sysfs_goal->target_value); if (!goal) return -ENOMEM; diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index 2111faa58153..5a924edb171f 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -1014,6 +1014,7 @@ static void damos_test_commit_quota_goals_for(struct kunit *test, * Make it kfree()-able. */ goal = damos_new_quota_goal(dst_goals[i].metric, + dst_goals[i].complement, dst_goals[i].target_value); if (!goal) goto out; @@ -2411,7 +2412,7 @@ static void damos_test_esz_goal_temporal(struct kunit *test) } damon_add_scheme(ctx, s); - goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, 10000); + goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, false, 10000); if (!goal) { damon_destroy_ctx(ctx); kunit_skip(test, "quota goal alloc fail"); diff --git a/samples/damon/mtier.c b/samples/damon/mtier.c index 27dc88bdf7a0..a2e311082cd4 100644 --- a/samples/damon/mtier.c +++ b/samples/damon/mtier.c @@ -163,7 +163,7 @@ static struct damon_ctx *damon_sample_mtier_build_ctx(bool promote) damon_set_schemes(ctx, &scheme, 1); quota_goal = damos_new_quota_goal( promote ? DAMOS_QUOTA_NODE_MEM_USED_BP : - DAMOS_QUOTA_NODE_MEM_FREE_BP, + DAMOS_QUOTA_NODE_MEM_FREE_BP, false, promote ? node0_mem_used_bp : node0_mem_free_bp); if (!quota_goal) goto free_out; From 9f3b27af2d41eab1914696a4d10f50c42fc7cbc2 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:06 -0700 Subject: [PATCH 058/213] mm/damon/sysfs-schemes: support quota goal complement flag Add a new sysfs file, complement, under the quota goal directory. It works for setting and getting the quota goal metric complement flag value. Link: https://lore.kernel.org/20260929080113.41708-4-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton --- mm/damon/sysfs-schemes.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/mm/damon/sysfs-schemes.c b/mm/damon/sysfs-schemes.c index 06af417bc9a2..8f083611741f 100644 --- a/mm/damon/sysfs-schemes.c +++ b/mm/damon/sysfs-schemes.c @@ -1220,6 +1220,7 @@ static const struct kobj_type damon_sysfs_watermarks_ktype = { struct damos_sysfs_quota_goal { struct kobject kobj; enum damos_quota_goal_metric metric; + bool complement; unsigned long target_value; unsigned long current_value; int nid; @@ -1316,6 +1317,30 @@ static ssize_t target_metric_store(struct kobject *kobj, return -EINVAL; } +static ssize_t complement_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct damos_sysfs_quota_goal *goal = container_of(kobj, + struct damos_sysfs_quota_goal, kobj); + + return sysfs_emit(buf, "%c\n", goal->complement ? 'Y' : 'N'); +} + +static ssize_t complement_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buf, size_t count) +{ + struct damos_sysfs_quota_goal *goal = container_of(kobj, + struct damos_sysfs_quota_goal, kobj); + bool complement; + int err = kstrtobool(buf, &complement); + + if (err) + return err; + + goal->complement = complement; + return count; +} + static ssize_t target_value_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -1424,6 +1449,9 @@ static void damos_sysfs_quota_goal_release(struct kobject *kobj) static struct kobj_attribute damos_sysfs_quota_goal_target_metric_attr = __ATTR_RW_MODE(target_metric, 0600); +static struct kobj_attribute damos_sysfs_quota_goal_complement_attr = + __ATTR_RW_MODE(complement, 0600); + static struct kobj_attribute damos_sysfs_quota_goal_target_value_attr = __ATTR_RW_MODE(target_value, 0600); @@ -1438,6 +1466,7 @@ static struct kobj_attribute damos_sysfs_quota_goal_path_attr = static struct attribute *damos_sysfs_quota_goal_attrs[] = { &damos_sysfs_quota_goal_target_metric_attr.attr, + &damos_sysfs_quota_goal_complement_attr.attr, &damos_sysfs_quota_goal_target_value_attr.attr, &damos_sysfs_quota_goal_current_value_attr.attr, &damos_sysfs_quota_goal_nid_attr.attr, @@ -2869,7 +2898,8 @@ static int damos_sysfs_add_quota_score( if (!sysfs_goal->target_value) continue; - goal = damos_new_quota_goal(sysfs_goal->metric, false, + goal = damos_new_quota_goal(sysfs_goal->metric, + sysfs_goal->complement, sysfs_goal->target_value); if (!goal) return -ENOMEM; From aeebce981d475466662140d55c18f609926ea522 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:07 -0700 Subject: [PATCH 059/213] mm/damon/tests/core-kunit: test quota_goal->complement commit Extend existing DAMOS quota goal commit unit test to test the complement flag. Set the source complement flag and confirm the destination is updated to the given input. Link: https://lore.kernel.org/20260929080113.41708-5-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton Cc: Brendan Higgins Cc: David Gow --- mm/damon/tests/core-kunit.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index 5a924edb171f..ef146ca2ae8a 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -861,6 +861,7 @@ static void damos_test_commit_quota_goal_for(struct kunit *test, damos_commit_quota_goal(dst, src); KUNIT_EXPECT_EQ(test, dst->metric, src->metric); + KUNIT_EXPECT_EQ(test, dst->complement, src->complement); KUNIT_EXPECT_EQ(test, dst->target_value, src->target_value); if (src->metric == DAMOS_QUOTA_USER_INPUT) KUNIT_EXPECT_EQ(test, dst->current_value, src->current_value); @@ -904,6 +905,7 @@ static void damos_test_commit_quota_goal(struct kunit *test) damos_test_commit_quota_goal_for(test, &dst, &(struct damos_quota_goal){ .metric = DAMOS_QUOTA_USER_INPUT, + .complement = true, .target_value = 789, .current_value = 12}); damos_test_commit_quota_goal_for(test, &dst, From 1aa52bdc4502086b253edd5fd6fb23d3727abb08 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:08 -0700 Subject: [PATCH 060/213] selftests/damon/sysfs.sh: test quota goal complement flag file Test the existence and the valid input acceptance of the newly added quota goal complement flag sysfs file. Link: https://lore.kernel.org/20260929080113.41708-6-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton Cc: Shuah Khan --- tools/testing/selftests/damon/sysfs.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tools/testing/selftests/damon/sysfs.sh b/tools/testing/selftests/damon/sysfs.sh index b66593c9ac47..1cc7ab2d7e22 100755 --- a/tools/testing/selftests/damon/sysfs.sh +++ b/tools/testing/selftests/damon/sysfs.sh @@ -212,6 +212,9 @@ test_goal() ensure_write_succ "$fpath" "node_eligible_mem_bp" "valid input" ensure_write_succ "$fpath" "hugepage_mem_bp" "valid input" ensure_write_fail "$fpath" "foo" "invalid input" + ensure_file "$goal_dir/complement" "exist" "600" + ensure_write_succ "$goal_dir/complement" "Y" "valid input" + ensure_write_succ "$goal_dir/complement" "N" "valid input" ensure_file "$goal_dir/nid" "exist" "600" ensure_file "$goal_dir/path" "exist" "600" } From 18ab75fb3cd27851a63757e1c7ea9c79abbe8cbe Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:09 -0700 Subject: [PATCH 061/213] Docs/mm/damon/design: document damos quota goal complement flag Update DAMON design document for the newly added damos quota goal metric value complement flag. Link: https://lore.kernel.org/20260929080113.41708-7-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton Cc: Liam R. Howlett Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Randy Dunlap Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka --- Documentation/mm/damon/design.rst | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/Documentation/mm/damon/design.rst b/Documentation/mm/damon/design.rst index e82390e77a70..5cd651b1b7aa 100644 --- a/Documentation/mm/damon/design.rst +++ b/Documentation/mm/damon/design.rst @@ -702,10 +702,10 @@ There are two such tuning algorithms that users can select as they need. that the zero quota is a valid quota, and therefore ``qt_exceeds`` :ref:`stat ` will keep increasing in this case. -The goal can be specified with five parameters, namely ``target_metric``, -``target_value``, ``current_value``, ``nid`` and ``path``. The auto-tuning -mechanism tries to make ``current_value`` of ``target_metric`` be same to -``target_value``. +The goal can be specified with six parameters, namely ``target_metric``, +``complement``, ``target_value``, ``current_value``, ``nid`` and ``path``. The +auto-tuning mechanism tries to make ``current_value`` of ``complement``-ed +``target_metric`` be same to ``target_value``. - ``user_input``: User-provided value. Users could use any metric that they has interest in for the value. Use space main workload's latency or @@ -733,6 +733,13 @@ mechanism tries to make ``current_value`` of ``target_metric`` be same to - ``hugepage_mem_bp``: Total huge page to total used memory ratio in bp (1/10,000). +``complement`` is a boolean parameter that determines whether to use +complemented value of the target metric. For example, if ``complement`` is set +and target metric is ``active_mem_bp``, it is effectively same to +``inactive_mem_bp``. ``complement`` is no-op when the target metric type is +``user_input``. For the metric type, the user should be able to emit +complemented metric values on their own. + ``nid`` is optionally required for ``node_mem_used_bp``, ``node_mem_free_bp``, ``node_memcg_used_bp``, ``node_memcg_free_bp`` and ``node_eligible_mem_bp`` to point the specific NUMA node. From 0a2965de3ddce823bfc6622307cf6f89295f1c30 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:10 -0700 Subject: [PATCH 062/213] Docs/admin-guide/mm/damon/usage: update for quota goal complement file Update DAMON usage document for the newly added quota goal metric complement sysfs file. Link: https://lore.kernel.org/20260929080113.41708-8-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton Cc: Liam R. Howlett Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Randy Dunlap Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka --- Documentation/admin-guide/mm/damon/usage.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst index ba4725544856..3d43954c7046 100644 --- a/Documentation/admin-guide/mm/damon/usage.rst +++ b/Documentation/admin-guide/mm/damon/usage.rst @@ -98,7 +98,8 @@ comma (","). │ │ │ │ │ │ │ fail_charge_num,fail_charge_denom │ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil │ │ │ │ │ │ │ │ :ref:`goals `/nr_goals - │ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid,path + │ │ │ │ │ │ │ │ │ 0/target_metric,complement,target_value, + │ │ │ │ │ │ │ │ │ current_value,nid,path │ │ │ │ │ │ │ :ref:`watermarks `/metric,interval_us,high,mid,low │ │ │ │ │ │ │ :ref:`{core_,ops_,}filters `/nr_filters │ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,damon_target_idx,min,max @@ -492,12 +493,12 @@ number (``N``) to the file creates the number of child directories named ``0`` to ``N-1``. Each directory represents each goal and current achievement. Among the multiple feedback, the best one is used. -Each goal directory contains five files, namely ``target_metric``, -``target_value``, ``current_value``, ``nid``, and ``path``. Users can set and -get the five parameters for the quota auto-tuning goals that specified on the -:ref:`design doc ` by writing to and -reading from each of the files. Because the kernel does not update -``current_value``, reading it only makes sense when ``target_metric`` is +Each goal directory contains six files, namely ``target_metric``, +``complement``, ``target_value``, ``current_value``, ``nid``, and ``path``. +Users can set and get the six parameters for the quota auto-tuning goals that +specified on the :ref:`design doc ` by +writing to and reading from each of the files. Because the kernel does not +update ``current_value``, reading it only makes sense when ``target_metric`` is ``user_input``. Note that users should further write ``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond directory ` to pass the feedback to DAMON. From b3d3ef5b2fc685da297ea1132e304a1b106d9e00 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 29 Sep 2026 01:01:11 -0700 Subject: [PATCH 063/213] Docs/ABI/damon: update for quota goal metric complement sysfs file Update DAMON ABI document for the newly added damos quota goal metric complement sysfs file. Link: https://lore.kernel.org/20260929080113.41708-9-sj@kernel.org Signed-off-by: SJ Park Signed-off-by: Andrew Morton Cc: Liam R. Howlett Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka --- Documentation/ABI/testing/sysfs-kernel-mm-damon | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-damon b/Documentation/ABI/testing/sysfs-kernel-mm-damon index 55df688ea596..4acb2015c2f4 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-damon +++ b/Documentation/ABI/testing/sysfs-kernel-mm-damon @@ -375,6 +375,12 @@ Contact: SJ Park Description: Writing to and reading from this file sets and gets the quota auto-tuning goal metric. +What: /sys/kernel/mm/damon/admin/kdamonds//contexts//schemes//quotas/goals//complement +Date: Sep 2026 +Contact: SJ Park +Description: Writing to and reading from this file sets and gets whether to + use the complement value of the quota auto-tuning goal metric. + What: /sys/kernel/mm/damon/admin/kdamonds//contexts//schemes//quotas/goals//target_value Date: Nov 2023 Contact: SJ Park From 372012202260db088b489ed00059c4571a602552 Mon Sep 17 00:00:00 2001 From: Pooyan Azad Date: Tue, 29 Sep 2026 09:18:46 +0200 Subject: [PATCH 064/213] zram: fix short reads from block_state read_block_state() formats each entry directly into the buffer supplied by read(). If the remaining buffer is too small for one complete record, snprintf() returns the full record length and the function stops without copying data or advancing the file position. A read from this debugfs file which is smaller than a record therefore returns zero at a non-EOF position and cannot make progress. Convert block_state to seq_file so formatted records are buffered independently of the userspace read size. Keep dev_lock held across each seq_file iteration and continue to protect individual entries with their slot locks. Link: https://lore.kernel.org/20260929071846.24829-1-pooyan.azadparvar@gmail.com Fixes: c0265342bff4 ("zram: introduce zram memory tracking") Signed-off-by: Pooyan Azad Signed-off-by: Andrew Morton Closes: https://lore.kernel.org/r/CANC3H+LdtoydSp+o2ecErAw7k6R2+gRf9LyxcaoHv_mGhJmyQQ@mail.gmail.com/ Reviewed-by: Sergey Senozhatsky Tested-by: Sergey Senozhatsky Cc: Minchan Kim Cc: Jens Axboe --- drivers/block/zram/zram_drv.c | 110 +++++++++++++++++----------------- 1 file changed, 56 insertions(+), 54 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 024402438bd1..ab064c84b092 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1553,68 +1553,70 @@ static void zram_debugfs_destroy(void) debugfs_remove_recursive(zram_debugfs_root); } -static ssize_t read_block_state(struct file *file, char __user *buf, - size_t count, loff_t *ppos) +static void *zram_block_state_start(struct seq_file *seq, loff_t *pos) { - char *kbuf; - unsigned long index; - ssize_t written = 0; - struct zram *zram = file->private_data; + struct zram *zram = seq->private; unsigned long nr_pages; - kbuf = kvmalloc(count, GFP_KERNEL); - if (!kbuf) - return -ENOMEM; - - guard(rwsem_read)(&zram->dev_lock); - if (!init_done(zram)) { - kvfree(kbuf); - return -EINVAL; - } + down_read(&zram->dev_lock); + if (!init_done(zram)) + return ERR_PTR(-EINVAL); nr_pages = zram->disksize >> PAGE_SHIFT; + if (*pos >= nr_pages) + return NULL; - for (index = *ppos; index < nr_pages; index++) { - int copied; - - slot_lock(zram, index); - if (!slot_allocated(zram, index)) - goto next; - - copied = snprintf(kbuf + written, count, - "%12lu %12u.%06d %c%c%c%c%c%c\n", - index, zram->table[index].attr.ac_time, 0, - test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.', - test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.', - test_slot_flag(zram, index, ZRAM_HUGE) ? 'h' : '.', - test_slot_flag(zram, index, ZRAM_IDLE) ? 'i' : '.', - get_slot_comp_priority(zram, index) ? 'r' : '.', - test_slot_flag(zram, index, - ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); - - if (count <= copied) { - slot_unlock(zram, index); - break; - } - written += copied; - count -= copied; -next: - slot_unlock(zram, index); - *ppos += 1; - } - - if (copy_to_user(buf, kbuf, written)) - written = -EFAULT; - kvfree(kbuf); - - return written; + return pos; } -static const struct file_operations proc_zram_block_state_op = { - .open = simple_open, - .read = read_block_state, - .llseek = default_llseek, +static void *zram_block_state_next(struct seq_file *seq, void *v, loff_t *pos) +{ + struct zram *zram = seq->private; + unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; + + ++*pos; + if (*pos >= nr_pages) + return NULL; + + return pos; +} + +static void zram_block_state_stop(struct seq_file *seq, void *v) +{ + struct zram *zram = seq->private; + + up_read(&zram->dev_lock); +} + +static int zram_block_state_show(struct seq_file *seq, void *v) +{ + struct zram *zram = seq->private; + unsigned long index = *(loff_t *)v; + + slot_lock(zram, index); + if (slot_allocated(zram, index)) { + seq_printf(seq, "%12lu %12u.%06d %c%c%c%c%c%c\n", + index, zram->table[index].attr.ac_time, 0, + test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.', + test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.', + test_slot_flag(zram, index, ZRAM_HUGE) ? 'h' : '.', + test_slot_flag(zram, index, ZRAM_IDLE) ? 'i' : '.', + get_slot_comp_priority(zram, index) ? 'r' : '.', + test_slot_flag(zram, index, + ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); + } + slot_unlock(zram, index); + + return 0; +} + +static const struct seq_operations zram_block_state_sops = { + .start = zram_block_state_start, + .next = zram_block_state_next, + .stop = zram_block_state_stop, + .show = zram_block_state_show, }; +DEFINE_SEQ_ATTRIBUTE(zram_block_state); static void zram_debugfs_register(struct zram *zram) { @@ -1624,7 +1626,7 @@ static void zram_debugfs_register(struct zram *zram) zram->debugfs_dir = debugfs_create_dir(zram->disk->disk_name, zram_debugfs_root); debugfs_create_file("block_state", 0400, zram->debugfs_dir, - zram, &proc_zram_block_state_op); + zram, &zram_block_state_fops); } static void zram_debugfs_unregister(struct zram *zram) From 76a8eb24482ed491c6d314ddfe98740d9f26963b Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:37 +0300 Subject: [PATCH 065/213] sh: remove CONFIG_NUMA and related configuration options Patch series "sh: remove NUMA and SPARSEMEM support", v2. NUMA support for SuperH was introduced a long time ago by commit b241cb0c885e ("sh: Support for multiple nodes.") "... for boards with many different memory blocks that are otherwise unused (SH7722/SH7785 URAM and so forth)" In reality, this added 128K of memory on sh7722 and sh7785 and 256K on shx3 at the expense of all the NUMA related code in the kernel. For build of v7.0-rc7 with defconfig and the same configuration with CONFIG_NUMA disabled, bloat-o-meter reports difference of ~76k. Disabling CONFIG_SPARSMEM on top increases the difference to ~94k. And that's only overhead in code and static data that does not take into the account data structures allocated at run time. And all this overhead has been there for nothing for almost 8 years because since commit ac21fc2dcb40 ("sh: switch to NO_BOOTMEM") those additional "nodes" could not be used by the core MM because the maximal pfn for ZONE_NORMAL was cut out at the end of the normal memory. This patch (of 10): NUMA support for SuperH was introduced a long time ago by commit b241cb0c885e ("sh: Support for multiple nodes.") "... for boards with many different memory blocks that are otherwise unused (SH7722/SH7785 URAM and so forth)" In reality, this added 128K of memory on sh7722 and sh7785 and 256K on shx3 at the expense of all the NUMA related code in the kernel. For build of v7.0-rc7 with defconfig and the same configuration with CONFIG_NUMA disabled, bloat-o-meter reports difference of ~76k. Disabling CONFIG_SPARSMEM on top increases the difference to ~94k. And that's only overhead in code and static data that does not take into the account data structures allocated at run time. And all this overhead has been there for nothing for almost 8 years because since commit ac21fc2dcb40 ("sh: switch to NO_BOOTMEM") those additional "nodes" could not be used by the core MM because the maximal pfn for ZONE_NORMAL was cut out at the end of the normal memory. Start removing NUMA support on sh with removing CONFIG_NUMA and related configuration options in arch/sh/Kconfig and arch/sh/mm/Kconfig. Link: https://lore.kernel.org/20260510135546.13554-1-rppt@kernel.org Link: https://lore.kernel.org/20260510135546.13554-2-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/Kconfig | 6 ------ arch/sh/configs/migor_defconfig | 1 - arch/sh/configs/sdk7786_defconfig | 1 - arch/sh/configs/se7722_defconfig | 1 - arch/sh/configs/shx3_defconfig | 1 - arch/sh/configs/ul2_defconfig | 1 - arch/sh/mm/Kconfig | 19 ------------------- 7 files changed, 30 deletions(-) diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig index fe859def918c..382a58fed557 100644 --- a/arch/sh/Kconfig +++ b/arch/sh/Kconfig @@ -107,9 +107,6 @@ config SYS_SUPPORTS_APM_EMULATION config SYS_SUPPORTS_SMP bool -config SYS_SUPPORTS_NUMA - bool - config STACKTRACE_SUPPORT def_bool y @@ -198,7 +195,6 @@ config CPU_SHX3 bool select DMA_COHERENT select SYS_SUPPORTS_SMP - select SYS_SUPPORTS_NUMA config ARCH_SHMOBILE bool @@ -444,7 +440,6 @@ config CPU_SUBTYPE_SH7785 select CPU_SH4A select CPU_SHX2 select ARCH_SPARSEMEM_ENABLE - select SYS_SUPPORTS_NUMA select PINCTRL config CPU_SUBTYPE_SH7786 @@ -480,7 +475,6 @@ config CPU_SUBTYPE_SH7722 select CPU_SHX2 select ARCH_SHMOBILE select ARCH_SPARSEMEM_ENABLE - select SYS_SUPPORTS_NUMA select SYS_SUPPORTS_SH_CMT select PINCTRL diff --git a/arch/sh/configs/migor_defconfig b/arch/sh/configs/migor_defconfig index 1d9d543eef4c..db993616f14e 100644 --- a/arch/sh/configs/migor_defconfig +++ b/arch/sh/configs/migor_defconfig @@ -9,7 +9,6 @@ CONFIG_MODULES=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SH7722=y CONFIG_MEMORY_START=0x0c000000 -CONFIG_NUMA=y # CONFIG_MIGRATION is not set CONFIG_SH_MIGOR=y # CONFIG_SH_TIMER_CMT is not set diff --git a/arch/sh/configs/sdk7786_defconfig b/arch/sh/configs/sdk7786_defconfig index d6fa3a422e24..36e6dd0622f7 100644 --- a/arch/sh/configs/sdk7786_defconfig +++ b/arch/sh/configs/sdk7786_defconfig @@ -41,7 +41,6 @@ CONFIG_CPU_SUBTYPE_SH7786=y CONFIG_MEMORY_START=0x40000000 CONFIG_MEMORY_SIZE=0x20000000 CONFIG_PMB=y -CONFIG_NUMA=y CONFIG_HUGETLB_PAGE_SIZE_64MB=y CONFIG_KSM=y CONFIG_SH_STORE_QUEUES=y diff --git a/arch/sh/configs/se7722_defconfig b/arch/sh/configs/se7722_defconfig index 85b8eb013b79..2908611b118e 100644 --- a/arch/sh/configs/se7722_defconfig +++ b/arch/sh/configs/se7722_defconfig @@ -10,7 +10,6 @@ CONFIG_MODULE_UNLOAD=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SH7722=y CONFIG_MEMORY_START=0x0c000000 -CONFIG_NUMA=y CONFIG_SH_STORE_QUEUES=y CONFIG_SH_7722_SOLUTION_ENGINE=y # CONFIG_SH_TIMER_CMT is not set diff --git a/arch/sh/configs/shx3_defconfig b/arch/sh/configs/shx3_defconfig index 0cb69a0b92cb..22c242cf89f7 100644 --- a/arch/sh/configs/shx3_defconfig +++ b/arch/sh/configs/shx3_defconfig @@ -29,7 +29,6 @@ CONFIG_MODULE_UNLOAD=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SHX3=y CONFIG_MEMORY_START=0x0c000000 -CONFIG_NUMA=y CONFIG_PAGE_SIZE_64KB=y CONFIG_SH_STORE_QUEUES=y CONFIG_SH_X3PROTO=y diff --git a/arch/sh/configs/ul2_defconfig b/arch/sh/configs/ul2_defconfig index 00a37944b043..1740a6465280 100644 --- a/arch/sh/configs/ul2_defconfig +++ b/arch/sh/configs/ul2_defconfig @@ -10,7 +10,6 @@ CONFIG_MODULE_UNLOAD=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SH7366=y CONFIG_MEMORY_SIZE=0x01f00000 -CONFIG_NUMA=y # CONFIG_MIGRATION is not set # CONFIG_SH_DSP is not set # CONFIG_SH_TIMER_CMT is not set diff --git a/arch/sh/mm/Kconfig b/arch/sh/mm/Kconfig index 1862411665ab..71fecbbb080e 100644 --- a/arch/sh/mm/Kconfig +++ b/arch/sh/mm/Kconfig @@ -112,27 +112,8 @@ config VSYSCALL For systems with an MMU that can afford to give up a page, (the default value) say Y. -config NUMA - bool "Non-Uniform Memory Access (NUMA) Support" - depends on MMU && SYS_SUPPORTS_NUMA - select ARCH_WANT_NUMA_VARIABLE_LOCALITY - default n - help - Some SH systems have many various memories scattered around - the address space, each with varying latencies. This enables - support for these blocks by binding them to nodes and allowing - memory policies to be used for prioritizing and controlling - allocation behaviour. - -config NODES_SHIFT - int - default "3" if CPU_SUBTYPE_SHX3 - default "1" - depends on NUMA - config ARCH_FLATMEM_ENABLE def_bool y - depends on !NUMA config ARCH_SPARSEMEM_ENABLE def_bool y From e800e2aa035268ec9e5b007b702518a7f28c845d Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:38 +0300 Subject: [PATCH 066/213] sh: mm: remove numa.c Since CONFIG_NUMA is removed, numa.c won't be ever compiled. Remove it. Link: https://lore.kernel.org/20260510135546.13554-3-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/mm/Makefile | 1 - arch/sh/mm/numa.c | 53 --------------------------------------------- 2 files changed, 54 deletions(-) delete mode 100644 arch/sh/mm/numa.c diff --git a/arch/sh/mm/Makefile b/arch/sh/mm/Makefile index f69ddc70b146..7033947955d6 100644 --- a/arch/sh/mm/Makefile +++ b/arch/sh/mm/Makefile @@ -37,7 +37,6 @@ endif obj-$(CONFIG_DEBUG_FS) += $(debugfs-y) obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PMB) += pmb.o -obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_IOREMAP_FIXED) += ioremap_fixed.o obj-$(CONFIG_UNCACHED_MAPPING) += uncached.o obj-$(CONFIG_HAVE_SRAM_POOL) += sram.o diff --git a/arch/sh/mm/numa.c b/arch/sh/mm/numa.c deleted file mode 100644 index 9bc212b5e762..000000000000 --- a/arch/sh/mm/numa.c +++ /dev/null @@ -1,53 +0,0 @@ -/* - * arch/sh/mm/numa.c - Multiple node support for SH machines - * - * Copyright (C) 2007 Paul Mundt - * - * This file is subject to the terms and conditions of the GNU General Public - * License. See the file "COPYING" in the main directory of this archive - * for more details. - */ -#include -#include -#include -#include -#include -#include - -/* - * On SH machines the conventional approach is to stash system RAM - * in node 0, and other memory blocks in to node 1 and up, ordered by - * latency. Each node's pgdat is node-local at the beginning of the node, - * immediately followed by the node mem map. - */ -void __init setup_bootmem_node(int nid, unsigned long start, unsigned long end) -{ - unsigned long start_pfn, end_pfn; - - /* Don't allow bogus node assignment */ - BUG_ON(nid >= MAX_NUMNODES || nid <= 0); - - start_pfn = PFN_DOWN(start); - end_pfn = PFN_DOWN(end); - - pmb_bolt_mapping((unsigned long)__va(start), start, end - start, - PAGE_KERNEL); - - memblock_add(start, end - start); - - __add_active_range(nid, start_pfn, end_pfn); - - /* Node-local pgdat */ - NODE_DATA(nid) = memblock_alloc_node(sizeof(struct pglist_data), - SMP_CACHE_BYTES, nid); - if (!NODE_DATA(nid)) - panic("%s: Failed to allocate %zu bytes align=0x%x nid=%d\n", - __func__, sizeof(struct pglist_data), SMP_CACHE_BYTES, - nid); - - NODE_DATA(nid)->node_start_pfn = start_pfn; - NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; - - /* It's up */ - node_set_online(nid); -} From 32dd0053f639468e549c161ad8c75e0fab2840a0 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:39 +0300 Subject: [PATCH 067/213] sh: mm: drop allocate_pgdat() allocate_pgdat() function used to allocate the node data structure and initialize its node_start_pfn and node_spanned_pages fields. Without CONFIG_NUMA there is no need to allocate the node data and setting node_start_pfn and node_spanned_pages are redundant because they are anyway overwritten later by core MM. Remove allocate_pgdat() function. Link: https://lore.kernel.org/20260510135546.13554-4-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/mm/init.c | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c index 93921109f4e6..f97f5c698364 100644 --- a/arch/sh/mm/init.c +++ b/arch/sh/mm/init.c @@ -201,20 +201,6 @@ void __init page_table_range_init(unsigned long start, unsigned long end, } #endif /* CONFIG_MMU */ -void __init allocate_pgdat(unsigned int nid) -{ - unsigned long start_pfn, end_pfn; - - get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); - -#ifdef CONFIG_NUMA - alloc_node_data(nid); -#endif - - NODE_DATA(nid)->node_start_pfn = start_pfn; - NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; -} - static void __init do_init_bootmem(void) { unsigned long start_pfn, end_pfn; @@ -224,8 +210,6 @@ static void __init do_init_bootmem(void) for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) __add_active_range(0, start_pfn, end_pfn); - /* All of system RAM sits in node 0 for the non-NUMA case */ - allocate_pgdat(0); node_set_online(0); plat_mem_setup(); From f979745fc660148e442b17d7500f0c5726a5b421 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:40 +0300 Subject: [PATCH 068/213] sh: remove setup_bootmem_node() and plat_mem_setup() setup_bootmem_node() and plat_mem_setup() were needed to setup NUMA "nodes" for URAM memory ranges. Since there is no NUMA support anymore, these functions are not needed. Remove them. Link: https://lore.kernel.org/20260510135546.13554-5-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/kernel/cpu/sh4a/setup-sh7366.c | 5 ----- arch/sh/kernel/cpu/sh4a/setup-sh7722.c | 6 ------ arch/sh/kernel/cpu/sh4a/setup-sh7757.c | 4 ---- arch/sh/kernel/cpu/sh4a/setup-sh7785.c | 6 ------ arch/sh/kernel/cpu/sh4a/setup-sh7786.c | 4 ---- arch/sh/kernel/cpu/sh4a/setup-shx3.c | 17 ----------------- arch/sh/mm/init.c | 7 ------- 7 files changed, 49 deletions(-) diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7366.c b/arch/sh/kernel/cpu/sh4a/setup-sh7366.c index 6676beef053e..0a4ceb9785f6 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7366.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7366.c @@ -381,8 +381,3 @@ void __init plat_irq_setup(void) { register_intc_controller(&intc_desc); } - -void __init plat_mem_setup(void) -{ - /* TODO: Register Node 1 */ -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7722.c b/arch/sh/kernel/cpu/sh4a/setup-sh7722.c index 0c6757ef63f4..2180819a1455 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7722.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7722.c @@ -658,9 +658,3 @@ void __init plat_irq_setup(void) { register_intc_controller(&intc_desc); } - -void __init plat_mem_setup(void) -{ - /* Register the URAM space as Node 1 */ - setup_bootmem_node(1, 0x055f0000, 0x05610000); -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7757.c b/arch/sh/kernel/cpu/sh4a/setup-sh7757.c index 2ad19a0c5e04..1f4396da00e5 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7757.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7757.c @@ -1239,7 +1239,3 @@ void __init plat_irq_setup_pins(int mode) BUG(); } } - -void __init plat_mem_setup(void) -{ -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7785.c b/arch/sh/kernel/cpu/sh4a/setup-sh7785.c index 3b4a414d60a9..95c3cc15a443 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7785.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7785.c @@ -600,9 +600,3 @@ void __init plat_irq_setup_pins(int mode) BUG(); } } - -void __init plat_mem_setup(void) -{ - /* Register the URAM space as Node 1 */ - setup_bootmem_node(1, 0xe55f0000, 0xe5610000); -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7786.c b/arch/sh/kernel/cpu/sh4a/setup-sh7786.c index c048842d8a58..a46d6c3241a9 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7786.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7786.c @@ -783,10 +783,6 @@ void __init plat_irq_setup_pins(int mode) } } -void __init plat_mem_setup(void) -{ -} - static int __init sh7786_devices_setup(void) { int ret, irq; diff --git a/arch/sh/kernel/cpu/sh4a/setup-shx3.c b/arch/sh/kernel/cpu/sh4a/setup-shx3.c index 60be8dca54f6..84569f163442 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-shx3.c +++ b/arch/sh/kernel/cpu/sh4a/setup-shx3.c @@ -377,20 +377,3 @@ void __init plat_irq_setup(void) { register_intc_controller(&intc_desc); } - -void __init plat_mem_setup(void) -{ - unsigned int nid = 1; - - /* Register CPU#0 URAM space as Node 1 */ - setup_bootmem_node(nid++, 0x145f0000, 0x14610000); /* CPU0 */ - -#if 0 - /* XXX: Not yet.. */ - setup_bootmem_node(nid++, 0x14df0000, 0x14e10000); /* CPU1 */ - setup_bootmem_node(nid++, 0x155f0000, 0x15610000); /* CPU2 */ - setup_bootmem_node(nid++, 0x15df0000, 0x15e10000); /* CPU3 */ -#endif - - setup_bootmem_node(nid++, 0x16000000, 0x16020000); /* CSM */ -} diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c index f97f5c698364..66b529e7acf0 100644 --- a/arch/sh/mm/init.c +++ b/arch/sh/mm/init.c @@ -38,11 +38,6 @@ void __init generic_mem_init(void) memblock_add(__MEMORY_START, __MEMORY_SIZE); } -void __init __weak plat_mem_setup(void) -{ - /* Nothing to see here, move along. */ -} - #ifdef CONFIG_MMU static pte_t *__get_pte_phys(unsigned long addr) { @@ -211,8 +206,6 @@ static void __init do_init_bootmem(void) __add_active_range(0, start_pfn, end_pfn); node_set_online(0); - - plat_mem_setup(); } static void __init early_reserve_mem(void) From acce12b992a423925096d63d1c56b2bbfa655bac Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:41 +0300 Subject: [PATCH 069/213] sh: drop dead code guarded by #ifdef CONFIG_NUMA and NUMA-related comments. Link: https://lore.kernel.org/20260510135546.13554-6-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/include/asm/topology.h | 13 ------------- arch/sh/kernel/setup.c | 1 - arch/sh/kernel/topology.c | 12 ------------ arch/sh/kernel/vmcore_info.c | 4 ---- arch/sh/mm/sram.c | 3 +-- 5 files changed, 1 insertion(+), 32 deletions(-) diff --git a/arch/sh/include/asm/topology.h b/arch/sh/include/asm/topology.h index 1db470e02456..a3c31754ba5f 100644 --- a/arch/sh/include/asm/topology.h +++ b/arch/sh/include/asm/topology.h @@ -2,19 +2,6 @@ #ifndef _ASM_SH_TOPOLOGY_H #define _ASM_SH_TOPOLOGY_H -#ifdef CONFIG_NUMA - -#define cpu_to_node(cpu) ((void)(cpu),0) - -#define cpumask_of_node(node) ((void)node, cpu_online_mask) - -#define pcibus_to_node(bus) ((void)(bus), -1) -#define cpumask_of_pcibus(bus) (pcibus_to_node(bus) == -1 ? \ - cpu_all_mask : \ - cpumask_of_node(pcibus_to_node(bus))) - -#endif - #define mc_capable() (1) const struct cpumask *cpu_coregroup_mask(int cpu); diff --git a/arch/sh/kernel/setup.c b/arch/sh/kernel/setup.c index 039a51291002..5e25c1db4d61 100644 --- a/arch/sh/kernel/setup.c +++ b/arch/sh/kernel/setup.c @@ -227,7 +227,6 @@ void __init __add_active_range(unsigned int nid, unsigned long start_pfn, /* * Also make sure that there is a PMB mapping that covers this * range before we attempt to activate it, to avoid reset by MMU. - * We can hit this path with NUMA or memory hot-add. */ pmb_bolt_mapping((unsigned long)__va(start), start, end - start, PAGE_KERNEL); diff --git a/arch/sh/kernel/topology.c b/arch/sh/kernel/topology.c index 2d2a7509b565..906b7b1d5443 100644 --- a/arch/sh/kernel/topology.c +++ b/arch/sh/kernel/topology.c @@ -57,18 +57,6 @@ static int __init topology_init(void) __func__, i, ret); } -#if defined(CONFIG_NUMA) && !defined(CONFIG_SMP) - /* - * In the UP case, make sure the CPU association is still - * registered under each node. Without this, sysfs fails - * to make the connection between nodes other than node0 - * and cpu0. - */ - for_each_online_node(i) - if (i != numa_node_id()) - register_cpu_under_node(raw_smp_processor_id(), i); -#endif - return 0; } subsys_initcall(topology_init); diff --git a/arch/sh/kernel/vmcore_info.c b/arch/sh/kernel/vmcore_info.c index a244a204a1b1..9f029b4a7e63 100644 --- a/arch/sh/kernel/vmcore_info.c +++ b/arch/sh/kernel/vmcore_info.c @@ -5,10 +5,6 @@ void arch_crash_save_vmcoreinfo(void) { -#ifdef CONFIG_NUMA - VMCOREINFO_SYMBOL(node_data); - VMCOREINFO_LENGTH(node_data, MAX_NUMNODES); -#endif #ifdef CONFIG_X2TLB VMCOREINFO_CONFIG(X2TLB); #endif diff --git a/arch/sh/mm/sram.c b/arch/sh/mm/sram.c index 2d8fa718d55e..208080f072df 100644 --- a/arch/sh/mm/sram.c +++ b/arch/sh/mm/sram.c @@ -16,8 +16,7 @@ * This provides a standard SRAM pool for tiny memories that can be * added either by the CPU or the platform code. Typical SRAM sizes * to be inserted in to the pool will generally be less than the page - * size, with anything more reasonably sized handled as a NUMA memory - * node. + * size. */ struct gen_pool *sram_pool; From 7c6420ecf636adc0b86966397efc0a320cd4845a Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:42 +0300 Subject: [PATCH 070/213] sh: drop include/asm/mmzone.h include/asm/mmzone.h was required for declarations of NUMA related functionality. The only function declared there that's not related to NUMA is __add_active_range(). Move __add_active_range() declaration to include/asm/setup.h and drop include/asm/mmzone.h Link: https://lore.kernel.org/20260510135546.13554-7-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/include/asm/mmzone.h | 42 -------------------------- arch/sh/include/asm/setup.h | 3 ++ arch/sh/kernel/cpu/sh4a/setup-sh7722.c | 1 - arch/sh/kernel/cpu/sh4a/setup-sh7723.c | 1 - arch/sh/kernel/cpu/sh4a/setup-sh7724.c | 1 - arch/sh/kernel/cpu/sh4a/setup-sh7757.c | 1 - arch/sh/kernel/cpu/sh4a/setup-sh7785.c | 1 - arch/sh/kernel/cpu/sh4a/setup-sh7786.c | 1 - arch/sh/kernel/cpu/sh4a/setup-shx3.c | 1 - arch/sh/kernel/setup.c | 2 -- arch/sh/mm/init.c | 1 - 11 files changed, 3 insertions(+), 52 deletions(-) delete mode 100644 arch/sh/include/asm/mmzone.h diff --git a/arch/sh/include/asm/mmzone.h b/arch/sh/include/asm/mmzone.h deleted file mode 100644 index 63f88b465e39..000000000000 --- a/arch/sh/include/asm/mmzone.h +++ /dev/null @@ -1,42 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __ASM_SH_MMZONE_H -#define __ASM_SH_MMZONE_H - -#ifdef CONFIG_NUMA -#include - -static inline int pfn_to_nid(unsigned long pfn) -{ - int nid; - - for (nid = 0; nid < MAX_NUMNODES; nid++) - if (pfn >= node_start_pfn(nid) && pfn <= node_end_pfn(nid)) - break; - - return nid; -} - -static inline struct pglist_data *pfn_to_pgdat(unsigned long pfn) -{ - return NODE_DATA(pfn_to_nid(pfn)); -} - -/* arch/sh/mm/numa.c */ -void __init setup_bootmem_node(int nid, unsigned long start, unsigned long end); -#else -static inline void -setup_bootmem_node(int nid, unsigned long start, unsigned long end) -{ -} -#endif /* CONFIG_NUMA */ - -/* Platform specific mem init */ -void __init plat_mem_setup(void); - -/* arch/sh/kernel/setup.c */ -void __init __add_active_range(unsigned int nid, unsigned long start_pfn, - unsigned long end_pfn); -/* arch/sh/mm/init.c */ -void __init allocate_pgdat(unsigned int nid); - -#endif /* __ASM_SH_MMZONE_H */ diff --git a/arch/sh/include/asm/setup.h b/arch/sh/include/asm/setup.h index 8488f76b48b4..6762612f456c 100644 --- a/arch/sh/include/asm/setup.h +++ b/arch/sh/include/asm/setup.h @@ -24,4 +24,7 @@ void check_for_initrd(void); void per_cpu_trap_init(void); void sh_fdt_init(phys_addr_t dt_phys); +void __add_active_range(unsigned int nid, unsigned long start_pfn, + unsigned long end_pfn); + #endif /* _SH_SETUP_H */ diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7722.c b/arch/sh/kernel/cpu/sh4a/setup-sh7722.c index 2180819a1455..5e2c24c518b4 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7722.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7722.c @@ -16,7 +16,6 @@ #include #include -#include #include #include diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7723.c b/arch/sh/kernel/cpu/sh4a/setup-sh7723.c index d64d28c4f059..5c24ff407c89 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7723.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7723.c @@ -17,7 +17,6 @@ #include #include -#include #include #include diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7724.c b/arch/sh/kernel/cpu/sh4a/setup-sh7724.c index ef4b26a4b3d6..9441e4a0f402 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7724.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7724.c @@ -24,7 +24,6 @@ #include #include #include -#include #include #include diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7757.c b/arch/sh/kernel/cpu/sh4a/setup-sh7757.c index 1f4396da00e5..3750b598d6bd 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7757.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7757.c @@ -21,7 +21,6 @@ #include #include -#include #include static struct plat_sci_port scif2_platform_data = { diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7785.c b/arch/sh/kernel/cpu/sh4a/setup-sh7785.c index 95c3cc15a443..13f08c44fb02 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7785.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7785.c @@ -13,7 +13,6 @@ #include #include #include -#include #include #include diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7786.c b/arch/sh/kernel/cpu/sh4a/setup-sh7786.c index a46d6c3241a9..79484c598b83 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7786.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7786.c @@ -22,7 +22,6 @@ #include #include #include -#include #include static struct plat_sci_port scif0_platform_data = { diff --git a/arch/sh/kernel/cpu/sh4a/setup-shx3.c b/arch/sh/kernel/cpu/sh4a/setup-shx3.c index 84569f163442..100a91034f2c 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-shx3.c +++ b/arch/sh/kernel/cpu/sh4a/setup-shx3.c @@ -13,7 +13,6 @@ #include #include #include -#include #include /* diff --git a/arch/sh/kernel/setup.c b/arch/sh/kernel/setup.c index 5e25c1db4d61..2f2764061003 100644 --- a/arch/sh/kernel/setup.c +++ b/arch/sh/kernel/setup.c @@ -23,7 +23,6 @@ #include #include #include -#include #include #include #include @@ -41,7 +40,6 @@ #include #include #include -#include #include #include #include diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c index 66b529e7acf0..b34a5c304abd 100644 --- a/arch/sh/mm/init.c +++ b/arch/sh/mm/init.c @@ -20,7 +20,6 @@ #include #include #include -#include #include #include #include From ab64cfe13cd7a0ee29deae0f2a0a9e971698ab63 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:43 +0300 Subject: [PATCH 071/213] init/Kconfig: drop ARCH_WANT_NUMA_VARIABLE_LOCALITY sh was the only architecture that selected ARCH_WANT_NUMA_VARIABLE_LOCALITY. With NUMA support on sh gone, there is no need in this configuration option. Remove it. Link: https://lore.kernel.org/20260510135546.13554-8-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- init/Kconfig | 7 ------- 1 file changed, 7 deletions(-) diff --git a/init/Kconfig b/init/Kconfig index ad592fdf29af..3b03a7f59b47 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1017,17 +1017,10 @@ config CC_STRINGOP_OVERFLOW config ARCH_SUPPORTS_INT128 bool -# For architectures that (ab)use NUMA to represent different memory regions -# all cpu-local but of different latencies, such as SuperH. -# -config ARCH_WANT_NUMA_VARIABLE_LOCALITY - bool - config NUMA_BALANCING bool "Memory placement aware NUMA scheduler" depends on ARCH_SUPPORTS_NUMA_BALANCING depends on ARCH_HAS_PTE_PROTNONE - depends on !ARCH_WANT_NUMA_VARIABLE_LOCALITY depends on SMP && NUMA_MIGRATION && !PREEMPT_RT help This option adds support for automatic NUMA aware memory/task placement. From 01640eb373cacf8bd3821725667e208fadb89b8c Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:44 +0300 Subject: [PATCH 072/213] sh: init: remove call the memblock_set_node() Without CONFIG_NUMA, the call to memblock_set_node() is not needed. Remove it. Link: https://lore.kernel.org/20260510135546.13554-9-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/kernel/setup.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/sh/kernel/setup.c b/arch/sh/kernel/setup.c index 2f2764061003..43d0f7b62b03 100644 --- a/arch/sh/kernel/setup.c +++ b/arch/sh/kernel/setup.c @@ -228,9 +228,6 @@ void __init __add_active_range(unsigned int nid, unsigned long start_pfn, */ pmb_bolt_mapping((unsigned long)__va(start), start, end - start, PAGE_KERNEL); - - memblock_set_node(PFN_PHYS(start_pfn), PFN_PHYS(end_pfn - start_pfn), - &memblock.memory, nid); } void __init __weak plat_early_device_setup(void) From c57f8d68a9bf18e143cf7c6a6f46265ff4b79ed1 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:45 +0300 Subject: [PATCH 073/213] sh: remove SPARSEMEM related entries from Kconfig Without NUMA support, SPARSEMEM related configuration options only create unnecessary overhead. Remove them. Link: https://lore.kernel.org/20260510135546.13554-10-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/Kconfig | 6 ------ arch/sh/mm/Kconfig | 10 ---------- 2 files changed, 16 deletions(-) diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig index 382a58fed557..d1c9d8e2f893 100644 --- a/arch/sh/Kconfig +++ b/arch/sh/Kconfig @@ -387,7 +387,6 @@ config CPU_SUBTYPE_SH7723 select CPU_SH4A select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE select SYS_SUPPORTS_SH_CMT select PINCTRL help @@ -398,7 +397,6 @@ config CPU_SUBTYPE_SH7724 select CPU_SH4A select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE select SYS_SUPPORTS_SH_CMT select PINCTRL help @@ -439,7 +437,6 @@ config CPU_SUBTYPE_SH7785 bool "Support SH7785 processor" select CPU_SH4A select CPU_SHX2 - select ARCH_SPARSEMEM_ENABLE select PINCTRL config CPU_SUBTYPE_SH7786 @@ -474,7 +471,6 @@ config CPU_SUBTYPE_SH7722 select CPU_SH4AL_DSP select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE select SYS_SUPPORTS_SH_CMT select PINCTRL @@ -483,8 +479,6 @@ config CPU_SUBTYPE_SH7366 select CPU_SH4AL_DSP select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE - select SYS_SUPPORTS_NUMA select SYS_SUPPORTS_SH_CMT endchoice diff --git a/arch/sh/mm/Kconfig b/arch/sh/mm/Kconfig index 71fecbbb080e..c7d889a1a9d7 100644 --- a/arch/sh/mm/Kconfig +++ b/arch/sh/mm/Kconfig @@ -115,16 +115,6 @@ config VSYSCALL config ARCH_FLATMEM_ENABLE def_bool y -config ARCH_SPARSEMEM_ENABLE - def_bool y - select SPARSEMEM_STATIC - -config ARCH_SPARSEMEM_DEFAULT - def_bool y - -config ARCH_SELECT_MEMORY_MODEL - def_bool y - config IOREMAP_FIXED def_bool y depends on X2TLB From 226593884252df75c61d0fb6221864969dffa3c8 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Sun, 10 May 2026 16:55:46 +0300 Subject: [PATCH 074/213] sh: drop include/asm/sparsemem.h Move MAX_PHYSMEM_BITS declaration to include/asm/mmu.h and drop now unneeded include/asm/sparsemem.h Link: https://lore.kernel.org/20260510135546.13554-11-rppt@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Reviewed-by: Arnd Bergmann Cc: John Paul Adrian Glaubitz Cc: Rich Felker Cc: Yoshinori Sato --- arch/sh/include/asm/mmu.h | 2 ++ arch/sh/include/asm/sparsemem.h | 12 ------------ arch/sh/kernel/setup.c | 1 - 3 files changed, 2 insertions(+), 13 deletions(-) delete mode 100644 arch/sh/include/asm/sparsemem.h diff --git a/arch/sh/include/asm/mmu.h b/arch/sh/include/asm/mmu.h index b9c9f91e6616..def7c1edd605 100644 --- a/arch/sh/include/asm/mmu.h +++ b/arch/sh/include/asm/mmu.h @@ -2,6 +2,8 @@ #ifndef __MMU_H #define __MMU_H +#define MAX_PHYSMEM_BITS 32 + /* * Privileged Space Mapping Buffer (PMB) definitions */ diff --git a/arch/sh/include/asm/sparsemem.h b/arch/sh/include/asm/sparsemem.h deleted file mode 100644 index 4703cbe23844..000000000000 --- a/arch/sh/include/asm/sparsemem.h +++ /dev/null @@ -1,12 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __ASM_SH_SPARSEMEM_H -#define __ASM_SH_SPARSEMEM_H - -/* - * SECTION_SIZE_BITS 2^N: how big each section will be - * MAX_PHYSMEM_BITS 2^N: how much physical address space we have - */ -#define SECTION_SIZE_BITS 26 -#define MAX_PHYSMEM_BITS 32 - -#endif /* __ASM_SH_SPARSEMEM_H */ diff --git a/arch/sh/kernel/setup.c b/arch/sh/kernel/setup.c index 43d0f7b62b03..99f52d888aa3 100644 --- a/arch/sh/kernel/setup.c +++ b/arch/sh/kernel/setup.c @@ -41,7 +41,6 @@ #include #include #include -#include #include /* From 07c81e0196cc8fbc261095c9f712e39b63e3c780 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:58:53 +0200 Subject: [PATCH 075/213] mm/sparse: move mem_section init to sparse_extreme_init() Patch series "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups", v2. SECTION_MARKED_PRESENT is really only needed during boot, where we can instead just rely on SECTION_IS_EARLY_BIT by setting that flag earlier. Some preparations for doing that conversion and some cleanups in the same (sparse) area. This patch (of 13): Let's just avoid another pair of ifdef inside a function. While at it, switch to INTERNODE_CACHE_BYTES by just defining a fallback in cache.h as well, given that the x86 variant already provides one. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-0-54d81d65e125@kernel.org Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-1-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Zi Yan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- include/linux/cache.h | 1 + mm/sparse.c | 19 ++++++++++++------- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/include/linux/cache.h b/include/linux/cache.h index e69768f50d53..b6e857b985bc 100644 --- a/include/linux/cache.h +++ b/include/linux/cache.h @@ -89,6 +89,7 @@ */ #ifndef INTERNODE_CACHE_SHIFT #define INTERNODE_CACHE_SHIFT L1_CACHE_SHIFT +#define INTERNODE_CACHE_BYTES (1 << INTERNODE_CACHE_SHIFT) #endif #if !defined(____cacheline_internodealigned_in_smp) diff --git a/mm/sparse.c b/mm/sparse.c index b75921c622ed..8cd2b06b231a 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -103,11 +103,22 @@ int __meminit sparse_index_init(unsigned long section_nr, int nid) return 0; } + +static void __init sparse_extreme_init(void) +{ + const unsigned long size = sizeof(struct mem_section *) * NR_SECTION_ROOTS; + + mem_section = memblock_alloc_or_panic(size, INTERNODE_CACHE_BYTES); +} #else /* !SPARSEMEM_EXTREME */ int __meminit sparse_index_init(unsigned long section_nr, int nid) { return 0; } + +static void __init sparse_extreme_init(void) +{ +} #endif /* @@ -196,13 +207,7 @@ void __init sparse_sections_init(void) unsigned long start, end; int i, nid; -#ifdef CONFIG_SPARSEMEM_EXTREME - unsigned long size, align; - - size = sizeof(struct mem_section *) * NR_SECTION_ROOTS; - align = 1 << (INTERNODE_CACHE_SHIFT); - mem_section = memblock_alloc_or_panic(size, align); -#endif + sparse_extreme_init(); for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) memory_present(nid, start, end); From 89e2b6e3d345cd4dca822a5ccea764ee7f9e0ae4 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:58:54 +0200 Subject: [PATCH 076/213] mm/sparse: refactor sparse_sections_init() memory_present() really identifies+prepares all early sections so the initialization in sparse_init() can properly iterating them to initialize metadata. Let's just inline memory_present() into sparse_sections_init() and cleaning up the code a bit while at it: make it clear that we are operating on pfns. Note that we call set_section_nid() now only if the section was not already created earlier. Now, there is no more inconsistency between what we (temporarily) store in ms->section_mem_map and what we store in our section->nid array. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-2-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- mm/sparse.c | 41 +++++++++++++++++------------------------ 1 file changed, 17 insertions(+), 24 deletions(-) diff --git a/mm/sparse.c b/mm/sparse.c index 8cd2b06b231a..2b285653abf1 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -179,40 +179,33 @@ static inline unsigned long first_present_section_nr(void) return next_present_section_nr(-1); } -/* Record a memory area against a node. */ -static void __init memory_present(int nid, unsigned long start, unsigned long end) +void __init sparse_sections_init(void) { - unsigned long pfn; + unsigned long pfn, start_pfn, end_pfn; + int i, nid; - start &= PAGE_SECTION_MASK; - mminit_validate_memmodel_limits(&start, &end); - for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) { - unsigned long section_nr = pfn_to_section_nr(pfn); - struct mem_section *ms; + sparse_extreme_init(); - sparse_index_init(section_nr, nid); - set_section_nid(section_nr, nid); + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { + start_pfn &= PAGE_SECTION_MASK; + mminit_validate_memmodel_limits(&start_pfn, &end_pfn); - ms = __nr_to_section(section_nr); - if (!ms->section_mem_map) { + for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) { + unsigned long section_nr = pfn_to_section_nr(pfn); + struct mem_section *ms; + + sparse_index_init(section_nr, nid); + ms = __nr_to_section(section_nr); + if (ms->section_mem_map) + continue; + + set_section_nid(section_nr, nid); ms->section_mem_map = sparse_encode_early_nid(nid) | SECTION_IS_ONLINE; __section_mark_present(ms, section_nr); } } } - -void __init sparse_sections_init(void) -{ - unsigned long start, end; - int i, nid; - - sparse_extreme_init(); - - for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) - memory_present(nid, start, end); -} - #ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, From d8ab71169ce579871d196a4178c51afd1e0d8b32 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:58:55 +0200 Subject: [PATCH 077/213] mm/sparse: move initialization of section metadata to sparse_metadata_init() Let's move the code responsible for initializing sparse metadata (usemap, memmap) into a helper. Cleanup the variable while at it (e.g., "map_count"). Drop the rather obvious code comments. No functional change intended. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-3-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- mm/sparse.c | 43 ++++++++++++++++++++++--------------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/mm/sparse.c b/mm/sparse.c index 2b285653abf1..71288daeb366 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -255,38 +255,39 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, } } +static void __init sparse_metadata_init(void) +{ + unsigned long start_section_nr = first_present_section_nr(); + int nid_begin = sparse_early_nid(__nr_to_section(start_section_nr)); + unsigned long section_nr, nr_sections = 1; + + for_each_present_section_nr(start_section_nr + 1, section_nr) { + const int nid = sparse_early_nid(__nr_to_section(section_nr)); + + if (nid == nid_begin) { + nr_sections++; + continue; + } + sparse_init_nid(nid_begin, start_section_nr, section_nr, nr_sections); + nid_begin = nid; + start_section_nr = section_nr; + nr_sections = 1; + } + sparse_init_nid(nid_begin, start_section_nr, section_nr, nr_sections); +} + /* * Allocate the accumulated non-linear sections, allocate a mem_map * for each and record the physical to section mapping. */ void __init sparse_init(void) { - unsigned long pnum_end, pnum_begin, map_count = 1; - int nid_begin; - if (compound_info_has_mask()) { VM_WARN_ON_ONCE(!IS_ALIGNED((unsigned long) pfn_to_page(0), MAX_FOLIO_VMEMMAP_ALIGN)); } - pnum_begin = first_present_section_nr(); - nid_begin = sparse_early_nid(__nr_to_section(pnum_begin)); - - for_each_present_section_nr(pnum_begin + 1, pnum_end) { - int nid = sparse_early_nid(__nr_to_section(pnum_end)); - - if (nid == nid_begin) { - map_count++; - continue; - } - /* Init node with sections in range [pnum_begin, pnum_end) */ - sparse_init_nid(nid_begin, pnum_begin, pnum_end, map_count); - nid_begin = nid; - pnum_begin = pnum_end; - map_count = 1; - } - /* cover the last node */ - sparse_init_nid(nid_begin, pnum_begin, pnum_end, map_count); + sparse_metadata_init(); sparse_init_subsection_map(); vmemmap_populate_print_last(); } From 31b3829703ec5a041c01162f8196fc49eb930b17 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:58:56 +0200 Subject: [PATCH 078/213] mm/sparse: rename and cleanup sparse_init_nid() Let's rename it to "sparse_metadata_init_nid", avoid the "pnum" terminology and drop the function comment. Further, rename the "map" variable to "mem_map" for consistency with other functions. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-4-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- mm/sparse.c | 41 ++++++++++++++++++++--------------------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/mm/sparse.c b/mm/sparse.c index 71288daeb366..3c19d4755909 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -221,36 +221,33 @@ void __weak __meminit vmemmap_populate_print_last(void) { } -/* - * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_end) - * And number of present sections in this node is map_count. - */ -static void __init sparse_init_nid(int nid, unsigned long pnum_begin, - unsigned long pnum_end, - unsigned long map_count) +static void __init sparse_metadata_init_nid(int nid, + unsigned long start_section_nr, unsigned long end_section_nr, + unsigned long nr_sections) { - unsigned long pnum; struct mem_section_usage *usage; + unsigned long section_nr; - usage = memblock_alloc_node(map_count * mem_section_usage_size(), + usage = memblock_alloc_node(nr_sections * mem_section_usage_size(), SMP_CACHE_BYTES, nid); if (!usage) panic("Failed to allocate usemap for node %d\n", nid); - for_each_present_section_nr(pnum_begin, pnum) { - unsigned long pfn = section_nr_to_pfn(pnum); - struct page *map; + for_each_present_section_nr(start_section_nr, section_nr) { + const unsigned long pfn = section_nr_to_pfn(section_nr); + struct page *mem_map; - if (pnum >= pnum_end) + if (section_nr >= end_section_nr) break; - map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) - panic("Failed to allocate memmap for section %lu\n", pnum); + mem_map = __populate_section_memmap(pfn, PAGES_PER_SECTION, nid, + NULL, NULL); + if (!mem_map) + panic("Failed to allocate memmap for section %lu\n", + section_nr); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); - sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, - SECTION_IS_EARLY); + sparse_init_one_section(__nr_to_section(section_nr), section_nr, + mem_map, usage, SECTION_IS_EARLY); usage = (void *)usage + mem_section_usage_size(); } } @@ -268,12 +265,14 @@ static void __init sparse_metadata_init(void) nr_sections++; continue; } - sparse_init_nid(nid_begin, start_section_nr, section_nr, nr_sections); + sparse_metadata_init_nid(nid_begin, start_section_nr, + section_nr, nr_sections); nid_begin = nid; start_section_nr = section_nr; nr_sections = 1; } - sparse_init_nid(nid_begin, start_section_nr, section_nr, nr_sections); + sparse_metadata_init_nid(nid_begin, start_section_nr, section_nr, + nr_sections); } /* From 2193260fd4c2173239dab9b7e7ebebdb95e180cc Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:58:57 +0200 Subject: [PATCH 079/213] mm/sparse: cleanup sparse_init_one_section() Let's avoid the "pnum" terminology. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-5-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- mm/sparse.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/sparse.h b/mm/sparse.h index 530692cdd516..4ee507c34b98 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -20,7 +20,7 @@ void sparse_sections_init(void); int sparse_index_init(unsigned long section_nr, int nid); static inline void sparse_init_one_section(struct mem_section *ms, - unsigned long pnum, struct page *mem_map, + unsigned long section_nr, struct page *mem_map, struct mem_section_usage *usage, unsigned long flags) { unsigned long coded_mem_map; @@ -32,7 +32,7 @@ static inline void sparse_init_one_section(struct mem_section *ms, * page_to_pfn() on !CONFIG_SPARSEMEM_VMEMMAP can simply subtract it * from the page pointer to obtain the PFN. */ - coded_mem_map = (unsigned long)(mem_map - section_nr_to_pfn(pnum)); + coded_mem_map = (unsigned long)(mem_map - section_nr_to_pfn(section_nr)); VM_WARN_ON_ONCE(coded_mem_map & ~SECTION_MAP_MASK); ms->section_mem_map &= ~SECTION_MAP_MASK; From a8c5020eeb963afa18f9f035fcea64f05080b96f Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:58:58 +0200 Subject: [PATCH 080/213] mm/sparse: rename __highest_present_section_nr to __highest_used_section_nr In preparation for getting rid of SECTION_MARKED_PRESENT, rename __highest_present_section_nr and clarify the comment. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-6-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- include/linux/mmzone.h | 6 +++--- mm/compaction.c | 2 +- mm/sparse.c | 12 ++++-------- mm/sparse.h | 4 ++-- 4 files changed, 10 insertions(+), 14 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 4a5c826daec7..628c13f19bec 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2155,7 +2155,7 @@ static inline struct mem_section *__pfn_to_section(unsigned long pfn) return __nr_to_section(pfn_to_section_nr(pfn)); } -extern unsigned long __highest_present_section_nr; +extern unsigned long __highest_used_section_nr; static inline int subsection_map_index(unsigned long pfn) { @@ -2254,7 +2254,7 @@ static inline unsigned long first_valid_pfn(unsigned long pfn, unsigned long end rcu_read_lock_sched(); - while (nr <= __highest_present_section_nr && pfn < end_pfn) { + while (nr <= __highest_used_section_nr && pfn < end_pfn) { struct mem_section *ms = __pfn_to_section(pfn); if (valid_section(ms) && @@ -2309,7 +2309,7 @@ static inline int pfn_in_present_section(unsigned long pfn) static inline unsigned long next_present_section_nr(unsigned long section_nr) { - while (++section_nr <= __highest_present_section_nr) { + while (++section_nr <= __highest_used_section_nr) { if (present_section_nr(section_nr)) return section_nr; } diff --git a/mm/compaction.c b/mm/compaction.c index 4994e200bbec..f1b2060eb201 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -216,7 +216,7 @@ static unsigned long skip_offline_sections(unsigned long start_pfn) if (online_section_nr(start_nr)) return 0; - while (++start_nr <= __highest_present_section_nr) { + while (++start_nr <= __highest_used_section_nr) { if (online_section_nr(start_nr)) return section_nr_to_pfn(start_nr); } diff --git a/mm/sparse.c b/mm/sparse.c index 3c19d4755909..bb89017254f4 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -164,15 +164,11 @@ static void __init mminit_validate_memmodel_limits(unsigned long *start_pfn, } /* - * There are a number of times that we loop over NR_MEM_SECTIONS, - * looking for section_present() on each. But, when we have very - * large physical address spaces, NR_MEM_SECTIONS can also be - * very large which makes the loops quite long. - * - * Keeping track of this gives us an easy way to break out of - * those loops early. + * Looping over all possible memory sections is expensive, especially if + * NR_MEM_SECTIONS is large but only a fraction is actually used. Keep track of + * the highest section number we ever used. */ -unsigned long __highest_present_section_nr; +unsigned long __highest_used_section_nr; static inline unsigned long first_present_section_nr(void) { diff --git a/mm/sparse.h b/mm/sparse.h index 4ee507c34b98..242b7bab0013 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -44,8 +44,8 @@ static inline void sparse_init_one_section(struct mem_section *ms, static inline void __section_mark_present(struct mem_section *ms, unsigned long section_nr) { - if (section_nr > __highest_present_section_nr) - __highest_present_section_nr = section_nr; + if (section_nr > __highest_used_section_nr) + __highest_used_section_nr = section_nr; ms->section_mem_map |= SECTION_MARKED_PRESENT; } From dff2d55bf7941677bd26f12344211ccc289a215d Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:58:59 +0200 Subject: [PATCH 081/213] mm/sparse: remove pfn_in_present_section() Unused, let's remove it. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-7-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- include/linux/mmzone.h | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 628c13f19bec..5c052c08d9aa 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2300,13 +2300,6 @@ static inline unsigned long next_valid_pfn(unsigned long pfn, unsigned long end_ #endif -static inline int pfn_in_present_section(unsigned long pfn) -{ - if (pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS) - return 0; - return present_section(__pfn_to_section(pfn)); -} - static inline unsigned long next_present_section_nr(unsigned long section_nr) { while (++section_nr <= __highest_used_section_nr) { @@ -2336,9 +2329,6 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) #else #define pfn_to_nid(pfn) (0) #endif - -#else -#define pfn_in_present_section pfn_valid #endif /* CONFIG_SPARSEMEM */ /* From 4af04bd8121fbd44891e6db2ca138c6d739b1f82 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:59:00 +0200 Subject: [PATCH 082/213] mm/sparse: move __highest_used_section_nr handling In preparation for removing __section_mark_present(), let's move __highest_used_section_nr handling into its callers. Verify in sparse_init_one_section() that it was properly updated. In sparse_sections_init() we can just set it to the last processed section_nr. No functional change intended. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-8-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reviewed-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- mm/sparse-vmemmap.c | 1 + mm/sparse.c | 5 +++-- mm/sparse.h | 4 +--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 23b31bc4bf94..d5fd61c83bac 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -904,6 +904,7 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, page_init_poison(memmap, sizeof(struct page) * nr_pages); __section_mark_present(ms, section_nr); + __highest_used_section_nr = max(section_nr, __highest_used_section_nr); /* Align memmap to section boundary in the subsection case */ if (section_nr_to_pfn(section_nr) != start_pfn) diff --git a/mm/sparse.c b/mm/sparse.c index bb89017254f4..a0f50ca5acf5 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -177,7 +177,7 @@ static inline unsigned long first_present_section_nr(void) void __init sparse_sections_init(void) { - unsigned long pfn, start_pfn, end_pfn; + unsigned long pfn, start_pfn, end_pfn, section_nr; int i, nid; sparse_extreme_init(); @@ -187,9 +187,9 @@ void __init sparse_sections_init(void) mminit_validate_memmodel_limits(&start_pfn, &end_pfn); for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) { - unsigned long section_nr = pfn_to_section_nr(pfn); struct mem_section *ms; + section_nr = pfn_to_section_nr(pfn); sparse_index_init(section_nr, nid); ms = __nr_to_section(section_nr); if (ms->section_mem_map) @@ -201,6 +201,7 @@ void __init sparse_sections_init(void) __section_mark_present(ms, section_nr); } } + __highest_used_section_nr = section_nr; } #ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, diff --git a/mm/sparse.h b/mm/sparse.h index 242b7bab0013..c396d5c05cfc 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -26,6 +26,7 @@ static inline void sparse_init_one_section(struct mem_section *ms, unsigned long coded_mem_map; BUILD_BUG_ON(SECTION_MAP_LAST_BIT > PFN_SECTION_SHIFT); + VM_WARN_ON_ONCE(section_nr > __highest_used_section_nr); /* * We encode the start PFN of the section into the mem_map such that @@ -44,9 +45,6 @@ static inline void sparse_init_one_section(struct mem_section *ms, static inline void __section_mark_present(struct mem_section *ms, unsigned long section_nr) { - if (section_nr > __highest_used_section_nr) - __highest_used_section_nr = section_nr; - ms->section_mem_map |= SECTION_MARKED_PRESENT; } From 4d42b10363f55baa72de3c6d46715394997d5ceb Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:59:01 +0200 Subject: [PATCH 083/213] scripts/gdb: mm.py: remove fallbacks for SECTION_HAS_MEM_MAP and SECTION_IS_EARLY There is no reason to handle the absence of the corresponding BIT values. So let's remove the fallbacks. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-9-54d81d65e125@kernel.org Link: https://lore.kernel.org/r/vpz5zqg5nwbpolxbkip45vxxdt3lni7j2haal7q42skc3b57tu@uh4cec4x32dq Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Cc: Seongjun Hong Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Oscar Salvador Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- scripts/gdb/linux/mm.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index 193a88d763ab..7398105ff3f6 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -71,12 +71,8 @@ class x86_page_ops(): self.NR_SECTION_ROOTS = DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.SECTIONS_PER_ROOT) - try: - self.SECTION_HAS_MEM_MAP = 1 << int(gdb.parse_and_eval('SECTION_HAS_MEM_MAP_BIT')) - self.SECTION_IS_EARLY = 1 << int(gdb.parse_and_eval('SECTION_IS_EARLY_BIT')) - except: - self.SECTION_HAS_MEM_MAP = 1 << 0 - self.SECTION_IS_EARLY = 1 << 3 + self.SECTION_HAS_MEM_MAP = 1 << int(gdb.parse_and_eval('SECTION_HAS_MEM_MAP_BIT')) + self.SECTION_IS_EARLY = 1 << int(gdb.parse_and_eval('SECTION_IS_EARLY_BIT')) self.SUBSECTION_SHIFT = 21 self.PAGES_PER_SUBSECTION = 1 << (self.SUBSECTION_SHIFT - self.PAGE_SHIFT) From da7acb8982edf70078b23f878c6b8f2b496abdb5 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:59:02 +0200 Subject: [PATCH 084/213] mm/sparse: remove SECTION_MARKED_PRESENT All present section iterators run before memory hotplug added any further memory sections, Therefore, we can simply use the SECTION_IS_EARLY flag by setting that flag earlier in sparse_sections_init(). Get rid of SECTION_MARKED_PRESENT entirely and rename for_each_present_section_nr() to for_each_early_section_nr(). Take care of the .clang-format for_each_present_section_nr() handling. No functional change intended. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-10-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- .clang-format | 2 +- drivers/base/memory.c | 2 +- include/linux/mmzone.h | 27 ++++++++++----------------- mm/sparse-vmemmap.c | 1 - mm/sparse.c | 17 ++++++++--------- mm/sparse.h | 6 ------ 6 files changed, 20 insertions(+), 35 deletions(-) diff --git a/.clang-format b/.clang-format index 5ef5743b77c9..395292ab2ac6 100644 --- a/.clang-format +++ b/.clang-format @@ -282,6 +282,7 @@ ForEachMacros: - 'for_each_dpcm_fe' - 'for_each_drhd_unit' - 'for_each_dss_dev' + - 'for_each_early_section_nr' - 'for_each_efi_memory_desc' - 'for_each_efi_memory_desc_in_map' - 'for_each_element' @@ -403,7 +404,6 @@ ForEachMacros: - 'for_each_possible_cpu_wrap' - 'for_each_present_blessed_reg' - 'for_each_present_cpu' - - 'for_each_present_section_nr' - 'for_each_prime_number' - 'for_each_prime_number_from' - 'for_each_probe_cache_entry' diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 5eead3346f1e..b0338de2f1d8 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -972,7 +972,7 @@ void __init memory_dev_init(void) * block so that it can be covered. */ block_id = ULONG_MAX; - for_each_present_section_nr(0, nr) { + for_each_early_section_nr(0, nr) { if (block_id != ULONG_MAX && memory_block_id(nr) == block_id) continue; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5c052c08d9aa..3b96d6c7123b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2059,7 +2059,6 @@ static inline struct mem_section *__nr_to_section(unsigned long nr) * accommodate SECTION_MAP_LAST_BIT. We use BUILD_BUG_ON() to ensure this. */ enum { - SECTION_MARKED_PRESENT_BIT, SECTION_HAS_MEM_MAP_BIT, SECTION_IS_ONLINE_BIT, SECTION_IS_EARLY_BIT, @@ -2069,7 +2068,6 @@ enum { SECTION_MAP_LAST_BIT, }; -#define SECTION_MARKED_PRESENT BIT(SECTION_MARKED_PRESENT_BIT) #define SECTION_HAS_MEM_MAP BIT(SECTION_HAS_MEM_MAP_BIT) #define SECTION_IS_ONLINE BIT(SECTION_IS_ONLINE_BIT) #define SECTION_IS_EARLY BIT(SECTION_IS_EARLY_BIT) @@ -2086,16 +2084,6 @@ static inline struct page *__section_mem_map_addr(struct mem_section *section) return (struct page *)map; } -static inline int present_section(const struct mem_section *section) -{ - return (section && (section->section_mem_map & SECTION_MARKED_PRESENT)); -} - -static inline int present_section_nr(unsigned long nr) -{ - return present_section(__nr_to_section(nr)); -} - static inline int valid_section(const struct mem_section *section) { return (section && (section->section_mem_map & SECTION_HAS_MEM_MAP)); @@ -2111,6 +2099,11 @@ static inline int valid_section_nr(unsigned long nr) return valid_section(__nr_to_section(nr)); } +static inline int early_section_nr(unsigned long nr) +{ + return early_section(__nr_to_section(nr)); +} + static inline int online_section(const struct mem_section *section) { return (section && (section->section_mem_map & SECTION_IS_ONLINE)); @@ -2300,20 +2293,20 @@ static inline unsigned long next_valid_pfn(unsigned long pfn, unsigned long end_ #endif -static inline unsigned long next_present_section_nr(unsigned long section_nr) +static inline unsigned long next_early_section_nr(unsigned long section_nr) { while (++section_nr <= __highest_used_section_nr) { - if (present_section_nr(section_nr)) + if (early_section_nr(section_nr)) return section_nr; } return -1; } -#define for_each_present_section_nr(start, section_nr) \ - for (section_nr = next_present_section_nr(start - 1); \ +#define for_each_early_section_nr(start, section_nr) \ + for (section_nr = next_early_section_nr(start - 1); \ section_nr != -1; \ - section_nr = next_present_section_nr(section_nr)) + section_nr = next_early_section_nr(section_nr)) /* * These are _only_ used during initialisation, therefore they diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index d5fd61c83bac..09376f5b1d92 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -903,7 +903,6 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (!section_vmemmap_optimizable(ms)) page_init_poison(memmap, sizeof(struct page) * nr_pages); - __section_mark_present(ms, section_nr); __highest_used_section_nr = max(section_nr, __highest_used_section_nr); /* Align memmap to section boundary in the subsection case */ diff --git a/mm/sparse.c b/mm/sparse.c index a0f50ca5acf5..5d2d9f95d402 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -170,13 +170,14 @@ static void __init mminit_validate_memmodel_limits(unsigned long *start_pfn, */ unsigned long __highest_used_section_nr; -static inline unsigned long first_present_section_nr(void) +static inline unsigned long first_early_section_nr(void) { - return next_present_section_nr(-1); + return next_early_section_nr(-1); } void __init sparse_sections_init(void) { + const unsigned long flags = SECTION_IS_EARLY | SECTION_IS_ONLINE; unsigned long pfn, start_pfn, end_pfn, section_nr; int i, nid; @@ -196,9 +197,7 @@ void __init sparse_sections_init(void) continue; set_section_nid(section_nr, nid); - ms->section_mem_map = sparse_encode_early_nid(nid) | - SECTION_IS_ONLINE; - __section_mark_present(ms, section_nr); + ms->section_mem_map = sparse_encode_early_nid(nid) | flags; } } __highest_used_section_nr = section_nr; @@ -230,7 +229,7 @@ static void __init sparse_metadata_init_nid(int nid, if (!usage) panic("Failed to allocate usemap for node %d\n", nid); - for_each_present_section_nr(start_section_nr, section_nr) { + for_each_early_section_nr(start_section_nr, section_nr) { const unsigned long pfn = section_nr_to_pfn(section_nr); struct page *mem_map; @@ -244,18 +243,18 @@ static void __init sparse_metadata_init_nid(int nid, section_nr); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); sparse_init_one_section(__nr_to_section(section_nr), section_nr, - mem_map, usage, SECTION_IS_EARLY); + mem_map, usage, 0); usage = (void *)usage + mem_section_usage_size(); } } static void __init sparse_metadata_init(void) { - unsigned long start_section_nr = first_present_section_nr(); + unsigned long start_section_nr = first_early_section_nr(); int nid_begin = sparse_early_nid(__nr_to_section(start_section_nr)); unsigned long section_nr, nr_sections = 1; - for_each_present_section_nr(start_section_nr + 1, section_nr) { + for_each_early_section_nr(start_section_nr + 1, section_nr) { const int nid = sparse_early_nid(__nr_to_section(section_nr)); if (nid == nid_begin) { diff --git a/mm/sparse.h b/mm/sparse.h index c396d5c05cfc..19664d702af8 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -42,12 +42,6 @@ static inline void sparse_init_one_section(struct mem_section *ms, ms->usage = usage; } -static inline void __section_mark_present(struct mem_section *ms, - unsigned long section_nr) -{ - ms->section_mem_map |= SECTION_MARKED_PRESENT; -} - static inline size_t mem_section_usage_size(void) { return struct_size_t(struct mem_section_usage, pageblock_flags, From 0632840dba0307d0959f374a79358127da9708ca Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:59:03 +0200 Subject: [PATCH 085/213] mm/sparse: remove flags parameter from sparse_init_one_section() The last user of the flags parameter that used to pass SECTION_IS_EARLY was removed, so let's remove the parameter. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-11-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- mm/sparse-vmemmap.c | 2 +- mm/sparse.c | 2 +- mm/sparse.h | 5 ++--- 3 files changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 09376f5b1d92..098975861523 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -908,7 +908,7 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, /* Align memmap to section boundary in the subsection case */ if (section_nr_to_pfn(section_nr) != start_pfn) memmap = pfn_to_page(section_nr_to_pfn(section_nr)); - sparse_init_one_section(ms, section_nr, memmap, ms->usage, 0); + sparse_init_one_section(ms, section_nr, memmap, ms->usage); return 0; } diff --git a/mm/sparse.c b/mm/sparse.c index 5d2d9f95d402..702904b41700 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -243,7 +243,7 @@ static void __init sparse_metadata_init_nid(int nid, section_nr); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); sparse_init_one_section(__nr_to_section(section_nr), section_nr, - mem_map, usage, 0); + mem_map, usage); usage = (void *)usage + mem_section_usage_size(); } } diff --git a/mm/sparse.h b/mm/sparse.h index 19664d702af8..fdbc4cacf91d 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -21,7 +21,7 @@ int sparse_index_init(unsigned long section_nr, int nid); static inline void sparse_init_one_section(struct mem_section *ms, unsigned long section_nr, struct page *mem_map, - struct mem_section_usage *usage, unsigned long flags) + struct mem_section_usage *usage) { unsigned long coded_mem_map; @@ -37,8 +37,7 @@ static inline void sparse_init_one_section(struct mem_section *ms, VM_WARN_ON_ONCE(coded_mem_map & ~SECTION_MAP_MASK); ms->section_mem_map &= ~SECTION_MAP_MASK; - ms->section_mem_map |= coded_mem_map; - ms->section_mem_map |= flags | SECTION_HAS_MEM_MAP; + ms->section_mem_map |= coded_mem_map | SECTION_HAS_MEM_MAP; ms->usage = usage; } From 3f4b13b06062a03530df98310fcd9e189b2cddfa Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:59:04 +0200 Subject: [PATCH 086/213] fs/proc/page: clarify comment in get_max_dump_pfn() pfn_to_online_page() will only succeed on some PFNs within the same section, not necessarily all. Let's make that clearer. While at it, rephrase it to "Allow inspection of". Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-12-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Oscar Salvador Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- fs/proc/page.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/fs/proc/page.c b/fs/proc/page.c index f90e1030825e..6aa99e42ec22 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -31,10 +31,9 @@ static inline unsigned long get_max_dump_pfn(void) { #ifdef CONFIG_SPARSEMEM /* - * The memmap of early sections is completely populated and marked - * online even if max_pfn does not fall on a section boundary - - * pfn_to_online_page() will succeed on all pages. Allow inspecting - * these memmaps. + * If max_pfn does not fall on a section boundary, pfn_to_online_page() + * can succeed on PFNs beyond max_pfn within the same section. Allow + * inspection of these memmaps. */ return round_up(max_pfn, PAGES_PER_SECTION); #else From 82b1ee68b5fdd3062aee988a6e5aa3608d2470e0 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 21 Sep 2026 21:59:05 +0200 Subject: [PATCH 087/213] mm/memory_hotplug: drop CONFIG_HAVE_ARCH_PFN_VALID handling from pfn_to_online_page() Drop CONFIG_HAVE_ARCH_PFN_VALID handling, as CONFIG_HAVE_ARCH_PFN_VALID is never used with CONFIG_MEMORY_HOTPLUG, as the latter depends on CONFIG_SPARSEMEM_VMEMMAP. Make sure it stays that way. Link: https://lore.kernel.org/20260921-b4-sparsemem_cleanups-v2-13-54d81d65e125@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Barry Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Zi Yan Cc: Jan Kiszka Cc: Kieran Bingham Cc: Greg Kroah-Hartman Cc: Rafael J. Wysocki Cc: Danilo Krummrich --- mm/memory_hotplug.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index d7a59167bec4..796af1028ee2 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -345,6 +345,8 @@ struct page *pfn_to_online_page(unsigned long pfn) struct dev_pagemap *pgmap; struct mem_section *ms; + BUILD_BUG_ON(IS_ENABLED(CONFIG_HAVE_ARCH_PFN_VALID)); + if (nr >= NR_MEM_SECTIONS) return NULL; @@ -352,13 +354,6 @@ struct page *pfn_to_online_page(unsigned long pfn) if (!online_section(ms)) return NULL; - /* - * Save some code text when online_section() + - * pfn_section_valid() are sufficient. - */ - if (IS_ENABLED(CONFIG_HAVE_ARCH_PFN_VALID) && !pfn_valid(pfn)) - return NULL; - if (!pfn_section_valid(ms, pfn)) return NULL; From 0b4af90344ac4d5ce7defcb39891c773da969132 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 28 Sep 2026 10:48:21 +0200 Subject: [PATCH 088/213] vmcoreinfo: export SECTION_HAS_MEM_MAP The crash tool currently hardcodes SECTION_HAS_MEM_MAP, and makedumpfile needs similar information (although still relying on SECTION_MARKED_PRESENT, it should switch to SECTION_HAS_MEM_MAP). Let's just export the value instead, so tools that work on vmcoreinfo will not have to guess. Link: https://lore.kernel.org/20260928-section_has_mem_map-v1-1-4129f6ba2838@kernel.org Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Acked-by: Baoquan He Reviewed-by: Anshuman Khandual Reviewed-by: Bradley Morgan Cc: David Carlier Cc: Dave Young Cc: Kazuhito Hagio --- kernel/vmcore_info.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/vmcore_info.c b/kernel/vmcore_info.c index 5a417f8a922a..7833a36064a8 100644 --- a/kernel/vmcore_info.c +++ b/kernel/vmcore_info.c @@ -182,6 +182,7 @@ static int __init crash_save_vmcoreinfo_init(void) VMCOREINFO_STRUCT_SIZE(mem_section); VMCOREINFO_OFFSET(mem_section, section_mem_map); VMCOREINFO_NUMBER(SECTION_SIZE_BITS); + VMCOREINFO_NUMBER(SECTION_HAS_MEM_MAP); VMCOREINFO_NUMBER(MAX_PHYSMEM_BITS); #endif VMCOREINFO_STRUCT_SIZE(page); From 4590752e20aa393cc0d4fe538c0747d14a98c97a Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Tue, 13 Oct 2026 13:41:24 +0200 Subject: [PATCH 089/213] mm: disallow ioremap'ing ordinary RAM in generic_access_phys() generic_access_phys() can end up getting called on CoW PFNMAP mappings, resulting in it trying to ioremap anonymous folios, which especially the x86 code doesn't enjoy, and which is in general just odd/wrong. E.g., with a CoW mapping of a PCI BAR: ioremap on RAM at 0x0000000045623000 - 0x0000000045623fff WARNING: arch/x86/mm/ioremap.c:216 at __ioremap_caller.isra.0+0x4c2/0x5f0 Call Trace: generic_access_phys+0x130/0x4d0 mm/memory.c:7178 kernfs_vma_access+0x1ce/0x280 fs/kernfs/file.c:437 __access_remote_vm+0x58f/0x890 mm/memory.c:7256 mem_rw+0x2a1/0x670 fs/proc/base.c:912 But more generically, we should disallow access to anything that looks like ordinary RAM in the system: including allocating memory (e.g., alloc_pages) and mapping them in a VM_PFNMAP. For pages with an online memmap, allow only pages marked as reserved (PageReserved). This also includes memblock allocations, however, distinguishing them from other MMIO is tricky, and nobody should really be mapping them in a PFNMAP (except someone abusing /dev/mem, which we do not care about). So only allow access if there is no online memmap, or if the pages are marked reserved. Reject other pages with -EINVAL. Note that, just like other pfn_to_online_page() users, we won't care about concurrent memory offlining; mapping memory that can concurrently get offlined is highly dubious. Link: https://lore.kernel.org/20261013-generic_access_phys-v1-1-b3963ef76e37@kernel.org Fixes: 28b2ee20c7cb ("access_process_vm device memory infrastructure") Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reported-by: Closes: https://syzkaller.appspot.com/bug?extid=49b1021becba70c1f3f6 Reported-by: Nguyen Ngoc Thang Closes: https://lore.kernel.org/r/20261001152524.171115-1-ngocthang2710.1999@gmail.com/ Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Mike Rapoport (Microsoft) Tested-by: Lance Yang Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Benjamin Herrenschmidt Cc: Peter Zijlstra Cc: Rik van Riel Cc: --- mm/memory.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index ed64455c73bd..34e60a5baf7c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7129,6 +7129,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; pgprot_t prot = __pgprot(0); void __iomem *maddr; + struct page *page; int offset = offset_in_page(addr); int ret = -EINVAL; bool writable; @@ -7143,6 +7144,19 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, retry: if (follow_pfnmap_start(&args)) return -EINVAL; + + /* + * Reject anything that looks like ordinary RAM, including anonymous + * folios in CoW mappings or allocated kernel memory. Note that we + * cannot distinguish memblock allocations from non-RAM reliably, so + * we cannot easily reject them here. + */ + page = pfn_to_online_page(args.pfn); + if (page && !PageReserved(page)) { + follow_pfnmap_end(&args); + return -EINVAL; + } + prot = args.pgprot; phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; writable = args.writable; From bbde9d5bdd2d7cf94707b2a8bb466f2e074f9db2 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Fri, 23 Oct 2026 11:21:31 +0100 Subject: [PATCH 090/213] ALSA: us122l: mark hwdep mappings VM_MIXEDMAP mm has recently moved to a strict model which requires any driver which clears VMA_MAYWRITE_BIT to be mapping non-mm-managed memory (VMA_PFNMAP_BIT or VMA_MIXEDMAP_BIT), failing any such mmap() with an error and a kernel warning - see "mm/vma: only allow mmap to clear VMA_MAYWRITE_BIT if not mm-managed". However, simultaneous to this, commit 71c610aeb177 ("ALSA: us122l: Prevent write upgrades for read mappings") updated us122l to clear VMA_MAYWRITE_BIT without marking the region with either VMA_PFNMAP_BIT or VMA_MIXEDMAP_BIT. vm_insert_pages() and friends set this automatically, however this driver maps its pre-allocated pages from a fault handler, so never calls these. Simply set the flag in usb_stream_hwdep_mmap() to resolve the issue. This correctly marks the mapping as one whose pages core mm does not manage, which has no meaningful impact for this driver beyond preventing the mmap() from failing. Link: https://lore.kernel.org/20261023-fix-us122l-mixedmap-v1-1-e7bdd053850f@kernel.org Fixes: 71c610aeb177 ("ALSA: us122l: Prevent write upgrades for read mappings") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reported-by: David Carlier Closes: https://lore.kernel.org/20261018025905.201296-1-devnexen@gmail.com Acked-by: Takashi Iwai Cc: Kazuki Hanai Cc: Jaroslav Kysela --- sound/usb/usx2y/us122l.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sound/usb/usx2y/us122l.c b/sound/usb/usx2y/us122l.c index a5db0d044ef9..2986a90ea335 100644 --- a/sound/usb/usx2y/us122l.c +++ b/sound/usb/usx2y/us122l.c @@ -193,7 +193,7 @@ static int usb_stream_hwdep_mmap(struct snd_hwdep *hw, } area->vm_ops = &usb_stream_hwdep_vm_ops; - vm_flags_set(area, VM_DONTDUMP); + vm_flags_set(area, VM_MIXEDMAP | VM_DONTDUMP); if (!read) vm_flags_set(area, VM_DONTEXPAND); area->vm_private_data = us122l; From f85eb0656c409aa5910c8eacc98374121e00e3e0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:41 +0100 Subject: [PATCH 091/213] mm/vma: fix mmap_prepare file handling, remove file_doesnt_need_get Patch series "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL", v6. The VM_SPECIAL / VMA_SPECIAL_FLAGS mask conflates several unrelated properties: * Is this managed by core mm, or by a driver mapping MMIO, kernel-allocated pages, or even ordinary pages itself? * Can it be expanded or merged? * Is this a 'weird' case like mlock where migration might race and we 'have' to set invalid flags to notify? * Is it another 'weird' case where we just want to stop GUP from touching it? Driver writers have often been confused about this, and who can blame them? It also interacts badly with the eternal edgecase known as hugetlb - which sets VMA_DONTEXPAND_BIT but doesn't also want to be treated like a 'special' flag. Another issue is that we cannot make sensible assumptions about flag use. It's not possible to assume VMA_IO_BIT means iommu because drivers abuse it and mlock abuses it. Special is also an overloaded term in mm. VDSO and VVAR mappings are also called 'special' but they're special in a... special way. Sometimes things are called special that are a subset of VMA_SPECIAL_FLAGS (VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT for instance when it comes to zapping or vm_normal_folio()). There's a specific kind of special for THP too, which considers PFN map, mixed map 'special' but DAX not. It's all rather a mess. This series brings some order to things by both limiting what drivers can do with VMA flags and switching to using predicates that describe behaviour, not arbitrary flags. It establishes the invariant that mm-managed mappings may neither set VMA_IO_BIT nor clear VMA_MAYWRITE_BIT in an mmap hook, enforcing this by validating VMA state after every mmap and mmap_prepare hook. It updates usbmon and sg to mmap_prepare in order to do so, adding a new mmap action for mapping discontiguous kernel pages, and has hfi1 and the ALSA PCM status page map their pages eagerly instead. It also establishes the invariant that VMA_MIXEDMAP_BIT be set when mapping kernel memory, something that is usually the case but happens not to be for some users - specifically defio, cmt_speech, uprobes and the bpf arena, all of which are updated to do the right thing. It replaces VM_SPECIAL and arbitrary flag tests with predicates that say what is actually being tested: vma_is_mm_managed() Are the VMA's contents established and managed by core mm, rather than a driver or other kernel component? vma_is_mm_backed() Is the VMA backed by memory core mm itself faults in, rather than a driver or device? vma_is_fixed_mapping() Is the VMA not permitted to be expanded or merged? vma_can_merge() Can the VMA be merged with a compatible neighbour? Remaining raw VMA_IO_BIT, VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT tests scattered across mm are converted to predicates where they express one, and left as explicit flag tests where they do not. And also the opportunity is taken to eliminate THP's vma_is_special_huge() which was an existing source of confusion. This patch (of 39): The map->file_doesnt_need_get flag is confusing and the existing implementation has holes. Drivers are permitted to change the owning file of a mapping. If they do so, they are required to take a reference on that file. The mmap() operation which ultimately invokes __mmap_region() is guaranteed to drop the refcount for the original file the mapping was made under, but this is not true for the replaced file. This has been addressed so far by tracking map->file_doesnt_need_get, which is rather poorly named and unfortunately fails to correctly track whether or not an additional put were needed in a number of cases. Make life easier by removing this flag, and instead drop the reference for both mmap_prepare and the deprecated mmap callback in a new function map_put_vm_file(). Track whether this needs to be done by aligning mmap_state with vm_area_desc and store the original file in the map->file field, keeping the updated file in map->vm_file. In order to have the same behaviour for both types of hooks, only drop the reference __mmap_new_file_vma() itself took in its error path, deferring the replaced file's reference to map_put_vm_file(). To make this work correctly, map->vm_file has to be updated before any error handling, so update __mmap_new_file_vma() and call_mmap_prepare() to set this field first. Also when mmap_prepare() changes the file and is then merged, the reference count also must be decremented, so update the logic to call map_put_vm_file() in this case too. Also update __compat_vma_mmap() to manually perform this step for stacked file systems using the compatibility layer, and update compat_set_vma_from_desc() to replace vma_set_file() with a correct refcount/file update. No in-tree driver is impacted by the incorrect implementation of this currently (no driver that does this is mergeable for one), so this does not need to be a fix. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-0-1aa3c9b139f9@kernel.org Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-1-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Reviewed-by: Gregory Price (Meta) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: David Hildenbrand Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/internal.h | 1 + mm/util.c | 5 +++- mm/vma.c | 81 +++++++++++++++++++++++++++++++-------------------- mm/vma.h | 6 ++-- 4 files changed, 58 insertions(+), 35 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 704f3e3ef2ae..fd1690c0cfde 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -7,6 +7,7 @@ #ifndef __MM_INTERNAL_H #define __MM_INTERNAL_H +#include #include #include #include diff --git a/mm/util.c b/mm/util.c index cddd55cbb61c..975cef080761 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1228,8 +1228,11 @@ int __compat_vma_mmap(struct vm_area_desc *desc, /* Perform any preparatory tasks for mmap action. */ err = mmap_action_prepare(desc); - if (err) + if (err) { + if (desc->vm_file != vma->vm_file) + fput(desc->vm_file); return err; + } /* Update the VMA from the descriptor. */ compat_set_vma_from_desc(vma, desc); /* Complete any specified mmap actions. */ diff --git a/mm/vma.c b/mm/vma.c index 5ab3fe59f06b..3eacc2a677a1 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -24,7 +24,8 @@ struct mmap_state { vm_flags_t vm_flags; vma_flags_t vma_flags; }; - struct file *file; + struct file *file; /* mmap()-specified file. */ + struct file *vm_file; /* May be updated by mmap_prepare. */ pgprot_t page_prot; /* User-defined fields, perhaps updated by .mmap_prepare(). */ @@ -43,8 +44,6 @@ struct mmap_state { /* Determine if we can check KSM flags early in mmap() logic. */ bool check_ksm_early :1; - /* If .mmap_prepare changed the file, we don't need to pin. */ - bool file_doesnt_need_get :1; }; #define MMAP_STATE(name, mm_, vmi_, addr_, len_, pgoff_, anon_pgoff_, vma_flags_, file_) \ @@ -58,6 +57,7 @@ struct mmap_state { .pglen = PHYS_PFN(len_), \ .vma_flags = vma_flags_, \ .file = file_, \ + .vm_file = file_, \ .page_prot = vma_flags_to_page_prot(vma_flags_), \ } @@ -70,7 +70,7 @@ struct mmap_state { .vma_flags = (map_)->vma_flags, \ .pgoff = (map_)->pgoff, \ .anon_pgoff = (map_)->anon_pgoff, \ - .file = (map_)->file, \ + .file = (map_)->vm_file, \ .prev = (map_)->prev, \ .middle = vma_, \ .next = (vma_) ? NULL : (map_)->next, \ @@ -2462,7 +2462,7 @@ void mm_drop_all_locks(struct mm_struct *mm) */ static bool accountable_mapping(struct mmap_state *map) { - const struct file *file = map->file; + const struct file *file = map->vm_file; /* * hugetlb has its own accounting separate from the core VM @@ -2511,7 +2511,7 @@ static void vms_abort_munmap_vmas(struct vma_munmap_struct *vms, static void update_ksm_flags(struct mmap_state *map) { - map->vma_flags = ksm_vma_flags(map->mm, map->file, map->vma_flags); + map->vma_flags = ksm_vma_flags(map->mm, map->vm_file, map->vma_flags); } static void set_desc_from_map(struct vm_area_desc *desc, @@ -2521,7 +2521,7 @@ static void set_desc_from_map(struct vm_area_desc *desc, desc->end = map->end; desc->pgoff = map->pgoff; - desc->vm_file = map->file; + desc->vm_file = map->vm_file; desc->vma_flags = map->vma_flags; desc->page_prot = map->page_prot; } @@ -2604,6 +2604,10 @@ static int __mmap_setup(struct mmap_state *map, struct vm_area_desc *desc, return 0; } +static bool map_same_file(struct mmap_state *map) +{ + return map->vm_file == map->file; +} static int __mmap_new_file_vma(struct mmap_state *map, struct vm_area_struct *vma) @@ -2611,20 +2615,23 @@ static int __mmap_new_file_vma(struct mmap_state *map, struct vma_iterator *vmi = map->vmi; int error; - vma->vm_file = map->file; - if (!map->file_doesnt_need_get) - get_file(map->file); + vma->vm_file = map->vm_file; + if (map_same_file(map)) + get_file(map->vm_file); - if (!map->file->f_op->mmap) + if (!map->vm_file->f_op->mmap) return 0; error = mmap_file(vma->vm_file, vma); + map->vm_file = vma->vm_file; + if (error) { UNMAP_STATE(unmap, vmi, vma, vma->vm_start, vma->vm_end, map->prev, map->next); - fput(vma->vm_file); - vma->vm_file = NULL; + if (map_same_file(map)) + fput(map->vm_file); + vma->vm_file = NULL; vma_iter_set(vmi, vma->vm_end); /* Undo any partial mapping done by a device driver. */ unmap_region(&unmap); @@ -2641,7 +2648,6 @@ static int __mmap_new_file_vma(struct mmap_state *map, !vma_flags_test(&map->vma_flags, VMA_MAYWRITE_BIT) && vma_test(vma, VMA_MAYWRITE_BIT)); - map->file = vma->vm_file; map->vma_flags = vma->flags; return 0; @@ -2649,7 +2655,7 @@ static int __mmap_new_file_vma(struct mmap_state *map, static void map_set_anon(struct mmap_state *map) { - map->file = NULL; + map->vm_file = NULL; map->vm_ops = NULL; map->pgoff = map->addr >> PAGE_SHIFT; } @@ -2661,7 +2667,7 @@ static bool map_is_private(const struct mmap_state *map) static bool map_is_anon(const struct mmap_state *map) { - return map_is_private(map) && !map->file; + return map_is_private(map) && !map->vm_file; } /* @@ -2706,7 +2712,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap, } /* Invoke callbacks. */ - if (map->file) + if (map->vm_file) error = __mmap_new_file_vma(map, vma); else if (!is_anon) error = shmem_zero_setup(vma); @@ -2815,10 +2821,14 @@ static int call_mmap_prepare(struct mmap_state *map, int err; /* Invoke the hook. */ - err = vfs_mmap_prepare(map->file, desc); + err = vfs_mmap_prepare(map->vm_file, desc); if (err) return err; + /* Update first so file refcount tracked correctly. */ + if (desc->vm_file != map->vm_file) + map->vm_file = desc->vm_file; + /* It's invalid for mmap_prepare hooks to clear vm_ops. */ if (!desc->vm_ops) return -EINVAL; @@ -2829,10 +2839,6 @@ static int call_mmap_prepare(struct mmap_state *map, /* Update fields permitted to be changed. */ map->pgoff = desc->pgoff; - if (desc->vm_file != map->file) { - map->file_doesnt_need_get = true; - map->file = desc->vm_file; - } map->vma_flags = desc->vma_flags; map->page_prot = desc->page_prot; /* User-defined fields. */ @@ -2844,7 +2850,7 @@ static int call_mmap_prepare(struct mmap_state *map, * anonymous mappings. Rather than allowing these mappings to be odd * outliers, simply make them truly anonymous. */ - if (map_is_private(map) && file_is_dev_zero(map->file)) + if (map_is_private(map) && file_is_dev_zero(map->vm_file)) map_set_anon(map); return 0; @@ -2863,7 +2869,7 @@ static void set_vma_user_defined_fields(struct vm_area_struct *vma, */ static bool can_set_ksm_flags_early(struct mmap_state *map) { - struct file *file = map->file; + struct file *file = map->vm_file; /* Anonymous mappings have no driver which can change them. */ if (!file) @@ -2886,6 +2892,20 @@ static bool can_set_ksm_flags_early(struct mmap_state *map) return false; } +static void map_put_vm_file(struct mmap_state *map) +{ + /* + * An error occurred or the VMA was merged. + * + * If the file was changed by the driver (which is required to increment + * the replacement file's reference count), drop its reference count. + * + * On error, the caller always drops the original file regardless. + */ + if (map->vm_file && !map_same_file(map)) + fput(map->vm_file); +} + static unsigned long __mmap_region(struct file *file, unsigned long addr, unsigned long len, vma_flags_t vma_flags, unsigned long pgoff, struct list_head *uf) @@ -2940,7 +2960,10 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, __mmap_complete(&map, vma); - if (have_mmap_prepare && allocated_new) { + if (!allocated_new) { + /* Merged, so need to drop refcount. */ + map_put_vm_file(&map); + } else if (have_mmap_prepare) { error = mmap_action_complete(vma, &desc.action, /*is_compat=*/false); if (error) @@ -2954,13 +2977,7 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, if (map.charged) vm_unacct_memory(map.charged); abort_munmap: - /* - * This indicates that .mmap_prepare has set a new file, differing from - * desc->vm_file. But since we're aborting the operation, only the - * original file will be cleaned up. Ensure we clean up both. - */ - if (map.file_doesnt_need_get) - fput(map.file); + map_put_vm_file(&map); vms_abort_munmap_vmas(&map.vms, &map.mas_detach); return error; } diff --git a/mm/vma.h b/mm/vma.h index f856d9ace3a6..4665b40163fa 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -394,8 +394,10 @@ static inline void compat_set_vma_from_desc(struct vm_area_struct *vma, /* Mutable fields. Populated with initial state. */ vma_set_pgoff(vma, desc->pgoff); - if (desc->vm_file != vma->vm_file) - vma_set_file(vma, desc->vm_file); + if (desc->vm_file != vma->vm_file) { + fput(vma->vm_file); + vma->vm_file = desc->vm_file; + } vma->flags = desc->vma_flags; vma->vm_page_prot = desc->page_prot; From 86a779c5320488b9142f4477435f235c9a04f7ae Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:42 +0100 Subject: [PATCH 092/213] mm/vma: introduce and use vma_[flags_]can_merge() Replace the open-coded VMA_SPECIAL_FLAGS check in the VMA merge logic with two new functions vma_flags_can_merge() and vma_can_merge() and update the merge logic to use the former. This abstracts the check and expresses it in terms of the desired behaviour rather than an arbitrary and confusing VMA flag. This also lays the groundwork for making further improvements in VMA flag usage. Also update the userland VMA tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-2-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Gregory Price (Meta) Reviewed-by: Zi Yan Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 21 +++++++++++++++++++++ mm/vma.c | 19 +++++++++++-------- tools/testing/vma/include/dup.h | 5 +++++ 3 files changed, 37 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 46a380aa1c84..10b5f08b8d21 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1613,6 +1613,27 @@ static inline bool vma_is_shared_maywrite(const struct vm_area_struct *vma) return is_shared_maywrite(&vma->flags); } +/** + * vma_flags_can_merge() - Do the specified VMA flags permit the VMA to be + * merged with another? + * @flags: The VMA flags to test. + * Returns: true if the flags permit merging, false otherwise. + */ +static inline bool vma_flags_can_merge(const vma_flags_t *flags) +{ + return !vma_flags_test_any_mask(flags, VMA_SPECIAL_FLAGS); +} + +/** + * vma_can_merge() - Do @vma's flags permit it to be merged with another VMA? + * @vma: The VMA to test. + * Returns: true if the flags permit merging, otherwise false. + */ +static inline bool vma_can_merge(const struct vm_area_struct *vma) +{ + return vma_flags_can_merge(&vma->flags); +} + /** * vma_kernel_pagesize - Default page size granularity for this VMA. * @vma: The user mapping. diff --git a/mm/vma.c b/mm/vma.c index 3eacc2a677a1..b6cd633bacfe 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -924,13 +924,14 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( vmg->state = VMA_MERGE_NOMERGE; + if (!vma_flags_can_merge(&vmg->vma_flags)) + return NULL; /* - * If a special mapping or if the range being modified is neither at the - * furthermost left or right side of the VMA, then we have no chance of - * merging and should abort. + * If the range being modified is neither at the furthermost left or + * right side of the VMA, then we have no chance of merging and should + * abort. */ - if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) || - (!left_side && !right_side)) + if (!left_side && !right_side) return NULL; if (left_side) @@ -1152,9 +1153,11 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg) vmg->state = VMA_MERGE_NOMERGE; - /* Special VMAs are unmergeable, also if no prev/next. */ - if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) || - (!prev && !next)) + if (!vma_flags_can_merge(&vmg->vma_flags)) + return NULL; + + /* VMAs with no prev/next are unmergeable. */ + if (!prev && !next) return NULL; can_merge_left = can_vma_merge_left(vmg); diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 16c09dac59d9..2fd422789717 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1647,3 +1647,8 @@ static inline bool file_is_dev_zero(const struct file *file) { return file && file->f_op == &zero_fops; } + +static inline bool vma_flags_can_merge(const vma_flags_t *flags) +{ + return !vma_flags_test_any_mask(flags, VMA_SPECIAL_FLAGS); +} From 9cae38dc11da28b4e902d59af3d5d32836f744b3 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:43 +0100 Subject: [PATCH 093/213] mm: consistently validate VMA state after mmap[_prepare] hooks When the f_op->mmap_prepare or deprecated f_op->mmap hooks are invoked, the driver might have done something crazy that is not permitted by the kernel. Currently we check for three such cases in __mmap_new_file_vma(), but only if the legacy f_op->mmap hook is used: * Did sparc ADI result in invalid flags? * Did the driver alter vma->vm_start? * Did the driver make a file-backed mapping on a read-only file writable? Generalise these checks for both mmap_prepare and mmap and apply to all invocations of mmap_file(), the f_op->mmap and f_op->mmap_prepare handling in the core VMA code and the mmap_prepare compatibility layer. Also extend the vm_start check to vm_end also - drivers must not change the VMA range at all. We also WARN_ON_ONCE() on these conditions as they are things that should simply not occur in the kernel and it's important to call it out when it does. We invoke mmap_prepare_validate() after mmap_action_prepare(), as mmap actions often manipulate state in the descriptor thus providing the final state the VMA will be derived from. Also call mmap_validate_vma_flags() in insert_vm_struct() to ensure that special regions which are inserted (such as a VDSO or VVAR) also satisfy the sanity checks. This way every VMA established through an mmap hook, whether via mmap() or the compatibility layer, or inserted via insert_vm_struct(), has been validated. brk() VMAs never pass through a driver hook and so need no such check. While we're here, also fixup a couple disjoint blocks of #ifdef CONFIG_MMU. Finally, update the VMA userland tests to reflect the change. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-3-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Gregory Price (Meta) Reviewed-by: Zi Yan Reviewed-by: Suren Baghdasaryan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: David Hildenbrand Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/internal.h | 72 +++++++++++--------- mm/util.c | 19 ++++-- mm/vma.c | 106 +++++++++++++++++++++++++----- mm/vma.h | 25 ++++++- tools/testing/vma/include/dup.h | 10 +++ tools/testing/vma/include/stubs.h | 4 ++ 6 files changed, 183 insertions(+), 53 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index fd1690c0cfde..95e686103320 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -213,36 +213,6 @@ static inline void *folio_raw_mapping(const struct folio *folio) return (void *)(mapping & ~FOLIO_MAPPING_FLAGS); } -/* - * This is a file-backed mapping, and is about to be memory mapped - invoke its - * mmap hook and safely handle error conditions. On error, VMA hooks will be - * mutated. - * - * @file: File which backs the mapping. - * @vma: VMA which we are mapping. - * - * Returns: 0 if success, error otherwise. - */ -static inline int mmap_file(struct file *file, struct vm_area_struct *vma) -{ - int err = vfs_mmap(file, vma); - - /* - * Either we tried to call the file hook for mmap() and an error arose - * or a driver set vma->vm_ops = NULL intending there to be no VMA - * operations. - * - * In the former case the VMA is in an inconsistent state and we mustn't - * invoke any further hooks on it, in the latter case the hook actually - * wanted no further hooks to be invoked, so fix both by setting dummy - * VMA ops. - */ - if (unlikely(err || !vma->vm_ops)) - vma->vm_ops = &vma_dummy_vm_ops; - - return err; -} - /* * If the VMA has a close hook then close it, and since closing it might leave * it in an inconsistent state which makes the use of any hooks suspect, clear @@ -261,6 +231,48 @@ static inline void vma_close(struct vm_area_struct *vma) } } +/* + * This is a file-backed mapping, and is about to be memory mapped - invoke its + * mmap hook and safely handle error conditions. On error, VMA hooks will be + * mutated. + * + * @file: File which backs the mapping. + * @vma: VMA which we are mapping. + * + * Returns: 0 if success, error otherwise. + */ +static inline int mmap_file(struct file *file, struct vm_area_struct *vma) +{ + const unsigned long orig_start = vma->vm_start; + const unsigned long orig_end = vma->vm_end; + const vma_flags_t orig_flags = vma->flags; + int err; + + err = vfs_mmap(file, vma); + /* + * Either we tried to call the file hook for mmap() and an error arose + * or a driver set vma->vm_ops = NULL intending there to be no VMA + * operations. + * + * In the former case the VMA is in an inconsistent state and we mustn't + * invoke any further hooks on it, in the latter case the hook actually + * wanted no further hooks to be invoked, so fix both by setting dummy + * VMA ops. + */ + if (unlikely(err || !vma->vm_ops)) + vma->vm_ops = &vma_dummy_vm_ops; + if (unlikely(err)) + return err; + + err = mmap_hook_validate(orig_start, orig_end, &orig_flags, vma); + if (unlikely(err)) { + vma->vm_start = orig_start; + vma->vm_end = orig_end; + } + + return err; +} + /* unmap_vmas is in mm/memory.c */ void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap); diff --git a/mm/util.c b/mm/util.c index 975cef080761..fd0d517c570e 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1224,19 +1224,28 @@ EXPORT_SYMBOL(compat_set_desc_from_vma); int __compat_vma_mmap(struct vm_area_desc *desc, struct vm_area_struct *vma) { + struct vm_area_desc orig_desc; int err; + /* Derive state prior to mmap_prepare hook. */ + compat_set_desc_from_vma(&orig_desc, desc->file, vma); /* Perform any preparatory tasks for mmap action. */ err = mmap_action_prepare(desc); - if (err) { - if (desc->vm_file != vma->vm_file) - fput(desc->vm_file); - return err; - } + if (err) + goto err_put; + /* Check the caller did nothing crazy. */ + err = mmap_prepare_validate(&orig_desc, desc); + if (err) + goto err_put; /* Update the VMA from the descriptor. */ compat_set_vma_from_desc(vma, desc); /* Complete any specified mmap actions. */ return mmap_action_complete(vma, &desc->action, /*is_compat=*/true); + +err_put: + if (desc->vm_file != vma->vm_file) + fput(desc->vm_file); + return err; } EXPORT_SYMBOL(__compat_vma_mmap); diff --git a/mm/vma.c b/mm/vma.c index b6cd633bacfe..4f0e848c7aff 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -2631,6 +2631,12 @@ static int __mmap_new_file_vma(struct mmap_state *map, if (error) { UNMAP_STATE(unmap, vmi, vma, vma->vm_start, vma->vm_end, map->prev, map->next); + + if (vma->vm_ops->close) { + zap_vma(vma); + /* Safe, because VMA zapped. */ + vma_close(vma); + } if (map_same_file(map)) fput(map->vm_file); @@ -2641,16 +2647,6 @@ static int __mmap_new_file_vma(struct mmap_state *map, return error; } - /* Drivers cannot alter the address of the VMA. */ - WARN_ON_ONCE(map->addr != vma->vm_start); - /* - * Drivers should not permit writability when previously it was - * disallowed. - */ - VM_WARN_ON_ONCE(!vma_flags_same_pair(&map->vma_flags, &vma->flags) && - !vma_flags_test(&map->vma_flags, VMA_MAYWRITE_BIT) && - vma_test(vma, VMA_MAYWRITE_BIT)); - map->vma_flags = vma->flags; return 0; @@ -2728,11 +2724,6 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap, vma->flags = map->vma_flags; } -#ifdef CONFIG_SPARC64 - /* TODO: Fix SPARC ADI! */ - WARN_ON_ONCE(!arch_validate_flags(map->vm_flags)); -#endif - /* Lock the VMA since it is modified after insertion into VMA tree */ vma_start_write(vma); vma_iter_store_new(vmi, vma); @@ -2795,6 +2786,80 @@ static void __mmap_complete(struct mmap_state *map, struct vm_area_struct *vma) vma_set_page_prot(vma); } +/* Check to ensure that the VMA flags of a newly mapped VMA are sane. */ +static int mmap_validate_vma_flags(const vma_flags_t *flags) +{ +#ifdef CONFIG_SPARC64 + const vm_flags_t legacy_flags = vma_flags_to_legacy(*flags); + + /* TODO: Fix SPARC ADI! */ + if (WARN_ON_ONCE(!arch_validate_flags(legacy_flags))) + return -EINVAL; +#endif + + return 0; +} + +/* Check to ensure a driver hasn't done something crazy. */ +static int mmap_validate(unsigned long orig_start, unsigned long orig_end, + unsigned long curr_start, unsigned long curr_end, + const vma_flags_t *orig_flags, + const vma_flags_t *curr_flags) +{ + bool was_maywrite, is_maywrite; + + /* Drivers cannot alter the range of the VMA. */ + if (WARN_ON_ONCE(orig_start != curr_start || orig_end != curr_end)) + return -EINVAL; + + was_maywrite = vma_flags_test(orig_flags, VMA_MAYWRITE_BIT); + is_maywrite = vma_flags_test(curr_flags, VMA_MAYWRITE_BIT); + + /* A driver may not make a previously unwritable mapping writable. */ + if (WARN_ON_ONCE(!was_maywrite && is_maywrite)) + return -EINVAL; + + return mmap_validate_vma_flags(curr_flags); +} + +/** + * mmap_prepare_validate() - Ensure the driver hasn't violated invariants in its + * f_op->mmap_prepare hook. + * @orig_desc: The VMA descriptor prior to the mmap_prepare hook being called. + * @desc: The VMA descriptor after the mmap_prepare hook has been called. + * + * Returns: 0 on success, otherwise an error. + */ +int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc) +{ + return mmap_validate(orig_desc->start, orig_desc->end, + desc->start, desc->end, + &orig_desc->vma_flags, &desc->vma_flags); +} + +/** + * mmap_hook_validate() - Ensure the driver hasn't violated invariants in + * its f_op->mmap hook. + * @orig_start: The start of the mapping prior to the mmap hook. + * @orig_end: The end of the mapping prior to the mmap hook. + * @orig_flags: The VMA flags set for the VMA prior to the mmap hook. + * @vma: The VMA after the hook has been applied. + * + * Returns: 0 on success, otherwise an error. + */ +int mmap_hook_validate(unsigned long orig_start, unsigned long orig_end, + const vma_flags_t *orig_flags, + const struct vm_area_struct *vma) +{ + const unsigned long start = vma->vm_start; + const unsigned long end = vma->vm_end; + const vma_flags_t *flags = &vma->flags; + + return mmap_validate(orig_start, orig_end, start, end, orig_flags, + flags); +} + static int call_action_prepare(struct mmap_state *map, struct vm_area_desc *desc) { @@ -2821,6 +2886,7 @@ static int call_action_prepare(struct mmap_state *map, static int call_mmap_prepare(struct mmap_state *map, struct vm_area_desc *desc) { + const struct vm_area_desc orig_desc = *desc; int err; /* Invoke the hook. */ @@ -2840,6 +2906,11 @@ static int call_mmap_prepare(struct mmap_state *map, if (err) return err; + /* Check the caller did nothing crazy. */ + err = mmap_prepare_validate(&orig_desc, desc); + if (err) + return err; + /* Update fields permitted to be changed. */ map->pgoff = desc->pgoff; map->vma_flags = desc->vma_flags; @@ -3475,10 +3546,15 @@ int __vm_munmap(unsigned long start, size_t len, bool unlock) int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) { unsigned long charged = vma_pages(vma); + int err; if (find_vma_intersection(mm, vma->vm_start, vma->vm_end)) return -ENOMEM; + err = mmap_validate_vma_flags(&vma->flags); + if (err) + return err; + if (vma_test(vma, VMA_ACCOUNT_BIT) && security_vm_enough_memory_mm(mm, charged)) return -ENOMEM; diff --git a/mm/vma.h b/mm/vma.h index 4665b40163fa..ba897cb96e41 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -787,14 +787,19 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm); struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig); void vm_area_free(struct vm_area_struct *vma); -/* vma_exec.c */ #ifdef CONFIG_MMU +int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc); + +int mmap_hook_validate(unsigned long orig_start, unsigned long orig_end, + const vma_flags_t *orig_flags, + const struct vm_area_struct *vma); + +/* vma_exec.c */ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap, unsigned long *top_mem_p); int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift); -#endif -#ifdef CONFIG_MMU /* * Denies creating a writable executable mapping or gaining executable permissions. * @@ -843,6 +848,20 @@ static inline bool map_deny_write_exec(const vma_flags_t *old, return false; } +#else +static inline int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc) +{ + return 0; +} + +static inline int mmap_hook_validate(unsigned long orig_start, + unsigned long orig_end, + const vma_flags_t *orig_flags, + const struct vm_area_struct *vma) +{ + return 0; +} #endif struct vm_area_struct *__install_special_mapping(struct mm_struct *mm, diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 2fd422789717..077df510a311 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1359,13 +1359,23 @@ static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc) return file->f_op->mmap_prepare(desc); } +int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc); + static inline int __compat_vma_mmap(struct vm_area_desc *desc, struct vm_area_struct *vma) { + struct vm_area_desc orig_desc; int err; + /* Derive state prior to mmap_prepare hook. */ + compat_set_desc_from_vma(&orig_desc, desc->file, vma); /* Perform any preparatory tasks for mmap action. */ err = mmap_action_prepare(desc); + if (err) + return err; + /* Check the caller did nothing crazy. */ + err = mmap_prepare_validate(&orig_desc, desc); if (err) return err; /* Update the VMA from the descriptor. */ diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h index d6136e19a8af..b35279a0ec78 100644 --- a/tools/testing/vma/include/stubs.h +++ b/tools/testing/vma/include/stubs.h @@ -329,6 +329,10 @@ static inline void vma_close(struct vm_area_struct *vma) { } +static inline void zap_vma(struct vm_area_struct *vma) +{ +} + static inline int mmap_file(struct file *file, struct vm_area_struct *vma) { return 0; From 6335c786299ffc0571506f385d8d748471929230 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:44 +0100 Subject: [PATCH 094/213] mm/vma: ensure mmap_prepare doesn't set actions on a mergeable vma When a user requests an mmap_action be performed in mmap_prepare, this involves populating the VMA range with data. However, if the VMA is mergeable, it might then mistakenly be merged with another VMA without having populated the range. Every mmap action currently available sets VMA flags such that the VMA cannot be merged. However, to ensure that no future mmap action falls foul of this, assert that this is the case upon mmap_prepare validation. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-4-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Reviewed-by: Zi Yan Reviewed-by: Gregory Price (Meta) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/vma.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/vma.c b/mm/vma.c index 4f0e848c7aff..106176ebe9fe 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -2833,6 +2833,18 @@ static int mmap_validate(unsigned long orig_start, unsigned long orig_end, int mmap_prepare_validate(const struct vm_area_desc *orig_desc, const struct vm_area_desc *desc) { + /* + * It is not valid to execute mmap actions for VMAs which can be merged, + * as any such merge would leave portions of the mapping incorrectly + * unmapped. + * + * This is checked after mmap_action_prepare(), as it may update the VMA + * flags and therefore whether the VMA can be merged. + */ + if (vma_flags_can_merge(&desc->vma_flags) && + WARN_ON_ONCE(desc->action.type != MMAP_NOTHING)) + return -EINVAL; + return mmap_validate(orig_desc->start, orig_desc->end, desc->start, desc->end, &orig_desc->vma_flags, &desc->vma_flags); From 2a234f79d416beeec9fb82dcc20c6794416a54b0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:45 +0100 Subject: [PATCH 095/213] mm: make map_kernel_pages_[prepare,complete] internal and unexported There's no reason to export the symbols for these functions which are only called from internal mm logic, additionally there's no reason for them to be declared in mm.h. This patch therefore removes the exports and moves the declarations to mm/internal.h. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-5-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Reviewed-by: Zi Yan Reviewed-by: Gregory Price (Meta) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 3 --- mm/internal.h | 3 +++ mm/memory.c | 2 -- 3 files changed, 3 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 10b5f08b8d21..64c52469eaaf 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4775,9 +4775,6 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, int vm_insert_page(struct vm_area_struct *, unsigned long addr, struct page *); int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, struct page **pages, unsigned long *num); -int map_kernel_pages_prepare(struct vm_area_desc *desc); -int map_kernel_pages_complete(struct vm_area_struct *vma, - struct mmap_action *action); int vm_map_pages(struct vm_area_struct *vma, struct page **pages, unsigned long num); int vm_map_pages_zero(struct vm_area_struct *vma, struct page **pages, diff --git a/mm/internal.h b/mm/internal.h index 95e686103320..a11ccda86fde 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1514,6 +1514,9 @@ int remap_pfn_range_prepare(struct vm_area_desc *desc); int remap_pfn_range_complete(struct vm_area_struct *vma, struct mmap_action *action); int simple_ioremap_prepare(struct vm_area_desc *desc); +int map_kernel_pages_prepare(struct vm_area_desc *desc); +int map_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action); static inline int io_remap_pfn_range_prepare(struct vm_area_desc *desc) { diff --git a/mm/memory.c b/mm/memory.c index 34e60a5baf7c..3e0dfab9a80e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2628,7 +2628,6 @@ int map_kernel_pages_prepare(struct vm_area_desc *desc) return 0; } -EXPORT_SYMBOL(map_kernel_pages_prepare); int map_kernel_pages_complete(struct vm_area_struct *vma, struct mmap_action *action) @@ -2640,7 +2639,6 @@ int map_kernel_pages_complete(struct vm_area_struct *vma, action->map_kernel.pages, &nr_pages, vma->vm_page_prot); } -EXPORT_SYMBOL(map_kernel_pages_complete); /** * vm_insert_page - insert single page into user vma From 4852f3421456f6ee2943726784f82b1db0028a18 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:46 +0100 Subject: [PATCH 096/213] mm/vma: tidy up map kernel pages enum values MMAP_MAP_KERNEL_PAGES is a mouthful, discard the MAP_ as that's implied by MMAP. Also while we're here delete useless comments for mmap actions whose names clearly indicate what they are for. Also update the userland VMA tests to reflect this change. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-6-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Reviewed-by: Zi Yan Reviewed-by: Gregory Price (Meta) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 8 ++++---- mm/util.c | 8 ++++---- tools/testing/vma/include/dup.h | 8 ++++---- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 64c52469eaaf..d724c7401798 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4630,7 +4630,7 @@ static inline void mmap_action_map_kernel_pages(struct vm_area_desc *desc, { struct mmap_action *action = &desc->action; - action->type = MMAP_MAP_KERNEL_PAGES; + action->type = MMAP_KERNEL_PAGES; action->map_kernel.start = start; action->map_kernel.pages = pages; action->map_kernel.nr_pages = nr_pages; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4b631fb4ff7a..bd5c8832d1af 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -815,11 +815,11 @@ struct pfnmap_track_ctx { /* What action should be taken after an .mmap_prepare call is complete? */ enum mmap_action_type { - MMAP_NOTHING, /* Mapping is complete, no further action. */ - MMAP_REMAP_PFN, /* Remap PFN range. */ - MMAP_IO_REMAP_PFN, /* I/O remap PFN range. */ + MMAP_NOTHING, + MMAP_REMAP_PFN, + MMAP_IO_REMAP_PFN, MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_MAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ }; /* diff --git a/mm/util.c b/mm/util.c index fd0d517c570e..441e21eb0e57 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1467,7 +1467,7 @@ int mmap_action_prepare(struct vm_area_desc *desc) return io_remap_pfn_range_prepare(desc); case MMAP_SIMPLE_IO_REMAP: return simple_ioremap_prepare(desc); - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: return map_kernel_pages_prepare(desc); } @@ -1498,7 +1498,7 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_REMAP_PFN: err = remap_pfn_range_complete(vma, action); break; - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: err = map_kernel_pages_complete(vma, action); break; case MMAP_IO_REMAP_PFN: @@ -1521,7 +1521,7 @@ int mmap_action_prepare(struct vm_area_desc *desc) case MMAP_REMAP_PFN: case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle these. */ break; } @@ -1542,7 +1542,7 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_REMAP_PFN: case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle this. */ err = -EINVAL; diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 077df510a311..023ad6092e9c 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -454,11 +454,11 @@ static __always_inline bool vma_flags_empty(const vma_flags_t *flags) /* What action should be taken after an .mmap_prepare call is complete? */ enum mmap_action_type { - MMAP_NOTHING, /* Mapping is complete, no further action. */ - MMAP_REMAP_PFN, /* Remap PFN range. */ - MMAP_IO_REMAP_PFN, /* I/O remap PFN range. */ + MMAP_NOTHING, + MMAP_REMAP_PFN, + MMAP_IO_REMAP_PFN, MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_MAP_KERNEL_PAGES, /* Map kernel page range from an array. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ }; /* From dad8e88fcfce855c691e65f2da3d1da326c69d7a Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:47 +0100 Subject: [PATCH 097/213] mm: assert mmap lock held, use new flags API in vm_insert_page[s] The existing logic asserts that an mmap write lock is held, bizarrely, by asserting that an mmap_read_trylock() fails (which is not the same thing). Assert the mmap write lock here and switch to using the new VMA flags API while here. Note that the vma_start_write() is explicitly broken out from vm_flags_set() here in order to use vma_set_flags() safely. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-7-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Gregory Price (Meta) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/memory.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3e0dfab9a80e..62c336ae3fa2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2599,10 +2599,11 @@ int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, if (!range_in_vma(vma, addr, end)) return -EFAULT; - if (!(vma->vm_flags & VM_MIXEDMAP)) { - VM_WARN_ON_ONCE(mmap_read_trylock(vma->vm_mm)); - VM_WARN_ON_ONCE(vma->vm_flags & VM_PFNMAP); - vm_flags_set(vma, VM_MIXEDMAP); + if (!vma_test(vma, VMA_MIXEDMAP_BIT)) { + mmap_assert_write_locked(vma->vm_mm); + VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT)); + vma_start_write(vma); + vma_set_flags(vma, VMA_MIXEDMAP_BIT); } /* Defer page refcount checking till we're about to map that page. */ return insert_pages(vma, addr, pages, num, vma->vm_page_prot); @@ -2675,10 +2676,11 @@ int vm_insert_page(struct vm_area_struct *vma, unsigned long addr, { if (addr < vma->vm_start || addr >= vma->vm_end) return -EFAULT; - if (!(vma->vm_flags & VM_MIXEDMAP)) { - BUG_ON(mmap_read_trylock(vma->vm_mm)); - BUG_ON(vma->vm_flags & VM_PFNMAP); - vm_flags_set(vma, VM_MIXEDMAP); + if (!vma_test(vma, VMA_MIXEDMAP_BIT)) { + mmap_assert_write_locked(vma->vm_mm); + VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT)); + vma_start_write(vma); + vma_set_flags(vma, VMA_MIXEDMAP_BIT); } return insert_page(vma, addr, page, vma->vm_page_prot, false); } From b7b4d8f37a4ff533dffeea9cbb08e9e4788b7742 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:48 +0100 Subject: [PATCH 098/213] mm: add mmap action for discontiguous kernel page mapping The existing kernel page mapping mmap actions allow for partial and full mapping of an array of struct page pointers. However some drivers require the mapping of discontiguous ranges. Permit this by providing discontig_kernel_page_ops which allows a driver to specify how the operation should begin and how batches of pages should be retrieved. It uses the minimum exposed interface to do so, providing address, page offset and both vm_private_data state and a local private state object. ops->init can establish state for the operation, and ops->get outputs the pages to map and their count. Should an error arise the core unmaps the VMA, invoking vm_ops->close, which is therefore where any state established by ops->init is released. Also update the compatibility layer to do this too via compat_mmap_action_abort(). Batches may not exceed the VMA, but may map less than its full range in case the driver wishes to allow the user to map an area larger than the available data. To use it, users invoke mmap_action_map_discontig_kernel_pages() with initial local private state and a set of operations. Users can then use one of the provided helper functions to perform an action: * discontig_kernel_map_abort() - Abort and leave the mapping as it has been accumulated so far. * discontig_kernel_map_page() - Map a single page, or a compound page given its head page, as 1..N contiguous pages. * discontig_kernel_map_pages() - Map a struct page ** array of a specified count, mapping only those pages which fit in the VMA. The userland VMA tests are updated accordingly. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-8-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 43 +++++++++++++ include/linux/mm_types.h | 42 ++++++++++++- mm/internal.h | 3 + mm/memory.c | 106 ++++++++++++++++++++++++++++++-- mm/util.c | 27 ++++++-- tools/testing/vma/include/dup.h | 11 +++- 6 files changed, 218 insertions(+), 14 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d724c7401798..fc0d4c0a0cb6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4654,10 +4654,53 @@ static inline void mmap_action_map_kernel_pages_full(struct vm_area_desc *desc, vma_desc_pages(desc)); } +static inline +void mmap_action_map_discontig_kernel_pages(struct vm_area_desc *desc, + void *init_private, const struct discontig_kernel_page_ops *ops) +{ + struct mmap_action *action = &desc->action; + + action->type = MMAP_DISCONTIG_KERNEL_PAGES; + action->map_kernel_discontig.init_private = init_private; + action->map_kernel_discontig.ops = ops; +} + int mmap_action_prepare(struct vm_area_desc *desc); int mmap_action_complete(struct vm_area_struct *vma, struct mmap_action *action, bool is_compat); +static inline void +discontig_kernel_map_abort(struct discontig_kernel_page_state *state) +{ + state->action = DISCONTIG_KERNEL_PAGE_ABORT; +} + +static inline void +discontig_kernel_map_page(struct discontig_kernel_page_state *state, + struct page *page) +{ + struct folio *folio = page_folio(page); + unsigned long nr_pages = 1; + + if (folio_test_large(folio)) { + VM_WARN_ON_ONCE(page != folio_page(folio, 0)); + nr_pages = folio_nr_pages(folio); + } + + state->action = DISCONTIG_KERNEL_PAGE_MAP_CONTIG_PAGES; + state->__page = page; + state->__nr_pages = min(nr_pages, state->nr_pages_remain); +} + +static inline void +discontig_kernel_map_pages(struct discontig_kernel_page_state *state, + struct page **page_arr, unsigned long nr_pages) +{ + state->action = DISCONTIG_KERNEL_PAGE_MAP_PAGES; + state->__page_arr = page_arr; + state->__nr_pages = min(nr_pages, state->nr_pages_remain); +} + /* Look up the first VMA which exactly match the interval vm_start ... vm_end */ static inline struct vm_area_struct *find_exact_vma(struct mm_struct *mm, unsigned long vm_start, unsigned long vm_end) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bd5c8832d1af..0dead656d3e3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -818,8 +818,42 @@ enum mmap_action_type { MMAP_NOTHING, MMAP_REMAP_PFN, MMAP_IO_REMAP_PFN, - MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_DISCONTIG_KERNEL_PAGES, /* Map kernel discontig page range. */ +}; + +enum discontig_kernel_page_action { + DISCONTIG_KERNEL_PAGE_ABORT, + DISCONTIG_KERNEL_PAGE_MAP_PAGES, /* Array of individual pages. */ + DISCONTIG_KERNEL_PAGE_MAP_CONTIG_PAGES, /* 1..N contiguous pages. */ +}; + +struct discontig_kernel_page_state { + /* Map state. */ + const unsigned long start; /* Start address of VMA. */ + const unsigned long end; /* End address of VMA. */ + unsigned long addr; /* The current address to be mapped. */ + pgoff_t pgoff; /* The current pgoff to be mapped. */ + unsigned long nr_pages_mapped; /* The number of pages mapped. */ + unsigned long nr_pages_remain; /* The number of pages remaining. */ + + /* User-defined state. */ + void *vm_private_data; /* VMA private data. */ + void *private; /* Mapping private data. */ + + /* Users should not touch these, use discontig_kernel_map_*() helpers. */ + enum discontig_kernel_page_action action; + union { + struct page *__page; + struct page **__page_arr; + }; + unsigned long __nr_pages; +}; + +struct discontig_kernel_page_ops { + int (*init)(void *vm_private_data, void **private); + int (*get)(struct discontig_kernel_page_state *state); }; /* @@ -844,6 +878,10 @@ struct mmap_action { unsigned long nr_pages; pgoff_t pgoff; } map_kernel; + struct { + void *init_private; + const struct discontig_kernel_page_ops *ops; + } map_kernel_discontig; }; enum mmap_action_type type; diff --git a/mm/internal.h b/mm/internal.h index a11ccda86fde..8cef7593003c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1517,6 +1517,9 @@ int simple_ioremap_prepare(struct vm_area_desc *desc); int map_kernel_pages_prepare(struct vm_area_desc *desc); int map_kernel_pages_complete(struct vm_area_struct *vma, struct mmap_action *action); +int map_discontig_kernel_pages_prepare(struct vm_area_desc *desc); +int map_discontig_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action); static inline int io_remap_pfn_range_prepare(struct vm_area_desc *desc) { diff --git a/mm/memory.c b/mm/memory.c index 62c336ae3fa2..1b91e2361f9d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2610,17 +2610,23 @@ int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, } EXPORT_SYMBOL(vm_insert_pages); +static void __map_kernel_pages_prepare(struct vm_area_desc *desc) +{ + if (vma_desc_test(desc, VMA_MIXEDMAP_BIT)) + return; + + mmap_assert_write_locked(desc->mm); + VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT)); + vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT); +} + int map_kernel_pages_prepare(struct vm_area_desc *desc) { const struct mmap_action *action = &desc->action; const unsigned long addr = action->map_kernel.start; unsigned long nr_pages, end; - if (!vma_desc_test(desc, VMA_MIXEDMAP_BIT)) { - VM_WARN_ON_ONCE(mmap_read_trylock(desc->mm)); - VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT)); - vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT); - } + __map_kernel_pages_prepare(desc); nr_pages = action->map_kernel.nr_pages; end = addr + PAGE_SIZE * nr_pages; @@ -2641,6 +2647,96 @@ int map_kernel_pages_complete(struct vm_area_struct *vma, &nr_pages, vma->vm_page_prot); } +int map_discontig_kernel_pages_prepare(struct vm_area_desc *desc) +{ + const struct mmap_action *action = &desc->action; + const struct discontig_kernel_page_ops *ops = + action->map_kernel_discontig.ops; + + /* At minimum need to be able to get pages. */ + if (WARN_ON_ONCE(!ops || !ops->get)) + return -EINVAL; + + __map_kernel_pages_prepare(desc); + return 0; +} + +static int apply_discontig_action(struct vm_area_struct *vma, + struct discontig_kernel_page_state *state) +{ + unsigned long nr_pages = state->__nr_pages; + unsigned long addr = state->addr; + unsigned long i; + + if (state->action == DISCONTIG_KERNEL_PAGE_MAP_PAGES) + return insert_pages(vma, addr, state->__page_arr, + &nr_pages, vma->vm_page_prot); + + VM_WARN_ON_ONCE(state->action != DISCONTIG_KERNEL_PAGE_MAP_CONTIG_PAGES); + /* Contiguous pages - have to iterate through each one. */ + for (i = 0; i < nr_pages; i++, addr += PAGE_SIZE) { + struct page *page = state->__page + i; + int err; + + err = insert_page(vma, addr, page, vma->vm_page_prot, + /*mkwrite=*/false); + if (err) + return err; + } + return 0; +} + +int map_discontig_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action) +{ + const struct discontig_kernel_page_ops *ops = + action->map_kernel_discontig.ops; + struct discontig_kernel_page_state state = { + .start = vma->vm_start, + .end = vma->vm_end, + .addr = vma->vm_start, + .pgoff = vma->vm_pgoff, + .nr_pages_mapped = 0, + .nr_pages_remain = vma_pages(vma), + .vm_private_data = vma->vm_private_data, + .private = action->map_kernel_discontig.init_private, + }; + int err = 0; + + if (ops->init) + err = ops->init(vma->vm_private_data, &state.private); + if (err) + return err; + + do { + unsigned long end, pgoff_end; + unsigned long nr_pages; + + /* Default to abort. */ + state.action = DISCONTIG_KERNEL_PAGE_ABORT; + err = ops->get(&state); + if (err || state.action == DISCONTIG_KERNEL_PAGE_ABORT) + return err; + nr_pages = state.__nr_pages; + + if (!nr_pages || nr_pages > state.nr_pages_remain) + return -EINVAL; + end = state.addr + PAGE_SIZE * nr_pages; + pgoff_end = state.pgoff + nr_pages; + + err = apply_discontig_action(vma, &state); + if (err) + return err; + + state.addr = end; + state.pgoff = pgoff_end; + state.nr_pages_mapped += nr_pages; + state.nr_pages_remain -= nr_pages; + } while (state.addr < vma->vm_end); + + return 0; +} + /** * vm_insert_page - insert single page into user vma * @vma: user vma to map to diff --git a/mm/util.c b/mm/util.c index 441e21eb0e57..7f374bb8bf36 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1401,6 +1401,15 @@ static int call_vma_mapped(struct vm_area_struct *vma) return 0; } +/* An mmap action failed under the compatibility layer - unmap and close. */ +static void compat_mmap_action_abort(struct vm_area_struct *vma) +{ +#ifdef CONFIG_MMU + zap_vma_range(vma, vma->vm_start, vma_pages(vma) << PAGE_SHIFT); +#endif + vma_close(vma); +} + static int mmap_action_finish(struct vm_area_struct *vma, struct mmap_action *action, int err, bool is_compat) @@ -1412,12 +1421,13 @@ static int mmap_action_finish(struct vm_area_struct *vma, /* do_munmap() might take rmap lock, so release if held. */ maybe_rmap_unlock_action(vma, action); - /* - * If this is invoked from the compatibility layer, post-mmap() hook - * logic will handle cleanup for us. - */ - if (!err || is_compat) + if (!err) + return 0; + + if (is_compat) { + compat_mmap_action_abort(vma); return err; + } /* * If an error occurs, unmap the VMA altogether and return an error. We @@ -1469,6 +1479,8 @@ int mmap_action_prepare(struct vm_area_desc *desc) return simple_ioremap_prepare(desc); case MMAP_KERNEL_PAGES: return map_kernel_pages_prepare(desc); + case MMAP_DISCONTIG_KERNEL_PAGES: + return map_discontig_kernel_pages_prepare(desc); } WARN_ON_ONCE(1); @@ -1501,6 +1513,9 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_KERNEL_PAGES: err = map_kernel_pages_complete(vma, action); break; + case MMAP_DISCONTIG_KERNEL_PAGES: + err = map_discontig_kernel_pages_complete(vma, action); + break; case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: /* Should have been delegated. */ @@ -1522,6 +1537,7 @@ int mmap_action_prepare(struct vm_area_desc *desc) case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: case MMAP_KERNEL_PAGES: + case MMAP_DISCONTIG_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle these. */ break; } @@ -1543,6 +1559,7 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: case MMAP_KERNEL_PAGES: + case MMAP_DISCONTIG_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle this. */ err = -EINVAL; diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 023ad6092e9c..a6a8e53c4b38 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -457,14 +457,17 @@ enum mmap_action_type { MMAP_NOTHING, MMAP_REMAP_PFN, MMAP_IO_REMAP_PFN, - MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_DISCONTIG_KERNEL_PAGES, /* Map kernel discontig page range. */ }; /* * Describes an action an mmap_prepare hook can instruct to be taken to complete * the mapping of a VMA. Specified in vm_area_desc. */ +struct discontig_kernel_page_ops; + struct mmap_action { union { struct { @@ -483,6 +486,10 @@ struct mmap_action { unsigned long nr_pages; pgoff_t pgoff; } map_kernel; + struct { + void *init_private; + const struct discontig_kernel_page_ops *ops; + } map_kernel_discontig; }; enum mmap_action_type type; From ae60723abb65b6304699eb7f5ab66cbc84bd3bb8 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:49 +0100 Subject: [PATCH 099/213] docs: filesystems: update mmap_prepare docs for discontig kernel pgs Describe the newly introduced discontiguous kernel page mapping mechanism, detailing how to use it sensibly and how the API looks. Explicitly detail the various discontiguous actions available and how to use them. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-9-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- Documentation/filesystems/mmap_prepare.rst | 81 ++++++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/Documentation/filesystems/mmap_prepare.rst b/Documentation/filesystems/mmap_prepare.rst index 82c99c95ad85..b5afbb33e5c8 100644 --- a/Documentation/filesystems/mmap_prepare.rst +++ b/Documentation/filesystems/mmap_prepare.rst @@ -164,5 +164,86 @@ pointer. These are: sufficient entries in the page array to cover the entire range of the described VMA. +* mmap_action_map_discontig_kernel_pages() - Maps a discontiguous range of + `struct page` pointers over the VMA. They must span from the start of the VMA, + but may terminate prior to the end (leaving the remainder unmapped). + **NOTE:** The ``action`` field should never normally be manipulated directly, rather you ought to use one of these helpers. + +Discontiguous Actions +===================== + +Some actions can be performed across discontiguous ranges. + +Map kernel pages +---------------- + +To map kernel pages discontiguously, you must provide hooks using ``struct +discontig_kernel_page_ops``: + +.. code-block:: C + + struct discontig_kernel_page_ops { + int (*init)(void *vm_private_data, void **private); + int (*get)(struct discontig_kernel_page_state *state); + }; + +The ``init`` hook is optional and allows state to be established before the +operation starts, for instance taking a reference count. Nothing is invoked +after the operation, so ``init`` must not leave locks held, and state that must +be released once the mapping goes away should be released in +``vm_ops->close``. + +The ``init`` hook, if provided, is invoked prior to the operation starting. It +may update what is pointed to by ``vm_private_data`` and/or ``private``. If an +error is returned, then the operation is aborted. The ``private`` field can be +reassigned. + +**NOTE:** The operation may sleep between invocations of ``get``, so locks +needed to stabilise state must be taken and released within each hook. + +The ``get`` handler is the key means through which the operation is +executed. The current state of the operation is provided through ``struct +discontig_kernel_page_state``: + +.. code-block:: C + + struct discontig_kernel_page_state { + /* Map state. */ + unsigned long start; /* Start address of VMA. */ + unsigned long end; /* End address of VMA. */ + unsigned long addr; /* The current address to be mapped. */ + pgoff_t pgoff; /* The current pgoff to be mapped. */ + unsigned long nr_pages_mapped; /* The number of pages mapped. */ + unsigned long nr_pages_remain; /* The number of pages remaining. */ + + /* User-defined state. */ + void *vm_private_data; /* VMA private data. */ + void *private; /* Mapping private data. */ + + /* Users should not touch these, use discontig_kernel_map_*() helpers. */ + ... internal fields ... + }; + +With ``private`` being an additional user-controllable state variable, +initialised via ``mmap_action_map_discontig_kernel_pages()``, and +``vm_private_data`` being equal to the ``desc->private_data`` field set in +the ``mmap_prepare()`` hook. + +In the ``get`` hook, the user must choose how to map kernel pages: + +* ``discontig_kernel_map_abort()`` - Call this to abort the operation, whatever + has been mapped so far will be retained, the rest of the mapping will SIGBUS + if accessed. +* ``discontig_kernel_map_page()`` - Maps a single page, correctly handling + compound pages (if the compound page is bigger than the remaining pages in the + VMA, then only those pages that fit will be mapped). For a compound page, the + head page must be passed. +* ``discontig_kernel_map_pages()`` - Map an array of pages of a specified + size. As with a compound page, if more pages are specified than remain in + the VMA then only those that fit will be mapped. + +If an error arises after ``init`` succeeded, the core unmaps the VMA, invoking +``vm_ops->close`` if set, which is therefore the place to release any state +that ``init`` established. From 5323bb5f6ef306197b5bc1dd3cbad1858afe3150 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:50 +0100 Subject: [PATCH 100/213] drivers/usb/mon: update to use mmap_prepare + map kernel pages Replace the deprecated .mmap hook with its replacement .mmap_prepare. As part of this change, additionally take the approach of mapping pages upon mmap rather than providing a fault handler. The page span cannot be mutated when an mmap mapping is in place, so this is safe to do in advance (the MON_IOCT_RING_SIZE ioctl operation exits -EBUSY if it's attempted, gated by the rp->mmap_active reference count). Utilise the newly introduced mmap_action_map_discontig_kernel_pages() to do this, which allows for iteration over pages in mon_bin_discontig_get(). mon_bin_discontig_init() increments the rp->mmap_active reference count to stabilise page spans. Should an error arise the core unmaps the VMA and mon_bin_vma_close() drops the reference again. The vm_ops->close hook implemented in mon_bin_vma_close() will ensure correct reference count arithmetic upon unmap (with mon_bin_vma_open() accounting for splitting). The existing semantics are all retained, including not mapping past the range of available pages, with a SIGBUS being raised in a userland process that attempts to access past this point. Ultimately insert_page() is invoked to insert each page, which increments the reference count on each mapped page. This mimics what was being done previously, only we pre-map the entire range rather than doing so on demand. The existing fault handler did nothing that required demand paging, and was presumably implemented this way for historical reasons. One behavioural difference: pages are no longer faulted in on demand, so a page discarded with MADV_DONTNEED is not repopulated and a subsequent access raises SIGBUS, as with other pre-populated kernel mappings. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-10-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Greg Kroah-Hartman Reviewed-by: Suren Baghdasaryan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: David Hildenbrand Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- drivers/usb/mon/mon_bin.c | 98 ++++++++++++++++++++++++--------------- 1 file changed, 61 insertions(+), 37 deletions(-) diff --git a/drivers/usb/mon/mon_bin.c b/drivers/usb/mon/mon_bin.c index 687f6a8981f3..9d00b21a8153 100644 --- a/drivers/usb/mon/mon_bin.c +++ b/drivers/usb/mon/mon_bin.c @@ -1219,13 +1219,8 @@ mon_bin_poll(struct file *file, struct poll_table_struct *wait) return mask; } -/* - * open and close: just keep track of how many times the device is - * mapped, to use the proper memory allocation function. - */ -static void mon_bin_vma_open(struct vm_area_struct *vma) +static void __mon_bin_vma_open(struct mon_reader_bin *rp) { - struct mon_reader_bin *rp = vma->vm_private_data; unsigned long flags; spin_lock_irqsave(&rp->b_lock, flags); @@ -1233,57 +1228,86 @@ static void mon_bin_vma_open(struct vm_area_struct *vma) spin_unlock_irqrestore(&rp->b_lock, flags); } -static void mon_bin_vma_close(struct vm_area_struct *vma) +/* + * open and close: just keep track of how many times the device is + * mapped, to use the proper memory allocation function. + */ +static void mon_bin_vma_open(struct vm_area_struct *vma) +{ + struct mon_reader_bin *rp = vma->vm_private_data; + + __mon_bin_vma_open(rp); +} + +static void __mon_bin_vma_close(struct mon_reader_bin *rp) { unsigned long flags; - struct mon_reader_bin *rp = vma->vm_private_data; spin_lock_irqsave(&rp->b_lock, flags); rp->mmap_active--; spin_unlock_irqrestore(&rp->b_lock, flags); } -/* - * Map ring pages to user space. - */ -static vm_fault_t mon_bin_vma_fault(struct vm_fault *vmf) +static void mon_bin_vma_close(struct vm_area_struct *vma) { - struct mon_reader_bin *rp = vmf->vma->vm_private_data; - unsigned long offset, chunk_idx; - struct page *pageptr; - unsigned long flags; + struct mon_reader_bin *rp = vma->vm_private_data; - spin_lock_irqsave(&rp->b_lock, flags); - offset = vmf->pgoff << PAGE_SHIFT; - if (offset >= rp->b_size) { - spin_unlock_irqrestore(&rp->b_lock, flags); - return VM_FAULT_SIGBUS; - } - chunk_idx = offset / CHUNK_SIZE; - pageptr = rp->b_vec[chunk_idx].pg; - get_page(pageptr); - vmf->page = pageptr; - spin_unlock_irqrestore(&rp->b_lock, flags); - return 0; + __mon_bin_vma_close(rp); } static const struct vm_operations_struct mon_bin_vm_ops = { .open = mon_bin_vma_open, .close = mon_bin_vma_close, - .fault = mon_bin_vma_fault, }; -static int mon_bin_mmap(struct file *filp, struct vm_area_struct *vma) +static int mon_bin_discontig_init(void *vm_private_data, void **private) { - /* don't do anything here: "fault" will set up page table entries */ - vma->vm_ops = &mon_bin_vm_ops; + struct mon_reader_bin *rp = vm_private_data; - if (vma->vm_flags & VM_WRITE) + /* Dropped by mon_bin_vma_close() on unmap, including on error. */ + __mon_bin_vma_open(rp); + return 0; +} + +static int mon_bin_discontig_get(struct discontig_kernel_page_state *state) +{ + struct mon_reader_bin *rp = state->vm_private_data; + unsigned long offset, chunk_idx; + unsigned long flags; + + spin_lock_irqsave(&rp->b_lock, flags); + + offset = state->pgoff << PAGE_SHIFT; + if (offset >= rp->b_size) { + spin_unlock_irqrestore(&rp->b_lock, flags); + discontig_kernel_map_abort(state); + return 0; + } + chunk_idx = offset / CHUNK_SIZE; + discontig_kernel_map_page(state, rp->b_vec[chunk_idx].pg); + + spin_unlock_irqrestore(&rp->b_lock, flags); + return 0; +} + +static const struct discontig_kernel_page_ops mon_discontig_ops = { + .init = mon_bin_discontig_init, + .get = mon_bin_discontig_get, +}; + +static int mon_bin_mmap_prepare(struct vm_area_desc *desc) +{ + const struct file *filp = desc->file; + + if (vma_desc_test(desc, VMA_WRITE_BIT)) return -EPERM; - vm_flags_mod(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_MAYWRITE); - vma->vm_private_data = filp->private_data; - mon_bin_vma_open(vma); + desc->vm_ops = &mon_bin_vm_ops; + vma_desc_clear_flags(desc, VMA_MAYWRITE_BIT); + vma_desc_set_flags(desc, VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT); + desc->private_data = filp->private_data; + + mmap_action_map_discontig_kernel_pages(desc, NULL, &mon_discontig_ops); return 0; } @@ -1298,7 +1322,7 @@ static const struct file_operations mon_fops_binary = { .compat_ioctl = mon_bin_compat_ioctl, #endif .release = mon_bin_release, - .mmap = mon_bin_mmap, + .mmap_prepare = mon_bin_mmap_prepare, }; static int mon_bin_wait_event(struct file *file, struct mon_reader_bin *rp) From 077b65ba0d8a27c2f3e9b51918fb26736d7d7942 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:51 +0100 Subject: [PATCH 101/213] infiniband: update hfi1 to use remap_vmalloc_range() In cases which map chip memory from vmalloc()'d ranges, the hfi1 infiniband drivers currently installs a fault handler, and then smuggles the kernel virtual address of this range in vma->vm_pgoff. This is exposing KASLR-sensitive internal kernel state in the VMA, and is entirely unnecessary. Instead, use remap_vmalloc_range() to remap the VMA to the span, and eliminate the fault handler altogether. remap_vmalloc_range() checks that the VMA does not extend beyond the vmalloc area, and the driver already requires the VMA to exactly match the span of the memory being mapped, so this has no impact. The memory is all preallocated so not having a fault handler has no impact either, other than pre-mapping the ranges which is beneficial. We also remove the VM_IO flag as it's not appropriate here, and the VM_DONTEXPAND flag as remap_vmalloc_range() will set it (and also mark the range correctly as a mixed map). We also update the vmalloc paths to place the virtual kernel address in memvirt, rather than overloading the physical address memaddr. We predicate the vmalloc handling on the vmalloc flag before we check memvirt for the virtual address-derived PFN remap path, so this works fine. remap_vmalloc_range() requires that the vmalloc()'d areas were all allocated using vmalloc_user() - each of cq->comps, uctxt->subctxt_rcvegrbuf, uctxt->subctxt_rcvhdr_base, uctxt->subctxt_uregbase and dd->events were allocated this way, so that requirement is satisfied. We also remove VM_IO and VM_DONTEXPAND from the STATUS command, as these are both set on remap. Finally, we remove VM_DONTEXPAND from the PIO_BUFS, PIO_BUFS_SOP and UREGS commands, as these are also all set on remap. PIO_CRED retains it, as dma_mmap_coherent() may map via vm_insert_page() on the IOMMU-DMA path, which sets only VM_MIXEDMAP. The RCV_HDRQ, RCV_EGRBUF and RTAIL commands also map via dma_mmap_coherent() and never set VM_DONTEXPAND, so set it for them for the same reason. Note that we retain expected behaviour throughout - the vmalloc remapped ranges set VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND for each range. VM_IO was never appropriate as the ranges are explicitly not MMIO, and the reference to the v3.7 VM_RESERVED semantics map on to VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND correctly - no core dump, unmergeable, no normal vm page for purposes of reclaim/migration/etc. There is a change in behaviour in that pages mapped using remap_vmalloc_range() will now have normal GUP-able pages, however this should have no impact as there is no reason not to allow this. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-11-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: David Hildenbrand Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- drivers/infiniband/hw/hfi1/file_ops.c | 83 ++++++++------------------- 1 file changed, 25 insertions(+), 58 deletions(-) diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c index 1a36f995c4f6..680364fda856 100644 --- a/drivers/infiniband/hw/hfi1/file_ops.c +++ b/drivers/infiniband/hw/hfi1/file_ops.c @@ -70,7 +70,6 @@ static int set_ctxt_pkey(struct hfi1_ctxtdata *uctxt, unsigned long arg); static int ctxt_reset(struct hfi1_ctxtdata *uctxt); static int manage_rcvq(struct hfi1_ctxtdata *uctxt, u16 subctxt, unsigned long arg); -static vm_fault_t vma_fault(struct vm_fault *vmf); static long hfi1_file_ioctl(struct file *fp, unsigned int cmd, unsigned long arg); @@ -85,10 +84,6 @@ static const struct file_operations hfi1_file_ops = { .llseek = noop_llseek, }; -static const struct vm_operations_struct vm_ops = { - .fault = vma_fault, -}; - /* * Types of memories mapped into user processes' space */ @@ -304,13 +299,13 @@ static ssize_t hfi1_write_iter(struct kiocb *kiocb, struct iov_iter *from) return reqs; } -static inline void mmap_cdbg(u16 ctxt, u8 subctxt, u8 type, u8 mapio, u8 vmf, +static inline void mmap_cdbg(u16 ctxt, u8 subctxt, u8 type, u8 mapio, u8 is_vmalloc, u64 memaddr, void *memvirt, dma_addr_t memdma, ssize_t memlen, struct vm_area_struct *vma) { hfi1_cdbg(PROC, - "%u:%u type:%u io/vf/dma:%d/%d/%d, addr:0x%llx, len:%lu(%lu), flags:0x%lx", - ctxt, subctxt, type, mapio, vmf, !!memdma, + "%u:%u type:%u io/vmalloc/dma:%d/%d/%d, addr:0x%llx, len:%lu(%lu), flags:0x%lx", + ctxt, subctxt, type, mapio, is_vmalloc, !!memdma, memaddr ?: (u64)memvirt, memlen, vma->vm_end - vma->vm_start, vma->vm_flags); } @@ -325,7 +320,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memaddr = 0; void *memvirt = NULL; dma_addr_t memdma = 0; - u8 subctxt, mapio = 0, vmf = 0, type; + u8 subctxt, mapio = 0, is_vmalloc = 0, type; size_t memdmalen = 0; ssize_t memlen = 0; int ret = 0; @@ -348,7 +343,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) /* * vm_pgoff is used as a buffer selector cookie. Always mmap from * the beginning. - */ + */ vma->vm_pgoff = 0; flags = vma->vm_flags; @@ -367,7 +362,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) */ memlen = PAGE_ALIGN(uctxt->sc->credits * PIO_BLOCK_SIZE); flags &= ~VM_MAYREAD; - flags |= VM_DONTCOPY | VM_DONTEXPAND; + flags |= VM_DONTCOPY; vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); mapio = 1; break; @@ -411,6 +406,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memlen = rcvhdrq_size(uctxt); memvirt = uctxt->rcvhdrq; memdma = uctxt->rcvhdrq_dma; + flags |= VM_DONTEXPAND; break; case RCV_EGRBUF: { unsigned long vm_start_save; @@ -432,7 +428,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) ret = -EPERM; goto done; } - vm_flags_clear(vma, VM_MAYWRITE); + vm_flags_mod(vma, VM_DONTEXPAND, VM_MAYWRITE); /* * Mmap multiple separate allocations into a single vma. From * here, dma_mmap_coherent() calls dma_direct_mmap(), which @@ -448,7 +444,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memvirt = uctxt->egrbufs.buffers[i].addr; memdma = uctxt->egrbufs.buffers[i].dma; vma->vm_end += memlen; - mmap_cdbg(ctxt, subctxt, type, mapio, vmf, memaddr, + mmap_cdbg(ctxt, subctxt, type, mapio, is_vmalloc, memaddr, memvirt, memdma, memlen, vma); ret = dma_mmap_coherent(&dd->pcidev->dev, vma, memvirt, memdma, memlen); @@ -477,7 +473,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) * user registers. */ memlen = PAGE_SIZE; - flags |= VM_DONTCOPY | VM_DONTEXPAND; + flags |= VM_DONTCOPY; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); mapio = 1; break; @@ -486,15 +482,10 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) * Use the page where this context's flags are. User level * knows where it's own bitmap is within the page. */ - memaddr = (unsigned long) - (dd->events + uctxt_offset(uctxt)) & PAGE_MASK; + memvirt = dd->events + uctxt_offset(uctxt); + memvirt = (void *)(((uintptr_t)memvirt) & PAGE_MASK); memlen = PAGE_SIZE; - /* - * v3.7 removes VM_RESERVED but the effect is kept by - * using VM_IO. - */ - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; case STATUS: if (flags & VM_WRITE) { @@ -503,7 +494,6 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) } memaddr = kvirt_to_phys((void *)dd->status); memlen = PAGE_SIZE; - flags |= VM_IO | VM_DONTEXPAND; break; case RTAIL: if (!HFI1_CAP_IS_USET(DMA_RTAIL)) { @@ -522,25 +512,23 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memvirt = (void *)hfi1_rcvhdrtail_kvaddr(uctxt); memdma = uctxt->rcvhdrqtailaddr_dma; flags &= ~VM_MAYWRITE; + flags |= VM_DONTEXPAND; break; case SUBCTXT_UREGS: - memaddr = (u64)uctxt->subctxt_uregbase; + memvirt = uctxt->subctxt_uregbase; memlen = PAGE_SIZE; - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; case SUBCTXT_RCV_HDRQ: - memaddr = (u64)uctxt->subctxt_rcvhdr_base; + memvirt = uctxt->subctxt_rcvhdr_base; memlen = rcvhdrq_size(uctxt) * uctxt->subctxt_cnt; - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; case SUBCTXT_EGRBUF: - memaddr = (u64)uctxt->subctxt_rcvegrbuf; + memvirt = uctxt->subctxt_rcvegrbuf; memlen = uctxt->egrbufs.size * uctxt->subctxt_cnt; - flags |= VM_IO | VM_DONTEXPAND; flags &= ~VM_MAYWRITE; - vmf = 1; + is_vmalloc = 1; break; case SDMA_COMP: { struct hfi1_user_sdma_comp_q *cq = fd->cq; @@ -549,10 +537,9 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) ret = -EFAULT; goto done; } - memaddr = (u64)cq->comps; + memvirt = cq->comps; memlen = PAGE_ALIGN(sizeof(*cq->comps) * cq->nentries); - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; } default: @@ -569,12 +556,10 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) } vm_flags_reset(vma, flags); - mmap_cdbg(ctxt, subctxt, type, mapio, vmf, memaddr, memvirt, memdma, + mmap_cdbg(ctxt, subctxt, type, mapio, is_vmalloc, memaddr, memvirt, memdma, memlen, vma); - if (vmf) { - vma->vm_pgoff = PFN_DOWN(memaddr); - vma->vm_ops = &vm_ops; - ret = 0; + if (is_vmalloc) { + ret = remap_vmalloc_range(vma, memvirt, 0); } else if (memdma) { ret = dma_mmap_coherent(&dd->pcidev->dev, vma, memvirt, memdma, @@ -599,24 +584,6 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) return ret; } -/* - * Local (non-chip) user memory is not mapped right away but as it is - * accessed by the user-level code. - */ -static vm_fault_t vma_fault(struct vm_fault *vmf) -{ - struct page *page; - - page = vmalloc_to_page((void *)(vmf->pgoff << PAGE_SHIFT)); - if (!page) - return VM_FAULT_SIGBUS; - - get_page(page); - vmf->page = page; - - return 0; -} - static __poll_t hfi1_poll(struct file *fp, struct poll_table_struct *pt) { struct hfi1_ctxtdata *uctxt; From c6ce4378f20a3bce1da53c9e2d2f74d853096ab8 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:52 +0100 Subject: [PATCH 102/213] selinux: reject writable opens of policy file, drop mmap shared/write check The policy file has no write method and is exposed read-only (S_IRUGO in selinux_files[]), yet sel_open_policy() performs no open mode check, so a CAP_DAC_OVERRIDE caller can open it O_RDWR. Reject FMODE_WRITE at open, as kernfs does. The file can then never be mapped with FMODE_WRITE, so do_mmap() always clears VM_MAYWRITE and VM_SHARED for MAP_SHARED mappings and the VM_SHARED check in sel_mmap_policy() cannot be reached. Remove it. This also stops sel_mmap_policy() clearing VM_MAYWRITE on a mapping that is neither a PFN map nor a mixed map, ahead of the core enforcing that only such mappings may do so. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-12-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Stephen Smalley Reviewed-by: Jann Horn Acked-by: Paul Moore Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- security/selinux/selinuxfs.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c index c7d91476971c..545a6f89f9e7 100644 --- a/security/selinux/selinuxfs.c +++ b/security/selinux/selinuxfs.c @@ -340,6 +340,9 @@ static int sel_open_policy(struct inode *inode, struct file *filp) struct policy_load_memory *plm = NULL; int rc; + if (filp->f_mode & FMODE_WRITE) + return -EACCES; + rc = avc_has_perm(current_sid(), SECINITSID_SECURITY, SECCLASS_SECURITY, SECURITY__READ_POLICY, NULL); if (rc) @@ -424,14 +427,6 @@ static const struct vm_operations_struct sel_mmap_policy_ops = { static int sel_mmap_policy(struct file *filp, struct vm_area_struct *vma) { - if (vma->vm_flags & VM_SHARED) { - /* do not allow mprotect to make mapping writable */ - vm_flags_clear(vma, VM_MAYWRITE); - - if (vma->vm_flags & VM_WRITE) - return -EACCES; - } - vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &sel_mmap_policy_ops; From ceb0de27371f8bebacd22374b99a076306a47760 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:53 +0100 Subject: [PATCH 103/213] ALSA: pcm: use vm_insert_page() to map PCM status page There's no need to keep a fault handler around for this, instead map on mmap. While we're here, rename area to vma to be consistent. This correctly makes the mapping a mixed map mapping. This works towards establishing the invariant that only PFN mapped or mixed map mappings may clear the VM_MAYWRITE flag. The status page mapping clears VM_MAYWRITE, so it must not be mm-managed; the control page mapping remains writable and is left fault-based. The assumption is made that the struct pcm_mmap_status structure is at most a page in size, which is asserted as a build bug. This is safe to assume, as the size of the structure is 56 bytes at most. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-13-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Takashi Iwai Reviewed-by: Suren Baghdasaryan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: David Hildenbrand Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- sound/core/pcm_native.c | 48 ++++++++++++++++------------------------- 1 file changed, 18 insertions(+), 30 deletions(-) diff --git a/sound/core/pcm_native.c b/sound/core/pcm_native.c index 6d32c12fb79b..af80488baaf4 100644 --- a/sound/core/pcm_native.c +++ b/sound/core/pcm_native.c @@ -3760,39 +3760,27 @@ static __poll_t snd_pcm_poll(struct file *file, poll_table *wait) /* * mmap status record */ -static vm_fault_t snd_pcm_mmap_status_fault(struct vm_fault *vmf) -{ - struct snd_pcm_substream *substream = vmf->vma->vm_private_data; - struct snd_pcm_runtime *runtime; - - if (substream == NULL) - return VM_FAULT_SIGBUS; - runtime = substream->runtime; - vmf->page = virt_to_page(runtime->status); - get_page(vmf->page); - return 0; -} - -static const struct vm_operations_struct snd_pcm_vm_ops_status = -{ - .fault = snd_pcm_mmap_status_fault, -}; - static int snd_pcm_mmap_status(struct snd_pcm_substream *substream, struct file *file, - struct vm_area_struct *area) + struct vm_area_struct *vma) { - long size; - if (!(area->vm_flags & VM_READ)) - return -EINVAL; - size = area->vm_end - area->vm_start; - if (size != PAGE_ALIGN(sizeof(struct snd_pcm_mmap_status))) - return -EINVAL; - area->vm_ops = &snd_pcm_vm_ops_status; - area->vm_private_data = substream; - vm_flags_mod(area, VM_DONTEXPAND | VM_DONTDUMP, - VM_WRITE | VM_MAYWRITE); + const unsigned long size = vma->vm_end - vma->vm_start; + struct snd_pcm_runtime *runtime; + struct page *page; - return 0; + BUILD_BUG_ON(sizeof(struct snd_pcm_mmap_status) > PAGE_SIZE); + + if (!(vma->vm_flags & VM_READ)) + return -EINVAL; + if (size != PAGE_SIZE) + return -EINVAL; + + vm_flags_mod(vma, VM_DONTEXPAND | VM_DONTDUMP, + VM_WRITE | VM_MAYWRITE); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + + runtime = substream->runtime; + page = virt_to_page(runtime->status); + return vm_insert_page(vma, vma->vm_start, page); } /* From 0c2c9885c5e90d2182118fe101d94c515f067ebc Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:54 +0100 Subject: [PATCH 104/213] bpf: arena: mark arena_map_mmap() mappings VM_MIXEDMAP The bpf_map->ops->map_mmap callback invoked by bpf_map_mmap() can be set to one of ringbuf_map_mmap_kern(), ringbuf_map_mmap_user(), array_map_mmap() or arena_map_mmap(). It is convention in mm to mark mappings whose pages the kernel manages itself with VM_MIXEDMAP, so core mm knows not to treat them as ordinary page cache or anonymous memory. The map_mmap callbacks ringbuf_map_mmap_kern() and ringbuf_map_mmap_user() use remap_vmalloc_range(), which ultimately invokes vm_insert_page() and so marks the ranges VM_MIXEDMAP, and array_map_mmap() sets VM_MIXEDMAP explicitly. However, the exception to this is arena_map_mmap(), which doesn't set the flag. This patch corrects this and updates the comment to reflect it. The pages are refcounted and vm_normal_page() finds them regardless of the flag, and VM_DONTEXPAND remains set (marking the memory as VM_SPECIAL and thus unmergeable). The one effect is that NUMA balancing now skips these VMAs, as it already does for the other bpf map mappings, which is the reason array_map_mmap() gives for setting the flag. The intent of this patch is to be able to establish the invariant that only PFN-mapped or mixed map ranges may clear the VM_MAYWRITE flag, as is done in bpf_map_mmap(). Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-14-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Emil Tsalapatis Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- kernel/bpf/arena.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 7b6847200b43..b69fe5e34339 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -620,8 +620,9 @@ static int arena_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) * clears VM_MAYEXEC. Set VM_DONTEXPAND to avoid potential change * of user_vm_start. Set VM_DONTCOPY to prevent arena VMA from * being copied into the child process on fork. + * This is a kernel page so set VM_MIXEDMAP. */ - vm_flags_set(vma, VM_DONTEXPAND | VM_DONTCOPY); + vm_flags_set(vma, VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTCOPY); vma->vm_ops = &arena_vm_ops; return 0; } From 7fdbc26b69c0241a2c1b8f426cf977e2dc4ed507 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:55 +0100 Subject: [PATCH 105/213] mm/vma: add vma[_flags]_is_mm_managed() predicates Rather than referring to VMA flags with uncertain meaning, add a new predicate that explicitly describes what absence of the VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT flags means, and then refer to that function for determining VMA mergeability. If neither of these flags are set, that means the contents of the mapping are managed by core mm. Otherwise they are instead owned by some other kernel component, usually a driver: the memory may be MMIO, kernel-allocated pages or even ordinary pages the driver maps itself, but core mm must not populate, reclaim, migrate, copy-on-write or merge the range itself. We initially also treat VMA_IO_BIT as implying a mapping is not mm-managed, as by implication it cannot be. (mlock() also sets VMA_IO_BIT transiently on ordinary VMAs while locking them, which is addressed later in this series.) However the intent is to in future remove this, as no mapping should be marked as an I/O mapping without also being marked with VMA_PFNMAP_BIT. This forms the basis of further work intended to improve how we express VMA properties such as this. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-15-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 53 ++++++++++++++++++++++++++++++++- tools/testing/vma/include/dup.h | 29 +++++++++++++++++- 2 files changed, 80 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fc0d4c0a0cb6..1275dd22bc0f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1613,6 +1613,41 @@ static inline bool vma_is_shared_maywrite(const struct vm_area_struct *vma) return is_shared_maywrite(&vma->flags); } +/** + * vma_flags_is_mm_managed() - Do the specified VMA flags indicate that the + * contents of the VMA are managed by core mm? + * @flags: The VMA flags to test. + * + * Such a mapping is one established by a simple mmap() or brk() with no driver + * or other kernel component involved. + * + * They are rmappable, and core mm services rmap operations including reclaim, + * as well as population, CoW and merge/split (if eligible). + * + * Mappings which fail this test are those where this is not the case - + * e.g. MMIO, user-mapped kernel pages, udmabuf (pfnmap shmem), etc. + * + * Returns: true if the flags indicate an mm-managed mapping. + */ +static inline bool vma_flags_is_mm_managed(const vma_flags_t *flags) +{ + return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT, + VMA_IO_BIT); +} + +/** + * vma_is_mm_managed() - Are the contents of @vma managed by core mm? + * @vma: The VMA to test. + * + * See vma_flags_is_mm_managed() for a description of this property. + * + * Returns: true if the VMA is mm-managed. + */ +static inline bool vma_is_mm_managed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_managed(&vma->flags); +} + /** * vma_flags_can_merge() - Do the specified VMA flags permit the VMA to be * merged with another? @@ -1621,7 +1656,23 @@ static inline bool vma_is_shared_maywrite(const struct vm_area_struct *vma) */ static inline bool vma_flags_can_merge(const vma_flags_t *flags) { - return !vma_flags_test_any_mask(flags, VMA_SPECIAL_FLAGS); + /* + * VMA merging assumes that a VMA's flags and fields completely describe + * its state. + * + * However, mappings which are not mm-managed may have established state + * upon mapping not embodied in any attribute of the VMA. + * + * Additionally, private (CoW) PFN maps encode the source PFN of the + * range in vma->vm_pgoff, which may otherwise cause spurious merges. + */ + if (!vma_flags_is_mm_managed(flags)) + return false; + /* VMA explicitly marked as being unmergeable. */ + if (vma_flags_test(flags, VMA_DONTEXPAND_BIT)) + return false; + + return true; } /** diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index a6a8e53c4b38..89c645a6ad60 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1665,7 +1665,34 @@ static inline bool file_is_dev_zero(const struct file *file) return file && file->f_op == &zero_fops; } +static inline bool vma_flags_is_mm_managed(const vma_flags_t *flags) +{ + return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT, + VMA_IO_BIT); +} + +static inline bool vma_is_mm_managed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_managed(&vma->flags); +} + static inline bool vma_flags_can_merge(const vma_flags_t *flags) { - return !vma_flags_test_any_mask(flags, VMA_SPECIAL_FLAGS); + /* + * VMA merging assumes that a VMA's flags and fields completely describe + * its state. + * + * However, mappings which are not mm-managed may have established state + * upon mapping not embodied in any attribute of the VMA. + * + * Additionally, private (CoW) PFN maps encode the source PFN of the + * range in vma->vm_pgoff, which may otherwise cause spurious merges. + */ + if (!vma_flags_is_mm_managed(flags)) + return false; + /* VMA explicitly marked as being unmergeable. */ + if (vma_flags_test(flags, VMA_DONTEXPAND_BIT)) + return false; + + return true; } From d99c4a302c5abcf4e46bdbefa5a3c6a620be230c Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:56 +0100 Subject: [PATCH 106/213] mm/vma: only allow mmap to clear VMA_MAYWRITE_BIT if not mm-managed For ordinary files the only way the VMA_MAYWRITE_BIT flag is cleared is if the underlying file is itself read-only. This means that mprotect() cannot mark a shared mapping of a read-only file as read/write, as doing so would violate the read only attribute, and permit writes. In general, we do not want file systems to be able to do this for read/write files. Doing so would violate fundamental user expectation of file attributes and likely break userspace. However, drivers pose a tricky problem here - the /dev/xxx file may be read/write but provide access to a resource which is fundamentally read-only. Therefore we must allow drivers to be able to clear VMA_MAYWRITE_BIT. To achieve both of these things, forbid this for mm-managed mappings as identified by vma_flags_is_mm_managed(). This constrains this ability to drivers which own the mapping's contents, whether memory-mapped I/O, kernel-allocated pages, or ordinary pages they map themselves, and so define its semantics. Every in-tree mmap hook which clears VMA_MAYWRITE_BIT, some twenty sites across drivers, filesystems and bpf, establishes a mapping which is not mm-managed, with usbmon and the ALSA PCM status page converted earlier in this series to do so. Note that drivers may, if they do not gate on VMA_SHARED_BIT, be able to disable MAP_PRIVATE-file-backed mapping CoW semantics. This is perhaps not always intended, but we retain this capacity to maintain existing behaviour. As no driver which clears VMA_MAYWRITE_BIT establishes an mm-managed mapping, no functional change is intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-16-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/vma.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/vma.c b/mm/vma.c index 106176ebe9fe..f334c87582d7 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -2819,6 +2819,11 @@ static int mmap_validate(unsigned long orig_start, unsigned long orig_end, if (WARN_ON_ONCE(!was_maywrite && is_maywrite)) return -EINVAL; + /* mm-managed mappings may not clear VMA_MAYWRITE_BIT. */ + if (vma_flags_is_mm_managed(curr_flags) && + WARN_ON_ONCE(was_maywrite && !is_maywrite)) + return -EINVAL; + return mmap_validate_vma_flags(curr_flags); } From 9201d0f5b9d55dad38201bf0561e7024039b429b Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:57 +0100 Subject: [PATCH 107/213] mm/vma: add and use vma_[flags]_is_fixed_mapping This determines whether a VMA cannot be expanded or merged because what they mapped was determined to be a set size at mmap time. This typically refers to mappings not managed by core mm, however VMA_DONTEXPAND_BIT is not reliably set alongside VMA_PFNMAP_BIT or VMA_MIXEDMAP_BIT, so we must explicitly test for this for now. We also explicitly test for VMA_PFNMAP_BIT as VMA_DONTEXPAND_BIT may not be set for VMA_PFNMAP_BIT's despite the one implying the other. Use this predicate in vma_flags_can_merge() and in check_prep_vma() in the mremap logic testing to see if mremap() can expand the VMA. The criteria for khugepaged and MADV_COLLAPSE eligibility in __thp_vma_allowable_orders() are precisely those for mergeability, so use vma_can_merge() there (with an expanded comment). This obviates the need for the VM_NO_KHUGEPAGED mask, so remove it. Hugetlb VMAs remain excluded from khugepaged as hugetlbfs always sets VMA_DONTEXPAND_BIT. Also update the userland VMA tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-17-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 39 +++++++++++++++++++++++++++++---- mm/huge_memory.c | 11 ++++++---- mm/mremap.c | 5 ++--- tools/testing/vma/include/dup.h | 16 +++++++++++++- 4 files changed, 59 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1275dd22bc0f..ef81e1fa7b03 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -601,9 +601,6 @@ enum { #define VMA_REMAP_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_PFNMAP_BIT, \ VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT) -/* This mask prevents VMA from being scanned with khugepaged */ -#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB) - /* This mask defines which mm->def_flags a process can inherit its parent */ #define VM_INIT_DEF_MASK VM_NOHUGEPAGE @@ -1648,6 +1645,40 @@ static inline bool vma_is_mm_managed(const struct vm_area_struct *vma) return vma_flags_is_mm_managed(&vma->flags); } +/** + * vma_flags_is_fixed_mapping() - Do the specified VMA flags indicate that this + * is a fixed mapping that cannot be expanded or merged? + * @flags: The VMA flags to test. + * + * Fixed mappings are those whose size is set at the point of mmap (for + * instance, a mapping of a fixed range of memory established by a driver), + * and thus cannot be expanded or merged. + * + * Returns: true if the flags indicate a fixed mapping. + */ +static inline bool vma_flags_is_fixed_mapping(const vma_flags_t *flags) +{ + /* + * VMA_PFNMAP_BIT should imply VMA_DONTEXPAND_BIT, but some callers set + * only the former. + */ + return vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT); +} + +/** + * vma_is_fixed_mapping() - Is this VMA a fixed mapping that cannot be + * expanded or merged? + * @vma: The VMA to test. + * + * See vma_flags_is_fixed_mapping() for a description of this property. + * + * Returns: true if the VMA maps a fixed mapping. + */ +static inline bool vma_is_fixed_mapping(const struct vm_area_struct *vma) +{ + return vma_flags_is_fixed_mapping(&vma->flags); +} + /** * vma_flags_can_merge() - Do the specified VMA flags permit the VMA to be * merged with another? @@ -1669,7 +1700,7 @@ static inline bool vma_flags_can_merge(const vma_flags_t *flags) if (!vma_flags_is_mm_managed(flags)) return false; /* VMA explicitly marked as being unmergeable. */ - if (vma_flags_test(flags, VMA_DONTEXPAND_BIT)) + if (vma_flags_is_fixed_mapping(flags)) return false; return true; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 67cf8ee898ba..4cf7fc461c67 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -212,11 +212,14 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, return in_pf ? orders : 0; /* - * khugepaged special VMA and hugetlb VMA. - * Must be checked after dax since some dax mappings may have - * VM_MIXEDMAP set. + * khugepaged moves data from VMAs once collapsed, after they have been + * faulted in, relying on refaulting for file-backed memory. + * + * Mappings not managed by core mm cannot be reliably reconstructed from + * page faults, and fixed mappings (including hugetlb) may well be + * mm-managed - precisely the mappings which cannot be merged. */ - if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED)) + if (!in_pf && !smaps && !vma_can_merge(vma)) return 0; /* diff --git a/mm/mremap.c b/mm/mremap.c index ace0c6a343b1..8983ccd44469 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -1822,8 +1822,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm) return -EINVAL; } - if ((vrm->flags & MREMAP_DONTUNMAP) && - vma_test_any(vma, VMA_DONTEXPAND_BIT, VMA_PFNMAP_BIT)) + if ((vrm->flags & MREMAP_DONTUNMAP) && vma_is_fixed_mapping(vma)) return -EINVAL; /* @@ -1861,7 +1860,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm) if (pgoff + (new_len >> PAGE_SHIFT) < pgoff) return -EINVAL; - if (vma_test_any(vma, VMA_DONTEXPAND_BIT, VMA_PFNMAP_BIT)) + if (vma_is_fixed_mapping(vma)) return -EFAULT; if (!mlock_future_ok(mm, vma_test(vma, VMA_LOCKED_BIT), vrm->delta)) diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 89c645a6ad60..442ea9eb098d 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1676,6 +1676,20 @@ static inline bool vma_is_mm_managed(const struct vm_area_struct *vma) return vma_flags_is_mm_managed(&vma->flags); } +static inline bool vma_flags_is_fixed_mapping(const vma_flags_t *flags) +{ + /* + * VMA_PFNMAP_BIT should imply VMA_DONTEXPAND_BIT, but some callers set + * only the former. + */ + return vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT); +} + +static inline bool vma_is_fixed_mapping(const struct vm_area_struct *vma) +{ + return vma_flags_is_fixed_mapping(&vma->flags); +} + static inline bool vma_flags_can_merge(const vma_flags_t *flags) { /* @@ -1691,7 +1705,7 @@ static inline bool vma_flags_can_merge(const vma_flags_t *flags) if (!vma_flags_is_mm_managed(flags)) return false; /* VMA explicitly marked as being unmergeable. */ - if (vma_flags_test(flags, VMA_DONTEXPAND_BIT)) + if (vma_flags_is_fixed_mapping(flags)) return false; return true; From abdd54cef5fcf6e50367224663035386d9f6bfb3 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:58 +0100 Subject: [PATCH 108/213] scsi: sg: convert mmap hook to mmap_prepare and rework Move from the deprecated mmap hook to the new mmap_prepare hook. We are mapping kernel pages here, so use the discontiguous kernel mapping mmap action to do so. Unwind the rather confusing loop and instead map as many pages as we can at one time. Note that we do not need to pay attention to rsv_schp->k_use_sg here, as the pages are populated for the length of the buffer at rsv_schp->page_order granularity as compound pages. The discontiguous kernel page mapping logic handles the compound pages for us. sfp->mmap_called keeps the buffer stable for us. As before it is never cleared, so a failed mmap also leaves it set. We also remove some useless vma, vma->vm_file NULL checks - these will always be non-NULL if you reached the mmap hook logic. We retain log output for consistency, but change what's output on page mapping to indicate that sg_discontig_get() does the work now. Note that we drop the VMA_IO_BIT flag for the VMA here. It was never necessary as we invoke alloc_pages() which gives us refcounted folios that are fine for GUP to access (VMA_IO_BIT would prevent that). Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-18-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: David Hildenbrand Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- drivers/scsi/sg.c | 115 ++++++++++++++++++++-------------------------- 1 file changed, 51 insertions(+), 64 deletions(-) diff --git a/drivers/scsi/sg.c b/drivers/scsi/sg.c index 5408f002e6c0..3f9e08725602 100644 --- a/drivers/scsi/sg.c +++ b/drivers/scsi/sg.c @@ -1212,85 +1212,72 @@ sg_fasync(int fd, struct file *filp, int mode) return fasync_helper(fd, filp, mode, &sfp->async_qp); } -static vm_fault_t -sg_vma_fault(struct vm_fault *vmf) +static int sg_discontig_init(void *vm_private_data, void **private) { - struct vm_area_struct *vma = vmf->vma; - Sg_fd *sfp; - unsigned long offset, len, sa; - Sg_scatter_hold *rsv_schp; - int k, length; + const unsigned long req_sz = (unsigned long)*private; + Sg_fd *sfp = vm_private_data; + Sg_scatter_hold *rsv_schp = &sfp->reserve; + int err = 0; - if ((NULL == vma) || (!(sfp = (Sg_fd *) vma->vm_private_data))) - return VM_FAULT_SIGBUS; - rsv_schp = &sfp->reserve; - offset = vmf->pgoff << PAGE_SHIFT; - if (offset >= rsv_schp->bufflen) - return VM_FAULT_SIGBUS; - SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sfp->parentdp, - "sg_vma_fault: offset=%lu, scatg=%d\n", - offset, rsv_schp->k_use_sg)); - sa = vma->vm_start; - length = 1 << (PAGE_SHIFT + rsv_schp->page_order); - for (k = 0; k < rsv_schp->k_use_sg && sa < vma->vm_end; k++) { - len = vma->vm_end - sa; - len = (len < length) ? len : length; - if (offset < len) { - struct page *page = rsv_schp->pages[k] + (offset >> PAGE_SHIFT); - get_page(page); /* increment page count */ - vmf->page = page; - return 0; /* success */ - } - sa += len; - offset -= len; + mutex_lock(&sfp->f_mutex); + if (req_sz > rsv_schp->bufflen) { + err = -ENOMEM; /* cannot map more than reserved buffer */ + goto out; } - - return VM_FAULT_SIGBUS; + sfp->mmap_called = 1; /* Prevents changes to buffer size. */ +out: + mutex_unlock(&sfp->f_mutex); + return err; } -static const struct vm_operations_struct sg_mmap_vm_ops = { - .fault = sg_vma_fault, +static int +sg_discontig_get(struct discontig_kernel_page_state *state) +{ + Sg_fd *sfp = state->vm_private_data; + Sg_scatter_hold *rsv_schp = &sfp->reserve; + const unsigned int order = rsv_schp->page_order; + const pgoff_t nr_pages = state->nr_pages_mapped; + + if (nr_pages >= (rsv_schp->bufflen >> PAGE_SHIFT)) { + discontig_kernel_map_abort(state); + return 0; + } + + SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sfp->parentdp, + "%s: offset=%lu, scatg=%d\n", __func__, + nr_pages << PAGE_SHIFT, rsv_schp->k_use_sg)); + + discontig_kernel_map_page(state, rsv_schp->pages[nr_pages >> order]); + return 0; +} + +static const struct discontig_kernel_page_ops sg_discontig_ops = { + .init = sg_discontig_init, + .get = sg_discontig_get, }; static int -sg_mmap(struct file *filp, struct vm_area_struct *vma) +sg_mmap_prepare(struct vm_area_desc *desc) { - Sg_fd *sfp; - unsigned long req_sz, len, sa; - Sg_scatter_hold *rsv_schp; - int k, length; - int ret = 0; + Sg_fd *sfp = desc->file->private_data; + const unsigned long req_sz = vma_desc_size(desc); - if ((!filp) || (!vma) || (!(sfp = (Sg_fd *) filp->private_data))) + if (!sfp) return -ENXIO; - req_sz = vma->vm_end - vma->vm_start; + SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sfp->parentdp, "sg_mmap starting, vm_start=%p, len=%d\n", - (void *) vma->vm_start, (int) req_sz)); - if (vma->vm_pgoff) + (void *) desc->start, (int) req_sz)); + + if (desc->pgoff) return -EINVAL; /* want no offset */ - rsv_schp = &sfp->reserve; - mutex_lock(&sfp->f_mutex); - if (req_sz > rsv_schp->bufflen) { - ret = -ENOMEM; /* cannot map more than reserved buffer */ - goto out; - } - sa = vma->vm_start; - length = 1 << (PAGE_SHIFT + rsv_schp->page_order); - for (k = 0; k < rsv_schp->k_use_sg && sa < vma->vm_end; k++) { - len = vma->vm_end - sa; - len = (len < length) ? len : length; - sa += len; - } + vma_desc_set_flags(desc, VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT); + desc->private_data = sfp; - sfp->mmap_called = 1; - vm_flags_set(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); - vma->vm_private_data = sfp; - vma->vm_ops = &sg_mmap_vm_ops; -out: - mutex_unlock(&sfp->f_mutex); - return ret; + mmap_action_map_discontig_kernel_pages(desc, (void *)req_sz, + &sg_discontig_ops); + return 0; } static void @@ -1415,7 +1402,7 @@ static const struct file_operations sg_fops = { .unlocked_ioctl = sg_ioctl, .compat_ioctl = compat_ptr_ioctl, .open = sg_open, - .mmap = sg_mmap, + .mmap_prepare = sg_mmap_prepare, .release = sg_release, .fasync = sg_fasync, }; From 19458d015fccbbf2657a8e6417bbaa915c2afdd1 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:04:59 +0100 Subject: [PATCH 109/213] fbdev: defio: assert FBINFO_VIRTFB, drop VM_IO, add VM_MIXEDMAP Currently all drivers which use defio allocate system memory. All of them also set FBINFO_VIRTFB, other than ssd1307fb, however this driver allocates system RAM, so simply failed to set this flag when it ought to. This patch sets FBINFO_VIRTFB on ssd1307fb probe, then drops setting VM_IO in fb_deferred_io_mmap() and instead requires FBINFO_VIRTFB to be set, erroring out with a kernel warning if not. The logic requires a page from the driver and since commit 1ecbc7dd2902 ("fbdev/deferred-io: Always call get_page() for framebuffer pages") has always required it to be refcounted, so this was implicitly already the case. Finally this patch sets VM_MIXEDMAP, as the logic is mapping kernel-allocated memory so this is appropriate. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-19-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Thomas Zimmermann Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- drivers/video/fbdev/core/fb_defio.c | 6 +++--- drivers/video/fbdev/ssd1307fb.c | 2 ++ 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/video/fbdev/core/fb_defio.c b/drivers/video/fbdev/core/fb_defio.c index fd00b86e1ae6..fb359ecc3966 100644 --- a/drivers/video/fbdev/core/fb_defio.c +++ b/drivers/video/fbdev/core/fb_defio.c @@ -366,13 +366,13 @@ int fb_deferred_io_mmap(struct fb_info *info, struct vm_area_struct *vma) { vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); + if (WARN_ON_ONCE(!(info->flags & FBINFO_VIRTFB))) + return -EINVAL; if (!try_module_get(THIS_MODULE)) return -EINVAL; vma->vm_ops = &fb_deferred_io_vm_ops; - vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP); - if (!(info->flags & FBINFO_VIRTFB)) - vm_flags_set(vma, VM_IO); + vm_flags_set(vma, VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = info->fbdefio_state; fb_deferred_io_state_get(info->fbdefio_state); /* released in vma->vm_ops->close() */ diff --git a/drivers/video/fbdev/ssd1307fb.c b/drivers/video/fbdev/ssd1307fb.c index 4d185c754284..db61d710fccb 100644 --- a/drivers/video/fbdev/ssd1307fb.c +++ b/drivers/video/fbdev/ssd1307fb.c @@ -767,6 +767,8 @@ static int ssd1307fb_probe(struct i2c_client *client) info->fix.smem_start = __pa(vmem); info->fix.smem_len = vmem_size; + info->flags = FBINFO_VIRTFB; + fb_deferred_io_init(info); i2c_set_clientdata(client, info); From 5605c40895fc5e76132f0538e1ee374d0f1a1733 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:00 +0100 Subject: [PATCH 110/213] HSI: cmt_speech: convert mmap hook to mmap_prepare, refactor Use the mmap_prepare in favour of the deprecated mmap hook as part of the work to convert one to another. Since this is simply a refcounted kernel page that has been allocated, it should not be marked VM_IO and should be inserted using the kernel page insertion mechanism, so convert it to do this instead. Use the VMA descriptor's private data field as a scratch buffer to store the page in - this stays valid throughout the kernel page mapping operation. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-20-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: David Hildenbrand Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- drivers/hsi/clients/cmt_speech.c | 35 +++++++++----------------------- 1 file changed, 10 insertions(+), 25 deletions(-) diff --git a/drivers/hsi/clients/cmt_speech.c b/drivers/hsi/clients/cmt_speech.c index 7226677ebde7..801697b74d4f 100644 --- a/drivers/hsi/clients/cmt_speech.c +++ b/drivers/hsi/clients/cmt_speech.c @@ -1084,22 +1084,6 @@ static void cs_hsi_stop(struct cs_hsi_iface *hi) kfree(hi); } -static vm_fault_t cs_char_vma_fault(struct vm_fault *vmf) -{ - struct cs_char *csdata = vmf->vma->vm_private_data; - struct page *page; - - page = virt_to_page((void *)csdata->mmap_base); - get_page(page); - vmf->page = page; - - return 0; -} - -static const struct vm_operations_struct cs_char_vm_ops = { - .fault = cs_char_vma_fault, -}; - static int cs_char_fasync(int fd, struct file *file, int on) { struct cs_char *csdata = file->private_data; @@ -1256,18 +1240,19 @@ static long cs_char_ioctl(struct file *file, unsigned int cmd, return r; } -static int cs_char_mmap(struct file *file, struct vm_area_struct *vma) +static int cs_char_mmap_prepare(struct vm_area_desc *desc) { - if (vma->vm_end < vma->vm_start) + struct file *file = desc->file; + struct cs_char *csdata = file->private_data; + struct page **pages = (struct page **)&desc->private_data; + + if (vma_desc_pages(desc) != 1) return -EINVAL; - if (vma_pages(vma) != 1) - return -EINVAL; - - vm_flags_set(vma, VM_IO | VM_DONTDUMP | VM_DONTEXPAND); - vma->vm_ops = &cs_char_vm_ops; - vma->vm_private_data = file->private_data; + vma_desc_set_flags(desc, VMA_DONTDUMP_BIT, VMA_DONTEXPAND_BIT); + *pages = virt_to_page((void *)csdata->mmap_base); + mmap_action_map_kernel_pages_full(desc, pages); return 0; } @@ -1353,7 +1338,7 @@ static const struct file_operations cs_char_fops = { .write = cs_char_write, .poll = cs_char_poll, .unlocked_ioctl = cs_char_ioctl, - .mmap = cs_char_mmap, + .mmap_prepare = cs_char_mmap_prepare, .open = cs_char_open, .release = cs_char_release, .fasync = cs_char_fasync, From 2d2b1c508f21496e4d7a0aa30eec0b863522479b Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:01 +0100 Subject: [PATCH 111/213] mm/gup: error out early on !VMA_MAYREAD_BIT VMAs When populating a VMA range via the aptly named populate_vma_page_range() an unreadable VMA will always eventually fail with -EFAULT. That a VMA is accessible is always checked, however VMA_MAYREAD_BIT is not. All user mappings always have VMA_MAYREAD_BIT set, so this check only impacts kernel mappings. It is implemented specifically to disallow population of uprobes XOL mappings which are exec-only. A nasty interaction with these mappings may occur if they are mlocked, so actively disallow this early. This allows a subsequent commit to remove the VM_IO check in __mm_populate() which otherwise requires non-MMIO mappings to be wrongly flagged simply as a workaround. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-21-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/gup.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index c2dfcb4744bc..e6310a7cc05b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1836,6 +1836,10 @@ long populate_vma_page_range(struct vm_area_struct *vma, if (!vma_is_accessible(vma)) return -EFAULT; + /* Unreadable VMAs also cannot be faulted in. */ + if (!vma_test(vma, VMA_MAYREAD_BIT)) + return -EFAULT; + gup_flags = FOLL_TOUCH; /* * We want to touch writable mappings with a write fault in order From 0d219bc3cb88b35e83b3b25d46374e8a5081a26e Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:02 +0100 Subject: [PATCH 112/213] uprobes: remove VM_IO, set VM_MIXEDMAP for mapped kernel pages These are not MMIO pages so VMA_IO_BIT is an inappropriate flag to set. Instead, set them VMA_MIXEDMAP_BIT as they are kernel mappings and this is the appropriate flag to set for those. This provides the semantics required - no VMA merging is permitted, but does not prevent GUP. However this has no meaningful impact as these are refcounted and thus can be GUPed. A previous commit already prevented __mm_populate() from being invoked on XOL areas, which VMA_IO_BIT was previously relied upon to do, so that is no longer required. Both VMAs set a VMA name, so always_dump_vma() returns true before vma_dump_size() reaches its VMA_IO_BIT check, and thus there is no change in core dump behaviour. Change this for both the core xol_add_vma() function and the x86-specific get_uprobe_trampoline() function. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-22-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- arch/x86/kernel/uprobes.c | 2 +- kernel/events/uprobes.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index 65a2de82ecd2..0f60c0d076b6 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -715,7 +715,7 @@ static struct vm_area_struct *get_uprobe_trampoline(struct mm_struct *mm, unsign *new_mapping = true; return _install_special_mapping(mm, vaddr, PAGE_SIZE, - VM_READ|VM_EXEC|VM_MAYEXEC|VM_MAYREAD|VM_IO, + VM_READ|VM_EXEC|VM_MAYEXEC|VM_MAYREAD|VM_MIXEDMAP, &tramp_mapping); } diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 7709ea882477..b89cc5cee002 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1726,8 +1726,8 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area) } vma = _install_special_mapping(mm, area->vaddr, PAGE_SIZE, - VM_EXEC|VM_MAYEXEC|VM_DONTCOPY|VM_IO| - VM_SEALED_SYSMAP, + VM_EXEC|VM_MAYEXEC|VM_DONTCOPY| + VM_MIXEDMAP|VM_SEALED_SYSMAP, &xol_mapping); if (IS_ERR(vma)) { ret = PTR_ERR(vma); From 61e012ab9a1cdec88ebcb55a061a09c4c6539058 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:03 +0100 Subject: [PATCH 113/213] mm/mlock: clear VMA_LOCKED_MASK over mmap callback Currently there's a confusing mess around VMA_LOCKED_BIT and VMA_LOCKONFAULT_BIT. It is permitted for drivers to set any flags they like, with the VMA already possessing lock flags. This results in the absurd situation of a VMA possessing both VMA_SPECIAL_FLAGS and VMA_LOCKED_MASK flags, which is not permitted. This has resulted in mlock_vma_folio() having a very silly check for this scenario to work around it. There is no need for this - just clear the flags before invoking the hook and reinstate them afterwards if they are required. Nothing relies upon this being set during the mmap operation. mmap_prepare is unaffected by this so requires no fix. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-23-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/internal.h | 9 +-------- mm/vma.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 8cef7593003c..93d8fa5ccabc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -970,14 +970,7 @@ void mlock_folio(struct folio *folio); static inline void mlock_vma_folio(struct folio *folio, struct vm_area_struct *vma) { - /* - * The VM_SPECIAL check here serves two purposes. - * 1) VM_IO check prevents migration from double-counting during mlock. - * 2) Although mmap_region() and mlock_fixup() take care that VM_LOCKED - * is never left set on a VM_SPECIAL vma, there is an interval while - * file->f_op->mmap() is using vm_insert_page(s), when VM_LOCKED may - * still be set while VM_SPECIAL bits are added: so ignore it then. - */ + /* The VM_IO check prevents migration from double-counting during mlock. */ if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED)) mlock_folio(folio); } diff --git a/mm/vma.c b/mm/vma.c index f334c87582d7..e422e06c7dfa 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -2625,6 +2625,11 @@ static int __mmap_new_file_vma(struct mmap_state *map, if (!map->vm_file->f_op->mmap) return 0; + /* + * Driver-specified flags may make the lock flags invalid, so clear + * VMA_LOCKED_MASK and reinstate it afterwards if appropriate. + */ + vma_clear_flags_mask(vma, VMA_LOCKED_MASK); error = mmap_file(vma->vm_file, vma); map->vm_file = vma->vm_file; @@ -2647,6 +2652,16 @@ static int __mmap_new_file_vma(struct mmap_state *map, return error; } + /* + * If the VMA is still eligible for mlock(), reinstate any original + * VMA_LOCKED_BIT and/or VMA_LOCKONFAULT_BIT flags. + */ + if (vma_supports_mlock(vma)) { + const vma_flags_t mask = vma_flags_and_mask(&map->vma_flags, VMA_LOCKED_MASK); + + vma_set_flags_mask(vma, mask); + } + map->vma_flags = vma->flags; return 0; From aea83f2694ed1f908401ffd2731deb14c2000ff0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:04 +0100 Subject: [PATCH 114/213] mm/mlock: eliminate weird VMA_IO_BIT abuse and simplify When performing mlock() or munlock() otherwise normal VMAs have VMA_IO_BIT solely to fix a race with migration which might otherwise double-count mlock VMAs. This is unnecessary - at the point of applying folio mlock state, whether setting or clearing PG_mlocked, we know whether or not we are locking. Solve this in two ways - thread a boolean through the page table walk indicating whether a lock or unlock is being performed, and run a locking walk with VMA_LOCKONFAULT_BIT set and VMA_LOCKED_BIT cleared. This state never occurs otherwise, as VMA_LOCKONFAULT_BIT always implies VMA_LOCKED_BIT. These are also always cleared together. Then, update folio_add_lru_vma() and mlock_folio() to check only for VMA_LOCKED_BIT, and update try_to_unmap_one() to check for VMA_LOCKED_MASK instead. Also remove the useless invocation of allow_mlock_munlock() which simply returns true if unlocking and instead rename it to allow_mlock() and only call it when locking. With the other mlock abuse of VMA_IO_BIT addressed, update mlock_vma_folio() and folio_add_lru_vma() to simply test for VMA_LOCKED_BIT. The munlock_vma_folio() function tests VMA_LOCKED_MASK instead, as an unmap racing with the locking walk must still munlock folios the walk has already counted. Additionally, in order to ensure a racing unmap/migration releases the reference munlock_folio() took (whose put was batched), check VMA_LOCKED_MASK before calling mlock_drain_local() in try_to_unmap_one() and try_to_migrate_one(). While here, also replace some deprecated VMA flag predicates. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-24-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/folio.c | 2 +- mm/internal.h | 10 ++++++--- mm/mlock.c | 61 ++++++++++++++++++++------------------------------- mm/rmap.c | 8 ++++--- 4 files changed, 37 insertions(+), 44 deletions(-) diff --git a/mm/folio.c b/mm/folio.c index 47a437e0f7fd..35e242b48870 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -505,7 +505,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) { VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED)) + if (vma_test(vma, VMA_LOCKED_BIT)) mlock_new_folio(folio); else folio_add_lru(folio); diff --git a/mm/internal.h b/mm/internal.h index 93d8fa5ccabc..56f2edbd9921 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -970,8 +970,7 @@ void mlock_folio(struct folio *folio); static inline void mlock_vma_folio(struct folio *folio, struct vm_area_struct *vma) { - /* The VM_IO check prevents migration from double-counting during mlock. */ - if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED)) + if (vma_test(vma, VMA_LOCKED_BIT)) mlock_folio(folio); } @@ -988,7 +987,12 @@ static inline void munlock_vma_folio(struct folio *folio, * always munlock the folio and page reclaim will correct it * if it's wrong. */ - if (unlikely(vma->vm_flags & VM_LOCKED)) + /* + * VMA_LOCKONFAULT_BIT alone marks an mlock walk in progress, see + * mlock_vma_pages_range(). An unmap racing with the walk must still + * munlock folios the walk has already counted. + */ + if (unlikely(vma_test_any_mask(vma, VMA_LOCKED_MASK))) munlock_folio(folio); } diff --git a/mm/mlock.c b/mm/mlock.c index 39215a3eab1f..4235a1518fc9 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -316,22 +316,10 @@ static inline unsigned int folio_mlock_step(struct folio *folio, return folio_pte_batch(folio, pte, ptent, count); } -static inline bool allow_mlock_munlock(struct folio *folio, +static inline bool allow_mlock(struct folio *folio, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned int step) { - /* - * For unlock, allow munlock large folio which is partially - * mapped to VMA. As it's possible that large folio is - * mlocked and VMA is split later. - * - * During memory pressure, such kind of large folio can - * be split. And the pages are not in VM_LOCKed VMA - * can be reclaimed. - */ - if (!vma_test(vma, VMA_LOCKED_BIT)) - return true; - /* folio_within_range() cannot take KSM, but any small folio is OK */ if (!folio_test_large(folio)) return true; @@ -352,6 +340,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, { struct vm_area_struct *vma = walk->vma; + const bool lock = walk->private; spinlock_t *ptl; pte_t *start_pte, *pte; pte_t ptent; @@ -368,7 +357,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, folio = pmd_folio(*pmd); if (folio_is_zone_device(folio)) goto out; - if (vma_test(vma, VMA_LOCKED_BIT)) + if (lock) mlock_folio(folio); else munlock_folio(folio); @@ -390,10 +379,10 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, continue; step = folio_mlock_step(folio, pte, addr, end); - if (!allow_mlock_munlock(folio, vma, start, end, step)) + if (lock && !allow_mlock(folio, vma, start, end, step)) goto next_entry; - if (vma_test(vma, VMA_LOCKED_BIT)) + if (lock) mlock_folio(folio); else munlock_folio(folio); @@ -428,31 +417,29 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, .pmd_entry = mlock_pte_range, .walk_lock = PGWALK_WRLOCK_VERIFY, }; + const bool lock = vma_flags_test(new_vma_flags, VMA_LOCKED_BIT); + vma_flags_t walk_flags = *new_vma_flags; /* - * There is a slight chance that concurrent page migration, - * or page reclaim finding a page of this now-VMA_LOCKED_BIT vma, - * will call mlock_vma_folio() and raise page's mlock_count: - * double counting, leaving the page unevictable indefinitely. - * Communicate this danger to mlock_vma_folio() with VMA_IO_BIT, - * which is a VMA_SPECIAL_FLAGS flag not allowed on VMA_LOCKED_BIT vmas. - * mmap_lock is held in write mode here, so this weird - * combination should not be visible to other mmap_lock users; - * but WRITE_ONCE so rmap walkers must see VMA_IO_BIT if VMA_LOCKED_BIT. + * LOCKONFAULT without LOCKED never otherwise occurs: it marks a walk in + * progress so that rmap-side callers, which test VMA_LOCKED_BIT, do not + * count folios, while try_to_unmap_one(), which tests VMA_LOCKED_MASK, + * still refuses to unmap them. */ - if (vma_flags_test(new_vma_flags, VMA_LOCKED_BIT)) - vma_flags_set(new_vma_flags, VMA_IO_BIT); - vma_start_write(vma); - vma_flags_reset_once(vma, new_vma_flags); - - lru_add_drain(); - walk_page_range_vma(vma, start, end, &mlock_walk_ops, NULL); - lru_add_drain(); - - if (vma_flags_test(new_vma_flags, VMA_IO_BIT)) { - vma_flags_clear(new_vma_flags, VMA_IO_BIT); - vma_flags_reset_once(vma, new_vma_flags); + if (lock) { + vma_flags_clear(&walk_flags, VMA_LOCKED_BIT); + vma_flags_set(&walk_flags, VMA_LOCKONFAULT_BIT); } + + vma_start_write(vma); + vma_flags_reset_once(vma, &walk_flags); + + lru_add_drain(); + walk_page_range_vma(vma, start, end, &mlock_walk_ops, (void *)lock); + lru_add_drain(); + + if (lock) + vma_flags_reset_once(vma, new_vma_flags); } /* diff --git a/mm/rmap.c b/mm/rmap.c index 5332c52909be..b4a4b3929044 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2239,9 +2239,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* * If the folio is in an mlock()d vma, we must not swap it out. + * VMA_LOCKONFAULT_BIT alone marks an mlock walk in progress, see + * mlock_vma_pages_range(). */ if (!(flags & TTU_IGNORE_MLOCK) && - (vma->vm_flags & VM_LOCKED)) { + vma_test_any_mask(vma, VMA_LOCKED_MASK)) { ptes++; /* @@ -2395,7 +2397,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } finish_unmap: folio_remove_rmap_ptes(folio, page, nr_pages, vma); - if (vma->vm_flags & VM_LOCKED) + if (vma_test_any_mask(vma, VMA_LOCKED_MASK)) mlock_drain_local(); folio_put_refs(folio, nr_pages); @@ -2770,7 +2772,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, hugetlb_remove_rmap(folio); else folio_remove_rmap_pte(folio, subpage, vma); - if (vma->vm_flags & VM_LOCKED) + if (vma_test_any_mask(vma, VMA_LOCKED_MASK)) mlock_drain_local(); folio_put(folio); } From 99fd6bf43d5d67ffb619c1fc61882c369d89599c Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:05 +0100 Subject: [PATCH 115/213] mm/vma: enforce that mm-managed mappings may not set VMA_IO_BIT It makes no sense for a mapping whose contents the kernel does not own to specify that the range is MMIO. Prior to this patch, all in-tree drivers which did so have been updated such that they are marked as not being mm-managed. The check WARNs and fails the mmap for any out-of-tree driver that still sets VMA_IO_BIT without a kernel mapping. Here VMA_PFNMAP_BIT, VMA_MIXEDMAP are tested individually rather than using vma_flags_is_mm_managed(). This is because vma_flags_is_mm_managed() still checks for VMA_IO_BIT. The very next commit in this series removes this, and the check is then converted to use vma_flags_is_mm_managed(). No functional change intended for in-tree code. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-25-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/vma.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/vma.c b/mm/vma.c index e422e06c7dfa..e46f02fb0503 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -2812,6 +2812,12 @@ static int mmap_validate_vma_flags(const vma_flags_t *flags) return -EINVAL; #endif + if (!vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT)) { + /* mm-managed mappings may not set VMA_IO_BIT. */ + if (WARN_ON_ONCE(vma_flags_test(flags, VMA_IO_BIT))) + return -EINVAL; + } + return 0; } From 88d77002e1a133c5d0071b2e7c738c5ab8aa1115 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:06 +0100 Subject: [PATCH 116/213] mm: remove VMA_IO_BIT check in vma[_flags]_is_mm_managed() We have now made it such that every driver which sets VMA_IO_BIT marks it as not mm-managed. However, vma_flags_is_mm_managed() still checks for VMA_IO_BIT and, if set, treats the mapping as not mm-managed. This was previously required to handle drivers which set VMA_IO_BIT but none of the other flags which identify a mapping as not mm-managed. We have now established that a driver which sets VMA_IO_BIT must also set one of those flags, so the VMA_IO_BIT check is redundant. So remove the VMA_IO_BIT check in vma_flags_is_mm_managed() and update mmap_validate_vma_flags() to use vma_flags_is_mm_managed() rather than open-coding the VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT check. This change means that vma[_flags]_can_merge() doesn't check VMA_IO_BIT any longer (which is now redundant) as it calls vma_flags_is_mm_managed(). Now that the predicate means precisely VMA_PFNMAP_BIT or VMA_MIXEDMAP_BIT, also use it at the other sites which open-code that pair, so the intent is stated rather than the flags, with no functional change: zap_special_vma_range() only zaps mappings which are not mm-managed, as drivers use it to tear down ranges they established themselves. The mprotect() arch PFN modification check applies to mappings not managed by core mm, which may map PFNs without struct pages. NUMA balancing skips VM_MIXEDMAP mappings having already excluded VM_IO and VM_PFNMAP mappings via vma_migratable(), so it skips exactly the mappings not managed by core mm - say so. Finally, update the VMA userland merge 'special' flag tests to no longer assert that VMA_IO_BIT prevents merge as VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT now suffices. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-26-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 3 +-- kernel/sched/fair.c | 2 +- mm/memory.c | 7 ++++--- mm/mprotect.c | 3 +-- mm/vma.c | 2 +- tools/testing/vma/include/dup.h | 3 +-- tools/testing/vma/tests/merge.c | 10 ++-------- 7 files changed, 11 insertions(+), 19 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ef81e1fa7b03..d3ca02e9aa28 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1628,8 +1628,7 @@ static inline bool vma_is_shared_maywrite(const struct vm_area_struct *vma) */ static inline bool vma_flags_is_mm_managed(const vma_flags_t *flags) { - return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT, - VMA_IO_BIT); + return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT); } /** diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 57360f5cdde4..1e6b1761fc8f 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4417,7 +4417,7 @@ static void task_numa_work(struct callback_head *work) for (; vma; vma = vma_next(&vmi)) { if (!vma_migratable(vma) || !vma_policy_mof(vma) || - is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + is_vm_hugetlb_page(vma) || !vma_is_mm_managed(vma)) { trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_UNSUITABLE); continue; } diff --git a/mm/memory.c b/mm/memory.c index 1b91e2361f9d..a8eaaa661206 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2343,19 +2343,20 @@ void zap_vma_range(struct vm_area_struct *vma, unsigned long address, } /** - * zap_special_vma_range - zap all page table entries in a special vma range + * zap_special_vma_range - zap all page table entries in a VMA which is not + * mm-managed * @vma: the vma covering the range to zap * @address: starting address of the range to zap * @size: number of bytes to zap * * This function does nothing when the provided address range is not fully - * contained in @vma, or when the @vma is not VM_PFNMAP or VM_MIXEDMAP. + * contained in @vma, or when @vma is mm-managed. */ void zap_special_vma_range(struct vm_area_struct *vma, unsigned long address, unsigned long size) { if (!range_in_vma(vma, address, address + size) || - !(vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))) + vma_is_mm_managed(vma)) return; zap_vma_range(vma, address, size); diff --git a/mm/mprotect.c b/mm/mprotect.c index 2888ee638d87..706c679d4281 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -783,8 +783,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, * uncommon case, so doesn't need to be very optimized. */ if (arch_has_pfn_modify_check() && - vma_flags_test_any(&old_vma_flags, VMA_PFNMAP_BIT, - VMA_MIXEDMAP_BIT) && + !vma_flags_is_mm_managed(&old_vma_flags) && !vma_flags_test_any_mask(&new_vma_flags, VMA_ACCESS_FLAGS)) { pgprot_t new_pgprot = vm_get_page_prot(newflags); diff --git a/mm/vma.c b/mm/vma.c index e46f02fb0503..6ac9424a7fe5 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -2812,7 +2812,7 @@ static int mmap_validate_vma_flags(const vma_flags_t *flags) return -EINVAL; #endif - if (!vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT)) { + if (vma_flags_is_mm_managed(flags)) { /* mm-managed mappings may not set VMA_IO_BIT. */ if (WARN_ON_ONCE(vma_flags_test(flags, VMA_IO_BIT))) return -EINVAL; diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 442ea9eb098d..94bf818ad0e1 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1667,8 +1667,7 @@ static inline bool file_is_dev_zero(const struct file *file) static inline bool vma_flags_is_mm_managed(const vma_flags_t *flags) { - return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT, - VMA_IO_BIT); + return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT); } static inline bool vma_is_mm_managed(const struct vm_area_struct *vma) diff --git a/tools/testing/vma/tests/merge.c b/tools/testing/vma/tests/merge.c index acaab282939c..b26f1a66a170 100644 --- a/tools/testing/vma/tests/merge.c +++ b/tools/testing/vma/tests/merge.c @@ -496,17 +496,11 @@ static bool test_vma_merge_special_flags(void) .mm = &mm, .vmi = &vmi, }; - vma_flag_t special_flags[] = { VMA_IO_BIT, VMA_DONTEXPAND_BIT, + vma_flag_t special_flags[] = { VMA_DONTEXPAND_BIT, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT }; - vma_flags_t all_special_flags = EMPTY_VMA_FLAGS; int i; struct vm_area_struct *vma_left, *vma; - /* Make sure there aren't new VM_SPECIAL flags. */ - for (i = 0; i < ARRAY_SIZE(special_flags); i++) - vma_flags_set(&all_special_flags, special_flags[i]); - ASSERT_FLAGS_SAME_MASK(&all_special_flags, VMA_SPECIAL_FLAGS); - /* * 01234 * AAA @@ -520,7 +514,7 @@ static bool test_vma_merge_special_flags(void) * 01234 * AAA* * - * This should merge if not for the VM_SPECIAL flag. + * This should merge if not for the 'special' flag. */ vmg_set_range(&vmg, 0x3000, 0x4000, 3, vma_flags); for (i = 0; i < ARRAY_SIZE(special_flags); i++) { From 94c86564538ea6a21485a3b06b4d33c23e6f8053 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:07 +0100 Subject: [PATCH 117/213] mm: remove hugetlb_inline.h This header really makes little sense - every place it is included mm.h is also included, and the header itself includes mm.h, so it does nothing to reduce header size. It also oddly does an #ifdef around checking VMA_HUGETLB_BIT, however VMA_HUGETLB_BIT is unconditionally available, and will never be set if hugetlb is not enabled. Simply remove the header, eliminate the odd ifdeffery and place the predicates in mm.h. The naming of these predicates is odd, but to keep changes separate, we will address this in a separate patch. The file was never put into MAINTAINERS so there's no change required there. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-27-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- drivers/gpu/drm/drm_gpusvm.c | 2 +- include/asm-generic/tlb.h | 2 +- include/linux/hugetlb.h | 1 - include/linux/hugetlb_inline.h | 28 ---------------------------- include/linux/mm.h | 11 +++++++++++ include/linux/pagemap.h | 1 - include/linux/userfaultfd_k.h | 1 - kernel/sched/fair.c | 1 - mm/vma_internal.h | 1 - 9 files changed, 13 insertions(+), 35 deletions(-) delete mode 100644 include/linux/hugetlb_inline.h diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c index a93eee7ddb9e..793dacec2100 100644 --- a/drivers/gpu/drm/drm_gpusvm.c +++ b/drivers/gpu/drm/drm_gpusvm.c @@ -9,9 +9,9 @@ #include #include #include -#include #include #include +#include #include #include diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 9d827076db19..bb7b05e10099 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -11,9 +11,9 @@ #ifndef _ASM_GENERIC__TLB_H #define _ASM_GENERIC__TLB_H +#include #include #include -#include #include #include diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 893ef4fe3ba5..3c16b1e823c2 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -7,7 +7,6 @@ #include #include #include -#include #include #include #include diff --git a/include/linux/hugetlb_inline.h b/include/linux/hugetlb_inline.h deleted file mode 100644 index 5c29cd3223a1..000000000000 --- a/include/linux/hugetlb_inline.h +++ /dev/null @@ -1,28 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _LINUX_HUGETLB_INLINE_H -#define _LINUX_HUGETLB_INLINE_H - -#include - -#ifdef CONFIG_HUGETLB_PAGE - -static inline bool is_vma_hugetlb_flags(const vma_flags_t *flags) -{ - return vma_flags_test(flags, VMA_HUGETLB_BIT); -} - -#else - -static inline bool is_vma_hugetlb_flags(const vma_flags_t *flags) -{ - return false; -} - -#endif - -static inline bool is_vm_hugetlb_page(const struct vm_area_struct *vma) -{ - return is_vma_hugetlb_flags(&vma->flags); -} - -#endif diff --git a/include/linux/mm.h b/include/linux/mm.h index d3ca02e9aa28..503210d0c572 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1610,6 +1610,17 @@ static inline bool vma_is_shared_maywrite(const struct vm_area_struct *vma) return is_shared_maywrite(&vma->flags); } +static inline bool is_vma_hugetlb_flags(const vma_flags_t *flags) +{ + return IS_ENABLED(CONFIG_HUGETLB_PAGE) && + vma_flags_test(flags, VMA_HUGETLB_BIT); +} + +static inline bool is_vm_hugetlb_page(const struct vm_area_struct *vma) +{ + return is_vma_hugetlb_flags(&vma->flags); +} + /** * vma_flags_is_mm_managed() - Do the specified VMA flags indicate that the * contents of the VMA are managed by core mm? diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index bcbb0afe1a68..73af18a37367 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,7 +14,6 @@ #include #include #include /* for in_interrupt() */ -#include struct folio_batch; diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index a4351cffc60c..a14b8a9ffb7b 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -18,7 +18,6 @@ #include #include #include -#include /* The set of all possible UFFD-related VM flags. */ #define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_MINOR | \ diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1e6b1761fc8f..e511a476fe92 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -22,7 +22,6 @@ */ #include #include -#include #include #include #include diff --git a/mm/vma_internal.h b/mm/vma_internal.h index 4d300e7bbaf4..4f73f0a4db79 100644 --- a/mm/vma_internal.h +++ b/mm/vma_internal.h @@ -18,7 +18,6 @@ #include #include #include -#include #include #include #include From c911e7ebebf58c06d2ea4a2d3858411066739e11 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:08 +0100 Subject: [PATCH 118/213] mm: rename is_vm_hugetlb_page() to vma_is_hugetlb() The is_vm_hugetlb_page() predicate is badly named - the mapping can span more than a page and it is inconsistent with other VMA predicates that typically are prefixed by vma_. Rename to vma_is_hugetlb() for consistency, and while we're here update some VM_BUG_ON_VMA() to VM_WARN_ON_ONCE_VMA() as to avoid unnecessary oopses. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-28-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Marc Zyngier Acked-by: Claudio Imbrenda Acked-by: Anup Patel Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- arch/arm64/kvm/mmu.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 6 +++--- arch/powerpc/mm/nohash/e500_hugetlbpage.c | 2 +- arch/powerpc/mm/nohash/tlb.c | 2 +- arch/riscv/kvm/mmu.c | 2 +- arch/riscv/mm/tlbflush.c | 2 +- arch/s390/mm/gmap_helpers.c | 6 +++--- arch/sparc/mm/init_64.c | 2 +- drivers/gpu/drm/drm_gpusvm.c | 2 +- fs/coredump.c | 2 +- fs/hugetlbfs/inode.c | 2 +- fs/proc/task_mmu.c | 8 +++---- include/asm-generic/tlb.h | 2 +- include/linux/hugetlb.h | 4 ++-- include/linux/mm.h | 19 ++++++++++++++--- include/linux/rmap.h | 2 +- kernel/events/core.c | 2 +- kernel/sched/fair.c | 2 +- mm/gup.c | 4 ++-- mm/huge_memory.c | 2 +- mm/hugetlb.c | 14 ++++++------ mm/internal.h | 2 +- mm/madvise.c | 4 ++-- mm/memory.c | 12 +++++------ mm/mempolicy.c | 2 +- mm/migrate_device.c | 2 +- mm/mmap.c | 2 +- mm/mmu_gather.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 6 +++--- mm/page_vma_mapped.c | 4 ++-- mm/pagewalk.c | 2 +- mm/swapfile.c | 2 +- mm/userfaultfd.c | 26 +++++++++++------------ mm/vma.c | 8 +++---- mm/vmscan.c | 2 +- tools/testing/vma/include/stubs.h | 2 +- 37 files changed, 91 insertions(+), 78 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index bf6d6526639f..b0ea1a9391da 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1470,7 +1470,7 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot, static int get_vma_page_shift(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return huge_page_shift(hstate_vma(vma)); return PAGE_SHIFT; diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 7de5760164a9..b4603a98224b 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -627,7 +627,7 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd { #ifdef CONFIG_HUGETLB_PAGE /* need the return fix for nohash.c */ - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return radix__local_flush_hugetlb_page(vma, vmaddr); #endif radix__local_flush_tlb_page_psize(vma->vm_mm, vmaddr, mmu_virtual_psize); @@ -945,7 +945,7 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, void radix__flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr) { #ifdef CONFIG_HUGETLB_PAGE - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return radix__flush_hugetlb_page(vma, vmaddr); #endif radix__flush_tlb_page_psize(vma->vm_mm, vmaddr, mmu_virtual_psize); @@ -1113,7 +1113,7 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, { #ifdef CONFIG_HUGETLB_PAGE - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return radix__flush_hugetlb_tlb_range(vma, start, end); #endif diff --git a/arch/powerpc/mm/nohash/e500_hugetlbpage.c b/arch/powerpc/mm/nohash/e500_hugetlbpage.c index a134d28a0e4d..b87623f04be5 100644 --- a/arch/powerpc/mm/nohash/e500_hugetlbpage.c +++ b/arch/powerpc/mm/nohash/e500_hugetlbpage.c @@ -180,7 +180,7 @@ book3e_hugetlb_preload(struct vm_area_struct *vma, unsigned long ea, pte_t pte) */ void __update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) book3e_hugetlb_preload(vma, address, *ptep); } diff --git a/arch/powerpc/mm/nohash/tlb.c b/arch/powerpc/mm/nohash/tlb.c index 0a650742f3a0..07a2db16c2b1 100644 --- a/arch/powerpc/mm/nohash/tlb.c +++ b/arch/powerpc/mm/nohash/tlb.c @@ -278,7 +278,7 @@ void __flush_tlb_page(struct mm_struct *mm, unsigned long vmaddr, void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr) { #ifdef CONFIG_HUGETLB_PAGE - if (vma && is_vm_hugetlb_page(vma)) + if (vma && vma_is_hugetlb(vma)) flush_hugetlb_page(vma, vmaddr); #endif diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 3e955d808743..371fccaf9df0 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -665,7 +665,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, return -EFAULT; } - is_hugetlb = is_vm_hugetlb_page(vma); + is_hugetlb = vma_is_hugetlb(vma); if (is_hugetlb) vma_pageshift = huge_page_shift(hstate_vma(vma)); else diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 962db300a166..a74a7d5258aa 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -149,7 +149,7 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, { unsigned long stride_size; - if (!is_vm_hugetlb_page(vma)) { + if (!vma_is_hugetlb(vma)) { stride_size = PAGE_SIZE; } else { stride_size = huge_page_size(hstate_vma(vma)); diff --git a/arch/s390/mm/gmap_helpers.c b/arch/s390/mm/gmap_helpers.c index ff63ffb1dbd2..3f6783b93e67 100644 --- a/arch/s390/mm/gmap_helpers.c +++ b/arch/s390/mm/gmap_helpers.c @@ -102,7 +102,7 @@ __context_unsafe(/* pte_unmap_unlock() not instrumented */) /* Find the vm address for the guest address */ vma = vma_lookup(mm, vmaddr); - if (!vma || is_vm_hugetlb_page(vma)) + if (!vma || vma_is_hugetlb(vma)) return; /* Get pointer to the page table entry */ @@ -139,7 +139,7 @@ void gmap_helper_discard(struct mm_struct *mm, unsigned long vmaddr, unsigned lo vma = find_vma_intersection(mm, vmaddr, end); if (!vma) return; - if (!is_vm_hugetlb_page(vma)) + if (!vma_is_hugetlb(vma)) zap_vma_range(vma, vmaddr, min(end, vma->vm_end) - vmaddr); vmaddr = vma->vm_end; } @@ -247,7 +247,7 @@ static int __gmap_helper_unshare_zeropages(struct mm_struct *mm) * proof to catch unexpected zeropages in other mappings and * fail. */ - if ((vma->vm_flags & VM_PFNMAP) || is_vm_hugetlb_page(vma)) + if ((vma->vm_flags & VM_PFNMAP) || vma_is_hugetlb(vma)) continue; addr = vma->vm_start; diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 103db4683b16..9bbccb5d23a8 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -413,7 +413,7 @@ void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, if (mm->context.hugetlb_pte_count || mm->context.thp_pte_count) { unsigned long hugepage_size = PAGE_SIZE; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugepage_size = huge_page_size(hstate_vma(vma)); if (hugepage_size >= PUD_SIZE) { diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c index 793dacec2100..a1d4989b0b61 100644 --- a/drivers/gpu/drm/drm_gpusvm.c +++ b/drivers/gpu/drm/drm_gpusvm.c @@ -1142,7 +1142,7 @@ drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, * have to change. */ migrate_devmem = ctx->devmem_possible && - vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas); + vma_is_anonymous(vas) && !vma_is_hugetlb(vas); chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier, vas, fault_addr, gpuva_start, diff --git a/fs/coredump.c b/fs/coredump.c index 6114839f5178..5820cb8ec88e 100644 --- a/fs/coredump.c +++ b/fs/coredump.c @@ -1608,7 +1608,7 @@ static unsigned long vma_dump_size(struct vm_area_struct *vma, } /* Hugetlb memory check */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { if ((vma->vm_flags & VM_SHARED) && FILTER(HUGETLB_SHARED)) goto whole; if (!(vma->vm_flags & VM_SHARED) && FILTER(HUGETLB_PRIVATE)) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 78e27ce0a6f6..e7ddb76905df 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -108,7 +108,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) * vma address alignment (but not the pgoff alignment) has * already been checked by prepare_hugepage_range. If you add * any error returns here, do so after setting VM_HUGETLB, so - * is_vm_hugetlb_page tests below unmap_region go the right + * vma_is_hugetlb tests below unmap_region go the right * way when do_mmap unwinds (may be important on powerpc * and ia64). */ diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index ee9fc2fd2b2f..44147ba0b899 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2986,7 +2986,7 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, * hugetlb differs, see pagemap_hugetlb_category(). */ categories = p->cur_vma_category; - if (userfaultfd_wp(vma) && !is_vm_hugetlb_page(vma)) + if (userfaultfd_wp(vma) && !vma_is_hugetlb(vma)) categories |= PAGE_IS_WRITTEN; if (!pagemap_scan_is_interesting_page(categories, p)) @@ -2999,7 +2999,7 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, if (~p->arg.flags & PM_SCAN_WP_MATCHING) return ret; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) err = pagemap_scan_hugetlb_hole_wp(vma, addr, end); else err = uffd_wp_range(vma, addr, end - addr, true); @@ -3441,7 +3441,7 @@ static int show_numa_map(struct seq_file *m, void *v) seq_puts(m, " stack"); } - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) seq_puts(m, " huge"); /* Skip walking pages if gate VMA */ @@ -3470,7 +3470,7 @@ static int show_numa_map(struct seq_file *m, void *v) if (md->swapcache) seq_printf(m, " swapcache=%lu", md->swapcache); - if (md->active < md->pages && !is_vm_hugetlb_page(vma)) + if (md->active < md->pages && !vma_is_hugetlb(vma)) seq_printf(m, " active=%lu", md->active); if (md->writeback) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index bb7b05e10099..fbe1114e9a44 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -438,7 +438,7 @@ tlb_update_vma_flags(struct mmu_gather *tlb, struct vm_area_struct *vma) * We rely on tlb_end_vma() to issue a flush, such that when we reset * these values the batch is empty. */ - tlb->vma_huge = is_vm_hugetlb_page(vma); + tlb->vma_huge = vma_is_hugetlb(vma); tlb->vma_exec = !!(vma->vm_flags & VM_EXEC); /* diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3c16b1e823c2..5029c7241863 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -252,14 +252,14 @@ extern void __hugetlb_zap_end(struct vm_area_struct *vma, static inline void hugetlb_zap_begin(struct vm_area_struct *vma, unsigned long *start, unsigned long *end) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) __hugetlb_zap_begin(vma, start, end); } static inline void hugetlb_zap_end(struct vm_area_struct *vma, struct zap_details *details) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) __hugetlb_zap_end(vma, details); } diff --git a/include/linux/mm.h b/include/linux/mm.h index 503210d0c572..8c1e9ae01491 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1610,15 +1610,28 @@ static inline bool vma_is_shared_maywrite(const struct vm_area_struct *vma) return is_shared_maywrite(&vma->flags); } -static inline bool is_vma_hugetlb_flags(const vma_flags_t *flags) +/** + * vma_flags_is_hugetlb() - Do the specified VMA flags indicate that the + * VMA is a hugetlb mapping? + * @flags: The VMA flags to test. + * + * Returns: true if the flags indicate a hugetlb mapping, false otherwise. + */ +static inline bool vma_flags_is_hugetlb(const vma_flags_t *flags) { return IS_ENABLED(CONFIG_HUGETLB_PAGE) && vma_flags_test(flags, VMA_HUGETLB_BIT); } -static inline bool is_vm_hugetlb_page(const struct vm_area_struct *vma) +/** + * vma_is_hugetlb() - Is @vma a hugetlb mapping? + * @vma: The VMA to test. + * + * Returns: true if @vma is a hugetlb mapping, false otherwise. + */ +static inline bool vma_is_hugetlb(const struct vm_area_struct *vma) { - return is_vma_hugetlb_flags(&vma->flags); + return vma_flags_is_hugetlb(&vma->flags); } /** diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0b332770abee..74cca0e3c726 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -888,7 +888,7 @@ struct page_vma_mapped_walk { static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) { /* HugeTLB pte is set to the relevant page table entry without pte_mapped. */ - if (pvmw->pte && !is_vm_hugetlb_page(pvmw->vma)) + if (pvmw->pte && !vma_is_hugetlb(pvmw->vma)) pte_unmap(pvmw->pte); if (pvmw->ptl) spin_unlock(pvmw->ptl); diff --git a/kernel/events/core.c b/kernel/events/core.c index 634d2ccbab82..601e8d944c24 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -9818,7 +9818,7 @@ static void perf_event_mmap_event(struct perf_mmap_event *mmap_event) if (vma->vm_flags & VM_LOCKED) flags |= MAP_LOCKED; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) flags |= MAP_HUGETLB; if (file) { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e511a476fe92..000d4cb7aca9 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4416,7 +4416,7 @@ static void task_numa_work(struct callback_head *work) for (; vma; vma = vma_next(&vmi)) { if (!vma_migratable(vma) || !vma_policy_mof(vma) || - is_vm_hugetlb_page(vma) || !vma_is_mm_managed(vma)) { + vma_is_hugetlb(vma) || !vma_is_mm_managed(vma)) { trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_UNSUITABLE); continue; } diff --git a/mm/gup.c b/mm/gup.c index e6310a7cc05b..f166acf794e3 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -621,7 +621,7 @@ static struct page *no_page_table(struct vm_area_struct *vma, * But we can only make this optimization where a hole would surely * be zero-filled if handle_mm_fault() actually did handle it. */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { struct hstate *h = hstate_vma(vma); if (!hugetlbfs_pagecache_present(h, vma, address)) @@ -1213,7 +1213,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; - if ((gup_flags & FOLL_SPLIT_PMD) && is_vm_hugetlb_page(vma)) + if ((gup_flags & FOLL_SPLIT_PMD) && vma_is_hugetlb(vma)) return -EOPNOTSUPP; if (vma_is_secretmem(vma)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4cf7fc461c67..37735b4d2f27 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4783,7 +4783,7 @@ static inline bool vma_not_suitable_for_thp_split(struct vm_area_struct *vma) return true; if (vma_test(vma, VMA_IO_BIT)) return true; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return true; return false; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fd41ff7432a6..58a807aa65ab 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1148,7 +1148,7 @@ static inline struct resv_map *inode_resv_map(struct inode *inode) static struct resv_map *vma_resv_map(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); if (vma->vm_flags & VM_MAYSHARE) { struct address_space *mapping = vma->vm_file->f_mapping; struct inode *inode = mapping->host; @@ -1163,7 +1163,7 @@ static struct resv_map *vma_resv_map(struct vm_area_struct *vma) static void set_vma_resv_map(struct vm_area_struct *vma, struct resv_map *map) { - VM_WARN_ON_ONCE_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); VM_WARN_ON_ONCE_VMA(vma_test(vma, VMA_MAYSHARE_BIT), vma); set_vma_private_data(vma, (unsigned long)map); @@ -1171,7 +1171,7 @@ static void set_vma_resv_map(struct vm_area_struct *vma, struct resv_map *map) static void set_vma_resv_flags(struct vm_area_struct *vma, unsigned long flags) { - VM_WARN_ON_ONCE_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); VM_WARN_ON_ONCE_VMA(vma_test(vma, VMA_MAYSHARE_BIT), vma); set_vma_private_data(vma, get_vma_private_data(vma) | flags); @@ -1179,7 +1179,7 @@ static void set_vma_resv_flags(struct vm_area_struct *vma, unsigned long flags) static int is_vma_resv_set(struct vm_area_struct *vma, unsigned long flag) { - VM_BUG_ON_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); return (get_vma_private_data(vma) & flag) != 0; } @@ -1193,7 +1193,7 @@ bool __vma_private_lock(struct vm_area_struct *vma) void hugetlb_dup_vma_private(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); /* * Clear vm_private_data * - For shared mappings this is a per-vma semaphore that may be @@ -5277,7 +5277,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long last_addr_mask; i_mmap_assert_write_locked(vma->vm_file->f_mapping); - WARN_ON(!is_vm_hugetlb_page(vma)); + WARN_ON(!vma_is_hugetlb(vma)); BUG_ON(start & ~huge_page_mask(h)); BUG_ON(end & ~huge_page_mask(h)); @@ -7563,7 +7563,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) */ void fixup_hugetlb_reservations(struct vm_area_struct *vma) { - if (!is_vm_hugetlb_page(vma)) + if (!vma_is_hugetlb(vma)) return; /* diff --git a/mm/internal.h b/mm/internal.h index 56f2edbd9921..498dbb71fcf1 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1116,7 +1116,7 @@ static inline bool vma_supports_mlock(const struct vm_area_struct *vma) return false; if (vma_test_single_mask(vma, VMA_DROPPABLE)) return false; - if (vma_is_dax(vma) || is_vm_hugetlb_page(vma)) + if (vma_is_dax(vma) || vma_is_hugetlb(vma)) return false; return vma != get_gate_vma(current->mm); } diff --git a/mm/madvise.c b/mm/madvise.c index 7a7bfcfb8f8f..4d7f398d9738 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -882,7 +882,7 @@ bool madvise_dontneed_free_valid_vma(struct madvise_behavior *madv_behavior) int behavior = madv_behavior->behavior; struct madvise_behavior_range *range = &madv_behavior->range; - if (!is_vm_hugetlb_page(vma)) { + if (!vma_is_hugetlb(vma)) { unsigned int forbidden = VM_PFNMAP; if (behavior != MADV_DONTNEED_LOCKED) @@ -1415,7 +1415,7 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior) new_flags |= VM_DONTDUMP; break; case MADV_DODUMP: - if ((!is_vm_hugetlb_page(vma) && (new_flags & VM_SPECIAL)) || + if ((!vma_is_hugetlb(vma) && (new_flags & VM_SPECIAL)) || (new_flags & VM_DROPPABLE)) return -EINVAL; new_flags &= ~VM_DONTDUMP; diff --git a/mm/memory.c b/mm/memory.c index a8eaaa661206..a142c778a4d8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1564,7 +1564,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) if (!vma_needs_copy(dst_vma, src_vma)) return 0; - if (is_vm_hugetlb_page(src_vma)) + if (vma_is_hugetlb(src_vma)) return copy_hugetlb_page_range(dst_mm, src_mm, dst_vma, src_vma); /* @@ -2178,7 +2178,7 @@ static void __zap_vma_range(struct mmu_gather *tlb, struct vm_area_struct *vma, if (vma->vm_file && !reaping) uprobe_munmap(vma, start, end); - if (unlikely(is_vm_hugetlb_page(vma))) { + if (unlikely(vma_is_hugetlb(vma))) { zap_flags_t zap_flags = details ? details->zap_flags : 0; VM_WARN_ON_ONCE(reaping); @@ -2313,7 +2313,7 @@ void zap_vma_range_batched(struct mmu_gather *tlb, */ __zap_vma_range(tlb, vma, address, end, details); mmu_notifier_invalidate_range_end(&range); - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { /* * flush tlb and free resources before hugetlb_zap_end(), to * avoid concurrent page faults' allocation failure. @@ -6937,7 +6937,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, lru_gen_enter_fault(vma); - if (unlikely(is_vm_hugetlb_page(vma))) + if (unlikely(vma_is_hugetlb(vma))) ret = hugetlb_fault(vma->vm_mm, vma, address, flags); else ret = __handle_mm_fault(vma, address, flags); @@ -7821,12 +7821,12 @@ void ptlock_free(struct ptdesc *ptdesc) void vma_pgtable_walk_begin(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_vma_lock_read(vma); } void vma_pgtable_walk_end(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_vma_unlock_read(vma); } diff --git a/mm/mempolicy.c b/mm/mempolicy.c index e9860fb9f73f..70298fded1b4 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2018,7 +2018,7 @@ bool vma_migratable(struct vm_area_struct *vma) if (vma_is_dax(vma)) return false; - if (is_vm_hugetlb_page(vma) && + if (vma_is_hugetlb(vma) && !hugepage_migration_supported(hstate_vma(vma))) return false; diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 0c437004329d..c38cbaaef5a4 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -743,7 +743,7 @@ int migrate_vma_setup(struct migrate_vma *args) args->start &= PAGE_MASK; args->end &= PAGE_MASK; - if (!args->vma || is_vm_hugetlb_page(args->vma) || + if (!args->vma || vma_is_hugetlb(args->vma) || (args->vma->vm_flags & VM_SPECIAL) || vma_is_dax(args->vma)) return -EINVAL; if (nr_pages <= 0) diff --git a/mm/mmap.c b/mm/mmap.c index 4bf26b0f1e6e..98449f364af1 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1786,7 +1786,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) /* * Copy/update hugetlb private vma information. */ - if (is_vm_hugetlb_page(tmp)) + if (vma_is_hugetlb(tmp)) hugetlb_dup_vma_private(tmp); /* diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 2a72a9686773..9f353f0e2ef4 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -480,7 +480,7 @@ void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma) { tlb_gather_mmu(tlb, vma->vm_mm); tlb_update_vma_flags(tlb, vma); - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) /* All entries have the same size. */ tlb_change_page_size(tlb, huge_page_size(hstate_vma(vma))); } diff --git a/mm/mprotect.c b/mm/mprotect.c index 706c679d4281..bef7b742d101 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -717,7 +717,7 @@ long change_protection(struct mmu_gather *tlb, (cp_flags & MM_CP_UFFD_RWP)) newprot = PAGE_NONE; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) pages = hugetlb_change_protection(vma, start, end, newprot, cp_flags); else diff --git a/mm/mremap.c b/mm/mremap.c index 8983ccd44469..d38ff332a13a 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -812,7 +812,7 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) if (!pmc->len_in) return 0; - if (is_vm_hugetlb_page(pmc->old)) + if (vma_is_hugetlb(pmc->old)) return move_hugetlb_page_tables(pmc->old, pmc->new, pmc->old_addr, pmc->new_addr, pmc->len_in); @@ -1769,7 +1769,7 @@ static bool vma_multi_allowed(struct vm_area_struct *vma) /* Known good. */ if (vma_is_shmem(vma)) return true; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return true; if (file->f_op->get_unmapped_area == thp_get_unmapped_area) return true; @@ -1792,7 +1792,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm) return -EPERM; /* Align to hugetlb page size, if required. */ - if (is_vm_hugetlb_page(vma) && !align_hugetlb(vrm)) + if (vma_is_hugetlb(vma) && !align_hugetlb(vrm)) return -EINVAL; vrm_set_delta(vrm); diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 28e306fdb3a5..8408aee7571b 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -109,7 +109,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) unsigned long pfn; pte_t ptent; - if (is_vm_hugetlb_page(pvmw->vma)) + if (vma_is_hugetlb(pvmw->vma)) ptent = huge_ptep_get(pvmw->vma->vm_mm, pvmw->address, pvmw->pte); else @@ -206,7 +206,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pmd && !pvmw->pte) return not_found(pvmw); - if (unlikely(is_vm_hugetlb_page(vma))) { + if (unlikely(vma_is_hugetlb(vma))) { struct hstate *hstate = hstate_vma(vma); unsigned long size = huge_page_size(hstate); /* The only possible mapping was handled on last iteration */ diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 7411702a37f5..e6493bbe6919 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -408,7 +408,7 @@ static int __walk_page_range(unsigned long start, unsigned long end, int err = 0; struct vm_area_struct *vma = walk->vma; const struct mm_walk_ops *ops = walk->ops; - bool is_hugetlb = is_vm_hugetlb_page(vma); + bool is_hugetlb = vma_is_hugetlb(vma); /* We do not support hugetlb PTE installation. */ if (ops->install_pte && is_hugetlb) diff --git a/mm/swapfile.c b/mm/swapfile.c index 16ceaa28db3a..0901cb8fa729 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2707,7 +2707,7 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type) if (check_stable_address_space(mm)) goto unlock; for_each_vma(vmi, vma) { - if (vma->anon_vma && !is_vm_hugetlb_page(vma)) { + if (vma->anon_vma && !vma_is_hugetlb(vma)) { ret = unuse_vma(vma, type); if (ret) break; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 79cc7b546f13..949017e60608 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -237,7 +237,7 @@ static int mfill_get_vma(struct mfill_state *state) if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP)) goto out_unlock; - if (is_vm_hugetlb_page(dst_vma)) + if (vma_is_hugetlb(dst_vma)) return 0; ops = vma_uffd_ops(dst_vma); @@ -804,7 +804,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( } err = -ENOENT; - if (!is_vm_hugetlb_page(dst_vma)) + if (!vma_is_hugetlb(dst_vma)) goto out_unlock_vma; err = -EINVAL; @@ -967,7 +967,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, /* * If this is a HUGETLB vma, pass off to appropriate routine */ - if (is_vm_hugetlb_page(state.vma)) + if (vma_is_hugetlb(state.vma)) return mfill_atomic_hugetlb(ctx, state.vma, dst_start, src_start, len, flags); @@ -1114,7 +1114,7 @@ static int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start, break; } - if (is_vm_hugetlb_page(dst_vma)) { + if (vma_is_hugetlb(dst_vma)) { err = -EINVAL; page_mask = vma_kernel_pagesize(dst_vma) - 1; if ((start & page_mask) || (len & page_mask)) @@ -1172,7 +1172,7 @@ int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start, if (!userfaultfd_rwp(dst_vma)) return -ENOENT; - if (is_vm_hugetlb_page(dst_vma)) { + if (vma_is_hugetlb(dst_vma)) { unsigned long page_mask; page_mask = vma_kernel_pagesize(dst_vma) - 1; @@ -2149,7 +2149,7 @@ static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, if (vma->vm_flags & (VM_DROPPABLE | VM_SHADOW_STACK)) return false; - if (!is_vm_hugetlb_page(vma) && (vma->vm_flags & VM_SPECIAL)) + if (!vma_is_hugetlb(vma) && (vma->vm_flags & VM_SPECIAL)) return false; vm_flags &= __VM_UFFD_FLAGS; @@ -2319,7 +2319,7 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx, */ userfaultfd_set_ctx(vma, ctx, vm_flags); - if (is_vm_hugetlb_page(vma) && uffd_disable_huge_pmd_share(vma)) + if (vma_is_hugetlb(vma) && uffd_disable_huge_pmd_share(vma)) hugetlb_unshare_all_pmds(vma); skip: @@ -2895,7 +2895,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) * (sleepable) vma lock can modify the current task state, that * must be before explicitly calling set_current_state(). */ - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_vma_lock_read(vma); spin_lock_irq(&ctx->fault_pending_wqh.lock); @@ -2912,7 +2912,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) set_current_state(blocking_state); spin_unlock_irq(&ctx->fault_pending_wqh.lock); - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { must_wait = userfaultfd_huge_must_wait(ctx, vmf, reason); hugetlb_vma_unlock_read(vma); } else { @@ -3744,7 +3744,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * If the first vma contains huge pages, make sure start address * is aligned to huge page size. */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { unsigned long vma_hpagesize = vma_kernel_pagesize(vma); if (start & (vma_hpagesize - 1)) @@ -3795,7 +3795,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * If this vma contains ending address, and huge pages * check alignment. */ - if (is_vm_hugetlb_page(cur) && end <= cur->vm_end && + if (vma_is_hugetlb(cur) && end <= cur->vm_end && end > cur->vm_start) { unsigned long vma_hpagesize = vma_kernel_pagesize(cur); @@ -3831,7 +3831,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, /* * Note vmas containing huge pages */ - if (is_vm_hugetlb_page(cur)) + if (vma_is_hugetlb(cur)) basic_ioctls = true; found = true; @@ -3917,7 +3917,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * If the first vma contains huge pages, make sure start address * is aligned to huge page size. */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { unsigned long vma_hpagesize = vma_kernel_pagesize(vma); if (start & (vma_hpagesize - 1)) diff --git a/mm/vma.c b/mm/vma.c index 6ac9424a7fe5..790f229d230f 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -599,7 +599,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, * boundary. */ vma_adjust_trans_huge(vma, vma->vm_start, addr, NULL); - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_split(vma, addr); if (new_below) { @@ -2251,7 +2251,7 @@ bool vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot) * Do we need to track softdirty? hugetlb does not support softdirty * tracking yet. */ - if (vma_soft_dirty_enabled(vma) && !is_vm_hugetlb_page(vma)) + if (vma_soft_dirty_enabled(vma) && !vma_is_hugetlb(vma)) return true; /* Do we need write faults for uffd-wp tracking? */ @@ -2370,7 +2370,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (signal_pending(current)) goto out_unlock; if (vma->vm_file && vma->vm_file->f_mapping && - is_vm_hugetlb_page(vma)) + vma_is_hugetlb(vma)) vm_lock_mapping(mm, vma->vm_file->f_mapping); } @@ -2379,7 +2379,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (signal_pending(current)) goto out_unlock; if (vma->vm_file && vma->vm_file->f_mapping && - !is_vm_hugetlb_page(vma)) + !vma_is_hugetlb(vma)) vm_lock_mapping(mm, vma->vm_file->f_mapping); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 37467004336b..f752dfd47ec2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3515,7 +3515,7 @@ static int should_skip_vma(unsigned long start, unsigned long end, struct mm_wal if (!vma_is_accessible(vma)) return true; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return true; if (!vma_has_recency(vma)) diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h index b35279a0ec78..83ff23226ddf 100644 --- a/tools/testing/vma/include/stubs.h +++ b/tools/testing/vma/include/stubs.h @@ -193,7 +193,7 @@ static inline bool mapping_can_writeback(struct address_space *mapping) return true; } -static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma) +static inline bool vma_is_hugetlb(struct vm_area_struct *vma) { return false; } From ba82d45b876400c3fbb16c49ab982fb8a9c5ec97 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:09 +0100 Subject: [PATCH 119/213] mm: drop some redundant checks around hugetlb VMAs Adjust code which inadvertently perform redundant checks on hugetlb VMAs and clean them up: * hugetlb VMAs have VMA_DONTEXPAND_BIT set so a VMA_SPECIAL_FLAGS check suffices. (migrate_vma_setup() regains an explicit hugetlb test later in the series, once VMA_SPECIAL_FLAGS is removed.) * hugetlb VMAs unconditionally set vma->vm_ops, so they are never anonymous. * hugetlb VMAs do not set VMA_PFNMAP_BIT so checking for this is redundant. While we're here also drop a VM_BUG_ON() which the simplified check above makes unreachable, and use the new VMA flag API. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-29-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Marc Zyngier Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- drivers/gpu/drm/drm_gpusvm.c | 3 +-- mm/migrate_device.c | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c index a1d4989b0b61..fab34fea99c2 100644 --- a/drivers/gpu/drm/drm_gpusvm.c +++ b/drivers/gpu/drm/drm_gpusvm.c @@ -1141,8 +1141,7 @@ drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, * limitations. If/when migrate_vma_* add more support, this logic will * have to change. */ - migrate_devmem = ctx->devmem_possible && - vma_is_anonymous(vas) && !vma_is_hugetlb(vas); + migrate_devmem = ctx->devmem_possible && vma_is_anonymous(vas); chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier, vas, fault_addr, gpuva_start, diff --git a/mm/migrate_device.c b/mm/migrate_device.c index c38cbaaef5a4..b9c453c28795 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -743,8 +743,8 @@ int migrate_vma_setup(struct migrate_vma *args) args->start &= PAGE_MASK; args->end &= PAGE_MASK; - if (!args->vma || vma_is_hugetlb(args->vma) || - (args->vma->vm_flags & VM_SPECIAL) || vma_is_dax(args->vma)) + if (!args->vma || vma_test_any_mask(args->vma, VMA_SPECIAL_FLAGS) || + vma_is_dax(args->vma)) return -EINVAL; if (nr_pages <= 0) return -EINVAL; From 44debd840ec959063fbaac24f3db7a08a5cba8dc Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:10 +0100 Subject: [PATCH 120/213] mm/vma: introduce vma[_flags]_is_mm_backed() Introduce vma[_flags]_is_mm_backed() for the purposes of identifying mappings which are mm-managed and whose contents were placed there by core mm's own fault path - rather than a custom ->fault handler. Mappings not managed by core mm do not fall into this category, as they are populated by their owner, and nor of course does memory-mapped I/O. We exclude fixed mappings as these have a layout core mm cannot assume ordinary fault semantics over. hugetlb mappings are fixed mappings, but core mm owns them entirely, so they are explicitly carved out as mm-backed, as the MADV_DODUMP check already does. Use this functionality to update the madvise MADV_DODUMP check to test for this rather than open-coding it. Also update vma_flags_can_merge() so it is written in terms of vma_flags_is_mm_backed() - all mm-backed mappings can be merged except hugetlb mappings, whose flags and fields do not completely describe them. Also update __thp_vma_allowable_orders() to use vma_is_mm_backed() rather than testing for mergeability, which is irrelevant to this function. This replaces the VM_SPECIAL check which means it no longer checks for VMA_IO_BIT, however this is safe as we have established the invariant that mm-managed mappings may not set VMA_IO_BIT, so we implicitly include these. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-30-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 66 ++++++++++++++++++++++++++++----- mm/huge_memory.c | 7 ++-- mm/madvise.c | 7 +++- tools/testing/vma/include/dup.h | 44 +++++++++++++++++----- 4 files changed, 100 insertions(+), 24 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8c1e9ae01491..8490d901c3e3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1702,6 +1702,54 @@ static inline bool vma_is_fixed_mapping(const struct vm_area_struct *vma) return vma_flags_is_fixed_mapping(&vma->flags); } +/** + * vma_flags_is_mm_backed() - Do the specified VMA flags imply the mapping is + * backed by core mm? + * @flags: The VMA flags to test. + * + * An mm-backed mapping is one managed by core mm (see + * vma_flags_is_mm_managed()) whose contents are established by core mm's own + * fault path rather than a custom ->fault handler. + * + * Mappings not managed by core mm are populated by their owner, and fixed + * mappings may be mm-managed but have their contents established by a custom + * ->fault handler, so neither are mm-backed. hugetlb mappings are fixed, but + * core mm owns them entirely, so they are. + * + * These are the only mappings which can sensibly be merged, locked, dumped or + * be subject to uffd faulting, as all of these assume core mm and core mm + * alone manages their contents. + * + * Returns: true if the flags imply an mm-backed mapping, otherwise false. + */ +static inline bool vma_flags_is_mm_backed(const vma_flags_t *flags) +{ + /* hugetlb is a fixed mapping, but core mm owns it entirely. */ + if (vma_flags_is_hugetlb(flags)) + return true; + /* + * Mappings not managed by core mm are populated by their owner. + * + * Fixed mappings may be mm-managed, but their contents are established + * by a custom ->fault handler, so core mm cannot assume ordinary fault + * semantics over them. + */ + return vma_flags_is_mm_managed(flags) && !vma_flags_is_fixed_mapping(flags); +} + +/** + * vma_is_mm_backed() - Is @vma backed by core mm? + * @vma: The VMA to test. + * + * See vma_flags_is_mm_backed() for details. + * + * Returns: true if the VMA is mm-backed, otherwise false. + */ +static inline bool vma_is_mm_backed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_backed(&vma->flags); +} + /** * vma_flags_can_merge() - Do the specified VMA flags permit the VMA to be * merged with another? @@ -1712,21 +1760,21 @@ static inline bool vma_flags_can_merge(const vma_flags_t *flags) { /* * VMA merging assumes that a VMA's flags and fields completely describe - * its state. + * its state, which is only the case for mm-backed mappings. * - * However, mappings which are not mm-managed may have established state - * upon mapping not embodied in any attribute of the VMA. + * Mappings which are not mm-managed may have established state upon + * mapping not embodied in any attribute of the VMA. * * Additionally, private (CoW) PFN maps encode the source PFN of the * range in vma->vm_pgoff, which may otherwise cause spurious merges. */ - if (!vma_flags_is_mm_managed(flags)) + if (!vma_flags_is_mm_backed(flags)) return false; - /* VMA explicitly marked as being unmergeable. */ - if (vma_flags_is_fixed_mapping(flags)) - return false; - - return true; + /* + * hugetlb mappings contain per-VMA state, so their flags and fields do + * not completely describe them. + */ + return !vma_flags_is_hugetlb(flags); } /** diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 37735b4d2f27..47dd2708d8ad 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -215,11 +215,10 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, * khugepaged moves data from VMAs once collapsed, after they have been * faulted in, relying on refaulting for file-backed memory. * - * Mappings not managed by core mm cannot be reliably reconstructed from - * page faults, and fixed mappings (including hugetlb) may well be - * mm-managed - precisely the mappings which cannot be merged. + * Only mm-backed mappings can be reliably reconstructed from page + * faults, and hugetlb mappings cannot be collapsed. */ - if (!in_pf && !smaps && !vma_can_merge(vma)) + if (!in_pf && !smaps && (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma))) return 0; /* diff --git a/mm/madvise.c b/mm/madvise.c index 4d7f398d9738..d19a8cf2d6b9 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1415,8 +1415,11 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior) new_flags |= VM_DONTDUMP; break; case MADV_DODUMP: - if ((!vma_is_hugetlb(vma) && (new_flags & VM_SPECIAL)) || - (new_flags & VM_DROPPABLE)) + /* Only mm-backed memory can be meaningfully dumped. */ + if (!vma_is_mm_backed(vma)) + return -EINVAL; + /* Core mm may discard droppable memory at any time. */ + if (new_flags & VM_DROPPABLE) return -EINVAL; new_flags &= ~VM_DONTDUMP; break; diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 94bf818ad0e1..19ce10faa9f3 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1689,23 +1689,49 @@ static inline bool vma_is_fixed_mapping(const struct vm_area_struct *vma) return vma_flags_is_fixed_mapping(&vma->flags); } +static inline bool vma_flags_is_hugetlb(const vma_flags_t *flags) +{ + return IS_ENABLED(CONFIG_HUGETLB_PAGE) && + vma_flags_test(flags, VMA_HUGETLB_BIT); +} + +static inline bool vma_flags_is_mm_backed(const vma_flags_t *flags) +{ + /* hugetlb is a fixed mapping, but core mm owns it entirely. */ + if (vma_flags_is_hugetlb(flags)) + return true; + /* + * Mappings not managed by core mm are populated by their owner. + * + * Fixed mappings may be mm-managed, but their contents are established + * by a custom ->fault handler, so core mm cannot assume ordinary fault + * semantics over them. + */ + return vma_flags_is_mm_managed(flags) && !vma_flags_is_fixed_mapping(flags); +} + +static inline bool vma_is_mm_backed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_backed(&vma->flags); +} + static inline bool vma_flags_can_merge(const vma_flags_t *flags) { /* * VMA merging assumes that a VMA's flags and fields completely describe - * its state. + * its state, which is only the case for mm-backed mappings. * - * However, mappings which are not mm-managed may have established state - * upon mapping not embodied in any attribute of the VMA. + * Mappings which are not mm-managed may have established state upon + * mapping not embodied in any attribute of the VMA. * * Additionally, private (CoW) PFN maps encode the source PFN of the * range in vma->vm_pgoff, which may otherwise cause spurious merges. */ - if (!vma_flags_is_mm_managed(flags)) + if (!vma_flags_is_mm_backed(flags)) return false; - /* VMA explicitly marked as being unmergeable. */ - if (vma_flags_is_fixed_mapping(flags)) - return false; - - return true; + /* + * hugetlb mappings contain per-VMA state, so their flags and fields do + * not completely describe them. + */ + return !vma_flags_is_hugetlb(flags); } From fc95d1cf4a136f51b257506c420164c74d2b2142 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:11 +0100 Subject: [PATCH 121/213] mm/madvise: update is_valid_guard_vma() to use vma_is_mm_backed() We currently disallow the installation of lightweight guard regions in VMAs whose flags intersect VMA_SPECIAL_FLAGS or VMA_HUGETLB_BIT, or VMA_LOCKED_BIT unless allow_locked is set. hugetlb VMAs set VMA_DONTEXPAND_BIT so this was already redundant, VMA_SPECIAL_FLAGS already sufficed. However, now that VMA_IO_BIT is only set if VMA_PFNMAP or VMA_MIXEDMAP_BIT is set, this check collapses to being the equivalent of !vma_is_mm_backed(), excluding hugetlb mappings in which guard markers cannot be installed. Update is_valid_guard_vma() to reflect this. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-31-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/madvise.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index d19a8cf2d6b9..7dbac9554fd8 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1057,19 +1057,25 @@ static long madvise_remove(struct madvise_behavior *madv_behavior) return error; } -static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked) +static bool is_valid_guard_vma(const struct vm_area_struct *vma, + bool allow_locked) { - vm_flags_t disallowed = VM_SPECIAL | VM_HUGETLB; - /* - * A user could lock after setting a guard range but that's fine, as + * A user could lock after setting a guard range but that's fine as * they'd not be able to fault in. The issue arises when we try to zap * existing locked VMAs. We don't want to do that. */ - if (!allow_locked) - disallowed |= VM_LOCKED; + if (!allow_locked && vma_test(vma, VMA_LOCKED_BIT)) + return false; + /* + * Guard regions require a VMA whose page tables are managed solely by + * core mm, so the VMA must be mm-backed (excluding hugetlb, in which + * guard markers cannot be installed). + */ + if (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma)) + return false; - return !(vma->vm_flags & disallowed); + return true; } static bool is_guard_pte_marker(pte_t ptent) From fb700342f506d0b3839259a499bd8cad673887f8 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:12 +0100 Subject: [PATCH 122/213] mm/uffd: use predicates for userfaultfd checks Rather than directly checking VMA flags, use the newly introduced vma_is_mm_managed() and vma_is_mm_backed() helpers in userfaultfd when assessing VMA suitability for userfaultfd and UFFDIO_MOVE. The droppable check is kept explicit, as userfaultfd cannot function correctly with droppable mappings. Update vma_move_compatible() so it's expressed in terms of VMA characteristics rather than arbitrary flags. Additionally, update the use of the deprecated VMA flag API when checking VMA_SHADOW_STACK_BIT. A VMA_IO_BIT check is no longer required but that is fine as a hard invariant has been established that mm-managed mappings may not set VMA_IO_BIT so the check is now redundant. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-32-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/userfaultfd.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 949017e60608..b3076d4a622a 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1754,10 +1754,16 @@ static inline bool move_splits_huge_pmd(unsigned long dst_addr, } #endif -static inline bool vma_move_compatible(struct vm_area_struct *vma) +static inline bool vma_move_compatible(const struct vm_area_struct *vma) { - return !(vma->vm_flags & (VM_PFNMAP | VM_IO | VM_HUGETLB | - VM_MIXEDMAP | VM_SHADOW_STACK)); + /* uffd is generally incompatible with mappings not managed by core mm. */ + if (!vma_is_mm_managed(vma)) + return false; + /* The shadow stack should not be written to by userspace. */ + if (vma_test_single_mask(vma, VMA_SHADOW_STACK)) + return false; + /* hugetlb mappings cannot be safely moved. */ + return !vma_is_hugetlb(vma); } static int validate_move_areas(struct userfaultfd_ctx *ctx, @@ -2146,10 +2152,14 @@ static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, { const struct vm_uffd_ops *ops = vma_uffd_ops(vma); - if (vma->vm_flags & (VM_DROPPABLE | VM_SHADOW_STACK)) + /* Only mm-backed memory can have its faults handled by userspace. */ + if (!vma_is_mm_backed(vma)) return false; - - if (!vma_is_hugetlb(vma) && (vma->vm_flags & VM_SPECIAL)) + /* Core mm may discard droppable memory at any time. */ + if (vma_test_single_mask(vma, VMA_DROPPABLE)) + return false; + /* The shadow stack should not be written to by userspace. */ + if (vma_test_single_mask(vma, VMA_SHADOW_STACK)) return false; vm_flags &= __VM_UFFD_FLAGS; From 1d1104fe6dc6130b6ca78f801aecfd050d6560d0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:13 +0100 Subject: [PATCH 123/213] mm/madvise: use predicates for madvise(..., MADV_DOFORK) Make it clear what we're blocking in MADV_DOFORK. Previously we simply disallowed VM_SPECIAL i.e. mappings not managed by core mm, fixed mappings and VMA_IO_BIT. Now the invariant is established that mm-managed mappings may not set VMA_IO_BIT, the VMA_IO_BIT check is redundant. The rest is equivalent to testing for a fixed or non-mm-managed mapping, i.e. a mapping which is not mm-backed, or hugetlb, which is specifically excluded. This was established by commit 0b2758f48f22 ("Require (reasonably) normal mappings for MADV_DOFORK") containing my hands-down favourite call out of all time. Express the same thing using the predicates - only mm-backed mappings may have their copy-on-fork behaviour changed, retaining the eternal exception to all rules, hugetlb. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-33-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/madvise.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/madvise.c b/mm/madvise.c index 7dbac9554fd8..828c28408615 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1402,7 +1402,7 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior) new_flags |= VM_DONTCOPY; break; case MADV_DOFORK: - if (new_flags & VM_SPECIAL) + if (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma)) return -EINVAL; new_flags &= ~VM_DONTCOPY; break; From 531e56359ffb9ea714033bb9a1aff15da0e108d1 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:14 +0100 Subject: [PATCH 124/213] mm: eliminate VMA_SPECIAL_FLAGS usage when hugetlb explicitly tested It is now an invariant that VMA_IO_BIT is not set by mm-managed mappings, so each existing VMA_SPECIAL_FLAGS test need only test for VMA_DONTEXPAND_BIT, VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT. This is precisely a test for a fixed or non-mm-managed mapping. Update a number of callsites which already explicitly handle hugetlb mappings. vma_supports_mlock(), ksm_compatible() and should_skip_vma() explicitly check for non-mm-managed or fixed mappings, i.e. for !vma_is_mm_backed(), with hugetlb having already been excluded separately. So update those to check for mm-backed (and add comments for the retained droppable checks). No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-34-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/internal.h | 3 ++- mm/ksm.c | 5 +++-- mm/vmscan.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 498dbb71fcf1..bef4ad142e2a 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1112,8 +1112,9 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf, static inline bool vma_supports_mlock(const struct vm_area_struct *vma) { - if (vma_test_any_mask(vma, VMA_SPECIAL_FLAGS)) + if (!vma_is_mm_backed(vma)) return false; + /* Core mm may discard droppable memory at any time. */ if (vma_test_single_mask(vma, VMA_DROPPABLE)) return false; if (vma_is_dax(vma) || vma_is_hugetlb(vma)) diff --git a/mm/ksm.c b/mm/ksm.c index 624f37975e12..df4c5199f625 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -747,9 +747,10 @@ static bool ksm_compatible(const struct file *file, vma_flags_t vma_flags) if (vma_flags_test_any(&vma_flags, VMA_SHARED_BIT, VMA_MAYSHARE_BIT, VMA_HUGETLB_BIT)) return false; - if (vma_flags_test_single_mask(&vma_flags, VMA_DROPPABLE)) + if (!vma_flags_is_mm_backed(&vma_flags)) return false; - if (vma_flags_test_any_mask(&vma_flags, VMA_SPECIAL_FLAGS)) + /* Core mm may discard droppable memory at any time. */ + if (vma_flags_test_single_mask(&vma_flags, VMA_DROPPABLE)) return false; if (file_is_dax(file)) return false; diff --git a/mm/vmscan.c b/mm/vmscan.c index f752dfd47ec2..c35b82a3069d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3521,7 +3521,7 @@ static int should_skip_vma(unsigned long start, unsigned long end, struct mm_wal if (!vma_has_recency(vma)) return true; - if (vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) + if (vma_test(vma, VMA_LOCKED_BIT) || !vma_is_mm_backed(vma)) return true; if (vma == get_gate_vma(vma->vm_mm)) From 5997da57a09b199bae688dfee7242f0f3f76b3bd Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:15 +0100 Subject: [PATCH 125/213] mm: eliminate VMA_SPECIAL_FLAGS check in lru_gen_look_around() A fixed or non-mm-managed mapping is one which sets VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT or VMA_DONTEXPAND_BIT, which is precisely what VMA_SPECIAL_FLAGS tests for other than VMA_IO_BIT, which is safe to drop as mm-managed mappings may not set it. This is equivalent to testing for an mm-backed mapping, except for hugetlb. However, hugetlb folios are never placed on the LRU and lru_gen_look_around() is only called from LRU shrinkers on reclaim so this is fine. Using vma_is_mm_backed() rather than checking VMA_SPECIAL_FLAGS makes the check self-documenting and helps eliminate confusion around 'special flags'. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-35-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c35b82a3069d..ac386422bc91 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4461,8 +4461,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) if (spin_is_contended(pvmw->ptl)) return true; - /* exclude special VMAs containing anon pages from COW */ - if (vma->vm_flags & VM_SPECIAL) + /* exclude non-mm-backed VMAs containing anon pages from COW */ + if (!vma_is_mm_backed(vma)) return true; /* avoid taking the LRU lock under the PTL when possible */ From 8bf792b7afcb5d1e4ec4e9ed4204a10b58cdbf77 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:16 +0100 Subject: [PATCH 126/213] mm: avoid use of VMA_SPECIAL_FLAGS in migrate_vma_setup() Now we have the expressive vma_is_mm_backed() predicate, use it to determine whether to proceed with migration. This drops the VMA_IO_BIT test, which is safe as mm-managed mappings may not set it. hugetlb mappings are mm-backed but remain explicitly excluded, as the migrate_vma page walk has no hugetlb support. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-36-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/migrate_device.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index b9c453c28795..9e19fadfa65c 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -739,19 +739,21 @@ static void migrate_vma_unmap(struct migrate_vma *migrate) */ int migrate_vma_setup(struct migrate_vma *args) { + const struct vm_area_struct *vma = args->vma; long nr_pages = (args->end - args->start) >> PAGE_SHIFT; args->start &= PAGE_MASK; args->end &= PAGE_MASK; - if (!args->vma || vma_test_any_mask(args->vma, VMA_SPECIAL_FLAGS) || - vma_is_dax(args->vma)) + if (!vma) + return -EINVAL; + /* The migrate_vma page walk has no hugetlb support. */ + if (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma) || vma_is_dax(vma)) return -EINVAL; if (nr_pages <= 0) return -EINVAL; - if (args->start < args->vma->vm_start || - args->start >= args->vma->vm_end) + if (args->start < vma->vm_start || args->start >= vma->vm_end) return -EINVAL; - if (args->end <= args->vma->vm_start || args->end > args->vma->vm_end) + if (args->end <= vma->vm_start || args->end > vma->vm_end) return -EINVAL; if (!args->src || !args->dst) return -EINVAL; From 72a993449cb0a2b7d7a0cf4fd3d5e7225ea89ea1 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:17 +0100 Subject: [PATCH 127/213] mm: eliminate VM_SPECIAL, VMA_SPECIAL_FLAGS Every user of the VM_SPECIAL or VMA_SPECIAL_FLAGS has now been converted to predicates which explicitly express what is actually being checked for rather than the nebulous concept of possessing 'special' VMA flags. In any case 'special' is not so special a term of art in mm - it includes VDSO/VVAR mappings, special in the sense of vm_normal_folio() and probably other cases too. Therefore make things less special by eliminating these now unused flags. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-37-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- include/linux/mm.h | 8 -------- tools/testing/vma/include/dup.h | 8 -------- 2 files changed, 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8490d901c3e3..3f95442667a9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -577,14 +577,6 @@ enum { #define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC) #define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT) -/* - * Special vmas that are non-mergable, non-mlock()able. - */ - -#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ - VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT) -#define VM_SPECIAL vma_flags_to_legacy(VMA_SPECIAL_FLAGS) - /* * Physically remapped pages are special. Tell the * rest of the world about it: diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 19ce10faa9f3..953895832901 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -352,14 +352,6 @@ enum { #define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC) #define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT) -/* - * Special vmas that are non-mergable, non-mlock()able. - */ -#define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP) - -#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ - VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT) - #define VMA_REMAP_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_PFNMAP_BIT, \ VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT) From b0d88bfa7515ed772afa286af0e2f635acad9f7e Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:18 +0100 Subject: [PATCH 128/213] fuse: dax: do not set VM_MIXEDMAP Commit e1fb4a086495 ("dax: remove VM_MIXEDMAP for fsdax and device dax") prevented fsdax and device-dax from setting VM_MIXEDMAP, as DAX no longer relies on it to direct core mm paths. The fuse DAX implementation, added later, copied the old pattern and still sets it. Fuse DAX maps pages the same way fsdax does, via dax_iomap_fault() and ultimately vmf_insert_page_mkwrite() and vmf_insert_folio_pmd(), which insert ordinary refcounted pages and so do not require VM_MIXEDMAP. Setting it only serves to mark the mapping as not mm-managed, making fuse DAX the sole DAX implementation whose mappings are unmergeable, cannot be mlock()'d, eagerly copy page tables on fork and reject MADV_DOFORK and MADV_DODUMP. It also requires vma_is_special_huge() in mm/huge_memory.c to carve DAX out of its check for mappings not managed by core mm explicitly. There is no reason for fuse DAX to keep on using this flag so drop it. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-38-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Zi Yan Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- fs/fuse/dax.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c index 85cdf0199bc0..a5994f1c637d 100644 --- a/fs/fuse/dax.c +++ b/fs/fuse/dax.c @@ -826,7 +826,7 @@ int fuse_dax_mmap(struct file *file, struct vm_area_struct *vma) { file_accessed(file); vma->vm_ops = &fuse_dax_vm_ops; - vm_flags_set(vma, VM_MIXEDMAP | VM_HUGEPAGE); + vma_set_flags(vma, VMA_HUGEPAGE_BIT); return 0; } From e2a1d080f51070fb4dcc74d352ae86eca5fb5bab Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 13 Oct 2026 16:05:19 +0100 Subject: [PATCH 129/213] mm/huge_memory: remove vma_is_special_huge() vma_is_special_huge() tests whether either the VMA_PFNMAP_BIT or VMA_MIXEDMAP_BIT is set (i.e. whether the VMA is not mm-managed), but with a DAX carve-out. DAX however no longer sets VMA_MIXEDMAP_BIT, so this carve-out is no longer required. Therefore test for !vma_is_mm_managed() instead and also drop the VMA_IO_BIT check, as it is now redundant since it is enforced that mm-managed mappings may not set VMA_IO_BIT. This also eliminates another overloaded use of 'special' within mm. No functional change intended. Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-39-1aa3c9b139f9@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Dennis Dalessandro Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Paul Moore Cc: Stephen Smalley Cc: Jaroslav Kysela Cc: Takashi Iwai Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Andrii Nakryiko Cc: Eduard Zingerman Cc: Kumar Kartikeya Dwivedi Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Doug Gilbert Cc: James Bottomley Cc: Martin K. Petersen Cc: Simona Vetter Cc: Helge Deller Cc: Sebastian Reichel Cc: John Hubbard Cc: Peter Xu Cc: Masami Hiramatsu Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Arnaldo Carvalho de Melo Cc: Namhyung Kim Cc: Mark Rutland Cc: Rik van Riel Cc: Harry Yoo Cc: Juri Lelli Cc: Vincent Guittot Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Will Deacon Cc: Aneesh Kumar K.V Cc: Nicholas Piggin Cc: Arnd Bergmann Cc: Muchun Song Cc: Oscar Salvador Cc: Matthew Wilcox (Oracle) Cc: Jan Kara Cc: Marc Zyngier Cc: Oliver Upton Cc: Catalin Marinas Cc: Madhavan Srinivasan Cc: Anup Patel Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: David S. Miller Cc: Andreas Larsson Cc: Al Viro Cc: Christian Brauner Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Youngjun Park Cc: Johannes Weiner Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Chengming Zhou Cc: Michal Hocko Cc: Miklos Szeredi Cc: Xu Xin Cc: Ihor Solodrai Cc: Bert Karwatzki --- mm/huge_memory.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 47dd2708d8ad..7bf35eeca6b6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -110,14 +110,6 @@ static inline bool file_thp_enabled(const struct vm_area_struct *vma) return S_ISREG(inode->i_mode); } -/* If returns true, we are unable to access the VMA's folios. */ -static bool vma_is_special_huge(const struct vm_area_struct *vma) -{ - if (vma_is_dax(vma)) - return false; - return vma_test_any(vma, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT); -} - static bool vma_file_bypass_thp_tuneables(const struct vm_area_struct *vma, enum tva_type type) { @@ -192,7 +184,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma) || vma_is_special_huge(vma)) + else if (vma_is_dax(vma) || !vma_is_mm_managed(vma)) supported_orders = THP_ORDERS_ALL_SPECIAL_DAX; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; @@ -3055,7 +3047,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); - if (vma_is_special_huge(vma)) { + if (!vma_is_mm_managed(vma)) { spin_unlock(ptl); /* No zero page support yet */ } else { @@ -3211,7 +3203,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, */ if (arch_needs_pgtable_deposit()) zap_deposited_table(mm, pmd); - if (vma_is_special_huge(vma)) + if (!vma_is_mm_managed(vma)) return; if (unlikely(pmd_is_migration_entry(old_pmd))) { const softleaf_t old_entry = softleaf_from_pmd(old_pmd); @@ -4778,9 +4770,7 @@ static inline bool vma_not_suitable_for_thp_split(struct vm_area_struct *vma) { if (vma_is_dax(vma)) return true; - if (vma_is_special_huge(vma)) - return true; - if (vma_test(vma, VMA_IO_BIT)) + if (!vma_is_mm_managed(vma)) return true; if (vma_is_hugetlb(vma)) return true; From 1a5c751385dc87a65235ea1511f70473994ededf Mon Sep 17 00:00:00 2001 From: Zhenghui Hao Date: Wed, 23 Sep 2026 10:28:58 +0800 Subject: [PATCH 130/213] hugetlbfs: fix stale comment in hugetlbfs_file_mmap() The comment above the VMA flag setup in hugetlbfs_file_mmap() says the VMA address alignment has already been checked by prepare_hugepage_range(). That function no longer exists: it was removed by commit eff41389d824 ("mm/hugetlb: remove prepare_hugepage_range()"), and the check now lives in hugetlb_get_unmapped_area(). The comment also mentions ia64, which was removed from the tree by commit cf8e8658100d ("arch: Remove Itanium (IA-64) architecture"). Point the comment at the current function and drop the ia64 reference. The rest of the comment, which tells future edits to add any error return only after VM_HUGETLB has been set, is still accurate and is left untouched. No functional change. Link: https://lore.kernel.org/tencent_AA61551D1F50E61F46C8542F5C8874512C05@qq.com Signed-off-by: Zhenghui Hao Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Oscar Salvador Cc: David Hildenbrand --- fs/hugetlbfs/inode.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index e7ddb76905df..ab1e4dec3f77 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -106,11 +106,10 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) /* * vma address alignment (but not the pgoff alignment) has - * already been checked by prepare_hugepage_range. If you add - * any error returns here, do so after setting VM_HUGETLB, so - * vma_is_hugetlb tests below unmap_region go the right - * way when do_mmap unwinds (may be important on powerpc - * and ia64). + * already been checked by hugetlb_get_unmapped_area(). If you + * add any error returns here, do so after setting VM_HUGETLB, + * so vma_is_hugetlb tests below unmap_region go the right + * way when do_mmap unwinds (may be important on powerpc). */ vma_set_flags(vma, VMA_HUGETLB_BIT, VMA_DONTEXPAND_BIT); vma->vm_ops = &hugetlb_vm_ops; From 6a9ef02c60bf6c06edf5e33f90e1eff70d7cf2af Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 17 Sep 2026 20:47:41 +0100 Subject: [PATCH 131/213] mm/vma: const-ify vma_assert_stabilised() and associated functions Patch series "mm: implement and use vma_has_anon_rmap(), silence KCSAN". Provide a function to abstract the common task of checking whether a VMA has an anonymous reverse mapping associated with it. In the first patch, const-ify vma_assert_stabilised() and related functions so that vma_has_anon_rmap() can reference a const vma pointer. In the second patch, introduce vma_has_anon_rmap(). Finally in the third patch, update comments referencing anon_vma to instead reference the anon rmap to abstract this conceptually to avoid confusion. There are still other functions which reference anon_vma directly, those can be addressed in a follow up. This patch (of 3): The vma pointers are not modified in any of these functions so make it official by const-ify them. Propagate this throughout the call stack. No functional change intended. Link: https://lore.kernel.org/20260917-vma-is-faulted-v3-0-5c22314a72e7@kernel.org Link: https://lore.kernel.org/20260917-vma-is-faulted-v3-1-5c22314a72e7@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Reviewed-by: Pedro Falcato Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: Lance Yang Cc: Alistair Popple Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Dev Jain Cc: Gregory Price Cc: Guilherme Giacomo Simoes Cc: Harry Yoo Cc: "Huang, Ying" Cc: Jann Horn Cc: Joshua Hahn Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Matthew Brost Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Oscar Salvador Cc: Peter Xu Cc: Rakie Kim Cc: Rik van Riel Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: xu xin --- include/linux/mmap_lock.h | 12 ++++++------ tools/testing/vma/include/dup.h | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 28e3696ce9ff..e5553f4a414c 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -273,7 +273,7 @@ static inline void vma_end_read(struct vm_area_struct *vma) vma_refcount_put(vma); } -static inline unsigned int __vma_raw_mm_seqnum(struct vm_area_struct *vma) +static inline unsigned int __vma_raw_mm_seqnum(const struct vm_area_struct *vma) { const struct mm_struct *mm = vma->vm_mm; @@ -288,7 +288,7 @@ static inline unsigned int __vma_raw_mm_seqnum(struct vm_area_struct *vma) * * Returns true if write-locked, otherwise false. */ -static inline bool __is_vma_write_locked(struct vm_area_struct *vma) +static inline bool __is_vma_write_locked(const struct vm_area_struct *vma) { /* * current task is holding mmap_write_lock, both vma->vm_lock_seq and @@ -344,7 +344,7 @@ int vma_start_write_killable(struct vm_area_struct *vma) * vma_assert_write_locked() - assert that @vma holds a VMA write lock. * @vma: The VMA to assert. */ -static inline void vma_assert_write_locked(struct vm_area_struct *vma) +static inline void vma_assert_write_locked(const struct vm_area_struct *vma) { if (!IS_ENABLED(CONFIG_MMU)) { mmap_assert_write_locked(vma->vm_mm); @@ -359,7 +359,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) * lock and is not detached. * @vma: The VMA to assert. */ -static inline void vma_assert_locked(struct vm_area_struct *vma) +static inline void vma_assert_locked(const struct vm_area_struct *vma) { unsigned int refcnt; @@ -410,7 +410,7 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) * With lockdep disabled we may sometimes race with other threads acquiring the * mmap read lock simultaneous with our VMA read lock. */ -static inline void vma_assert_stabilised(struct vm_area_struct *vma) +static inline void vma_assert_stabilised(const struct vm_area_struct *vma) { /* * If another thread owns an mmap lock, it may go away at any time, and @@ -445,7 +445,7 @@ static inline void vma_assert_stabilised(struct vm_area_struct *vma) vma_assert_locked(vma); } -static inline bool vma_is_attached(struct vm_area_struct *vma) +static inline bool vma_is_attached(const struct vm_area_struct *vma) { return refcount_read(&vma->vm_refcnt); } diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 953895832901..718132580fac 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1178,7 +1178,7 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) return mas_find(&vmi->mas, ULONG_MAX); } -static inline bool vma_is_attached(struct vm_area_struct *vma) +static inline bool vma_is_attached(const struct vm_area_struct *vma) { return refcount_read(&vma->vm_refcnt); } From 38564710f4436156c017419abefb33eee0376f19 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 17 Sep 2026 20:47:42 +0100 Subject: [PATCH 132/213] mm: implement and use vma_has_anon_rmap(), silence KCSAN Provide a function to abstract the common task of checking whether a VMA has an anonymous reverse mapping associated with it. If the VMA is attached, a VMA or mmap lock must be held when calling this function. For an attached, anonymous, VMA: Transition | VMA/mmap Lock state -----------------------------|------------------------------------------- No anon rmap to anon rmap | Write lock/read lock + mm->page_table_lock Anon rmap to no anon rmap | Write lock vma_has_anon_rmap() never provides a false positive (the lock precludes it), but if only a read lock is held, a negative result must be re-checked with mm->page_table_lock held. A VMA obtains an anonymous reverse mapping when first faulted or forked and it is removed when it is freed. Detached VMAs cannot be concurrently manipulated as they are removed from the maple tree so require no guarantees. Use data_race() to silence KCSAN about non-existent data races between concurrent vma->anon_vma read/write on optimistic fault tests. Update the core VMA merge/split, rmap, mremap, KSM, fork, khugepaged and fault preparation callers which test vma->anon_vma directly to use vma_has_anon_rmap() instead. Finally, update comments that reference anon_vma to reference the anon rmap instead. Since the lockless read in reusable_anon_vma() is doing more than checking whether the VMA has anon rmap - it is returning the anon_vma to be used on fault - do not alter it. There is one odd one out - file_backed_vma_is_retractable() - which holds neither a VMA nor mmap lock and is stabilised by the file rmap lock only, so simply add a comment to explain why it's necessary. Link: https://lore.kernel.org/20260917-vma-is-faulted-v3-2-5c22314a72e7@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reported-by: Guilherme Giacomo Simoes Closes: https://lore.kernel.org/all/20260829100034.423064-1-trintaeoitogc@gmail.com/ Closes: https://lore.kernel.org/all/20260909115723.528501-1-trintaeoitogc@gmail.com/ Reviewed-by: Pedro Falcato Reviewed-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) Reviewed-by: Lance Yang Cc: Alistair Popple Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Dev Jain Cc: Gregory Price Cc: Harry Yoo Cc: "Huang, Ying" Cc: Jann Horn Cc: Joshua Hahn Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Matthew Brost Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Oscar Salvador Cc: Peter Xu Cc: Rakie Kim Cc: Rik van Riel Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan --- mm/huge_memory.c | 4 ++-- mm/internal.h | 2 +- mm/khugepaged.c | 5 ++++- mm/ksm.c | 10 +++++----- mm/madvise.c | 4 ++-- mm/memory.c | 4 ++-- mm/mprotect.c | 2 +- mm/mremap.c | 4 ++-- mm/rmap.c | 22 +++++++++++----------- mm/swapfile.c | 2 +- mm/userfaultfd.c | 2 +- mm/vma.c | 29 +++++++++++++++-------------- mm/vma.h | 29 ++++++++++++++++++++++++++++- tools/testing/vma/include/stubs.h | 4 ++++ 14 files changed, 79 insertions(+), 44 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7bf35eeca6b6..194a1ed97865 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -258,7 +258,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, * Allow page fault since anon_vma may be not initialized until * the first page fault. */ - if (!vma->anon_vma) + if (!vma_has_anon_rmap(vma)) return (smaps || in_pf) ? orders : 0; return orders; @@ -2173,7 +2173,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) pmd_t orig_pmd = vmf->orig_pmd; vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd); - VM_BUG_ON_VMA(!vma->anon_vma, vma); + VM_BUG_ON_VMA(!vma_has_anon_rmap(vma), vma); if (is_huge_zero_pmd(orig_pmd)) { vm_fault_t ret = do_huge_zero_wp_pmd(vmf); diff --git a/mm/internal.h b/mm/internal.h index bef4ad142e2a..b1b0f29d2216 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -338,7 +338,7 @@ void unlink_anon_vmas(struct vm_area_struct *vma); static inline int anon_vma_prepare(struct vm_area_struct *vma) { - if (likely(vma->anon_vma)) + if (likely(vma_has_anon_rmap(vma))) return 0; return __anon_vma_prepare(vma); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 85ea095906fb..73a0a2249f67 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1094,7 +1094,7 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l * thp_vma_allowable_orders() may return true for qualified file * vmas. */ - if (expect_anon && (!(*vmap)->anon_vma || !vma_is_anonymous(*vmap))) + if (expect_anon && (!vma_has_anon_rmap(vma) || !vma_is_anonymous(vma))) return SCAN_PAGE_ANON; return SCAN_SUCCEED; } @@ -2137,6 +2137,9 @@ static bool file_backed_vma_is_retractable(struct vm_area_struct *vma) * Check vma->anon_vma to exclude MAP_PRIVATE mappings that * got written to. These VMAs are likely not worth removing * page tables from, as PMD-mapping is likely to be split later. + * + * Can't use vma_has_anon_rmap() here as the VMA may be stabilised + * by the file rmap lock. */ if (READ_ONCE(vma->anon_vma)) return false; diff --git a/mm/ksm.c b/mm/ksm.c index df4c5199f625..fe0eb13e914d 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -778,7 +778,7 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm, if (ksm_test_exit(mm)) return NULL; vma = vma_lookup(mm, addr); - if (!vma || !(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) + if (!vma || !(vma->vm_flags & VM_MERGEABLE) || !vma_has_anon_rmap(vma)) return NULL; return vma; } @@ -1242,7 +1242,7 @@ static int unmerge_and_remove_all_rmap_items(void) goto mm_exiting; for_each_vma(vmi, vma) { - if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) + if (!(vma->vm_flags & VM_MERGEABLE) || !vma_has_anon_rmap(vma)) continue; err = break_ksm(vma, vma->vm_start, vma->vm_end, false); if (err) @@ -2692,7 +2692,7 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) continue; if (ksm_scan.address < vma->vm_start) ksm_scan.address = vma->vm_start; - if (!vma->anon_vma) + if (!vma_has_anon_rmap(vma)) ksm_scan.address = vma->vm_end; while (ksm_scan.address < vma->vm_end) { @@ -2883,7 +2883,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma) if (!(vma->vm_flags & VM_MERGEABLE)) return 0; - if (vma->anon_vma) { + if (vma_has_anon_rmap(vma)) { err = break_ksm(vma, vma->vm_start, vma->vm_end, true); if (err) return err; @@ -3035,7 +3035,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start, if (!(*vm_flags & VM_MERGEABLE)) return 0; /* just ignore the advice */ - if (vma->anon_vma) { + if (vma_has_anon_rmap(vma)) { err = break_ksm(vma, start, end, true); if (err) return err; diff --git a/mm/madvise.c b/mm/madvise.c index 828c28408615..6b48a3f6c559 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1164,7 +1164,7 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior) * as part of the VMA lock logic. */ if (vma_is_anonymous(vma)) { - VM_WARN_ON_ONCE(!vma->anon_vma && + VM_WARN_ON_ONCE(!vma_has_anon_rmap(vma) && madv_behavior->lock_mode != MADVISE_MMAP_READ_LOCK); err = anon_vma_prepare(vma); @@ -1630,7 +1630,7 @@ static bool is_vma_lock_sufficient(struct vm_area_struct *vma, * check overly paranoid which is safe. */ if (vma_is_anonymous(vma) && - prepares_anon_vma(madv_behavior->behavior) && !vma->anon_vma) + prepares_anon_vma(madv_behavior->behavior) && !vma_has_anon_rmap(vma)) return false; return true; diff --git a/mm/memory.c b/mm/memory.c index a142c778a4d8..c030c208749b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1536,7 +1536,7 @@ vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * The presence of an anon_vma indicates an anonymous VMA has page * tables which naturally cannot be reconstituted on page fault. */ - if (src_vma->anon_vma) + if (vma_has_anon_rmap(src_vma)) return true; /* @@ -4015,7 +4015,7 @@ vm_fault_t __vmf_anon_prepare(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret = 0; - if (likely(vma->anon_vma)) + if (likely(vma_has_anon_rmap(vma))) return 0; if (vmf->flags & FAULT_FLAG_VMA_LOCK) { if (!mmap_read_trylock(vma->vm_mm)) diff --git a/mm/mprotect.c b/mm/mprotect.c index bef7b742d101..1318e44ef43e 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -816,7 +816,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, vma_flags_set(&new_vma_flags, VMA_ACCOUNT_BIT); } } else if (vma_flags_test(&old_vma_flags, VMA_ACCOUNT_BIT) && - vma_is_anonymous(vma) && !vma->anon_vma) { + vma_is_anonymous(vma) && !vma_has_anon_rmap(vma)) { vma_flags_clear(&new_vma_flags, VMA_ACCOUNT_BIT); } diff --git a/mm/mremap.c b/mm/mremap.c index d38ff332a13a..845f7b5844d4 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -144,13 +144,13 @@ static void take_rmap_locks(struct vm_area_struct *vma) { if (vma->vm_file) i_mmap_lock_write(vma->vm_file->f_mapping); - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) anon_vma_lock_write(vma->anon_vma); } static void drop_rmap_locks(struct vm_area_struct *vma) { - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) anon_vma_unlock_write(vma->anon_vma); if (vma->vm_file) i_mmap_unlock_write(vma->vm_file->f_mapping); diff --git a/mm/rmap.c b/mm/rmap.c index b4a4b3929044..805db93fe042 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -208,7 +208,7 @@ int __anon_vma_prepare(struct vm_area_struct *vma) anon_vma_lock_write(anon_vma); /* page_table_lock to protect against threads */ spin_lock(&mm->page_table_lock); - if (likely(!vma->anon_vma)) { + if (likely(!vma_has_anon_rmap(vma))) { /* * Make anon_vma fields visible before anon_vma is published. * Paired with an address dependency in reusable_anon_vma(). @@ -246,21 +246,21 @@ static void check_anon_vma_clone(struct vm_area_struct *dst, VM_WARN_ON_ONCE(operation != VMA_OP_FORK && dst->vm_mm != src->vm_mm); /* If we have anything to do src->anon_vma must be provided. */ - VM_WARN_ON_ONCE(!src->anon_vma && !list_empty(&src->anon_vma_chain)); - VM_WARN_ON_ONCE(!src->anon_vma && dst->anon_vma); + VM_WARN_ON_ONCE(!vma_has_anon_rmap(src) && !list_empty(&src->anon_vma_chain)); + VM_WARN_ON_ONCE(!vma_has_anon_rmap(src) && vma_has_anon_rmap(dst)); /* We are establishing a new anon_vma_chain. */ VM_WARN_ON_ONCE(!list_empty(&dst->anon_vma_chain)); /* * On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma * must be the same across dst and src. */ - VM_WARN_ON_ONCE(dst->anon_vma && dst->anon_vma != src->anon_vma); + VM_WARN_ON_ONCE(vma_has_anon_rmap(dst) && dst->anon_vma != src->anon_vma); /* * Essentially equivalent to above - if not a no-op, we should expect * dst->anon_vma to be set for everything except a fork. */ - VM_WARN_ON_ONCE(operation != VMA_OP_FORK && src->anon_vma && - !dst->anon_vma); + VM_WARN_ON_ONCE(operation != VMA_OP_FORK && vma_has_anon_rmap(src) && + !vma_has_anon_rmap(dst)); /* For the anon_vma to be compatible, it can only be singular. */ VM_WARN_ON_ONCE(operation == VMA_OP_MERGE_UNFAULTED && !list_is_singular(&src->anon_vma_chain)); @@ -273,7 +273,7 @@ static void maybe_reuse_anon_vma(struct vm_area_struct *dst, struct anon_vma *anon_vma) { /* If already populated, nothing to do.*/ - if (dst->anon_vma) + if (vma_has_anon_rmap(dst)) return; /* @@ -327,7 +327,7 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, check_anon_vma_clone(dst, src, operation); - if (!active_anon_vma) + if (!vma_has_anon_rmap(src)) return 0; /* @@ -384,7 +384,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) int rc; /* Don't bother if the parent process has no anon_vma here. */ - if (!pvma->anon_vma) + if (!vma_has_anon_rmap(pvma)) return 0; /* Drop inherited anon_vma, we'll reuse existing or allocate new. */ @@ -405,7 +405,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) */ rc = anon_vma_clone(vma, pvma, VMA_OP_FORK); /* An error arose or an existing anon_vma was reused, all done then. */ - if (rc || vma->anon_vma) { + if (rc || vma_has_anon_rmap(vma)) { put_anon_vma(anon_vma); anon_vma_chain_free(avc); return rc; @@ -864,7 +864,7 @@ unsigned long page_address_in_vma(const struct folio *folio, * Note: swapoff's unuse_vma() is more efficient with this * check, and needs it to match anon_vma when KSM is active. */ - if (!vma->anon_vma || !anon_vma || + if (!vma_has_anon_rmap(vma) || !anon_vma || vma->anon_vma->root != anon_vma->root) return -EFAULT; /* KSM folios don't reach here because of the !anon_vma check */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 0901cb8fa729..6187c02ec5ec 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2707,7 +2707,7 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type) if (check_stable_address_space(mm)) goto unlock; for_each_vma(vmi, vma) { - if (vma->anon_vma && !vma_is_hugetlb(vma)) { + if (vma_has_anon_rmap(vma) && !vma_is_hugetlb(vma)) { ret = unuse_vma(vma, type); if (ret) break; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index b3076d4a622a..3103f36c9916 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -145,7 +145,7 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, * We know we're going to need to use anon_vma, so check * that early. */ - if (!(vma->vm_flags & VM_SHARED) && unlikely(!vma->anon_vma)) + if (!(vma->vm_flags & VM_SHARED) && unlikely(!vma_has_anon_rmap(vma))) vma_end_read(vma); else return vma; diff --git a/mm/vma.c b/mm/vma.c index 790f229d230f..0b9abf3a85f9 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -100,7 +100,8 @@ static bool vma_is_fork_child(struct vm_area_struct *vma) * parents. This can improve scalability caused by the anon_vma root * lock. */ - return vma && vma->anon_vma && !list_is_singular(&vma->anon_vma_chain); + return vma && vma_has_anon_rmap(vma) && + !list_is_singular(&vma->anon_vma_chain); } static inline bool is_mergeable_vma(struct vma_merge_struct *vmg, bool merge_next) @@ -140,7 +141,7 @@ static bool is_mergeable_anon_vma(struct vma_merge_struct *vmg, bool merge_next) VM_WARN_ON(src && src_anon != src->anon_vma); /* Case 1 - we will dup_anon_vma() from src into tgt. */ - if (!tgt_anon && src_anon) { + if (!vma_has_anon_rmap(tgt) && src_anon) { struct vm_area_struct *copied_from = vmg->copied_from; if (vma_is_fork_child(src)) @@ -151,7 +152,7 @@ static bool is_mergeable_anon_vma(struct vma_merge_struct *vmg, bool merge_next) return true; } /* Case 2 - we will simply use tgt's anon_vma. */ - if (tgt_anon && !src_anon) + if (vma_has_anon_rmap(tgt) && !src_anon) return !vma_is_fork_child(tgt); /* Case 3 - the anon_vma's are already shared. */ return src_anon == tgt_anon; @@ -190,10 +191,10 @@ static void init_multi_vma_prep(struct vma_prepare *vp, adjust = NULL; vp->adj_next = adjust; - if (!vp->anon_vma && adjust) + if (!vma_has_anon_rmap(vma) && adjust) vp->anon_vma = adjust->anon_vma; - VM_WARN_ON(vp->anon_vma && adjust && adjust->anon_vma && + VM_WARN_ON(vma_has_anon_rmap(vma) && adjust && vma_has_anon_rmap(adjust) && vp->anon_vma != adjust->anon_vma); vp->file = vma->vm_file; @@ -430,7 +431,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, vp->remove->vm_end); fput(vp->file); } - if (vp->remove->anon_vma) + if (vma_has_anon_rmap(vp->remove)) unlink_anon_vmas(vp->remove); mm->map_count--; mpol_put(vma_policy(vp->remove)); @@ -500,7 +501,7 @@ static bool can_vma_merge_right(struct vma_merge_struct *vmg, * We therefore check this in addition to mergeability to either side. */ prev = vmg->prev; - return !prev->anon_vma || !next->anon_vma || + return !vma_has_anon_rmap(prev) || !vma_has_anon_rmap(next) || prev->anon_vma == next->anon_vma; } @@ -670,7 +671,7 @@ static int dup_anon_vma(struct vm_area_struct *dst, * that is it is unfaulted, we need to ensure that the newly merged * range is referenced by the anon_vma's of the source. */ - if (src->anon_vma && !dst->anon_vma) { + if (vma_has_anon_rmap(src) && !vma_has_anon_rmap(dst)) { int ret; vma_assert_write_locked(dst); @@ -720,7 +721,7 @@ void validate_mm(struct mm_struct *mm) } #ifdef CONFIG_DEBUG_VM_RB - if (anon_vma) { + if (vma_has_anon_rmap(vma)) { anon_vma_lock_read(anon_vma); list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) anon_rmap_tree_verify(avc); @@ -1020,7 +1021,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( * simply a case of, if prev has no anon_vma object, which of * next or middle contains the anon_vma we must duplicate. */ - err = dup_anon_vma(prev, next->anon_vma ? next : middle, + err = dup_anon_vma(prev, vma_has_anon_rmap(next) ? next : middle, &anon_dup); } else if (merge_left) { /* @@ -1960,7 +1961,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, * If a vma has not yet been faulted, update its anonymous pgoff to * match the new location to increase its chance of merging. */ - if (!vma->anon_vma) { + if (!vma_has_anon_rmap(vma)) { anon_pgoff = addr >> PAGE_SHIFT; if (vma_is_anonymous(vma)) { @@ -2387,7 +2388,7 @@ int mm_take_all_locks(struct mm_struct *mm) for_each_vma(vmi, vma) { if (signal_pending(current)) goto out_unlock; - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_lock_anon_vma(mm, avc->anon_vma); } @@ -2449,7 +2450,7 @@ void mm_drop_all_locks(struct mm_struct *mm) BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); for_each_vma(vmi, vma) { - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_unlock_anon_vma(avc->anon_vma); if (vma->vm_file && vma->vm_file->f_mapping) @@ -3610,7 +3611,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) * Similarly in do_mmap and in do_brk_flags. */ if (vma_is_anonymous(vma)) { - WARN_ON_ONCE(vma->anon_vma); + WARN_ON_ONCE(vma_has_anon_rmap(vma)); vma_set_pgoff(vma, vma->vm_start >> PAGE_SHIFT); } vma_set_anon_pgoff(vma, vma->vm_start >> PAGE_SHIFT); diff --git a/mm/vma.h b/mm/vma.h index ba897cb96e41..ec73a872091e 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -255,6 +255,33 @@ static inline pgoff_t vmg_end_pgoff(const struct vma_merge_struct *vmg) return vmg_start_pgoff(vmg) + vmg_pages(vmg); } +/** + * vma_has_anon_rmap() - does @vma possess an anonymous reverse mapping? + * @vma: The VMA to be checked. + * + * If the VMA is attached, a VMA or mmap lock must be held. + * + * This state is only possible for CoW mappings, see the comment for + * vma_flags_is_cow_mapping() for details. + * + * Importantly, a VMA which possesses an anonymous rmap may map anonymous + * folios. + * + * This function will not result in a false positive. + * + * However, if only a read lock is held, it may give a false negative, in which + * case it should be re-checked with mm->page_table_lock held. + * + * Returns: true if @vma has an anonymous reverse mapping, otherwise false. + */ +static inline bool vma_has_anon_rmap(const struct vm_area_struct *vma) +{ + if (vma_is_attached(vma)) + vma_assert_stabilised(vma); + /* KCSAN gets confused about the optimistic check. Silence it. */ + return data_race(vma->anon_vma); +} + static inline void assert_sane_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) { /* nommu doesn't set a virtual pgoff for anon VMAs. */ @@ -268,7 +295,7 @@ static inline void assert_sane_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) if (!vma_is_anonymous(vma)) return; /* If faulted in, could have been remapped. */ - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) return; /* OK this is really an anon VMA - expect virtual page offset. */ VM_WARN_ON_ONCE(pgoff != vma->vm_start >> PAGE_SHIFT); diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h index 83ff23226ddf..0e64d5416359 100644 --- a/tools/testing/vma/include/stubs.h +++ b/tools/testing/vma/include/stubs.h @@ -302,6 +302,10 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) { } +static inline void vma_assert_stabilised(const struct vm_area_struct *vma) +{ +} + static inline void ksm_add_vma(struct vm_area_struct *vma) { } From 14a5c975b1382225e283c867bb6d173cf90233cc Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 17 Sep 2026 20:47:43 +0100 Subject: [PATCH 133/213] mm: update comments to refer to anon rmap rather than anon_vma Now that vma_has_anon_rmap() abstracts whether a VMA has an anonymous reverse mapping, remove references to anon_vma and instead reference the anon rmap. The anon_vma is an implementation detail and should be treated as such. Do not update mm/rmap.c which implements the anon_vma mechanism as it is reasonable to directly reference it there. No functional change intended. [akpm@linux-foundation.org: fix comment, per Zi Yan] Link: https://lore.kernel.org/20260917-vma-is-faulted-v3-3-5c22314a72e7@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Acked-by: Pedro Falcato Reviewed-by: Lance Yang Cc: Alistair Popple Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Dev Jain Cc: Gregory Price Cc: Guilherme Giacomo Simoes Cc: Harry Yoo Cc: "Huang, Ying" Cc: Jann Horn Cc: Joshua Hahn Cc: Kairui Song Cc: Kemeng Shi Cc: Kiryl Shutsemau Cc: Liam R. Howlett Cc: Matthew Brost Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Oscar Salvador Cc: Peter Xu Cc: Rakie Kim Cc: Rik van Riel Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan --- mm/huge_memory.c | 8 ++-- mm/hugetlb.c | 2 +- mm/khugepaged.c | 12 +++--- mm/ksm.c | 6 +-- mm/madvise.c | 6 +-- mm/memory.c | 12 +++--- mm/migrate.c | 12 +++--- mm/mmap.c | 6 +-- mm/mprotect.c | 4 +- mm/mremap.c | 6 +-- mm/pgtable-generic.c | 2 +- mm/userfaultfd.c | 10 ++--- mm/vma.c | 98 ++++++++++++++++++++++---------------------- 13 files changed, 92 insertions(+), 92 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 194a1ed97865..bd44d58e8684 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -253,10 +253,10 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* * THPeligible bit of smaps should show 1 for proper VMAs even - * though anon_vma is not initialized yet. + * though they don't have an anon rmap yet. * - * Allow page fault since anon_vma may be not initialized until - * the first page fault. + * Allow page fault since the VMA may not have an anon rmap until the + * first page fault. */ if (!vma_has_anon_rmap(vma)) return (smaps || in_pf) ? orders : 0; @@ -4372,7 +4372,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, * THP pages in the middle of migration, due to allocation issues on either * side. * - * anon_vma_lock is not required to be held, mmap_read_lock() or + * The anon rmap lock is not required to be held, mmap_read_lock() or * mmap_write_lock() should be held. @folio is expected to be locked by the * caller. device-private and non device-private folios are supported along * with folios that are in the swapcache. @folio should also be unmapped and diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 58a807aa65ab..b41a4bd054f8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5692,7 +5692,7 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf) /* * When the original hugepage is shared one, it does not have - * anon_vma prepared. + * an anon rmap prepared. */ ret = __vmf_anon_prepare(vmf); if (unlikely(ret)) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 73a0a2249f67..911f562a232e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1365,7 +1365,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s /* * Prevent all access to pagetables with the exception of * gup_fast later handled by the pmdp_collapse_flush() and the VM - * handled by the anon_vma lock + folio lock. + * handled by the anon rmap lock + folio lock. * * UFFDIO_MOVE is prevented to race as well thanks to the * mmap_lock. @@ -1432,8 +1432,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s } /* - * For PMD collapse all pages are isolated and locked so anon_vma - * rmap can't run anymore. For mTHP collapse the PMD entry has been + * For PMD collapse all pages are isolated and locked so the anon + * rmap walk can't run anymore. For mTHP collapse the PMD entry has been * removed and not all pages are isolated and locked, so we must hold * the lock to prevent neighboring folios from attempting to access * this PMD until its reinstalled. @@ -2226,9 +2226,9 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) /* * Huge page lock is still held, so normally the page table must - * remain empty; and we have already skipped anon_vma and - * userfaultfd_wp() vmas. But since the mmap_lock is not held, - * it is still possible for a racing userfaultfd_ioctl() or + * remain empty; and we have already skipped vmas with an anon + * rmap and userfaultfd_wp() vmas. But since the mmap_lock is not + * held, it is still possible for a racing userfaultfd_ioctl() or * madvise() to have inserted ptes or markers. Now that we hold * ptlock, repeating the retractable checks protects us from * races against the prior checks. diff --git a/mm/ksm.c b/mm/ksm.c index fe0eb13e914d..dbfb8f98b189 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -796,7 +796,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item) /* * It is not an accident that whenever we want to break COW - * to undo, we also need to drop a reference to the anon_vma. + * to undo, we also need to drop a reference to the anon rmap. */ put_anon_vma(rmap_item->anon_vma); /* @@ -1416,7 +1416,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, goto out; /* * Some THP functions use the sequence pmdp_huge_clear_flush(), set_pmd_at() - * without holding anon_vma lock for write. So when looking for a + * without holding the anon rmap lock for write. So when looking for a * genuine pmde (in which to find pte), test present and !THP together. */ pmde = pmdp_get_lockless(pmd); @@ -1620,7 +1620,7 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item, /* * We can consider the VMA only while still holding the mmap lock, - * so lock, so reference the anon_vma and calculate the linear + * so reference the anon rmap and calculate the linear * page index early, before stable_tree_append(). If anything goes * wrong that prevents the rmap_item from being added to the * stable_tree, break_cow() will clean it up. diff --git a/mm/madvise.c b/mm/madvise.c index 6b48a3f6c559..8f407610c089 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1158,7 +1158,7 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior) /* * If anonymous and we are establishing page tables the VMA ought to - * have an anon_vma associated with it. + * have an anon rmap associated with it. * * We will hold an mmap read lock if this is necessary, this is checked * as part of the VMA lock logic. @@ -1626,8 +1626,8 @@ static bool is_vma_lock_sufficient(struct vm_area_struct *vma, * anon_vma_prepare() explicitly requires an mmap lock for * serialisation, so we cannot use a VMA lock in this case. * - * Note we might race with anon_vma being set, however this makes this - * check overly paranoid which is safe. + * Note we might race with the anon rmap being assigned, however this + * makes this check overly paranoid which is safe. */ if (vma_is_anonymous(vma) && prepares_anon_vma(madv_behavior->behavior) && !vma_has_anon_rmap(vma)) diff --git a/mm/memory.c b/mm/memory.c index c030c208749b..5d6d695675bc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1291,8 +1291,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, * copy_pmd_range()'s prior pmd_none_or_clear_bad(src_pmd), and the * error handling here, assume that exclusive mmap_lock on dst and src * protects anon from unexpected THP transitions; with shmem and file - * protected by mmap_lock-less collapse skipping areas with anon_vma - * (whereas vma_needs_copy() skips areas without anon_vma). A rework + * protected by mmap_lock-less collapse skipping areas with an anon rmap + * (whereas vma_needs_copy() skips areas without one). A rework * can remove such assumptions later, but this is good enough for now. */ dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl); @@ -1533,8 +1533,8 @@ vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) if (dst_vma->vm_flags & VM_COPY_ON_FORK) return true; /* - * The presence of an anon_vma indicates an anonymous VMA has page - * tables which naturally cannot be reconstituted on page fault. + * The presence of an anon rmap indicates the VMA may map anonymous + * folios which naturally cannot be reconstituted on page fault. */ if (vma_has_anon_rmap(src_vma)) return true; @@ -4001,10 +4001,10 @@ static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf) * * When preparing to insert an anonymous page into a VMA from a * fault handler, call this function rather than anon_vma_prepare(). - * If this vma does not already have an associated anon_vma and we are + * If this vma does not already have an anon rmap and we are * only protected by the per-VMA lock, the caller must retry with the * mmap_lock held. __anon_vma_prepare() will look at adjacent VMAs to - * determine if this VMA can share its anon_vma, and that's not safe to + * determine if this VMA can share its anon rmap, and that's not safe to * do with only the per-VMA lock held for this VMA. * * Return: 0 if fault handling can proceed. Any other value should be diff --git a/mm/migrate.c b/mm/migrate.c index 7e3a81f06974..7bdcdb57652f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1174,7 +1174,7 @@ static void migrate_folio_undo_src(struct folio *src, int was_mapped, { if (was_mapped) remove_migration_ptes(src, src, 0); - /* Drop an anon_vma reference if we took one */ + /* Drop an anon rmap reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); if (locked) @@ -1281,15 +1281,15 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, /* * By try_to_migrate(), src->mapcount goes down to 0 here. In this case, - * we cannot notice that anon_vma is freed while we migrate a page. - * This get_anon_vma() delays freeing anon_vma pointer until the end + * we cannot notice that the anon rmap is freed while we migrate a page. + * This get_anon_vma() delays freeing the anon rmap until the end * of migration. File cache pages are no problem because of page_lock() * File Caches may use write_page() or lock_page() in migration, then, * just care Anon page here. * * Only folio_get_anon_vma() understands the subtleties of - * getting a hold on an anon_vma from outside one of its mms. - * But if we cannot get anon_vma, then we won't need it anyway, + * getting a hold on an anon rmap from outside one of its mms. + * But if we cannot get the anon rmap, then we won't need it anyway, * because that implies that the anon page is no longer mapped * (and cannot be remapped so long as we hold the page lock). */ @@ -1432,7 +1432,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, * and will be freed. */ list_del(&src->lru); - /* Drop an anon_vma reference if we took one */ + /* Drop an anon rmap reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); diff --git a/mm/mmap.c b/mm/mmap.c index 98449f364af1..148ba01c0373 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -547,7 +547,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, } case MAP_PRIVATE: /* - * Set pgoff according to addr for anon_vma. + * Set pgoff according to addr for the anon rmap. */ pgoff = addr >> PAGE_SHIFT; break; @@ -1774,8 +1774,8 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) if (vma_test(tmp, VMA_WIPEONFORK_BIT)) { /* * VMA_WIPEONFORK_BIT gets a clean slate in the child. - * Don't prepare anon_vma until fault since we don't - * copy page for current vma. + * Don't prepare the anon rmap until fault since we + * don't copy pages for the current vma. */ tmp->anon_vma = NULL; } else if (anon_vma_fork(tmp, mpnt)) diff --git a/mm/mprotect.c b/mm/mprotect.c index 1318e44ef43e..754333698c1e 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -796,8 +796,8 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, /* * If we make a private mapping writable we increase our commit; * but (without finer accounting) cannot reduce our commit if we - * make it unwritable again except in the anonymous case where no - * anon_vma has yet to be assigned. + * make it unwritable again except in the anonymous case where the + * VMA's anon rmap has yet to be assigned. * * hugetlb mapping were accounted for even if read-only so there is * no need to account for them here. diff --git a/mm/mremap.c b/mm/mremap.c index 845f7b5844d4..6970783fc445 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -214,7 +214,7 @@ static int move_ptes(struct pagetable_move_control *pmc, int err = 0; /* - * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma + * When need_rmap_locks is true, we take the i_mmap_rwsem and anon rmap * locks to ensure that rmap will always observe either the old or the * new ptes. This is the easiest way to avoid races with * truncate_pagecache(), page migration, etc... @@ -1373,8 +1373,8 @@ static void dontunmap_complete(struct vma_remap_struct *vrm, vma_clear_flags_mask(vma, VMA_LOCKED_MASK); /* - * anon_vma links of the old vma is no longer needed after its page - * table has been moved. + * The anon rmap links of the old vma are no longer needed after its + * page table has been moved. */ if (start == old_start && end == old_end) { const pgoff_t pgoff_unfaulted = vma->vm_start >> PAGE_SHIFT; diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 26643d76bfb0..6e83ce3801b8 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -349,7 +349,7 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * pte_offset_map_lock(mm, pmd, addr, ptlp) is usually called with the pmd * pointer for addr, reached by walking down the mm's pgd, p4d, pud for addr: * either while holding mmap_lock or vma lock for read or for write; or in - * truncate or rmap context, while holding file's i_mmap_lock or anon_vma lock + * truncate or rmap context, while holding file's i_mmap_lock or anon rmap lock * for read (or for write). In a few cases, it may be used with pmd pointing to * a pmd_t already copied to or constructed on the stack. * diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 3103f36c9916..da67d35cd954 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -130,7 +130,7 @@ struct vm_area_struct *find_vma_and_prepare_anon(struct mm_struct *mm, * Should be called without holding mmap_lock. * * Return: A locked vma containing @address, -ENOENT if no vma is found, - * -ENOMEM if anon_vma couldn't be allocated, or -EAGAIN if vma refcount + * -ENOMEM if the anon rmap couldn't be allocated, or -EAGAIN if vma refcount * overflow happened due to high number of readers and the caller should * retry later. */ @@ -142,8 +142,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, vma = lock_vma_under_rcu(mm, address); if (vma) { /* - * We know we're going to need to use anon_vma, so check - * that early. + * We know we're going to need an anon rmap, so check that + * early. */ if (!(vma->vm_flags & VM_SHARED) && unlikely(!vma_has_anon_rmap(vma))) vma_end_read(vma); @@ -1680,7 +1680,7 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd /* * Verify the existence of the swapcache. If present, the folio's * index and mapping must be updated even when the PTE is a swap - * entry. The anon_vma lock is not taken during this process since + * entry. The anon rmap lock is not taken during this process since * the folio has already been unmapped, and the swap entry is * exclusive, preventing rmap walks. * @@ -1915,7 +1915,7 @@ static void uffd_move_unlock(struct vm_area_struct *dst_vma, * * move_pages() remaps arbitrary anonymous pages atomically in zero * copy. It only works on non shared anonymous pages because those can - * be relocated without generating non linear anon_vmas in the rmap + * be relocated without generating non linear anon rmaps in the rmap * code. * * It provides a zero copy mechanism to handle userspace page faults. diff --git a/mm/vma.c b/mm/vma.c index 0b9abf3a85f9..152c405643d2 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -97,7 +97,7 @@ static bool vma_is_fork_child(struct vm_area_struct *vma) { /* * The list_is_singular() test is to avoid merging VMA cloned from - * parents. This can improve scalability caused by the anon_vma root + * parents. This can improve scalability caused by the anon rmap root * lock. */ return vma && vma_has_anon_rmap(vma) && @@ -135,7 +135,7 @@ static bool is_mergeable_anon_vma(struct vma_merge_struct *vmg, bool merge_next) /* * We _can_ have !src, vmg->anon_vma via copy_vma(). In this instance we - * will remove the existing VMA's anon_vma's so there's no scalability + * will remove the existing VMA's anon rmap so there's no scalability * concerns. */ VM_WARN_ON(src && src_anon != src->anon_vma); @@ -151,10 +151,10 @@ static bool is_mergeable_anon_vma(struct vma_merge_struct *vmg, bool merge_next) return true; } - /* Case 2 - we will simply use tgt's anon_vma. */ + /* Case 2 - we will simply use tgt's anon rmap. */ if (vma_has_anon_rmap(tgt) && !src_anon) return !vma_is_fork_child(tgt); - /* Case 3 - the anon_vma's are already shared. */ + /* Case 3 - src and tgt already share an anon rmap. */ return src_anon == tgt_anon; } @@ -228,8 +228,8 @@ static bool needs_adjacent_anon_pgoff(const struct vma_merge_struct *vmg) * Return true if we can merge this (vma_flags,anon_vma,file,vm_pgoff) * in front of (at a lower virtual address and file offset than) the vma. * - * We cannot merge two vmas if they have differently assigned (non-NULL) - * anon_vmas, nor if same anon_vma is assigned but offsets incompatible. + * We cannot merge two vmas if they have differently assigned anon rmaps, + * nor if the same anon rmap is assigned but offsets incompatible. * * We don't check here for the merged mmap wrapping around the end of pagecache * indices (16TB on ia32) because do_mmap() does not permit mmap's which @@ -255,8 +255,8 @@ static bool can_vma_merge_before(struct vma_merge_struct *vmg) * Return true if we can merge this (vma_flags,anon_vma,file,vm_pgoff) * beyond (at a higher virtual address and file offset than) the vma. * - * We cannot merge two vmas if they have differently assigned (non-NULL) - * anon_vmas, nor if same anon_vma is assigned but offsets incompatible. + * We cannot merge two vmas if they have differently assigned anon rmaps, + * nor if the same anon rmap is assigned but offsets incompatible. * * We assume that vma is not removed as part of the merge. */ @@ -300,18 +300,18 @@ static void __remove_shared_vm_struct(struct vm_area_struct *vma, } /* - * vma has some anon_vma assigned, and is already inserted on that - * anon_vma's interval trees. + * vma has an anon rmap assigned, and is already inserted on its interval + * trees. * * Before updating the vma's vm_start / vm_end / vm_pgoff fields, the - * vma must be removed from the anon_vma's interval trees using + * vma must be removed from the anon rmap's interval trees using * anon_rmap_tree_pre_update_vma(). * * After the update, the vma will be reinserted using * anon_rmap_tree_post_update_vma(). * * The entire update must be protected by exclusive mmap_lock and by - * the root anon_vma's mutex. + * the anon rmap root lock. */ static void anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma) @@ -479,7 +479,7 @@ static bool can_vma_merge_left(struct vma_merge_struct *vmg) * account the end position of the proposed range. * * In addition, if we can merge with the left VMA, ensure that left and right - * anon_vma's are also compatible. + * anon rmaps are also compatible. */ static bool can_vma_merge_right(struct vma_merge_struct *vmg, bool can_merge_left) @@ -495,7 +495,7 @@ static bool can_vma_merge_right(struct vma_merge_struct *vmg, /* * If we can merge with prev (left) and next (right), indicating that - * each VMA's anon_vma is compatible with the proposed anon_vma, this + * each VMA's anon rmap is compatible with the proposed anon rmap, this * does not mean prev and next are compatible with EACH OTHER. * * We therefore check this in addition to mergeability to either side. @@ -645,8 +645,8 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, } /* - * dup_anon_vma() - Helper function to duplicate anon_vma on VMA merge in the - * instance that the destination VMA has no anon_vma but the source does. + * dup_anon_vma() - Helper function to duplicate the anon rmap on VMA merge in + * the instance that the destination VMA has no anon rmap but the source does. * * @dst: The destination VMA * @src: The source VMA @@ -659,17 +659,17 @@ static int dup_anon_vma(struct vm_area_struct *dst, { /* * There are three cases to consider for correctly propagating - * anon_vma's on merge. + * anon rmaps on merge. * - * The first is trivial - neither VMA has anon_vma, we need not do + * The first is trivial - neither VMA has an anon rmap, we need not do * anything. * - * The second where both have anon_vma is also a no-op, as they must + * The second where both have an anon rmap is also a no-op, as they must * then be the same, so there is simply nothing to copy. * - * Here we cover the third - if the destination VMA has no anon_vma, + * Here we cover the third - if the destination VMA has no anon rmap, * that is it is unfaulted, we need to ensure that the newly merged - * range is referenced by the anon_vma's of the source. + * range is referenced by the anon rmap of the source. */ if (vma_has_anon_rmap(src) && !vma_has_anon_rmap(dst)) { int ret; @@ -1017,9 +1017,9 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( vmg->anon_pgoff = vma_start_anon_pgoff(prev); /* - * We already ensured anon_vma compatibility above, so now it's - * simply a case of, if prev has no anon_vma object, which of - * next or middle contains the anon_vma we must duplicate. + * We already ensured anon rmap compatibility above, so now it's + * simply a case of, if prev has no anon rmap, which of next or + * middle contains the anon rmap we must duplicate. */ err = dup_anon_vma(prev, vma_has_anon_rmap(next) ? next : middle, &anon_dup); @@ -1085,7 +1085,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( unlink_anon_vmas(anon_dup); /* - * This means we have failed to clone anon_vma's correctly, but no + * This means we have failed to clone the anon rmap correctly, but no * actual changes to VMAs have occurred, so no harm no foul - if the * user doesn't want this reported and instead just wants to give up on * the merge, allow it. @@ -1282,7 +1282,7 @@ int vma_expand(struct vma_merge_struct *vmg) /* * If we are removing the next VMA or copying from a VMA - * (e.g. mremap()'ing), we must propagate anon_vma state. + * (e.g. mremap()'ing), we must propagate anon rmap state. * * Note that, by convention, callers ignore OOM for this case, so * we don't need to account for vmg->give_up_on_mm here. @@ -2059,16 +2059,16 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, /* * Rough compatibility check to quickly see if it's even worth looking - * at sharing an anon_vma. + * at sharing an anon rmap. * * They need to have the same vm_file, and the flags can only differ * in things that mprotect may change. * - * NOTE! The fact that we share an anon_vma doesn't _have_ to mean that + * NOTE! The fact that we share an anon rmap doesn't _have_ to mean that * we can merge the two vma's. For example, we refuse to merge a vma if * there is a vm_ops->close() function, because that indicates that the * driver is doing some kind of reference counting. But that doesn't - * really matter for the anon_vma sharing case. + * really matter for the anon rmap sharing case. */ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct *b) { @@ -2101,13 +2101,13 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * } /* - * Do some basic sanity checking to see if we can re-use the anon_vma + * Do some basic sanity checking to see if we can re-use the anon rmap * from 'old'. The 'a'/'b' vma's are in VM order - one of them will be * the same as 'old', the other will be the new one that is trying - * to share the anon_vma. + * to share the anon rmap. * * NOTE! This runs with mmap_lock held for reading, so it is possible that - * the anon_vma of 'old' is concurrently in the process of being set up + * the anon rmap of 'old' is concurrently in the process of being set up * by another page fault trying to merge _that_. But that's ok: if it * is being set up, that automatically means that it will be a singleton * acceptable for merging, so we can do all of this optimistically. But @@ -2121,8 +2121,8 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * * accessing an uninitialised anon_vma's fields may result in a UAF. * * IOW: that the "list_is_singular()" test on the anon_vma_chain only - * matters for the 'stable anon_vma' case (ie the thing we want to avoid - * is to return an anon_vma that is "complex" due to having gone through + * matters for the 'stable anon rmap' case (ie the thing we want to avoid + * is to return an anon rmap that is "complex" due to having gone through * a fork). * * We also make sure that the two vma's are compatible (adjacent, @@ -2145,10 +2145,10 @@ static struct anon_vma *reusable_anon_vma(struct vm_area_struct *old, /* * find_mergeable_anon_vma is used by anon_vma_prepare, to check - * neighbouring vmas for a suitable anon_vma, before it goes off - * to allocate a new anon_vma. It checks because a repetitive + * neighbouring vmas for a suitable anon rmap, before it goes off + * to allocate a new anon rmap. It checks because a repetitive * sequence of mprotects and faults may otherwise lead to distinct - * anon_vmas being allocated, preventing vma merge in subsequent + * anon rmaps being allocated, preventing vma merge in subsequent * mprotect. */ struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma) @@ -2174,13 +2174,13 @@ struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma) /* * We might reach here with anon_vma == NULL if we can't find - * any reusable anon_vma. + * any reusable anon rmap. * There's no absolute need to look only at touching neighbours: - * we could search further afield for "compatible" anon_vmas. + * we could search further afield for "compatible" anon rmaps. * But it would probably just be a waste of time searching, - * or lead to too many vmas hanging off the same anon_vma. + * or lead to too many vmas hanging off the same anon rmap. * We're trying to allow mprotect remerging later on, - * not trying to minimize memory used for anon_vmas. + * not trying to minimize memory used for anon rmaps. */ return anon_vma; } @@ -2276,7 +2276,7 @@ static void vm_lock_anon_vma(struct mm_struct *mm, struct anon_vma *anon_vma) /* * We can safely modify head.next after taking the * anon_vma->root->rwsem. If some other vma in this mm shares - * the same anon_vma we won't take it again. + * the same anon rmap we won't take it again. * * No need of atomic instructions here, head.next * can't change from under us thanks to the @@ -2318,14 +2318,14 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * mmap_lock in write mode is required in order to block all operations * that could modify pagetables and free pages without need of * altering the vma layout. It's also needed in write mode to avoid new - * anon_vmas to be associated with existing vmas. + * anon rmaps being associated with existing vmas. * * A single task can't take more than one mm_take_all_locks() in a row * or it would deadlock. * * The LSB in anon_vma->rb_root.rb_node and the AS_MM_ALL_LOCKS bitflag in * mapping->flags avoid to take the same lock twice, if more than one - * vma in this mm is backed by the same anon_vma or address_space. + * vma in this mm is backed by the same anon rmap or address_space. * * We take locks in following order, accordingly to comment at beginning * of mm/rmap.c: @@ -3432,7 +3432,7 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) if (next && vma_is_accessible(next)) { if (!vma_test(next, VMA_GROWSUP_BIT)) return -ENOMEM; - /* Check that both stack segments have the same anon_vma? */ + /* Check that both stack segments have the same anon rmap? */ } if (next) @@ -3442,7 +3442,7 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) if (vma_iter_prealloc(&vmi, vma)) return -ENOMEM; - /* We must make sure the anon_vma is allocated. */ + /* We must make sure the anon rmap is allocated. */ if (unlikely(anon_vma_prepare(vma))) { vma_iter_free(&vmi); return -ENOMEM; @@ -3505,7 +3505,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) /* Enforce stack_guard_gap */ prev = vma_prev(&vmi); - /* Check that both stack segments have the same anon_vma? */ + /* Check that both stack segments have the same anon rmap? */ if (prev) { if (!vma_test(prev, VMA_GROWSDOWN_BIT) && vma_is_accessible(prev) && @@ -3520,7 +3520,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) if (vma_iter_prealloc(&vmi, vma)) return -ENOMEM; - /* We must make sure the anon_vma is allocated. */ + /* We must make sure the anon rmap is allocated. */ if (unlikely(anon_vma_prepare(vma))) { vma_iter_free(&vmi); return -ENOMEM; @@ -3600,7 +3600,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) /* * The vm_pgoff of a purely anonymous vma should be irrelevant - * until its first write fault, when page's anon_vma and index + * until its first write fault, when page's anon rmap and index * are set. But now set the vm_pgoff it will almost certainly * end up with (unless mremap moves it elsewhere before that * first wfault), so /proc/pid/maps tells a consistent story. From dc0b17144d62c1e05043bd11ef2bf5c23cc3b500 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Wed, 30 Sep 2026 18:53:36 +0100 Subject: [PATCH 134/213] mm/vma: don't remove VMA from rmap if pgoff unchanged When updating a VMA, vma_prepare() unconditionally removes it from its rmap interval trees under the rmap lock, and vma_complete() reinserts it before releasing the lock. This is wholly unnecessary if its page offset (file rmap) or anonymous page offset (anon rmap) is unchanged. So, track whether they will change in the newly introduced vp->file_pgoff_unchanged and vp->anon_pgoff_unchanged fields, and use them to determine whether to remove the VMA or not. The rmap lock keeps things safe as no rmap walks can concurrently occur during the operation. Additionally, some architectures (arm, parisc, nios2, csky) have dcache flush rmap walkers which take only flush_dcache_mmap_lock(), which is likewise held across the operation. If the VMA remains in the tree, it's necessary to keep the augmented rb_subtree_last field updated to reflect its changed range. Provide mapping_rmap_tree_[pre, post]_update() and anon_rmap_tree_[pre, post]_update_vma() (replacing the existing logic in the anonymous case) to handle both the changed and unchanged cases. For the anon rmap case, with CONFIG_DEBUG_VM_RB set, avc->cached_vma_last is also updated when propagating in place. When performing a VMA shrink or a split where the VMA is the lower one, the page offset cannot change, so set the flags unconditionally in these cases. When merging VMAs the page offset is unchanged only in some cases, so update init_multi_vma_prep() to set the flags only if the page offsets remain the same. Finally, while we're here, also update expand_upwards() similarly. These changes ultimately result in less rmap lock contention. Pan Deng reported results using the UnixBench/excel benchmark on a 2-socket 192 core, 384 thread x86-64 system for v7.3-rc4 with/without the patch applied: Execl Throughput, index score: avg %stdev min max v7.3-rc4 3511.5 0.44% 3494.5 3543.4 + patch 4069.0 0.48% 4047.4 4109.5 (+15.9%) Average wait on file rmap lock in ms, 5 runs per kernel: avg %stdev min max v7.3-rc4 9.470 2.91% 9.070 9.820 + patch 8.420 3.34% 8.100 8.770 (-11.1%) Profiling data obtained during the operation highlighted the file rmap lock as the primary source of contention. Link: https://lore.kernel.org/20260930-speed-up-inplace-rmap-v2-1-ac1aa19708aa@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Suggested-by: Pan Deng Reviewed-by: Rik van Riel Acked-by: Lance Yang Tested-by: Lance Yang Reviewed-by: Pedro Falcato Cc: David Hildenbrand Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Harry Yoo Cc: Jann Horn --- include/linux/mm.h | 10 +++ mm/interval_tree.c | 101 ++++++++++++++++++++++++++++++ mm/vma.c | 74 +++++++++------------- mm/vma.h | 2 + tools/testing/vma/include/stubs.h | 22 +++++++ 5 files changed, 164 insertions(+), 45 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3f95442667a9..6756f7fc1806 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4330,6 +4330,12 @@ void mapping_rmap_tree_insert_after(struct vm_area_struct *vma, struct address_space *mapping); void mapping_rmap_tree_remove(struct vm_area_struct *vma, struct address_space *mapping); +void mapping_rmap_tree_pre_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged); +void mapping_rmap_tree_post_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged); struct vm_area_struct * mapping_rmap_tree_iter_first(struct address_space *mapping, pgoff_t pgoff_start, pgoff_t pgoff_last); @@ -4347,6 +4353,10 @@ void anon_rmap_tree_insert(struct anon_vma_chain *avc, struct anon_vma *anon_vma); void anon_rmap_tree_remove(struct anon_vma_chain *avc, struct anon_vma *anon_vma); +void anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged); +void anon_rmap_tree_post_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged); struct anon_vma_chain * anon_rmap_tree_iter_first(struct anon_vma *anon_vma, pgoff_t pgoff_start, pgoff_t pgoff_last); diff --git a/mm/interval_tree.c b/mm/interval_tree.c index 7bbbf15cfbf0..9e24bb99fd80 100644 --- a/mm/interval_tree.c +++ b/mm/interval_tree.c @@ -64,6 +64,51 @@ void mapping_rmap_tree_remove(struct vm_area_struct *vma, __mapping_rmap_tree_remove(vma, &mapping->i_mmap); } +static void mapping_rmap_tree_update_inplace(struct vm_area_struct *vma) +{ + /* Propagate all the way up the tree. */ + __mapping_rmap_tree_augment.propagate(&vma->shared.rb, NULL); +} + +/** + * mapping_rmap_tree_pre_update() - Prepare the file rmap tree for a change to + * be made to @vma. + * @vma: The VMA about to be updated. + * @mapping: The file rmap to which @vma belongs. + * @pgoff_unchanged: Whether @vma's page offset will remain unchanged. + * + * The file rmap lock must be held across the entire update. + */ +void mapping_rmap_tree_pre_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ + /* If the pgoff has changed, then remove and reinsert afterwards. */ + if (!pgoff_unchanged) + mapping_rmap_tree_remove(vma, mapping); +} + +/** + * mapping_rmap_tree_post_update() - Update the file rmap tree to reflect a + * change that has been made to @vma. + * @vma: The VMA that has been updated. + * @mapping: The file rmap to which @vma belongs. + * @pgoff_unchanged: Whether @vma's page offset remained unchanged. + * + * mapping_rmap_tree_pre_update() must have been called prior to this. + * + * The file rmap lock must be held across the entire update. + */ +void mapping_rmap_tree_post_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ + if (pgoff_unchanged) + mapping_rmap_tree_update_inplace(vma); + else + mapping_rmap_tree_insert(vma, mapping); +} + struct vm_area_struct * mapping_rmap_tree_iter_first(struct address_space *mapping, pgoff_t pgoff_start, pgoff_t pgoff_last) @@ -111,6 +156,62 @@ void anon_rmap_tree_remove(struct anon_vma_chain *avc, __anon_rmap_tree_remove(avc, &anon_vma->rb_root); } +static void anon_rmap_tree_update_inplace(struct anon_vma_chain *avc) +{ +#ifdef CONFIG_DEBUG_VM_RB + avc->cached_vma_last = avc_last_pgoff(avc); +#endif + /* Propagate all the way up the tree. */ + __anon_rmap_tree_augment.propagate(&avc->rb, NULL); +} + +/** + * anon_rmap_tree_pre_update_vma() - Prepare the anon rmap trees for a change + * to be made to @vma. + * @vma: The VMA about to be updated, which has an anon rmap assigned and is + * already inserted on its interval trees. + * @anon_pgoff_unchanged: Whether @vma's anonymous page offset will remain + * unchanged. + * + * The anon rmap lock must be held across the entire update. + */ +void anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ + struct anon_vma_chain *avc; + + if (anon_pgoff_unchanged) + return; + + /* If the pgoff has changed, then remove and reinsert afterwards. */ + list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) + anon_rmap_tree_remove(avc, avc->anon_vma); +} + +/** + * anon_rmap_tree_post_update_vma() - Update the anon rmap trees to reflect a + * change that has been made to @vma. + * @vma: The VMA that has been updated. + * @anon_pgoff_unchanged: Whether @vma's anonymous page offset remained + * unchanged. + * + * anon_rmap_tree_pre_update_vma() must have been called prior to this. + * + * The anon rmap lock must be held across the entire update. + */ +void anon_rmap_tree_post_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ + struct anon_vma_chain *avc; + + list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) { + if (anon_pgoff_unchanged) + anon_rmap_tree_update_inplace(avc); + else + anon_rmap_tree_insert(avc, avc->anon_vma); + } +} + struct anon_vma_chain * anon_rmap_tree_iter_first(struct anon_vma *anon_vma, pgoff_t pgoff_start, pgoff_t pgoff_last) diff --git a/mm/vma.c b/mm/vma.c index 152c405643d2..de5ebc57e7a5 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -201,8 +201,15 @@ static void init_multi_vma_prep(struct vma_prepare *vp, if (vp->file) vp->mapping = vma->vm_file->f_mapping; - if (vmg && vmg->skip_vma_uprobe) + if (!vmg) + return; + + if (vmg->skip_vma_uprobe) vp->skip_vma_uprobe = true; + if (vma_start_pgoff(vma) == vmg_start_pgoff(vmg)) + vp->file_pgoff_unchanged = true; + if (vma_start_anon_pgoff(vma) == vmg_start_anon_pgoff(vmg)) + vp->anon_pgoff_unchanged = true; } /* @@ -299,38 +306,6 @@ static void __remove_shared_vm_struct(struct vm_area_struct *vma, flush_dcache_mmap_unlock(mapping); } -/* - * vma has an anon rmap assigned, and is already inserted on its interval - * trees. - * - * Before updating the vma's vm_start / vm_end / vm_pgoff fields, the - * vma must be removed from the anon rmap's interval trees using - * anon_rmap_tree_pre_update_vma(). - * - * After the update, the vma will be reinserted using - * anon_rmap_tree_post_update_vma(). - * - * The entire update must be protected by exclusive mmap_lock and by - * the anon rmap root lock. - */ -static void -anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma) -{ - struct anon_vma_chain *avc; - - list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) - anon_rmap_tree_remove(avc, avc->anon_vma); -} - -static void -anon_rmap_tree_post_update_vma(struct vm_area_struct *vma) -{ - struct anon_vma_chain *avc; - - list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) - anon_rmap_tree_insert(avc, avc->anon_vma); -} - /* * vma_prepare() - Helper function for handling locking VMAs prior to altering * @vp: The initialized vma_prepare struct @@ -359,16 +334,19 @@ static void vma_prepare(struct vma_prepare *vp) if (vp->anon_vma) { anon_vma_lock_write(vp->anon_vma); - anon_rmap_tree_pre_update_vma(vp->vma); + anon_rmap_tree_pre_update_vma(vp->vma, vp->anon_pgoff_unchanged); + /* The adjacent VMA's start is moved, so its page offset changes. */ if (vp->adj_next) - anon_rmap_tree_pre_update_vma(vp->adj_next); + anon_rmap_tree_pre_update_vma(vp->adj_next, false); } if (vp->file) { flush_dcache_mmap_lock(vp->mapping); - mapping_rmap_tree_remove(vp->vma, vp->mapping); + mapping_rmap_tree_pre_update(vp->vma, vp->mapping, + vp->file_pgoff_unchanged); if (vp->adj_next) - mapping_rmap_tree_remove(vp->adj_next, vp->mapping); + mapping_rmap_tree_pre_update(vp->adj_next, vp->mapping, + false); } } @@ -386,8 +364,10 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, { if (vp->file) { if (vp->adj_next) - mapping_rmap_tree_insert(vp->adj_next, vp->mapping); - mapping_rmap_tree_insert(vp->vma, vp->mapping); + mapping_rmap_tree_post_update(vp->adj_next, vp->mapping, + false); + mapping_rmap_tree_post_update(vp->vma, vp->mapping, + vp->file_pgoff_unchanged); flush_dcache_mmap_unlock(vp->mapping); } @@ -406,9 +386,9 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, } if (vp->anon_vma) { - anon_rmap_tree_post_update_vma(vp->vma); + anon_rmap_tree_post_update_vma(vp->vma, vp->anon_pgoff_unchanged); if (vp->adj_next) - anon_rmap_tree_post_update_vma(vp->adj_next); + anon_rmap_tree_post_update_vma(vp->adj_next, false); anon_vma_unlock_write(vp->anon_vma); } @@ -593,6 +573,8 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, init_vma_prep(&vp, vma); vp.insert = new; + vp.file_pgoff_unchanged = !new_below; + vp.anon_pgoff_unchanged = !new_below; vma_prepare(&vp); /* @@ -1346,6 +1328,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, vma_start_write(vma); init_vma_prep(&vp, vma); + vp.file_pgoff_unchanged = true; + vp.anon_pgoff_unchanged = true; vma_prepare(&vp); vma_adjust_trans_huge(vma, vma->vm_start, end, NULL); @@ -3466,11 +3450,11 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) if (vma_test(vma, VMA_LOCKED_BIT)) mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); - anon_rmap_tree_pre_update_vma(vma); + anon_rmap_tree_pre_update_vma(vma, true); vma->vm_end = address; /* Overwrite old entry in mtree. */ vma_iter_store_overwrite(&vmi, vma); - anon_rmap_tree_post_update_vma(vma); + anon_rmap_tree_post_update_vma(vma, true); perf_event_mmap(vma); } @@ -3543,12 +3527,12 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) if (vma_test(vma, VMA_LOCKED_BIT)) mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); - anon_rmap_tree_pre_update_vma(vma); + anon_rmap_tree_pre_update_vma(vma, false); vma->vm_start = address; vma_sub_pgoff(vma, grow); /* Overwrite old entry in mtree. */ vma_iter_store_overwrite(&vmi, vma); - anon_rmap_tree_post_update_vma(vma); + anon_rmap_tree_post_update_vma(vma, false); perf_event_mmap(vma); } diff --git a/mm/vma.h b/mm/vma.h index ec73a872091e..3fc3b47a954b 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -28,6 +28,8 @@ struct vma_prepare { struct vm_area_struct *remove2; bool skip_vma_uprobe :1; + bool file_pgoff_unchanged :1; + bool anon_pgoff_unchanged :1; }; struct unlink_vma_file_batch { diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h index 0e64d5416359..be8f5161f92a 100644 --- a/tools/testing/vma/include/stubs.h +++ b/tools/testing/vma/include/stubs.h @@ -267,6 +267,18 @@ static inline void mapping_rmap_tree_remove(struct vm_area_struct *vma, { } +static inline void mapping_rmap_tree_pre_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ +} + +static inline void mapping_rmap_tree_post_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ +} + static inline void flush_dcache_mmap_unlock(struct address_space *mapping) { } @@ -281,6 +293,16 @@ static inline void anon_rmap_tree_remove(struct anon_vma_chain *avc, { } +static inline void anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ +} + +static inline void anon_rmap_tree_post_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ +} + static inline void uprobe_mmap(struct vm_area_struct *vma) { } From ae2514c95518b39b36fa463d855c74f29c91d064 Mon Sep 17 00:00:00 2001 From: Ren Tamura Date: Wed, 30 Sep 2026 15:03:17 +0900 Subject: [PATCH 135/213] tools/cgroup: sum shrinker object counts across NUMA nodes Each row of a memcg-aware shrinker's debugfs count file contains the cgroup ID followed by one object count per NUMA node. memcg_shrinker.py uses only the first count when sorting and displaying entries. This undercounts objects on NUMA systems. An entry with counts of 0 and 23 is treated as zero and omitted from the output, even though its total is 23. Sum all count fields after the cgroup ID so that the displayed totals and sort order account for every node. Single-node input retains the same result. Link: https://lore.kernel.org/179074810112.139422.9274126921568167859@gmail.com Fixes: d261ea23533b ("tools: add memcg_shrinker.py") Signed-off-by: Ren Tamura Signed-off-by: Andrew Morton Assisted-by: LLM Cc: Roman Gushchin --- tools/cgroup/memcg_shrinker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/cgroup/memcg_shrinker.py b/tools/cgroup/memcg_shrinker.py index e81c3017ada9..714772449c6b 100644 --- a/tools/cgroup/memcg_shrinker.py +++ b/tools/cgroup/memcg_shrinker.py @@ -31,7 +31,7 @@ def scan_shrinkers(shrinker_debugfs): items = line.split(' ') ino = int(items[0]) # (count, shrinker, memcg ino) - shrinkers.append((int(items[1]), shrinker, ino)) + shrinkers.append((sum(map(int, items[1:])), shrinker, ino)) return shrinkers From f9168f2f0b06b22dcfe75b2f4c677c8017d95bb5 Mon Sep 17 00:00:00 2001 From: Alexandre Ghiti Date: Thu, 1 Oct 2026 10:57:12 +0200 Subject: [PATCH 136/213] mm/swap: submit the last readahead batch before unplugging Since block swap I/O is batched in struct swap_iocb, swap_add_folio() only appends a folio to the pending batch when its slot directly follows the previous one, and submits the pending batch otherwise. Batches submitted inside the readahead loop go through the plug, where the block layer can still merge them, but the final batch is submitted after blk_finish_plug() and always becomes a separate request. So submit the final batch before finishing the plug. Link: https://lore.kernel.org/20261001085730.2029773-1-alex@ghiti.fr Fixes: dda8fb68b590 ("mm/swap: also use struct swap_iocb for block I/O") Signed-off-by: Alexandre Ghiti Signed-off-by: Andrew Morton Reviewed-by: Joshua Hahn Reviewed-by: Christoph Hellwig Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Barry Song Cc: Youngjun Park --- mm/swap_state.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 2475ba29126d..ebef568cd62e 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -906,8 +906,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, continue; folio_put(folio); } - blk_finish_plug(&plug); swap_read_submit(&ctx); + blk_finish_plug(&plug); skip: return swap_cache_read_folio_sync(entry, gfp_mask, mpol, ilx); } @@ -1019,8 +1019,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, } if (pte) pte_unmap(pte); - blk_finish_plug(&plug); swap_read_submit(&ctx); + blk_finish_plug(&plug); skip: /* The folio was likely read above, so no need for plugging here */ return swap_cache_read_folio_sync(targ_entry, gfp_mask, mpol, targ_ilx); From 879cd218870a1ae3e0177989207760a50a2fbb7c Mon Sep 17 00:00:00 2001 From: Enze Li Date: Sat, 3 Oct 2026 14:00:28 +0800 Subject: [PATCH 137/213] MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM The HUGETLB SUBSYSTEM F: entries list include/linux/hugetlb.h explicitly, but other hugetlb headers are not covered, so get_maintainer.pl does not attribute them to the hugetlb maintainers. Enumerating headers by hand is also fragile: any header not listed (such as hugetlb_cgroup.h) falls outside the subsystem. Joe Perches suggested using a glob F: entry, which covers all current and future hugetlb headers in one line and avoids the gap. Link: https://lore.kernel.org/20261003060029.2152489-1-lienze@kylinos.cn Signed-off-by: Enze Li Signed-off-by: Andrew Morton Suggested-by: Joe Perches Acked-by: David Hildenbrand (Arm) Reviewed-by: SJ Park Acked-by: Muchun Song Cc: Oscar Salvador --- MAINTAINERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index d2fc3d8364b3..39abe7378b6f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12097,7 +12097,7 @@ F: Documentation/admin-guide/mm/hugetlbpage.rst F: Documentation/mm/hugetlbfs_reserv.rst F: Documentation/mm/vmemmap_dedup.rst F: fs/hugetlbfs/ -F: include/linux/hugetlb.h +F: include/linux/hugetlb* F: include/linux/vmemmap-optimization.h F: include/trace/events/hugetlbfs.h F: mm/hugetlb.c From b3d752e23cbccbc4bc56a1673655c621eb5f3710 Mon Sep 17 00:00:00 2001 From: Enze Li Date: Sat, 3 Oct 2026 14:00:29 +0800 Subject: [PATCH 138/213] mm/hugetlb_cgroup: add trailing #endif comment for include guard The final #endif that closes the _LINUX_HUGETLB_CGROUP_H include guard is left without a comment. Add the macro name to the comment for consistency and easier navigation. Link: https://lore.kernel.org/20261003060029.2152489-2-lienze@kylinos.cn Signed-off-by: Enze Li Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Acked-by: Muchun Song Reviewed-by: SJ Park Cc: Oscar Salvador --- include/linux/hugetlb_cgroup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/hugetlb_cgroup.h b/include/linux/hugetlb_cgroup.h index 16d72c8c71f6..b4e0958eabf3 100644 --- a/include/linux/hugetlb_cgroup.h +++ b/include/linux/hugetlb_cgroup.h @@ -268,4 +268,4 @@ static inline void hugetlb_cgroup_migrate(struct folio *old_folio, } #endif /* CONFIG_CGROUP_HUGETLB */ -#endif +#endif /* _LINUX_HUGETLB_CGROUP_H */ From 8930760d468d94b47b013bd5f0256ff2e1776b78 Mon Sep 17 00:00:00 2001 From: Aveline Noir Date: Wed, 30 Sep 2026 23:00:27 -0700 Subject: [PATCH 139/213] selftests/mm: mrelease_test: fix retry limit mrelease_test doubles the child allocation when process_mrelease() returns ESRCH. When size is already MAX_SIZE_MB, the current <= check still allows another retry, causing the allocation to grow from 1024 MB to 2048 MB. Use < instead of <= so the largest allocation attempted remains MAX_SIZE_MB. LLM used in discovering bug. Changes were made and reviewed manually. Link: https://lore.kernel.org/CANOyQmFzsssM_BXHUDrV+UuVD5SZMBmSkg3UQnmw9Ns1PV7GCQ@mail.gmail.com Fixes: 33776141b812 ("selftests: vm: add process_mrelease tests") Signed-off-by: Aveline Noir Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: SJ Park Assisted-by: LLM Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Michal Hocko Cc: Christian Brauner Cc: Muhammad Usama Anjum --- tools/testing/selftests/mm/mrelease_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/mrelease_test.c b/tools/testing/selftests/mm/mrelease_test.c index 64e8d00ae944..a5124c9e9f43 100644 --- a/tools/testing/selftests/mm/mrelease_test.c +++ b/tools/testing/selftests/mm/mrelease_test.c @@ -157,7 +157,7 @@ int main(void) * Retry until we succeed or reach MAX_SIZE_MB. */ if (errno == ESRCH) { - retry = (size <= MAX_SIZE_MB); + retry = (size < MAX_SIZE_MB); } else { waitpid(pid, NULL, 0); ksft_exit_fail_msg("process_mrelease: %s\n", strerror(errno)); From 101fea09a860c7cd3ef53faefaab1f4a03132446 Mon Sep 17 00:00:00 2001 From: Dima Koziuk Date: Fri, 2 Oct 2026 23:05:07 +0300 Subject: [PATCH 140/213] mm: kmsan: fix iounmap metadata teardown kmsan_vmalloc_to_page_or_null() rejects addresses outside the regular vmalloc and module ranges, including the shadow and origin addresses passed by its callers. As a result, iounmap does not free the metadata backing pages. Also, the first iteration of the teardown loop unmaps the entire metadata range, preventing subsequent iterations from finding their pages even if the address lookup is fixed. Factor the page-table walk out of vmalloc_to_page() into __vmalloc_to_page(), keeping the address check in the public wrapper. Use the unchecked helper in kmsan_vmalloc_to_page_or_null() and check for NULL before converting the returned page to a PFN. Mark __vmalloc_to_page() as __always_inline to avoid an additional function call in vmalloc_to_page(). Move metadata teardown into kmsan_iounmap_pages(). Free the backing blocks while their mappings are still available, then unmap each metadata range once and flush its TLB entries. Link: https://lore.kernel.org/20261002200508.546-1-dmytrokoziuk68@gmail.com Fixes: b073d7f8aee4 ("mm: kmsan: maintain KMSAN metadata for page operations") Signed-off-by: Dima Koziuk Signed-off-by: Andrew Morton Suggested-by: Alexander Potapenko Link: https://lkml.iu.edu/2609.3/12748.html Cc: Marco Elver Cc: Dmitry Vyukov Cc: Uladzislau Rezki (Sony) --- mm/kmsan/core.c | 8 +++----- mm/kmsan/hooks.c | 51 +++++++++++++++++++++++++++++++----------------- mm/vmalloc.c | 21 ++++++++++++-------- mm/vmalloc.h | 1 + 4 files changed, 50 insertions(+), 31 deletions(-) diff --git a/mm/kmsan/core.c b/mm/kmsan/core.c index 90f427b95a21..3ad5f31d56b6 100644 --- a/mm/kmsan/core.c +++ b/mm/kmsan/core.c @@ -27,6 +27,7 @@ #include #include "../slab.h" +#include "../vmalloc.h" #include "kmsan.h" bool kmsan_enabled __read_mostly; @@ -240,11 +241,8 @@ struct page *kmsan_vmalloc_to_page_or_null(void *vaddr) { struct page *page; - if (!kmsan_internal_is_vmalloc_addr(vaddr) && - !kmsan_internal_is_module_addr(vaddr)) - return NULL; - page = vmalloc_to_page(vaddr); - if (pfn_valid(page_to_pfn(page))) + page = __vmalloc_to_page(vaddr); + if (page && pfn_valid(page_to_pfn(page))) return page; else return NULL; diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 5f1b8053f9fa..24f71bc65896 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -20,6 +20,8 @@ #include #include +#include + #include "../internal.h" #include "../vmalloc.h" #include "../slab.h" @@ -142,6 +144,36 @@ void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end) flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); } +#define KMSAN_IOREMAP_META_ORDER 1 + +static void kmsan_iounmap_pages(unsigned long start, unsigned long end) +{ + unsigned long shadow_start = vmalloc_shadow(start), + shadow_end = vmalloc_shadow(end); + unsigned long origin_start = vmalloc_origin(start), + origin_end = vmalloc_origin(end); + unsigned long v_shadow, v_origin; + struct page *shadow, *origin; + int nr; + + nr = (end - start) / PAGE_SIZE; + v_shadow = shadow_start; + v_origin = origin_start; + for (int i = 0; i < nr; + i++, v_shadow += PAGE_SIZE, v_origin += PAGE_SIZE) { + shadow = kmsan_vmalloc_to_page_or_null((void *)v_shadow); + origin = kmsan_vmalloc_to_page_or_null((void *)v_origin); + if (shadow) + __free_pages(shadow, KMSAN_IOREMAP_META_ORDER); + if (origin) + __free_pages(origin, KMSAN_IOREMAP_META_ORDER); + } + __vunmap_range_noflush(shadow_start, shadow_end); + __vunmap_range_noflush(origin_start, origin_end); + flush_tlb_kernel_range(shadow_start, shadow_end); + flush_tlb_kernel_range(origin_start, origin_end); +} + /* * This function creates new shadow/origin pages for the physical pages mapped * into the virtual memory. If those physical pages already had shadow/origin, @@ -219,28 +251,11 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end, void kmsan_iounmap_page_range(unsigned long start, unsigned long end) { - unsigned long v_shadow, v_origin; - struct page *shadow, *origin; - int nr; - if (!kmsan_enabled || kmsan_in_runtime()) return; - nr = (end - start) / PAGE_SIZE; kmsan_enter_runtime(); - v_shadow = (unsigned long)vmalloc_shadow(start); - v_origin = (unsigned long)vmalloc_origin(start); - for (int i = 0; i < nr; - i++, v_shadow += PAGE_SIZE, v_origin += PAGE_SIZE) { - shadow = kmsan_vmalloc_to_page_or_null((void *)v_shadow); - origin = kmsan_vmalloc_to_page_or_null((void *)v_origin); - __vunmap_range_noflush(v_shadow, vmalloc_shadow(end)); - __vunmap_range_noflush(v_origin, vmalloc_origin(end)); - if (shadow) - __free_pages(shadow, 1); - if (origin) - __free_pages(origin, 1); - } + kmsan_iounmap_pages(start, end); flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); kmsan_leave_runtime(); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 81cad5095889..64411d9140c2 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -817,9 +817,10 @@ EXPORT_SYMBOL_GPL(is_vmalloc_or_module_addr); /* * Walk a vmap address to the struct page it maps. Huge vmap mappings will * return the tail page that corresponds to the base page address, which - * matches small vmap mappings. + * matches small vmap mappings. Unlike vmalloc_to_page(), this also accepts + * addresses outside the vmalloc and module ranges, such as KMSAN metadata. */ -struct page *vmalloc_to_page(const void *vmalloc_addr) +__always_inline struct page *__vmalloc_to_page(const void *vmalloc_addr) { unsigned long addr = (unsigned long) vmalloc_addr; struct page *page = NULL; @@ -829,12 +830,6 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) pmd_t *pmd; pte_t *ptep, pte; - /* - * XXX we might need to change this if we add VIRTUAL_BUG_ON for - * architectures that do not vmalloc module space - */ - VIRTUAL_BUG_ON(!is_vmalloc_or_module_addr(vmalloc_addr)); - if (pgd_none(*pgd)) return NULL; if (WARN_ON_ONCE(pgd_leaf(*pgd))) @@ -873,6 +868,16 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) return page; } + +struct page *vmalloc_to_page(const void *vmalloc_addr) +{ + /* + * XXX we might need to change this if we add VIRTUAL_BUG_ON for + * architectures that do not vmalloc module space + */ + VIRTUAL_BUG_ON(!is_vmalloc_or_module_addr(vmalloc_addr)); + return __vmalloc_to_page(vmalloc_addr); +} EXPORT_SYMBOL(vmalloc_to_page); /* diff --git a/mm/vmalloc.h b/mm/vmalloc.h index 211869f36509..d8372a4f65c4 100644 --- a/mm/vmalloc.h +++ b/mm/vmalloc.h @@ -8,6 +8,7 @@ #include #ifdef CONFIG_MMU +struct page *__vmalloc_to_page(const void *vmalloc_addr); void __init vmalloc_init(void); int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, From 29f8fcb55cd113daa14006d974f549781e488e1b Mon Sep 17 00:00:00 2001 From: Dima Koziuk Date: Fri, 2 Oct 2026 23:05:08 +0300 Subject: [PATCH 141/213] mm: kmsan: fix ioremap error cleanup Looking further at kmsan_ioremap_page_range(), I found three cases where error cleanup leaks metadata blocks. 1. If the first iteration fails, clean is zero and cleanup is skipped, leaking any allocations that succeeded in that iteration. 2. If shadow mapping succeeds but origin mapping fails, the shadow pointer has already been cleared. Removing its mapping loses the backing block. 3. On failures after completed iterations, cleanup removes the earlier metadata mappings without freeing their backing blocks. The cleanup needed here is the same as for iounmap, so it makes sense to reuse kmsan_iounmap_pages(). Track the end of installed mappings with mapped_end and advance it after each successful shadow mapping. This includes the current shadow block if origin mapping subsequently fails, while the helper skips the missing origin mapping. Run cleanup whenever err is non-zero. Free allocations that have not been mapped directly, and use the shared helper to unmap and free the installed metadata, including blocks from completed iterations. Link: https://lore.kernel.org/20261002200508.546-2-dmytrokoziuk68@gmail.com Fixes: fdea03e12aa2 ("mm: kmsan: handle alloc failures in kmsan_ioremap_page_range()") Signed-off-by: Dima Koziuk Signed-off-by: Andrew Morton Reviewed-by: Alexander Potapenko Cc: Marco Elver Cc: Dmitry Vyukov Cc: Uladzislau Rezki (Sony) --- mm/kmsan/hooks.c | 35 ++++++++++++----------------------- 1 file changed, 12 insertions(+), 23 deletions(-) diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 24f71bc65896..a706666db097 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -186,16 +186,17 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end, gfp_t gfp_mask = GFP_KERNEL | __GFP_ZERO; struct page *shadow, *origin; unsigned long off = 0; - int nr, err = 0, clean = 0, mapped; + unsigned long mapped_end = start; + int nr, err = 0, mapped; if (!kmsan_enabled || kmsan_in_runtime()) return 0; nr = (end - start) / PAGE_SIZE; kmsan_enter_runtime(); - for (int i = 0; i < nr; i++, off += PAGE_SIZE, clean = i) { - shadow = alloc_pages(gfp_mask, 1); - origin = alloc_pages(gfp_mask, 1); + for (int i = 0; i < nr; i++, off += PAGE_SIZE) { + shadow = alloc_pages(gfp_mask, KMSAN_IOREMAP_META_ORDER); + origin = alloc_pages(gfp_mask, KMSAN_IOREMAP_META_ORDER); if (!shadow || !origin) { err = -ENOMEM; goto ret; @@ -209,39 +210,27 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end, goto ret; } shadow = NULL; + mapped_end = start + off + PAGE_SIZE; mapped = __vmap_pages_range_noflush( vmalloc_origin(start + off), vmalloc_origin(start + off + PAGE_SIZE), prot, &origin, PAGE_SHIFT); if (mapped) { - __vunmap_range_noflush( - vmalloc_shadow(start + off), - vmalloc_shadow(start + off + PAGE_SIZE)); err = mapped; goto ret; } origin = NULL; } - /* Page mapping loop finished normally, nothing to clean up. */ - clean = 0; ret: - if (clean > 0) { - /* - * Something went wrong. Clean up shadow/origin pages allocated - * on the last loop iteration, then delete mappings created - * during the previous iterations. - */ + if (err) { if (shadow) - __free_pages(shadow, 1); + __free_pages(shadow, KMSAN_IOREMAP_META_ORDER); if (origin) - __free_pages(origin, 1); - __vunmap_range_noflush( - vmalloc_shadow(start), - vmalloc_shadow(start + clean * PAGE_SIZE)); - __vunmap_range_noflush( - vmalloc_origin(start), - vmalloc_origin(start + clean * PAGE_SIZE)); + __free_pages(origin, KMSAN_IOREMAP_META_ORDER); + + if (mapped_end > start) + kmsan_iounmap_pages(start, mapped_end); } flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); From bf033d04366e1375698b01f77f8cf3fa6d039679 Mon Sep 17 00:00:00 2001 From: Wen Jiang Date: Sun, 4 Oct 2026 15:16:39 +0800 Subject: [PATCH 142/213] mm/vmalloc: bail out early on invalid page_shift in __vmap_pages_range_noflush() __vmap_pages_range_noflush() currently only warns when page_shift is smaller than PAGE_SHIFT, then continues with the invalid shift. Return -EINVAL instead so the bogus argument is caught immediately. Link: https://lore.kernel.org/20261004071639.119857-1-jiangwenxiaomi@gmail.com Signed-off-by: Wen Jiang Signed-off-by: Andrew Morton Suggested-by: Uladzislau Rezki (Sony) Reviewed-by: Barry Song Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Vishal Moola (Fractile) Cc: Dev Jain --- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 64411d9140c2..f7aa4686fe5b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -684,7 +684,8 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; - WARN_ON(page_shift < PAGE_SHIFT); + if (WARN_ON_ONCE(page_shift < PAGE_SHIFT)) + return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || page_shift == PAGE_SHIFT) From 2dc36cf60dde233d864a29d3703541f8e530c0a6 Mon Sep 17 00:00:00 2001 From: Enze Li Date: Sun, 4 Oct 2026 00:11:25 +0800 Subject: [PATCH 143/213] docs: hugetlbpage.rst: fix typo in per-node attribute description The per-node hugepages section writes the free_hugepages and surplus_hugepages file names with stray trailing quotes. Quote the real names as inline literals instead. Link: https://lore.kernel.org/20261003161125.40485-1-lienze@kylinos.cn Signed-off-by: Enze Li Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Oscar Salvador Cc: David Hildenbrand Cc: Jonathan Corbet --- Documentation/admin-guide/mm/hugetlbpage.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index 3cc15d800be1..2075f1f14a42 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -373,9 +373,9 @@ contains the following attribute files:: free_hugepages surplus_hugepages -The free\_' and surplus\_' attribute files are read-only. They return the number -of free and surplus [overcommitted] huge pages, respectively, on the parent -node. +The ``free_hugepages`` and ``surplus_hugepages`` attribute files are read-only. +They return the number of free and surplus [overcommitted] huge pages, +respectively, on the parent node. The ``nr_hugepages`` attribute returns the total number of huge pages on the specified node. When this attribute is written, the number of persistent huge From ecaccc8c889e7cefcde88b70663d2a8704c41ca7 Mon Sep 17 00:00:00 2001 From: Jaeyeon Lee Date: Sun, 4 Oct 2026 22:54:58 +0200 Subject: [PATCH 144/213] selftests/mm: hugetlb-read-hwpoison: add setup of HugeTLB pages Every test is expected to reserve the HugeTLB pages it needs. hugetlb-read-hwpoison was not converted, so on a system without pre-reserved huge pages the MAP_POPULATE mmap() fails with ENOMEM and every test case is skipped. Reserve the pages with hugetlb_setup_default(). Each chunk size runs two HWPOISON tests and a poisoned huge page can't be reused, so reserve two pages per chunk size. The pool size is restored on exit. The poisoned pages remain allocated until reboot. With the patch, all 12 cases pass on a system with no huge pages reserved (tested on Fedora, x86_64, CONFIG_MEMORY_FAILURE=y). Link: https://lore.kernel.org/20261004205458.119608-1-jaeyeon.lee.dev@gmail.com Signed-off-by: Jaeyeon Lee Signed-off-by: Andrew Morton Tested-by: Sarthak Sharma Reviewed-by: Sarthak Sharma Assisted-by: LLM Cc: David Hildenbrand Cc: Shuah Khan Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko --- tools/testing/selftests/mm/hugetlb-read-hwpoison.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tools/testing/selftests/mm/hugetlb-read-hwpoison.c b/tools/testing/selftests/mm/hugetlb-read-hwpoison.c index 70b24e3660c4..31748a24448f 100644 --- a/tools/testing/selftests/mm/hugetlb-read-hwpoison.c +++ b/tools/testing/selftests/mm/hugetlb-read-hwpoison.c @@ -13,6 +13,7 @@ #include #include "kselftest.h" +#include "vm_util.h" #define MAX_WRITE_READ_CHUNK_SIZE (getpagesize() * 16) #define MAX(a, b) (((a) > (b)) ? (a) : (b)) @@ -281,6 +282,13 @@ int main(void) size_t i; ksft_print_header(); + + if (!hugetlb_available()) + ksft_exit_skip("HugeTLB is not available\n"); + + if (!hugetlb_setup_default(ARRAY_SIZE(wr_chunk_sizes) * 2)) + ksft_exit_skip("Not enough free huge pages\n"); + ksft_set_plan(ARRAY_SIZE(wr_chunk_sizes) * 3); signal(SIGBUS, sigbus_handler); From fa71cee86530039392094103aab713ba8ba9e301 Mon Sep 17 00:00:00 2001 From: Dheeraj Kumar Srivastava Date: Tue, 6 Oct 2026 21:31:28 +0530 Subject: [PATCH 145/213] mm/hugetlb: use kvmalloc_objs for hugetlb_fault_mutex_table On systems/VMs with a large number of CPUs (e.g. 4096), hugetlb_init() computes num_fault_mutexes = roundup_pow_of_two(8 * num_possible_cpus()) = 32768. The subsequent kmalloc for the mutex table can exceed MAX_PAGE_ORDER when struct mutex is enlarged by debug options like CONFIG_DEBUG_MUTEXES and CONFIG_DEBUG_LOCK_ALLOC, resulting in: WARNING: at __alloc_frozen_pages_noprof (order > MAX_PAGE_ORDER) kernel BUG at mm/hugetlb.c (BUG_ON(!hugetlb_fault_mutex_table)) Kernel panic - not syncing: Fatal exception Switch to kvmalloc_objs() so the allocation falls back to vmalloc when the contiguous physical allocation is too large. The table is only accessed by index, so virtual contiguity is sufficient. Link: https://lore.kernel.org/20261006160128.31447-1-dheerajkumar.srivastava@amd.com Fixes: 8382d914ebf7 ("mm, hugetlb: improve page-fault scalability") Signed-off-by: Dheeraj Kumar Srivastava Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Oscar Salvador Cc: David Hildenbrand Cc: Davidlohr Bueso Cc: Vasant Hegde Cc: Bharata B Rao --- mm/hugetlb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b41a4bd054f8..0a68eb99bc41 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4231,7 +4231,7 @@ static int __init hugetlb_init(void) num_fault_mutexes = 1; #endif hugetlb_fault_mutex_table = - kmalloc_objs(struct mutex, num_fault_mutexes); + kvmalloc_objs(struct mutex, num_fault_mutexes); BUG_ON(!hugetlb_fault_mutex_table); for (i = 0; i < num_fault_mutexes; i++) From ccedc5062f969846b61c4c6f61f765c6a99d5eb7 Mon Sep 17 00:00:00 2001 From: Hajime Tazaki Date: Tue, 6 Oct 2026 17:56:41 +0900 Subject: [PATCH 146/213] selftests: run tests on nommu architecture Patch series "support kselftest on nommu platform", v5. This patchset adds an ability to execute kselftest on nommu platforms. Currently there are several issues if we wish to run kselftests on nommu targets: - it cannot compile/build test binaries because the current files mainly assume to build with glibc, - some of the tests are not able to run on nommu targets as there are no fork(2) syscall. The first issue can be avoided if we can build static PIE binaries (if targets support it), but in our case (build on ubuntu/glibc and run on alpine/musl-libc), it fails to invoke due to lack of the GNU ifunc mechanism. Thus, we need to cross-compile with musl toolchain, which needs to be solved the first issue. The second issue is the lack of fork(2) syscall on those platforms. Especially the test harness helper (kselftest_harness.h) uses the syscall, which cannot be simply with vfork(2). `timeout` command used in `runner.sh` never works for nommu platform as it uses fork(2). nommu component in the mm subsystem has several known issues and having test cases should help this situation, thus this patchset is very first step toward enriching test environment which has not been well tested for a while. The test cases is implemented based on the document (Documentation/admin-guide/mm/nommu-mmap.rst). Initial shape for this patches is implemented as an opt-in TARGET, not built/invoked automatically if users don't specify this nommu target. This patch (of 2): Architectures lacking MMU don't support fork(2) syscall and only vfork(2) is available with limitations. Thus, we cannot run kselftest on nommu architecture as is. This commit addresses this issue with the following changes: - Detect NOMMU at runtime from the MmapCopy field in /proc/meminfo and avoid timeout, which requires fork(2). - Document how to build and run the NOMMU tests, including the procfs prerequisite for runtime detection. Build and run the tests with: $ make ARCH=um O=build kselftest-all TARGETS=nommu $ make ARCH=um O=build kselftest-install TARGETS=nommu $ ./build/kselftest/kselftest_install/run_kselftest.sh -p -c nommu Link: https://lore.kernel.org/20261006085642.3852152-1-thehajime@gmail.com Link: https://lore.kernel.org/20261006085642.3852152-2-thehajime@gmail.com Signed-off-by: Hajime Tazaki Signed-off-by: Andrew Morton Closes: https://sashiko.dev/#/patchset/20260813063401.1786548-1-thehajime%40gmail.com Acked-by: David Hildenbrand (Arm) Cc: Shuah Khan Cc: Jonathan Corbet Cc: Kees Cook Cc: Andy Lutomirski Cc: Will Drewry Cc: Mark Brown Cc: Brendan Jackman Cc: Hangbin Liu Cc: "Ricardo B. Marliere" Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Geert Uytterhoeven Cc: Daniel Palmer --- Documentation/dev-tools/kselftest.rst | 18 ++++++++++++++++++ tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 64c0ec7428a2..fc4dc1f9dc93 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -230,6 +230,24 @@ section:: .. _tar's auto-compress: https://www.gnu.org/software/tar/manual/html_node/gzip.html#auto_002dcompress +Build and test on nommu target +============================== + +The nommu selftests detect NOMMU at runtime by checking for the ``MmapCopy`` +entry in ``/proc/meminfo``. Ensure procfs is mounted at ``/proc`` before +building and running the tests. + +:: + + $ make ARCH=um O=build kselftest-all TARGETS=nommu # <= build-only + $ make ARCH=um O=build kselftest-install TARGETS=nommu + $ ./build/kselftest/kselftest_install/run_kselftest.sh -p -c nommu + +The C tests report a skip if ``/proc/meminfo`` is unavailable. On a NOMMU +system, the runner needs that file to avoid wrapping the tests in ``timeout`` +before they can report a skip. Mounting procfs is therefore a prerequisite +for running this target through the kselftest runner. + Contributing new tests ====================== diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 311811dc55a0..45fb8ec38493 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -36,10 +36,19 @@ tap_prefix() fi } +is_nommu() +{ + grep -q '^MmapCopy:' /proc/meminfo 2>/dev/null +} + tap_timeout() { + # nommu doesn't support timeout command (missing fork(2)) + if is_nommu; then + echo "timeout isn't supported for NOMMU" + $1 # Make sure tests will time out if utility is available. - if [ -x /usr/bin/timeout ] ; then + elif [ -x /usr/bin/timeout ]; then /usr/bin/timeout --foreground "$kselftest_timeout" \ /usr/bin/timeout "$kselftest_timeout" $1 else From e4dce9a62bc2bf0a211a9a3d12b050c26feed634 Mon Sep 17 00:00:00 2001 From: Hajime Tazaki Date: Tue, 6 Oct 2026 17:56:42 +0900 Subject: [PATCH 147/213] selftests/nommu: add nommu mmap and mremap behavior tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce a kselftest utility to validate memory mapping capabilities under nommu kernels, aligned with Documentation/admin-guide/mm/nommu-mmap.rst. The test implements basic checks into a generic architecture-agnostic test matrix applicable across nommu targets. It evaluates: 1. MAP_FIXED allocation rejections. 2. Standard MAP_PRIVATE and MAP_ANONYMOUS allocation resilience. 3. Regular file mappings via standard filesystem storage. 4. Block device subsystem mappings (gracefully skipping if node is missing). 5. Shared vs Private backing discrepancies under nommu conditions. 6. mremap limits, ensuring non-expandable restrictions behave properly. Link: https://lore.kernel.org/20261006085642.3852152-3-thehajime@gmail.com Signed-off-by: Hajime Tazaki Signed-off-by: Andrew Morton Reviewed-by: Lorenzo Stoakes (ARM) Closes: https://sashiko.dev/#/patchset/20260825015945.141739-1-thehajime@gmail.com Assisted-by: Gemini:Pro [AI_Reviewer] [Sashiko_Linter] Assisted-by: cubic.dev:unspecified Cc: David Hildenbrand Cc: "Liam R. Howlett" Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan Cc: Ricardo B . Marlière Cc: Andy Lutomirski Cc: Brendan Jackman Cc: Hangbin Liu Cc: Jonathan Corbet Cc: Kees Cook Cc: Mark Brown Cc: Will Drewry Cc: Geert Uytterhoeven Cc: Daniel Palmer --- tools/testing/selftests/nommu/Makefile | 7 + tools/testing/selftests/nommu/nommu.h | 33 ++ .../testing/selftests/nommu/nommu_mmap_test.c | 264 +++++++++++++ .../selftests/nommu/nommu_mremap_test.c | 356 ++++++++++++++++++ 4 files changed, 660 insertions(+) create mode 100644 tools/testing/selftests/nommu/Makefile create mode 100644 tools/testing/selftests/nommu/nommu.h create mode 100644 tools/testing/selftests/nommu/nommu_mmap_test.c create mode 100644 tools/testing/selftests/nommu/nommu_mremap_test.c diff --git a/tools/testing/selftests/nommu/Makefile b/tools/testing/selftests/nommu/Makefile new file mode 100644 index 000000000000..87bc3bbfa2fd --- /dev/null +++ b/tools/testing/selftests/nommu/Makefile @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +# Makefile for nommu selftests + +TEST_GEN_PROGS += nommu_mmap_test +TEST_GEN_PROGS += nommu_mremap_test + +include ../lib.mk diff --git a/tools/testing/selftests/nommu/nommu.h b/tools/testing/selftests/nommu/nommu.h new file mode 100644 index 000000000000..2c9a2f4e013a --- /dev/null +++ b/tools/testing/selftests/nommu/nommu.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef NOMMU_H +#define NOMMU_H + +#include +#include + +/* Returns 1 for NOMMU, 0 for MMU, and -1 if /proc/meminfo is unavailable. */ +static inline int ksft_is_nommu(void) +{ + FILE *fp; + char line[256]; + int nommu = 0; + + fp = fopen("/proc/meminfo", "r"); + if (!fp) + return -1; + + while (fgets(line, sizeof(line), fp)) { + if (strncmp(line, "MmapCopy:", sizeof("MmapCopy:") - 1) == 0) { + nommu = 1; + break; + } + } + + if (ferror(fp)) + nommu = -1; + fclose(fp); + + return nommu; +} + +#endif diff --git a/tools/testing/selftests/nommu/nommu_mmap_test.c b/tools/testing/selftests/nommu/nommu_mmap_test.c new file mode 100644 index 000000000000..2b3639cf72ef --- /dev/null +++ b/tools/testing/selftests/nommu/nommu_mmap_test.c @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include "kselftest.h" +#include "nommu.h" + +#include +#ifndef RAMFS_MAGIC +#define RAMFS_MAGIC 0x858458f6 +#endif + +static size_t ps; +static int nommu; + +struct test_case_t { + const char *name; + const char *pathname; + int open_flags; + int mmap_prot; + int mmap_flags; + int exp_err; + int (*resolve_exp_err)(const char *path); +}; + +static int get_shm_expected_error(const char *path) +{ + struct statfs fs; + + if (!nommu) + return 0; + + if (statfs(path, &fs) == 0) { + if (fs.f_type == RAMFS_MAGIC) + return 0; /* ramfs succeed with contiguous memory */ + } + /* hostfs, etc returns ENODEV due to lack of contiguous allocation */ + return ENODEV; +} + +static struct test_case_t test_cases[] = { + { + .name = "anonymous private allocation", + .pathname = NULL, + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_ANONYMOUS | MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = NULL, + }, + { + .name = "non-anonymous private file mapping (rw-)", + .pathname = "/tmp/ksft.nommu-reg-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = NULL, + }, + { + .name = "non-anonymous private file mapping (r--)", + .pathname = "/tmp/ksft.nommu-reg-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = NULL, + }, + { + .name = "non-anonymous shared file mapping (rw-)", + .pathname = "/tmp/ksft.nommu-shm-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_SHARED, + .exp_err = 0, + .resolve_exp_err = get_shm_expected_error, + }, + { + .name = "non-anonymous shared file mapping (r--)", + .pathname = "/tmp/ksft.nommu-shm-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_SHARED, + .exp_err = 0, + .resolve_exp_err = get_shm_expected_error, + }, +}; + +static int run_mapping_matrix_test(struct test_case_t *tcase) +{ + int fd; + void *ptr = MAP_FAILED; + char path_buf[PATH_MAX]; + const char *path = tcase->pathname; + int rc = KSFT_PASS; + int expected_error; + + ksft_print_msg("[RUN] %s\n", tcase->name); + + if (tcase->pathname == NULL) { + fd = -1; + } else if (strstr(tcase->pathname, "XXXXXX")) { + strncpy(path_buf, tcase->pathname, sizeof(path_buf) - 1); + path_buf[sizeof(path_buf) - 1] = '\0'; + fd = mkstemp(path_buf); + if (fd < 0) { + ksft_print_msg("Failed to setup temp node: %s\n", + tcase->pathname); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + if (ftruncate(fd, ps) != 0) { + ksft_print_msg("ftruncate failed for: %s\n", tcase->pathname); + rc = KSFT_FAIL; + goto cleanup; + } + path = path_buf; + } else { + fd = open(tcase->pathname, tcase->open_flags, 0600); + if (fd < 0) { + ksft_print_msg("Device node not accessible: %s\n", + tcase->pathname); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + } + + expected_error = tcase->exp_err; + if (tcase->resolve_exp_err && fd >= 0) + expected_error = tcase->resolve_exp_err(path); + + ptr = mmap(NULL, ps, tcase->mmap_prot, tcase->mmap_flags, fd, 0); + + if (expected_error != 0) { + if (ptr != MAP_FAILED) { + ksft_print_msg("mmap unexpectedly succeeded (exp error %d)\n", + expected_error); + rc = KSFT_FAIL; + goto cleanup; + } + if (errno != expected_error) { + ksft_print_msg("mmap failed with %d (%s), but expected %d\n", + errno, strerror(errno), expected_error); + rc = KSFT_FAIL; + goto cleanup; + } + ksft_print_msg("Correctly rejected with expected error %s(%d)\n", + strerror(expected_error), expected_error); + rc = KSFT_PASS; + goto cleanup; + } + + if (ptr == MAP_FAILED) { + ksft_print_msg("mmap failed unexpectedly: %s\n", strerror(errno)); + rc = KSFT_FAIL; + goto cleanup; + } + +cleanup: + if (ptr != MAP_FAILED && munmap(ptr, ps)) { + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + rc = KSFT_FAIL; + } + if (fd >= 0) { + close(fd); + if (tcase->pathname && strstr(tcase->pathname, "XXXXXX") && + unlink(path_buf)) { + ksft_print_msg("unlink failed for %s: %s\n", + path_buf, strerror(errno)); + rc = KSFT_FAIL; + } + } + ksft_test_result_report(rc, "%s\n", tcase->name); + return rc; +} + +static int test_map_fixed(void) +{ + void *fixed_addr; + void *ptr; + + ksft_print_msg("[RUN] %s\n", __func__); + + fixed_addr = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (fixed_addr == MAP_FAILED) { + ksft_print_msg("Unable to reserve test address: %s\n", + strerror(errno)); + ksft_test_result_skip("MAP_FIXED behavior\n"); + return KSFT_SKIP; + } + + if (munmap(fixed_addr, ps)) { + ksft_print_msg("Unable to release test address: %s\n", + strerror(errno)); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; + } + + ptr = mmap(fixed_addr, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0); + + if (nommu) { + if (ptr == MAP_FAILED && (errno == ENODEV || errno == EINVAL)) { + ksft_print_msg("MAP_FIXED correctly rejected under nommu\n"); + ksft_test_result_pass("MAP_FIXED behavior\n"); + return KSFT_PASS; + } + if (ptr != MAP_FAILED) { + ksft_print_msg("MAP_FIXED unexpectedly allowed under nommu\n"); + ksft_test_result_fail("MAP_FIXED behavior\n"); + if (munmap(ptr, ps)) + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + return KSFT_FAIL; + } + ksft_print_msg("MAP_FIXED failed under NOMMU: %s\n", + strerror(errno)); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; + } + if (ptr != MAP_FAILED) { + ksft_print_msg("MAP_FIXED successfully allocated under MMU\n"); + if (munmap(ptr, ps)) { + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; + } + ksft_test_result_pass("MAP_FIXED behavior\n"); + return KSFT_PASS; + } + ksft_print_msg("MAP_FIXED failed allocation under MMU\n"); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; +} + +int main(int argc, char **argv) +{ + int i; + + nommu = ksft_is_nommu(); + if (nommu < 0) + ksft_exit_skip("Cannot read /proc/meminfo; mount procfs at /proc\n"); + + ps = sysconf(_SC_PAGESIZE); + ksft_print_header(); + ksft_set_plan(ARRAY_SIZE(test_cases) + 1); + + if (nommu) + ksft_print_msg("Running strict MMAP test criteria under nommu architecture\n"); + else + ksft_print_msg("Running MMAP test criteria under MMU architecture\n"); + + test_map_fixed(); + for (i = 0; i < (int)ARRAY_SIZE(test_cases); i++) + run_mapping_matrix_test(&test_cases[i]); + + ksft_finished(); +} diff --git a/tools/testing/selftests/nommu/nommu_mremap_test.c b/tools/testing/selftests/nommu/nommu_mremap_test.c new file mode 100644 index 000000000000..6ee89cb25f61 --- /dev/null +++ b/tools/testing/selftests/nommu/nommu_mremap_test.c @@ -0,0 +1,356 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include "kselftest.h" +#include "nommu.h" + +#include +#ifndef RAMFS_MAGIC +#define RAMFS_MAGIC 0x858458f6 +#endif + +static size_t ps; +static int nommu; + +static long get_fs_type(const char *path) +{ + struct statfs fs; + + if (statfs(path, &fs) == 0) + return fs.f_type; + + return 0; +} + +static void munmap_shrink_test(void) +{ + void *addr; + int ret; + + /* munmap shrink test */ + for (int i = 0; i < 4; i++) { + addr = mmap(NULL, ps * 4, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (addr == MAP_FAILED) { + ksft_print_msg("mmap failed: %s(%d)\n", strerror(errno), errno); + ksft_test_result_fail("munmap shrink\n"); + return; + } + ret = munmap((char *)addr + ps * i, ps); + if (ret != 0) { + ksft_print_msg("memory %p isn't unmapped at %p\n", + addr, (char *)addr + ps * i); + goto error; + } + + if (i == 0) { + if (munmap(addr + ps, ps * 3)) + goto error; + } else if (i == 1) { + if (munmap(addr, ps) || munmap(addr + (ps * 2), ps * 2)) + goto error; + } else if (i == 2) { + if (munmap(addr, ps * 2) || munmap(addr + (ps * 3), ps)) + goto error; + } else if (i == 3) { + if (munmap(addr, ps * 3)) + goto error; + } + } + + ksft_test_result_pass("munmap shrink\n"); + return; +error: + for (int j = 0; j < 4; j++) { + if (munmap((char *)addr + j * ps, ps)) + ksft_print_msg("cleanup munmap failed: %s\n", strerror(errno)); + } + ksft_print_msg("clean up failures\n"); + ksft_test_result_fail("munmap shrink\n"); +} + +static size_t page_align(size_t len) +{ + return (len + ps - 1) / ps * ps; +} + +static void mremap_shrink_test(void) +{ + void *addr, *addr2; + size_t current_len; + size_t old_len, new_len; + struct param { + size_t old; + size_t new; + } params[] = { + /* should not happen any shrink */ + { .old = ps * 4 - 1, .new = ps * 4 - 2 }, + /* should not happen any shrink */ + { .old = ps * 4 - 1, .new = ps * 4 }, + { .old = ps * 4, .new = ps * 2 }, + /* should not happen any shrink */ + { .old = ps * 2, .new = ps * 2 - 2 }, + { .old = ps * 2 - 2, .new = ps * 1 }, + }; + + /* mremap shrink test */ + current_len = page_align(ps * 4 - 1); + addr = mmap(NULL, ps * 4 - 1, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (addr == MAP_FAILED) { + ksft_print_msg("mmap failed: %s(%d)\n", strerror(errno), errno); + ksft_test_result_fail("mremap shrink\n"); + return; + } + + for (int i = 0; i < ARRAY_SIZE(params); i++) { + old_len = params[i].old; + new_len = params[i].new; + current_len = page_align(new_len); + addr2 = mremap(addr, old_len, new_len, MREMAP_MAYMOVE); + if (addr2 == MAP_FAILED) { + ksft_print_msg("memory %p isn't remapped at %p\n", addr, addr2); + ksft_test_result_fail("mremap shrink\n"); + munmap(addr, page_align(old_len)); + return; + } + + addr = addr2; + } + + if (munmap(addr, current_len)) { + ksft_print_msg("cleanup failed: %s\n", strerror(errno)); + ksft_test_result_fail("mremap shrink\n"); + return; + } + ksft_test_result_pass("mremap shrink\n"); +} + +static int get_shared_writable_file_expected_error(const char *path) +{ + if (nommu && get_fs_type(path) == RAMFS_MAGIC) + return EPERM; /* ramfs failed */ + + return 0; +} + +struct mremap_case_t { + const char *name; + const char *pathname; + int open_flags; + int mmap_prot; + int mmap_flags; + int exp_err; + int nommu_exp_err; + int (*resolve_exp_err)(const char *path); + unsigned int old_pages; + unsigned int new_pages; +}; + +static struct mremap_case_t mremap_cases[] = { + { + .name = "anonymous shrink (r--)", + .pathname = NULL, + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_ANONYMOUS | MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = 0, + }, + { + .name = "shared file shrink (r--)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_SHARED, + .exp_err = 0, + .resolve_exp_err = get_shared_writable_file_expected_error, + }, + { + .name = "private file unchanged length (r-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .nommu_exp_err = EPERM, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 4, + }, + { + .name = "private file unchanged length (rw-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 4, + }, + { + .name = "private file growth (r-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .nommu_exp_err = EPERM, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 8, + }, + { + .name = "private file growth (rw-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .nommu_exp_err = ENOMEM, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 8, + }, +}; + +static int run_mremap_test(struct mremap_case_t *tcase) +{ + int fd = -1; + void *addr, *addr2; + char pb[PATH_MAX]; + const char *path = tcase->pathname; + int rc = KSFT_PASS; + int expected_error; + unsigned int old_pages = tcase->old_pages ?: 4; + unsigned int new_pages = tcase->new_pages ?: 2; + unsigned int file_pages = old_pages > new_pages ? + old_pages : new_pages; + + ksft_print_msg("[RUN] Testing mremap: %s\n", tcase->name); + + if (tcase->pathname && strstr(tcase->pathname, "XXXXXX")) { + strncpy(pb, tcase->pathname, sizeof(pb) - 1); + pb[sizeof(pb) - 1] = '\0'; + fd = mkstemp(pb); + if (fd < 0) { + ksft_print_msg("Failed to setup file backing\n"); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + if (ftruncate(fd, ps * file_pages) != 0) { + ksft_print_msg("Failed to setup file backing\n"); + rc = KSFT_FAIL; + goto out; + } + + if (nommu && (tcase->mmap_flags & MAP_SHARED) && + get_fs_type(pb) != RAMFS_MAGIC) { + ksft_print_msg("Skip the test under non-ramfs filesystem (%s)\n", + pb); + rc = KSFT_SKIP; + goto out; + } + path = pb; + } else if (tcase->pathname) { + fd = open(tcase->pathname, tcase->open_flags, 0600); + if (fd < 0) { + ksft_print_msg("Backing node not accessible\n"); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + + if (nommu && (tcase->mmap_flags & MAP_SHARED) && + get_fs_type(tcase->pathname) != RAMFS_MAGIC) { + ksft_print_msg("Skip the test under non-ramfs filesystem (%s)\n", + tcase->pathname); + rc = KSFT_SKIP; + goto out; + } + } + + addr = mmap(NULL, ps * old_pages, tcase->mmap_prot, + tcase->mmap_flags, fd, 0); + if (addr == MAP_FAILED) { + ksft_print_msg("mmap mapping failed %s(%d)\n", strerror(errno), errno); + rc = KSFT_FAIL; + goto out; + } + + expected_error = nommu ? tcase->nommu_exp_err : tcase->exp_err; + if (tcase->resolve_exp_err && fd >= 0) + expected_error = tcase->resolve_exp_err(path); + + addr2 = mremap(addr, ps * old_pages, ps * new_pages, + MREMAP_MAYMOVE); + + if (expected_error != 0) { + if (addr2 != MAP_FAILED) { + ksft_print_msg("Expected error %d, but mremap unexpectedly succeeded\n", + expected_error); + rc = KSFT_FAIL; + } else if (errno != expected_error) { + ksft_print_msg("Expected error %d, got %s(%d)\n", + expected_error, strerror(errno), errno); + rc = KSFT_FAIL; + } else { + ksft_print_msg("%s: Handled expected error path (errno=%d)\n", + tcase->name, expected_error); + } + } else if (addr2 == MAP_FAILED) { + ksft_print_msg("mremap shrink failed unexpectedly: %s\n", + strerror(errno)); + rc = KSFT_FAIL; + } else { + ksft_print_msg("%s step successful\n", tcase->name); + } + + /* clean up */ + if (munmap(addr2 == MAP_FAILED ? addr : addr2, + addr2 == MAP_FAILED ? ps * old_pages : ps * new_pages)) { + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + rc = KSFT_FAIL; + } + +out: + if (fd >= 0) { + close(fd); + if (tcase->pathname && strstr(tcase->pathname, "XXXXXX") && + unlink(pb)) { + ksft_print_msg("unlink failed for %s: %s\n", pb, strerror(errno)); + rc = KSFT_FAIL; + } + } + + ksft_test_result_report(rc, "%s\n", tcase->name); + return rc; +} + +int main(int argc, char **argv) +{ + int i; + + nommu = ksft_is_nommu(); + if (nommu < 0) + ksft_exit_skip("Cannot read /proc/meminfo; mount procfs at /proc\n"); + + ps = sysconf(_SC_PAGESIZE); + ksft_print_header(); + ksft_set_plan(ARRAY_SIZE(mremap_cases) + 2); + + munmap_shrink_test(); + mremap_shrink_test(); + + for (i = 0; i < (int)ARRAY_SIZE(mremap_cases); i++) + run_mremap_test(&mremap_cases[i]); + + ksft_finished(); +} From 0284dff76e87fc47a1055080d4e8540b41d096b2 Mon Sep 17 00:00:00 2001 From: Jaeyeon Lee Date: Mon, 5 Oct 2026 23:52:05 -0700 Subject: [PATCH 148/213] mm/damon/ops-common: fix age_in_sec overflow on 32-bit Patch series "mm/damon: fix a minor bug, clean code, and add a test". Patch 1 from Jaeyeon Lee fixes a minor overflow issue in DAMOS. Patch 2 from Gregory Price (Meta) cleans up the code for handling of hugetlb pages. Patch 3 from Liew Rui Yan adds a kunit test for a recently fixed bug. This patch (of 3): damon_hot_score() calculates the age of a region by multiplying r->age and the aggregation interval. On 32-bit architectures, unsigned long is 32 bits, so the multiplication overflows once the age exceeds about 4295 seconds. Then old regions are scored as if they were young. Fix it by doing the calculation in 64-bit arithmetic using div_u64(). Also make age_in_sec u64, so the result is not truncated back to 32 bits. The issue was discovered [1] by Sashiko. Link: https://lore.kernel.org/20261006065209.41507-1-sj@kernel.org Link: https://lore.kernel.org/20261006065209.41507-2-sj@kernel.org Link: https://lore.kernel.org/20260623011652.1354-1-sj@kernel.org [1] Fixes: 198f0f4c58b9 ("mm/damon/vaddr,paddr: support pageout prioritization") Signed-off-by: Jaeyeon Lee Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Assisted-by: LLM --- mm/damon/ops-common.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 77366f42b3e5..7a7429359dc8 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -164,7 +164,7 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, struct damos *s) { int freq_subscore; - unsigned int age_in_sec; + u64 age_in_sec; int age_in_log, age_subscore; unsigned int freq_weight = s->quota.weight_nr_accesses; unsigned int age_weight = s->quota.weight_age; @@ -174,7 +174,8 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, DAMON_MAX_SUBSCORE, damon_nr_samples_per_aggr(&c->attrs)); - age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000; + age_in_sec = div_u64((u64)r->age * c->attrs.aggr_interval, + USEC_PER_SEC); if (age_in_sec) age_in_log = min_t(int, ilog2(age_in_sec) + 1, DAMON_MAX_AGE_IN_LOG); From 8cb5bfe457dfd430907c5561739a76e54a735c63 Mon Sep 17 00:00:00 2001 From: Gregory Price Date: Mon, 5 Oct 2026 23:52:06 -0700 Subject: [PATCH 149/213] mm/damon: use damon_get_monitor_folio() for hugetlb entries Hugetlb folio acquisition is open-coded in three places: - damon_hugetlb_mkold() - damon_young_hugetlb_entry() - damon_va_probe_hugetlb_entry() DAMON already has damon_get_monitor_folio(), which paddr uses to get monitored LRU and hugetlb folios and to filter them through damon_folio_acceptable(). These paths bypass that check. Use that helper in these paths too, so that DAMON acquires hugetlb folios and applies its acceptance policy in one place. No functional change intended. All callers reach these paths with a present hugetlb PTE under the page table lock, so the folio is always a mapped hugetlb folio and the helper always returns it. Link: https://lore.kernel.org/20261006065209.41507-3-sj@kernel.org Signed-off-by: Gregory Price (Meta) Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park --- mm/damon/ops-common.c | 6 ++++-- mm/damon/vaddr.c | 10 ++++++---- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 7a7429359dc8..2f3bf86a221b 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -140,9 +140,11 @@ void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, { bool referenced = false; pte_t entry = huge_ptep_get(mm, addr, pte); - struct folio *folio = pfn_folio(pte_pfn(entry)); + struct folio *folio; - folio_get(folio); + folio = damon_get_monitor_folio(pte_pfn(entry)); + if (!folio) + return; referenced = damon_hugetlb_ptep_mkold(pte, mm, vma, addr, &entry); if (mmu_notifier_clear_young(mm, addr, diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index d5dde97b3cd0..b549496ea8e2 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -411,8 +411,9 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, if (!pte_present(entry)) goto out; - folio = pfn_folio(pte_pfn(entry)); - folio_get(folio); + folio = damon_get_monitor_folio(pte_pfn(entry)); + if (!folio) + goto out; if (pte_young(entry) || !folio_test_idle(folio) || mmu_notifier_test_young(walk->mm, addr)) @@ -660,8 +661,9 @@ static int damon_va_probe_hugetlb_entry(pte_t *pte, unsigned long hmask, if (!pte_present(entry)) goto out; - folio = pfn_folio(pte_pfn(entry)); - folio_get(folio); + folio = damon_get_monitor_folio(pte_pfn(entry)); + if (!folio) + goto out; damon_va_probe_folio(priv->ctx, priv->r, folio, &entry, NULL, walk->vma->vm_mm); folio_put(folio); From 20a0f2b4705c3f2e476270c54e79637893b5e50b Mon Sep 17 00:00:00 2001 From: Liew Rui Yan Date: Mon, 5 Oct 2026 23:52:07 -0700 Subject: [PATCH 150/213] mm/damon/tests/core-kunit: add test for unconditionally skipping the last region Commit b3723b596b54 ("mm/damon/core: fix unconditionally skip last region") fixed a minor bug. Add a KUnit test case to cover this behavior and prevent regressions in the future. Link: https://lore.kernel.org/20261006065209.41507-4-sj@kernel.org Signed-off-by: Liew Rui Yan Signed-off-by: SJ Park Signed-off-by: Andrew Morton Reviewed-by: SJ Park Cc: Brendan Higgins Cc: David Gow --- mm/damon/tests/core-kunit.h | 42 +++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index ef146ca2ae8a..3fbb4e4e36fa 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -2439,6 +2439,47 @@ static void damos_test_esz_goal_temporal(struct kunit *test) damon_destroy_ctx(ctx); } +static void damon_test_last_region_quota_reset(struct kunit *test) +{ + struct damos s = {0}; + struct damon_target *t; + struct damon_region *r1, *r2; + unsigned long min_region_sz = 1; + bool skipped; + + t = damon_new_target(); + if (!t) + kunit_skip(test, "target alloc fail"); + + r1 = damon_new_region(0, 100); + if (!r1) { + damon_free_target(t); + kunit_skip(test, "region 1 alloc fail"); + } + damon_add_region(r1, t); + + r2 = damon_new_region(100, 200); + if (!r2) { + damon_free_target(t); + kunit_skip(test, "region 2 alloc fail"); + } + damon_add_region(r2, t); + + s.quota.charge_target_from = t; + s.quota.charge_addr_from = r1->ar.end; + + skipped = damos_skip_charged_region(t, r2, &s, min_region_sz); + + /* 'r2' is not processed, it should not skip */ + KUNIT_EXPECT_EQ(test, skipped, false); + + /* 'r2' is last region, it should reset charge_{target,addr}_from */ + KUNIT_EXPECT_PTR_EQ(test, s.quota.charge_target_from, NULL); + KUNIT_EXPECT_EQ(test, s.quota.charge_addr_from, 0); + + damon_free_target(t); +} + static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_target), KUNIT_CASE(damon_test_regions), @@ -2482,6 +2523,7 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_walk_control_obsolete), KUNIT_CASE(damon_test_rand), KUNIT_CASE(damos_test_esz_goal_temporal), + KUNIT_CASE(damon_test_last_region_quota_reset), {}, }; From 50ee565c1e900e247e439e4139839b060c968343 Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Thu, 3 Sep 2026 11:01:34 +0800 Subject: [PATCH 151/213] mm/memfd: fix hugetlb reservation accounting in error paths If hugetlb_add_to_page_cache() in memfd_alloc_folio() fails with -EEXIST, a concurrent fault has already instantiated the folio in the page cache, and the reservation now belongs to that folio. Calling hugetlb_unreserve_pages() in that case incorrectly removes the region backing the cached folio and releases one reservation more than it should, leaving that folio in the page cache with no region recording it. That leaves resv_huge_pages one page short until that folio is removed from the page cache, so available_huge_pages() reports a page that is not actually free and the pool can grant one reservation more than it can back. Applications using hugetlb memfds can then fail to allocate or fault in a page they already reserved. They see -ENOMEM or -ENOSPC from the allocation or fault path even though the reservation and HugePages_Free still look healthy. So hold the hugetlb fault mutex from hugetlb_reserve_pages() until the error-path unreserve completes to make the reserve, allocate and instantiate steps atomic against concurrent faults. With the mutex held from the start, a concurrent fault can no longer consume the reservation between reserve and allocate/instantiate. If a fault completed before the mutex was taken, it has already added the region for that index, so hugetlb_reserve_pages() returns 0 and the error path leaves the region in place. Link: https://lore.kernel.org/20260927094757.31665-1-hongfu.li@linux.dev Link: https://lore.kernel.org/20260903030134.7407-1-hongfu.li@linux.dev Fixes: 717cf9357325 ("mm/memfd: reserve hugetlb folios before allocation") Signed-off-by: Hongfu Li Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Hugh Dickins Cc: Muchun Song Cc: Oscar Salvador Cc: Vivek Kasireddy Cc: --- mm/memfd.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/mm/memfd.c b/mm/memfd.c index c708d92533f4..0f6fff004f5e 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -82,22 +82,31 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) struct hstate *h = hstate_file(memfd); int err = -ENOMEM; long nr_resv; + u32 hash; gfp_mask = htlb_alloc_mask(h); gfp_mask &= ~(__GFP_HIGHMEM | __GFP_MOVABLE); idx >>= huge_page_order(h); + /* + * Serialize hugepage allocation and instantiation to prevent + * races with concurrent allocations, as required by all other + * callers of hugetlb_add_to_page_cache(). + */ + hash = hugetlb_fault_mutex_hash(memfd->f_mapping, idx); + mutex_lock(&hugetlb_fault_mutex_table[hash]); + nr_resv = hugetlb_reserve_pages(inode, idx, idx + 1, NULL, EMPTY_VMA_FLAGS); - if (nr_resv < 0) - return ERR_PTR(nr_resv); + if (nr_resv < 0) { + err = nr_resv; + goto out_unlock; + } folio = alloc_hugetlb_folio_reserve(h, numa_node_id(), NULL, gfp_mask); if (folio) { - u32 hash; - /* * Zero the folio to prevent information leaks to userspace. * Use folio_zero_user() which is optimized for huge/gigantic @@ -112,20 +121,9 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) */ __folio_mark_uptodate(folio); - /* - * Serialize hugepage allocation and instantiation to prevent - * races with concurrent allocations, as required by all other - * callers of hugetlb_add_to_page_cache(). - */ - hash = hugetlb_fault_mutex_hash(memfd->f_mapping, idx); - mutex_lock(&hugetlb_fault_mutex_table[hash]); - err = hugetlb_add_to_page_cache(folio, memfd->f_mapping, idx); - - mutex_unlock(&hugetlb_fault_mutex_table[hash]); - if (err) { folio_put(folio); goto err_unresv; @@ -133,11 +131,14 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) hugetlb_set_folio_subpool(folio, subpool_inode(inode)); folio_unlock(folio); + mutex_unlock(&hugetlb_fault_mutex_table[hash]); return folio; } err_unresv: if (nr_resv > 0) hugetlb_unreserve_pages(inode, idx, idx + 1, 0); +out_unlock: + mutex_unlock(&hugetlb_fault_mutex_table[hash]); return ERR_PTR(err); } #endif From 03825d869f3f6f94b6820e5efd8c58947b1555da Mon Sep 17 00:00:00 2001 From: David Carlier Date: Thu, 30 Apr 2026 20:56:51 +0100 Subject: [PATCH 152/213] mm/swap, PM: hibernate: atomically replace hibernation pin snapshot_set_swap_area() unpins the previously selected swap device and pins the new one in two separate swap_lock critical sections. In the gap between them, swapoff() observes SWP_HIBERNATION cleared, bypasses the guard, and tears down the device, reopening the race the SWP_HIBERNATION pin was meant to close. The window is reachable on any SNAPSHOT_SET_SWAP_AREA call after the snapshot device is opened for hibernation, and on any retry after the resume path's first selection. Add repin_hibernation_swap_type(), which looks up the new device, clears the old SWP_HIBERNATION flag and sets the new one under a single swap_lock acquisition. The same-device case is short- circuited so userspace can re-select the same swap area without tripping WARN_ON_ONCE and -EBUSY. Switch snapshot_set_swap_area() to the new helper. A failed lookup now preserves the previous pin instead of dropping it, so a bad SNAPSHOT_SET_SWAP_AREA leaves the prior selection intact. The open and release paths keep using pin_hibernation_swap_type() and unpin_hibernation_swap_type(). The race was identified during AI-assisted review of the SWP_HIBERNATION pinning series. [akpm@linux-foundation.org: coding-style cleanups] Link: https://lore.kernel.org/20260430195651.287659-1-devnexen@gmail.com Fixes: 8e6e0d845823 ("mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device") Signed-off-by: David Carlier Signed-off-by: Andrew Morton Assisted-by: Codex (gpt-5-codex) Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Len Brown Cc: Nhat Pham Cc: "Rafael J. Wysocki" Cc: DaeMyung Kang Cc: Youngjun Park --- include/linux/swap.h | 2 ++ kernel/power/user.c | 17 ++++-------- mm/swapfile.c | 61 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+), 12 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index cb434cccd653..4f686709d71f 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -395,6 +395,8 @@ static inline long get_nr_swap_pages(void) extern void si_swapinfo(struct sysinfo *); extern int pin_hibernation_swap_type(dev_t device, sector_t offset); extern void unpin_hibernation_swap_type(int type); +extern int repin_hibernation_swap_type(int old_type, dev_t device, + sector_t offset); extern int find_hibernation_swap_type(dev_t device, sector_t offset); int find_first_swap(dev_t *device); extern unsigned int count_swap_pages(int, int); diff --git a/kernel/power/user.c b/kernel/power/user.c index d0fcfba7ac23..6e4f40e49319 100644 --- a/kernel/power/user.c +++ b/kernel/power/user.c @@ -218,6 +218,7 @@ static int snapshot_set_swap_area(struct snapshot_data *data, { sector_t offset; dev_t swdev; + int new_type; if (swsusp_swap_in_use()) return -EPERM; @@ -238,19 +239,11 @@ static int snapshot_set_swap_area(struct snapshot_data *data, offset = swap_area.offset; } - /* - * Unpin the swap device if a swap area was already - * set by SNAPSHOT_SET_SWAP_AREA. - */ - unpin_hibernation_swap_type(data->swap); + new_type = repin_hibernation_swap_type(data->swap, swdev, offset); + if (new_type < 0) + return new_type; - /* - * User space encodes device types as two-byte values, - * so we need to recode them - */ - data->swap = pin_hibernation_swap_type(swdev, offset); - if (data->swap < 0) - return swdev ? -ENODEV : -EINVAL; + data->swap = new_type; data->dev = swdev; return 0; } diff --git a/mm/swapfile.c b/mm/swapfile.c index 6187c02ec5ec..254ce86fa923 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2312,6 +2312,67 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset) return type; } +/** + * repin_hibernation_swap_type - Atomically replace the hibernation pin + * @old_type: Swap type currently pinned (or < 0 if none). + * @device: Block device of the new resume image. + * @offset: Offset identifying the new swap area. + * + * Look up the swap device for @device/@offset and atomically transfer + * the SWP_HIBERNATION pin from @old_type (if valid) to the new device, + * all under a single swap_lock critical section. This closes the + * swapoff() window that exists when callers unpin and re-pin in two + * separate operations. + * + * If the new device cannot be located, the existing pin on @old_type + * is preserved and an error is returned. If @old_type already refers + * to the same swap_info_struct as the new lookup, no flag changes are + * made and @old_type is returned. + * + * Return: + * >= 0 on success (new swap type). + * -EINVAL if @device is invalid. + * -ENODEV if the swap device is not found. + * -EBUSY if the new device is already pinned by another context. + */ +int repin_hibernation_swap_type(int old_type, dev_t device, sector_t offset) +{ + struct swap_info_struct *old_si, *new_si; + int new_type; + + spin_lock(&swap_lock); + + new_type = __find_hibernation_swap_type(device, offset); + if (new_type < 0) { + spin_unlock(&swap_lock); + return new_type; + } + + new_si = swap_type_to_info(new_type); + if (WARN_ON_ONCE(!new_si)) { + spin_unlock(&swap_lock); + return -ENODEV; + } + + old_si = swap_type_to_info(old_type); + if (new_si == old_si) { + spin_unlock(&swap_lock); + return new_type; + } + + if (WARN_ON_ONCE(new_si->flags & SWP_HIBERNATION)) { + spin_unlock(&swap_lock); + return -EBUSY; + } + + if (old_si) + old_si->flags &= ~SWP_HIBERNATION; + new_si->flags |= SWP_HIBERNATION; + + spin_unlock(&swap_lock); + return new_type; +} + /** * unpin_hibernation_swap_type - Unpin the swap device for hibernation * @type: Swap type previously returned by pin_hibernation_swap_type() From 4d1a248a38c648978743fb0f5488894e24cc438c Mon Sep 17 00:00:00 2001 From: Enze Li Date: Sat, 10 Oct 2026 17:23:33 +0800 Subject: [PATCH 153/213] mm/hugetlb: prefix log messages via pr_fmt Pretty much every pr_* message in this file starts with "HugeTLB: ", typed out by hand, and one of them even drifted to lowercase ("hugetlb:"). hugetlb_vmemmap.c already handles this with pr_fmt, so let's do the same here instead of repeating the prefix over and over. The output stays the same for almost everything; the only tweaks are that the lowercase message now matches the others, and the ratelimited "killed due to inadequate hugepage pool" warning picks up the prefix like everything else. Link: https://lore.kernel.org/20261010092333.137126-1-lienze@kylinos.cn Signed-off-by: Enze Li Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Oscar Salvador Cc: David Hildenbrand Cc: Enze Li --- mm/hugetlb.c | 52 +++++++++++++++++++++++++++------------------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0a68eb99bc41..e3367b319bc5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3,6 +3,8 @@ * Generic hugetlb support. * (C) Nadia Yvette Chambers, April 2004 */ +#define pr_fmt(fmt) "HugeTLB: " fmt + #include #include #include @@ -458,7 +460,7 @@ static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma) * until the file is removed. Warn in the unlikely case of * allocation failure. */ - pr_warn_once("HugeTLB: unable to allocate vma specific lock\n"); + pr_warn_once("unable to allocate vma specific lock\n"); return; } @@ -987,7 +989,7 @@ void hugetlb_fix_reserve_counts(struct inode *inode) } if (!reserved) - pr_warn("hugetlb: Huge Page Reserved count may go negative.\n"); + pr_warn("Huge Page Reserved count may go negative.\n"); } /* @@ -3408,7 +3410,7 @@ static unsigned long __init hugetlb_free_cross_zone_pages(struct hstate *h, int char buf[32]; string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); - pr_warn("HugeTLB: freed %lu cross-zone hugepages of size %s on node %d.\n", + pr_warn("freed %lu cross-zone hugepages of size %s on node %d.\n", freed, buf, nid); } @@ -3454,7 +3456,7 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) return; string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n", + pr_warn("allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n", h->max_huge_pages_node[nid], buf, nid, i); h->max_huge_pages -= (h->max_huge_pages_node[nid] - i); h->max_huge_pages_node[nid] = i; @@ -3481,7 +3483,7 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, char buf[32]; string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", + pr_warn("allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", h->max_huge_pages, buf, allocated); h->max_huge_pages = allocated; } @@ -3596,7 +3598,7 @@ static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) jiffies_end = jiffies; - pr_info("HugeTLB: allocation took %dms with hugepage_allocation_threads=%ld\n", + pr_info("allocation took %dms with hugepage_allocation_threads=%ld\n", jiffies_to_msecs(jiffies_end - jiffies_start), hugepage_allocation_threads); @@ -3624,7 +3626,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) */ if (hstate_is_gigantic(h) && hugetlb_cma_total_size() && !hugetlb_early_cma(h)) { - pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n"); + pr_warn_once("hugetlb_cma is enabled, skip boot time allocation\n"); return; } @@ -3692,9 +3694,9 @@ static void __init report_hugepages(void) char buf[32]; string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_info("HugeTLB: registered %s page size, pre-allocated %ld pages\n", + pr_info("registered %s page size, pre-allocated %ld pages\n", buf, h->nr_huge_pages); - pr_info("HugeTLB: %d KiB vmemmap can be freed for a %s page\n", + pr_info("%d KiB vmemmap can be freed for a %s page\n", hugetlb_vmemmap_optimizable_size(h) / SZ_1K, buf); } } @@ -4034,7 +4036,7 @@ long demote_pool_huge_page(struct hstate *src, nodemask_t *nodes_allowed, /* We should never get here if no demote order */ if (!src->demote_order) { - pr_warn("HugeTLB: NULL demote order passed to demote_pool_huge_page.\n"); + pr_warn("NULL demote order passed to demote_pool_huge_page.\n"); return -EINVAL; /* internal error */ } dst = size_to_hstate(PAGE_SIZE << src->demote_order); @@ -4179,7 +4181,7 @@ static int __init hugetlb_init(void) if (!hugepages_supported()) { if (hugetlb_max_hstate || default_hstate_max_huge_pages) - pr_warn("HugeTLB: huge pages not supported, ignoring associated command-line parameters\n"); + pr_warn("huge pages not supported, ignoring associated command-line parameters\n"); return 0; } @@ -4204,9 +4206,9 @@ static int __init hugetlb_init(void) string_get_size(huge_page_size(&default_hstate), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: Ignoring hugepages=%lu associated with %s page size\n", + pr_warn("Ignoring hugepages=%lu associated with %s page size\n", default_hstate.max_huge_pages, buf); - pr_warn("HugeTLB: Using hugepages=%lu for number of default huge pages\n", + pr_warn("Using hugepages=%lu for number of default huge pages\n", default_hstate_max_huge_pages); } default_hstate.max_huge_pages = @@ -4346,12 +4348,12 @@ static int __init hugepages_setup(char *s) char *p = s; if (!hugepages_supported()) { - pr_warn("HugeTLB: hugepages unsupported, ignoring hugepages=%s cmdline\n", s); + pr_warn("hugepages unsupported, ignoring hugepages=%s cmdline\n", s); return 0; } if (!parsed_valid_hugepagesz) { - pr_warn("HugeTLB: hugepages=%s does not follow a valid hugepagesz, ignoring\n", s); + pr_warn("hugepages=%s does not follow a valid hugepagesz, ignoring\n", s); parsed_valid_hugepagesz = true; return -EINVAL; } @@ -4368,7 +4370,7 @@ static int __init hugepages_setup(char *s) mhp = &parsed_hstate->max_huge_pages; if (mhp == last_mhp) { - pr_warn("HugeTLB: hugepages= specified twice without interleaving hugepagesz=, ignoring hugepages=%s\n", s); + pr_warn("hugepages= specified twice without interleaving hugepagesz=, ignoring hugepages=%s\n", s); return 1; } @@ -4379,7 +4381,7 @@ static int __init hugepages_setup(char *s) /* Parameter is node format */ if (p[count] == ':') { if (!hugetlb_node_alloc_supported()) { - pr_warn("HugeTLB: architecture can't support node specific alloc, ignoring!\n"); + pr_warn("architecture can't support node specific alloc, ignoring!\n"); return 1; } if (tmp >= MAX_NUMNODES || !node_online(tmp)) @@ -4412,7 +4414,7 @@ static int __init hugepages_setup(char *s) return 0; invalid: - pr_warn("HugeTLB: Invalid hugepages parameter %s\n", p); + pr_warn("Invalid hugepages parameter %s\n", p); hugepages_clear_pages_in_node(); return -EINVAL; } @@ -4431,7 +4433,7 @@ static int __init hugepagesz_setup(char *s) struct hstate *h; if (!hugepages_supported()) { - pr_warn("HugeTLB: hugepages unsupported, ignoring hugepagesz=%s cmdline\n", s); + pr_warn("hugepages unsupported, ignoring hugepagesz=%s cmdline\n", s); return 0; } @@ -4439,7 +4441,7 @@ static int __init hugepagesz_setup(char *s) size = (unsigned long)memparse(s, NULL); if (!arch_hugetlb_valid_size(size)) { - pr_err("HugeTLB: unsupported hugepagesz=%s\n", s); + pr_err("unsupported hugepagesz=%s\n", s); return -EINVAL; } @@ -4454,7 +4456,7 @@ static int __init hugepagesz_setup(char *s) */ if (!parsed_default_hugepagesz || h != &default_hstate || default_hstate.max_huge_pages) { - pr_warn("HugeTLB: hugepagesz=%s specified twice, ignoring\n", s); + pr_warn("hugepagesz=%s specified twice, ignoring\n", s); return -EINVAL; } @@ -4484,21 +4486,21 @@ static int __init default_hugepagesz_setup(char *s) int i; if (!hugepages_supported()) { - pr_warn("HugeTLB: hugepages unsupported, ignoring default_hugepagesz=%s cmdline\n", + pr_warn("hugepages unsupported, ignoring default_hugepagesz=%s cmdline\n", s); return 0; } parsed_valid_hugepagesz = false; if (parsed_default_hugepagesz) { - pr_err("HugeTLB: default_hugepagesz previously specified, ignoring %s\n", s); + pr_err("default_hugepagesz previously specified, ignoring %s\n", s); return -EINVAL; } size = (unsigned long)memparse(s, NULL); if (!arch_hugetlb_valid_size(size)) { - pr_err("HugeTLB: unsupported default_hugepagesz=%s\n", s); + pr_err("unsupported default_hugepagesz=%s\n", s); return -EINVAL; } @@ -4763,7 +4765,7 @@ static void hugetlb_vm_op_open(struct vm_area_struct *vma) vma->vm_private_data = NULL; hugetlb_vma_lock_alloc(vma); } else { - pr_warn("HugeTLB: vma_lock already exists in %s.\n", __func__); + pr_warn("vma_lock already exists in %s.\n", __func__); } } else { hugetlb_vma_lock_alloc(vma); From f5cfc8e3b20ad71949a1e5623ff45882c063a71b Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Sat, 10 Oct 2026 10:40:42 +0800 Subject: [PATCH 154/213] mm/madvise: add cond_resched() in THP branch of madvise_cold_or_pageout_pte_range() During stress testing with stress-ng on a 256-CPU system, a softlockup was observed when madvise(MADV_COLD) was called on a 31GB memory region backed by transparent hugepages. The region contains 15935 THP PMD entries, and the softlockup was triggered after 14485 of them had been processed. The THP branch of madvise_cold_or_pageout_pte_range() has no scheduling point. walk_pmd_range() invokes it once per 2MB PMD entry with no cond_resched() anywhere in the loop. Each iteration calls pmdp_invalidate() which issues a synchronous TLB shootdown (dsb ish) waiting for all CPUs to acknowledge. This operation can be time-consuming under heavy system load, causing the accumulated time across all PMD entries to exceed the softlockup threshold (20s) and panic when softlockup_panic=1. On a CONFIG_PREEMPT_NONE=y kernel, there is no other opportunity to reschedule during this entire traversal. The regular PTE path already has an unconditional cond_resched() at the function tail (introduced by commit 9c276cc65a58 ("mm: introduce MADV_COLD") and commit 1a4e58cce84e ("mm: introduce MADV_PAGEOUT") when MADV_COLD and MADV_PAGEOUT were first introduced) and a conditional one inside the loop (commit b2f557a21bc8 ("mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()")). The THP branch never received equivalent treatment. Fix by adding cond_resched() at the huge_unlock exit path after spin_unlock(ptl) and reclaim_pages(), which is the single exit point for all goto paths in the THP branch. Crash log: watchdog: BUG: soft lockup - CPU#63 stuck for 23s! [stress-ng-vm:1665914] ... Call trace: pmdp_invalidate+0x48/0x68 madvise_cold_or_pageout_pte_range+0x84c/0xb00 walk_pmd_range.isra.0+0x1ac/0x3b0 walk_pud_range+0x120/0x190 walk_pgd_range+0x170/0x1b8 __walk_page_range+0x184/0x198 walk_page_range_vma+0x94/0x128 madvise_cold+0xd8/0x198 madvise_vma_behavior+0x234/0x638 madvise_walk_vmas.constprop.0+0xdc/0x210 madvise_do_behavior+0xa8/0x328 do_madvise+0xd8/0x140 __arm64_sys_madvise+0x2c/0xe0 Link: https://lore.kernel.org/20261010024042.4070048-1-albin_yang@163.com Signed-off-by: Wei Yang Signed-off-by: Andrew Morton Reviewed-by: Gregory Price (Meta) Acked-by: David Hildenbrand (Arm) Acked-by: Lorenzo Stoakes (ARM) Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: --- mm/madvise.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/madvise.c b/mm/madvise.c index 8f407610c089..8702b082a02d 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -454,6 +454,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, spin_unlock(ptl); if (pageout) reclaim_pages(&folio_list); + cond_resched(); return 0; } From 984c0700854e35779a2bcce6b02783e709b6efa2 Mon Sep 17 00:00:00 2001 From: Zi Yan Date: Mon, 12 Oct 2026 22:20:16 -0400 Subject: [PATCH 155/213] mm/rmap: don't discard lazyfree THPs at PMD level __discard_anon_folio_pmd_locked() clears a lazyfree THP's PMD before it determines whether the folio can be discarded, and restores it if the folio is dirty or has extra references. It is done under only the PMD lock and the anon_vma read lock inside try_to_unmap_one() and no mmap lock is held. The temporarily none PMD causes UAF. A parallel munmap() or mremap() is supposed to serialize with it on the PMD lock, but zap_pmd_range() (called by munmap()) and move_page_tables() (called by mremap()) do not take the lock when they see the temporarily none PMD and skip the PMD. There is no more page table walk after the none PMD is restored: - munmap(): free_pgtables() waits in unlink_anon_vmas() for the anon_vma lock, then free_pmd_range() drops the restored PMD as bad without removing the rmap, so the folio keeps a stale mapcount. - whole-VMA MREMAP_DONTUNMAP: dontunmap_complete() unlinks the source VMA from its anon_vma while the restored PMD still maps the folio. - plain mremap(): the THP is not moved but unmapped, so the THP's data is lost. In the first two cases, the folio stays mapped, but no VMA that maps it is linked to its anon_vma. When the anon_vma is freed, it is still reachable when one rmap walks the folio. Racing lazyfree reclaim against MREMAP_DONTUNMAP as an unprivileged user reproduces a KASAN use-after-free in folio_lock_anon_vma_read() as reported in the Closes link. One solution is to use an invalidated PMD instead of a cleared one until the discard is committed, but it needs arch fixes first. 1. GUP-fast still follows an invalidated PMD on riscv, LoongArch and s390. 2. sparc64 miscounts thp_pte_count when an invalidated PMD is set again or cleared. To keep the fix simple for backport, stop discarding lazyfree THPs at PMD level instead. try_to_unmap_one() splits the PMD and discards the folio at PTE level. unmap_huge_pmd_locked() and __discard_anon_folio_pmd_locked() are left in place, unused, to minimize this fix for backport and the invalidated PMD approach is planned as a follow-up series to use them again. Link: https://lore.kernel.org/20261012-lazyfree-stable-fix-v2-1-6387d6e30063@nvidia.com Fixes: 735ecdfaf4e8 ("mm/vmscan: avoid split lazyfree THP during shrink_folio_list()") Signed-off-by: Zi Yan Signed-off-by: Andrew Morton Reported-by: Kyle Zeng Closes: https://lore.kernel.org/r/20261009165214.40212-2-kylebot@openai.com Reviewed-by: Kyle Zeng Reviewed-by: Lance Yang Acked-by: David Hildenbrand (Arm) Reviewed-by: Lorenzo Stoakes (ARM) Tested-by: JongHyuk Kim(MalHyuk) Link: https://lore.kernel.org/CADQRuqvXQ=YE=7T3oR66vTxE33pi=ygEeZKPTOU=cALZL0Figw@mail.gmail.com Assisted-by: LLM Cc: Rik van Riel Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Harry Yoo Cc: Jann Horn Cc: --- mm/rmap.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 805db93fe042..1131b76bbbc2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2275,17 +2275,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } if (!pvmw.pte) { - if (folio_test_lazyfree(folio)) { - if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) - goto walk_done; - /* - * unmap_huge_pmd_locked has either already marked - * the folio as swap-backed or decided to retain it - * due to GUP or speculative references. - */ - goto walk_abort; - } - if (flags & TTU_SPLIT_HUGE_PMD) { /* * We temporarily have to drop the PTL and From cbe319858c1d20f503f0e806e94d7df495427cef Mon Sep 17 00:00:00 2001 From: "Jose A. Perez de Azpillaga" Date: Tue, 6 Oct 2026 08:39:01 +0200 Subject: [PATCH 156/213] selftests/mm: check MREMAP_DONTUNMAP mlock accounting MREMAP_DONTUNMAP keeps the source VMA in place, but clears its mlock flags while setting them on the destination VMA. The "mm/mremap: fix two issues with MREMAP_DONTUNMAP" series fixed two cases where this leaked mm->locked_vm: - an unfaulted mlock-on-fault VMA moved behind itself self-merges, so the single resulting VMA loses the flags without the accounting being dropped; - a partial mremap() moves only part of the range, leaving the pages which are not moved accounted as locked in a VMA whose flags were cleared. Add a test which mlock2()s a source mapping, moves all or part of it with MREMAP_DONTUNMAP, unmaps everything and checks that VmLck is back to its value from before, and run it for the self-merge case and for a partial move with and without MLOCK_ONFAULT. Verified on x86_64: the three cases fail on mm-unstable with the two fixes reverted and pass with them applied. Link: https://lore.kernel.org/20261006064120.6796-1-azpijr@gmail.com Signed-off-by: Jose A. Perez de Azpillaga Signed-off-by: Andrew Morton Acked-by: Lorenzo Stoakes (ARM) Cc: David Hildenbrand Cc: Shuah Khan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko --- tools/testing/selftests/mm/mremap_dontunmap.c | 63 ++++++++++++++++++- 1 file changed, 62 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/mremap_dontunmap.c b/tools/testing/selftests/mm/mremap_dontunmap.c index 96ba537facf7..5ca33d9d42af 100644 --- a/tools/testing/selftests/mm/mremap_dontunmap.c +++ b/tools/testing/selftests/mm/mremap_dontunmap.c @@ -7,6 +7,7 @@ */ #define _GNU_SOURCE #include +#include #include #include #include @@ -15,6 +16,7 @@ #include #include "kselftest.h" +#include "vm_util.h" unsigned long page_size; char *page_buffer; @@ -335,6 +337,62 @@ static void mremap_dontunmap_partial_mapping_overwrite(void) ksft_test_result_pass("%s\n", __func__); } +/* VmLck from /proc/self/status, which is mm->locked_vm in kB. */ +static unsigned long locked_vm_kb(void) +{ + unsigned long kb; + char buf[256]; + FILE *fp; + + fp = fopen("/proc/self/status", "r"); + BUG_ON(!fp, "unable to open /proc/self/status"); + BUG_ON(!check_for_pattern(fp, "VmLck:", buf, sizeof(buf)) || + sscanf(buf, "VmLck: %lu kB", &kb) != 1, "unable to read VmLck"); + fclose(fp); + + return kb; +} + +/* + * MREMAP_DONTUNMAP clears the mlock flags of the source VMA and sets them on + * the destination, and mm->locked_vm has to follow. mlock2() a source of + * num_pages and move its first move_pages to gap pages past its end: + * + * |------ source ------|... gap ...|-- dest --| + * + * then unmap everything and check that VmLck is back where it started. With + * no gap the destination is adjacent to the source and could merge with it, + * otherwise the gap is left PROT_NONE so that it cannot. + */ +static void mremap_dontunmap_mlock(const char *desc, unsigned long num_pages, + unsigned long move_pages, unsigned long gap, + int mlock_flags) +{ + unsigned long size = (num_pages + gap + move_pages) * page_size; + unsigned long locked = locked_vm_kb(); + void *source, *dest; + + source = mmap(NULL, size, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + BUG_ON(source == MAP_FAILED, "mmap"); + BUG_ON(mprotect(source, num_pages * page_size, + PROT_READ | PROT_WRITE) == -1, "mprotect"); + dest = source + (num_pages + gap) * page_size; + + if (syscall(__NR_mlock2, source, num_pages * page_size, mlock_flags)) { + ksft_test_result_skip("%s: %s: mlock2: %s\n", __func__, desc, + strerror(errno)); + BUG_ON(munmap(source, size) == -1, "unable to unmap mappings"); + return; + } + + BUG_ON(mremap(source, move_pages * page_size, move_pages * page_size, + MREMAP_DONTUNMAP | MREMAP_MAYMOVE | MREMAP_FIXED, + dest) != dest, "mremap"); + + BUG_ON(munmap(source, size) == -1, "unable to unmap mappings"); + ksft_test_result(locked_vm_kb() == locked, "%s: %s\n", __func__, desc); +} + int main(void) { ksft_print_header(); @@ -348,7 +406,7 @@ int main(void) ksft_finished(); } - ksft_set_plan(5); + ksft_set_plan(8); // Keep a page sized buffer around for when we need it. page_buffer = @@ -361,6 +419,9 @@ int main(void) mremap_dontunmap_simple_fixed(); mremap_dontunmap_partial_mapping(); mremap_dontunmap_partial_mapping_overwrite(); + mremap_dontunmap_mlock("onfault self-merge", 1, 1, 0, MLOCK_ONFAULT); + mremap_dontunmap_mlock("partial", 3, 2, 1, 0); + mremap_dontunmap_mlock("onfault partial", 3, 2, 1, MLOCK_ONFAULT); BUG_ON(munmap(page_buffer, page_size) == -1, "unable to unmap page buffer"); From 1b0f9f70a30feb9271aad2639c67f44c2129c84d Mon Sep 17 00:00:00 2001 From: Enze Li Date: Fri, 9 Oct 2026 22:20:10 +0800 Subject: [PATCH 157/213] mm/hugetlb: move hugetlb_sysfs_initialized into hugetlb_register_all_nodes() hugetlb_sysfs_init() guards the "hugetlb_sysfs_initialized = true" assignment with an #ifdef CONFIG_NUMA block, while the following hugetlb_register_all_nodes() call is left unguarded and relies on the empty !CONFIG_NUMA stub. Move the assignment into the NUMA version of hugetlb_register_all_nodes(), before it registers the online nodes, so the #ifdef in hugetlb_sysfs_init() can go away. Keep the !CONFIG_NUMA stub, so the call site is unconditional in both configurations. No functional change; this is an #ifdef cleanup. Link: https://lore.kernel.org/20261009142010.552614-1-lienze@kylinos.cn Signed-off-by: Enze Li Signed-off-by: Andrew Morton Suggested-by: Muchun Song Acked-by: David Hildenbrand (Arm) Reviewed-by: Muchun Song Cc: Oscar Salvador Cc: Enze Li --- mm/hugetlb_sysfs.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c index 326a54b4d991..c89e4c847575 100644 --- a/mm/hugetlb_sysfs.c +++ b/mm/hugetlb_sysfs.c @@ -462,6 +462,8 @@ static void __init hugetlb_register_all_nodes(void) { int nid; + hugetlb_sysfs_initialized = true; + for_each_online_node(nid) hugetlb_register_node(node_devices[nid]); } @@ -495,8 +497,5 @@ void __init hugetlb_sysfs_init(void) pr_err("HugeTLB: Unable to add hstate %s\n", h->name); } -#ifdef CONFIG_NUMA - hugetlb_sysfs_initialized = true; -#endif hugetlb_register_all_nodes(); } From 9f5c2aec33884405606f1590f4700dea3e04f43d Mon Sep 17 00:00:00 2001 From: "Zenghui Yu (Huawei)" Date: Mon, 12 Oct 2026 00:35:27 +0800 Subject: [PATCH 158/213] selftests/mm: fix uffd move-pmd test crash on 64K base page systems uffd_move_pmd_test()/uffd_move_test_common() splits the test area into chunks of chunk_size and moves them one by one. On arm64 with 64K base pages the chunk_size (read_pmd_pagesize()) is 512MB, which is larger than the whole 32MB (UFFD_TEST_MEM_SIZE) test area: step_count = nr_pages / (chunk_size / page_size) == 0 Because chunk_size > page_size, the test then aligns both areas up to 512MB, which points far outside of the 32MB mappings, and decrements step_count, which underflows to ULONG_MAX. The verification loop then runs away and the first dereference of area_src faults with a SIGSEGV. Skip the test when a single chunk cannot fit into the test area. Link: https://lore.kernel.org/20261011163527.95988-1-zenghui.yu@linux.dev Fixes: a2bf6a9ca805 ("selftests/mm: add UFFDIO_MOVE ioctl test") Signed-off-by: Zenghui Yu (Huawei) Signed-off-by: Andrew Morton Reviewed-by: Jose A. Perez de Azpillaga Tested-by: Sarthak Sharma Cc: Mike Rapoport Cc: Peter Xu Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko --- tools/testing/selftests/mm/uffd-unit-tests.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/selftests/mm/uffd-unit-tests.c index ef9b3956bdcf..2a97ba1d6feb 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -1937,6 +1937,11 @@ uffd_move_test_common(uffd_global_test_opts_t *gopts, args.gopts = gopts; + if (chunk_size > gopts->nr_pages * gopts->page_size) { + uffd_test_skip("chunk size does not fit in the test area"); + return; + } + /* Prevent source pages from being mapped more than once */ if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DONTFORK)) err("madvise(MADV_DONTFORK) failure"); From 4cd78a2dcfbb13f049f841b98763896a3d1d7750 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Tue, 13 Oct 2026 18:37:20 +0100 Subject: [PATCH 159/213] mm: page_alloc: make defrag_mode retries follow the promoted order Since commit 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm in defrag_mode"), direct reclaim and compaction for non-movable requests under defrag_mode run at pageblock_order, to produce the whole blocks that ALLOC_NOFRAGMENT needs. The retry decisions that follow still use the request order. An order-0 request can therefore retry indefinitely without ever reaching the ALLOC_NOFRAGMENT fallback: - Reclaim at pageblock_order gives up after one pass as soon as a zone looks compaction_ready(), and do_try_to_free_pages() then returns 1 even though nothing was reclaimed. It returns before the retry that would reclaim memory.low-protected cgroups, so when most memory is protected, the pass that did run finds next to nothing. - Compaction at pageblock_order fails or is deferred. - should_reclaim_retry() takes the reported progress as progress for the order-0 request and resets no_progress_loops. The request retries. Order 1-3 requests loop the same way, and should_compact_retry() also checks their pageblock_order compaction result against the request order. On a production host (64G, defrag_mode, memory.low covering most of the workload), 95% of direct reclaim runs were order-9 runs that returned 1 with nothing reclaimed, at up to 60k runs per second. Across ~200M should_reclaim_retry() calls in a day, no_progress_loops never left 0. The spinning allocations were SLUB slab refills for inode and dentry caches. The time spent registers as memory pressure, and pressure-based OOM killing takes down both workloads and system services. For promoted requests: - Reclaim progress does not reset no_progress_loops, as for costly orders. - should_compact_retry() checks the compaction result at the promoted order and does not retry COMPACT_SKIPPED, since the request can fall back. The compaction priority floor and the COMPACT_SUCCESS retry limit stay those of the request order, so a non-costly request still gets its COMPACT_PRIO_SYNC_FULL pass before it falls back. When a promoted request takes the fallback, reset the retry counters, so that the fallback attempt gets a full retry budget before the OOM killer is considered. A request whose reclaim was not promoted spent its budget on the request itself and keeps its counters, as Usama Arif pointed out. __alloc_pages_slowpath() computes the promoted order once per iteration and keeps it in alloc_context for direct reclaim, direct compaction and the two retry helpers, so no callee has to recompute it. Harry Yoo asked for the promoted order to be explicit rather than derived in each callee, and Vlastimil Babka suggested alloc_context over new function parameters. In a VM reproducer (32G, defrag_mode, inode churn under memory.low): before after should_reclaim_retry() calls 63M 293k peak memory pressure (PSI some avg10) 99% 12% File creation runs 5.7x faster. [kas@kernel.org: s/order_nofrag_promoted/order_defrag_promoted/, per hannes] Link: https://lore.kernel.org/atDL-nSFP2KVU7cI@thinkstation Link: https://lore.kernel.org/as5q14OsbK2CmOrh@thinkstation Fixes: 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm in defrag_mode") Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Reviewed-by: Vlastimil Babka (SUSE) Assisted-by: LLM Cc: Johannes Weiner Cc: David Hildenbrand Cc: Harry Yoo Cc: Usama Arif Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Brendan Jackman Cc: Zi Yan Cc: Shakeel Butt Cc: --- mm/page_alloc.c | 65 +++++++++++++++++++++++++++++-------------------- mm/page_alloc.h | 2 ++ 2 files changed, 41 insertions(+), 26 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ff7b9acf0144..914e3c9375f2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4117,6 +4117,12 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, return page; } +static inline bool order_defrag_promoted(unsigned int order, + const struct alloc_context *ac) +{ + return ac->reclaim_compact_order != order; +} + /* * Maximum number of compaction retries with a progress before OOM * killer is consider as the only way to move forward. @@ -4139,22 +4145,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, .order = order, .page = NULL, }; - int compact_order = order; - - /* - * If fallbacks are not permitted (defrag_mode), we either - * need to reclaim space in a block of matching type, or clear - * out an entire block to allow __rmqueue_claim() to convert. - * - * Reclaim by itself is primarily freeing space in movable - * blocks, since that's where the LRU pages live. So this - * works for movable requests, but not for others. - * - * For those, promote the order to help make blocks, instead - * of spinning in reclaim alone unproductively. - */ - if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) - compact_order = max(order, pageblock_order); + int compact_order = ac->reclaim_compact_order; if (!compact_order) return NULL; @@ -4247,7 +4238,7 @@ should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order, int retries = *compaction_retries; enum compact_priority priority = *compact_priority; - if (!order) + if (!ac->reclaim_compact_order) return false; if (fatal_signal_pending(current)) @@ -4256,10 +4247,12 @@ should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order, /* * Compaction was skipped due to a lack of free order-0 * migration targets. Continue if reclaim can help. + * Promoted requests fall back instead. */ if (compact_result == COMPACT_SKIPPED) { - ret = compaction_zonelist_suitable(ac, order, alloc_flags, - gfp_mask); + if (!order_defrag_promoted(order, ac)) + ret = compaction_zonelist_suitable(ac, order, + alloc_flags, gfp_mask); goto out; } @@ -4458,11 +4451,7 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; bool drained = false; - int reclaim_order = order; - - /* Match the slowpath compaction promotion in __alloc_pages_direct_compact */ - if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) - reclaim_order = max(order, pageblock_order); + int reclaim_order = ac->reclaim_compact_order; psi_memstall_enter(&pflags); *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac); @@ -4638,9 +4627,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, /* * Costly allocations might have made a progress but this doesn't mean * their order will become available due to high fragmentation so - * always increment the no progress counter for them + * always increment the no progress counter for them. The same goes for + * promoted requests: their reclaim reports progress when it backs off + * for compaction without freeing anything. */ - if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) + if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER && + !order_defrag_promoted(order, ac)) *no_progress_loops = 0; else (*no_progress_loops)++; @@ -4924,6 +4916,22 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* If allocation has taken excessively long, warn about it */ check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time); + /* + * If fallbacks are not permitted (defrag_mode), we either + * need to reclaim space in a block of matching type, or clear + * out an entire block to allow __rmqueue_claim() to convert. + * + * Reclaim by itself is primarily freeing space in movable + * blocks, since that's where the LRU pages live. So this + * works for movable requests, but not for others. + * + * For those, promote the order to help make blocks, instead + * of spinning in reclaim alone unproductively. + */ + ac->reclaim_compact_order = order; + if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) + ac->reclaim_compact_order = max(order, pageblock_order); + /* Try direct reclaim and then allocating */ if (!compact_first) { page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, @@ -5005,6 +5013,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Reclaim/compaction failed to prevent the fallback */ if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT)) { alloc_flags &= ~ALLOC_NOFRAGMENT; + /* Retries went into making blocks, start the fallback afresh */ + if (order_defrag_promoted(order, ac)) { + no_progress_loops = 0; + compaction_retries = 0; + } goto retry; } diff --git a/mm/page_alloc.h b/mm/page_alloc.h index b9259deddb59..2b1017a91928 100644 --- a/mm/page_alloc.h +++ b/mm/page_alloc.h @@ -93,6 +93,8 @@ struct alloc_context { bool spread_dirty_pages; /* Only flags that are global to the whole allocation go here. */ unsigned int alloc_flags; + /* Set by __alloc_pages_slowpath() before reclaim and compaction */ + unsigned int reclaim_compact_order; }; /* From bd918fca02076763269a7cba474d1cff86ec5954 Mon Sep 17 00:00:00 2001 From: Enze Li Date: Wed, 14 Oct 2026 11:11:47 +0800 Subject: [PATCH 160/213] mm/hugetlb: use sizeof(buf) in string_get_size() calls Replace the hardcoded 32 with sizeof(buf) at the remaining four call sites, for consistency with the rest of the file. Link: https://lore.kernel.org/20261014031147.444733-1-lienze@kylinos.cn Signed-off-by: Enze Li Signed-off-by: Andrew Morton Acked-by: Muchun Song Cc: Oscar Salvador Cc: David Hildenbrand --- mm/hugetlb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e3367b319bc5..27e8136b1815 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3455,7 +3455,7 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) if (i == h->max_huge_pages_node[nid]) return; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); pr_warn("allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n", h->max_huge_pages_node[nid], buf, nid, i); h->max_huge_pages -= (h->max_huge_pages_node[nid] - i); @@ -3482,7 +3482,7 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, if (allocated < h->max_huge_pages) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); pr_warn("allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", h->max_huge_pages, buf, allocated); h->max_huge_pages = allocated; @@ -3693,7 +3693,7 @@ static void __init report_hugepages(void) for_each_hstate(h) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); pr_info("registered %s page size, pre-allocated %ld pages\n", buf, h->nr_huge_pages); pr_info("%d KiB vmemmap can be freed for a %s page\n", @@ -4205,7 +4205,7 @@ static int __init hugetlb_init(void) char buf[32]; string_get_size(huge_page_size(&default_hstate), - 1, STRING_UNITS_2, buf, 32); + 1, STRING_UNITS_2, buf, sizeof(buf)); pr_warn("Ignoring hugepages=%lu associated with %s page size\n", default_hstate.max_huge_pages, buf); pr_warn("Using hugepages=%lu for number of default huge pages\n", From 514c9247db305922591d806ec65d3c3fe420538d Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Wed, 14 Oct 2026 13:53:11 +0200 Subject: [PATCH 161/213] mm/userfaultfd: only allocate anon_vma in CoW mappings find_vma_and_prepare_anon() can end up allocating an anon_vma in VMAs that are not CoW mappings, which is wrong. If we have a MAP_SHARED R/O file mapped, we lack VM_SHARED (but do have VM_MAYSHARE) and lack VM_MAYWRITE. While uffd later rejects such mappings (uffd registration requires VM_MAYWRITE, see comment in validate_dst_vma) find_vma_and_prepare_anon() still wrongly allocates an anon vma before it validates the vma. This in turn later confuses copy_page_range()->vma_needs_copy() such that we end up calling copy_hugetlb_page_range() with ranges that actually do not contain *any* per-pagetable information that would require copying, which is the basis for PMD-table sharing in the first place. copy_hugetlb_page_range() should not have to worry about shared PMD tables at all: it doesn't make any sense to copy page tables that are shared, as they can be fully recreated later. Ending up in the function with shareable page tables is the real issue. If a destination PMD table is shared in copy_hugetlb_page_range(), it might get concurrently unshared as we are not holding relevant locks, resulting in us using a page table that is no longer mapped and possibly freed after calling: dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz); To fix it, do not allocate an anon_vma in non-CoW mappings and add a safety guard to copy_hugetlb_page_range(), making sure that we never end up messing with shared PMD tables in there. Before commit 867a43a34ff8a ("userfaultfd: use per-vma locks in userfaultfd operations") we would always validate the VMA before allocating an anon_vma, and consequently never wrongly allocate an anon_vma. As uffd_lock_vma() is part of the same lookup machinery and also contains a misleading VM_SHARED check, change that one as well directly for consistency. A follow-up cleanup will stop taking the hugetlb_vma_lock_read(src_vma) lock in copy_hugetlb_page_range() entirely. Further, the usage of VM_SHARED in userfaultfd code should be cleaned up later to focus on CoW vs. non-CoW mappings. Link: https://lore.kernel.org/20261014-copy_hugetlb_page_range-v1-1-5b68e03fe7fe@kernel.org Fixes: 867a43a34ff8 ("userfaultfd: use per-vma locks in userfaultfd operations") Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton Reported-by: Kyle Zeng Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Kyle Zeng Tested-by: Kyle Zeng Acked-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Tested-by: Lance Yang Cc: Oscar Salvador Cc: Peter Xu Cc: Miaohe Lin Cc: Lokesh Gidra Cc: Liam R. Howlett Cc: --- mm/hugetlb.c | 24 ++++++++++++------------ mm/userfaultfd.c | 5 ++--- 2 files changed, 14 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 27e8136b1815..da5d04edf7c1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4973,10 +4973,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, raw_write_seqcount_begin(&src->write_protect_seq); } else { /* - * For shared mappings the vma lock must be held before - * calling hugetlb_walk() in the src vma. Otherwise, the - * returned ptep could go away if part of a shared pmd and - * another thread calls huge_pmd_unshare. + * Shared mappings only need copying when we could have + * information stored in page tables (e.g., userfaultfd PTE + * bits or markers), which is incompatible with PMD sharing. + * Still take the hugetlb VMA lock for now to satisfy + * hugetlb_walk()'s locking requirements. */ hugetlb_vma_lock_read(src_vma); } @@ -4984,6 +4985,13 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask = hugetlb_mask_last_page(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; + + if (WARN_ON_ONCE(want_pmd_share(src_vma, addr) || + want_pmd_share(dst_vma, addr))) { + ret = -EINVAL; + break; + } + src_pte = hugetlb_walk(src_vma, addr, sz); if (!src_pte) { addr |= last_addr_mask; @@ -4995,14 +5003,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, break; } -#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING - /* If the pagetables are shared, there is nothing to do */ - if (ptdesc_pmd_is_shared(virt_to_ptdesc(dst_pte))) { - addr |= last_addr_mask; - continue; - } -#endif - dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index da67d35cd954..405f2d72f204 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -115,8 +115,7 @@ struct vm_area_struct *find_vma_and_prepare_anon(struct mm_struct *mm, vma = vma_lookup(mm, addr); if (!vma) vma = ERR_PTR(-ENOENT); - else if (!(vma->vm_flags & VM_SHARED) && - unlikely(anon_vma_prepare(vma))) + else if (vma_is_cow_mapping(vma) && unlikely(anon_vma_prepare(vma))) vma = ERR_PTR(-ENOMEM); return vma; @@ -145,7 +144,7 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, * We know we're going to need an anon rmap, so check that * early. */ - if (!(vma->vm_flags & VM_SHARED) && unlikely(!vma_has_anon_rmap(vma))) + if (vma_is_cow_mapping(vma) && unlikely(!vma_has_anon_rmap(vma))) vma_end_read(vma); else return vma; From b83970d6962d12f0d5a625baafddaa44456bec28 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:34 +0100 Subject: [PATCH 162/213] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse() Patch series "mm/collapse: separate a collapse from its callers", v5. There is no line between the collapse engine and the callers that ask for a collapse. khugepaged.c holds both, and they reach into each other. - Sixteen tests through the collapse path read cc->is_khugepaged to work out what they are allowed to do, when every one of those decisions was made by the caller before it asked. - collapse_single_pmd() does both halves of a collapse behind one call. Where it gives up mmap_lock between them is buried in the anonymous scan on one side and ahead of the file scan on the other. Start drawing the line. The caller states what it allows in a policy, and the collapse code reads the policy instead of asking who called. The scan of a PTE table and the collapse that follows get a function each, with the mmap_lock drop between them in one place. This patch (of 9): madvise_collapse() holds an mmgrab() reference across its work. It is redundant. Every caller already holds mm_users: - madvise(2) works on current->mm, which lives as long as the task is in the syscall; - process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns; - io_uring passes current->mm; - DAMON takes one with get_task_mm() and drops it after the call. Drop the mmgrab()/mmdrop() pair. Link: https://lore.kernel.org/20261014171145.3292736-1-kas@kernel.org Link: https://lore.kernel.org/20261014171145.3292736-2-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/khugepaged.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 911f562a232e..68ae96eac97e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -3255,7 +3255,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, cc->is_khugepaged = false; cc->progress = 0; - mmgrab(mm); lru_add_drain_all(); for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) { @@ -3311,7 +3310,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, } out_nolock: mmap_assert_locked(mm); - mmdrop(mm); kfree(cc); return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 From 47cda827a05f3dc1e1c05646db7acd430f2cc6d5 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:35 +0100 Subject: [PATCH 163/213] mm/khugepaged: count collapses where khugepaged makes them collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and tests cc->is_khugepaged to know whether it should: the counter belongs to the daemon, and MADV_COLLAPSE must not touch it. The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called. Link: https://lore.kernel.org/20261014171145.3292736-3-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/khugepaged.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 68ae96eac97e..cb13264a4928 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2841,10 +2841,8 @@ static enum scan_result collapse_single_pmd(unsigned long addr, mmap_assert_locked(mm); - if (vma_is_anonymous(vma)) { - result = collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); - goto end; - } + if (vma_is_anonymous(vma)) + return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); file = get_file(vma->vm_file); pgoff = linear_page_index(vma, addr); @@ -2880,9 +2878,6 @@ static enum scan_result collapse_single_pmd(unsigned long addr, result = SCAN_SUCCEED; mmap_read_unlock(mm); } -end: - if (cc->is_khugepaged && result == SCAN_SUCCEED) - ++khugepaged_pages_collapsed; return result; } @@ -2959,6 +2954,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, *result = collapse_single_pmd(khugepaged_scan.address, vma, &lock_dropped, cc); + if (*result == SCAN_SUCCEED) + khugepaged_pages_collapsed++; /* move to next address */ khugepaged_scan.address += HPAGE_PMD_SIZE; if (lock_dropped) From d950ec79fcbb000459b46820e8f7ed69b58a6dc3 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:36 +0100 Subject: [PATCH 164/213] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source. mthp_collapse() then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either. Name it for what a set bit means, and update the comments that named it. No functional change. Link: https://lore.kernel.org/20261014171145.3292736-4-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/khugepaged.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index cb13264a4928..e30f210040b4 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -115,8 +115,8 @@ struct collapse_control { /* nodemask for allocation fallback */ nodemask_t alloc_nmask; - /* Each bit represents a single occupied (!none/zero) page. */ - DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE); + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); }; /** @@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); - bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE); + bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE); } static void release_pte_folio(struct folio *folio) @@ -1512,15 +1512,15 @@ static unsigned int max_order_from_offset(unsigned int offset) * mthp_collapse() consumes the bitmap that is generated during * collapse_scan_pmd() to determine what regions and mTHP orders fit best. * - * Each bit in cc->mthp_present_ptes represents a single occupied (!none/zero) - * page. We start at the PMD order and check if it is eligible for collapse; + * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse + * source. We start at the PMD order and check if it is eligible for collapse; * if not, we check the left and right halves of the PTE page table we are * examining at a lower order. * - * For each of these, we determine how many PTE entries are occupied in the - * range of PTE entries we propose to collapse, then we compare this to a - * threshold number of PTE entries which would need to be occupied for a - * collapse to be permitted at that order (accounting for max_ptes_none). + * For each of these, we count the eligible PTEs in the range we propose to + * collapse, then we compare this to the number of eligible PTEs the range + * would need for a collapse to be permitted at that order (accounting for + * max_ptes_none). * * If a collapse is permitted, we attempt to collapse the PTE range into a * mTHP. @@ -1529,7 +1529,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long address, int referenced, int unmapped, struct collapse_control *cc, unsigned long enabled_orders) { - unsigned int nr_occupied_ptes, nr_ptes, max_ptes_none; + unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none; enum scan_result last_result = SCAN_FAIL; int collapsed = 0; bool alloc_failed = false; @@ -1544,18 +1544,18 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, goto next_order; max_ptes_none = collapse_max_ptes_none(cc, NULL, order); - nr_occupied_ptes = bitmap_weight_from(cc->mthp_present_ptes, offset, + nr_eligible_ptes = bitmap_weight_from(cc->eligible_ptes, offset, offset + nr_ptes); /* * Swap PTEs accepted during the scan are counted in @unmapped, - * not in the present-PTE bitmap. Account them for the PMD-order + * not in cc->eligible_ptes. Account them for the PMD-order * candidate. */ if (is_pmd_order(order)) - nr_occupied_ptes += unmapped; + nr_eligible_ptes += unmapped; - if (nr_occupied_ptes >= nr_ptes - max_ptes_none) { + if (nr_eligible_ptes >= nr_ptes - max_ptes_none) { enum scan_result ret; collapse_address = address + offset * PAGE_SIZE; @@ -1761,8 +1761,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, } } - /* Set bit for occupied pages */ - __set_bit(i, cc->mthp_present_ptes); + /* The scan accepted this PTE as a collapse source */ + __set_bit(i, cc->eligible_ptes); /* * Record which node the original page is from and save this * information to cc->node_load[]. From 81f8bfda35004f544f65d3d150540b512f9b68c3 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:37 +0100 Subject: [PATCH 165/213] mm/collapse: add collapse.h for the collapse interface khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins. Start drawing that line. Add mm/collapse.h for what the two sides have to agree on: - enum scan_result - what the engine hands back; - struct collapse_control - the state a request carries. And two constants move with them: - KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT; - KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER. Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix. No functional change. Link: https://lore.kernel.org/20261014171145.3292736-5-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- MAINTAINERS | 1 + mm/collapse.h | 64 +++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 78 insertions(+), 66 deletions(-) create mode 100644 mm/collapse.h diff --git a/MAINTAINERS b/MAINTAINERS index 39abe7378b6f..5c38da7090db 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17452,6 +17452,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.h diff --git a/mm/collapse.h b/mm/collapse.h new file mode 100644 index 000000000000..b115034d9018 --- /dev/null +++ b/mm/collapse.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_COLLAPSE_H +#define __MM_COLLAPSE_H + +#include +#include +#include +#include + +#define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) +#define COLLAPSE_MIN_MTHP_ORDER 2 + +enum scan_result { + SCAN_FAIL, + SCAN_SUCCEED, + SCAN_NO_PTE_TABLE, + SCAN_PMD_MAPPED, + SCAN_EXCEED_NONE_PTE, + SCAN_EXCEED_SWAP_PTE, + SCAN_EXCEED_SHARED_PTE, + SCAN_PTE_NON_PRESENT, + SCAN_PTE_UFFD, + SCAN_PTE_MAPPED_HUGEPAGE, + SCAN_LACK_REFERENCED_PAGE, + SCAN_PAGE_NULL, + SCAN_SCAN_ABORT, + SCAN_PAGE_COUNT, + SCAN_PAGE_LRU, + SCAN_PAGE_LOCK, + SCAN_PAGE_ANON, + SCAN_PAGE_LAZYFREE, + SCAN_PAGE_COMPOUND, + SCAN_ANY_PROCESS, + SCAN_VMA_NULL, + SCAN_VMA_CHECK, + SCAN_ADDRESS_RANGE, + SCAN_DEL_PAGE_LRU, + SCAN_ALLOC_HUGE_PAGE_FAIL, + SCAN_CGROUP_CHARGE_FAIL, + SCAN_TRUNCATED, + SCAN_PAGE_HAS_PRIVATE, + SCAN_STORE_FAILED, + SCAN_COPY_MC, + SCAN_PAGE_FILLED, + SCAN_PAGE_DIRTY_OR_WRITEBACK, +}; + +struct collapse_control { + bool is_khugepaged; + + /* Num pages scanned per node */ + u32 node_load[MAX_NUMNODES]; + + /* Num pages scanned (see khugepaged_pages_to_scan) */ + unsigned int progress; + + /* nodemask for allocation fallback */ + nodemask_t alloc_nmask; + + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); +}; + +#endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index e30f210040b4..a6f59c54bc82 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -26,44 +26,10 @@ #include #include +#include "collapse.h" #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" - -enum scan_result { - SCAN_FAIL, - SCAN_SUCCEED, - SCAN_NO_PTE_TABLE, - SCAN_PMD_MAPPED, - SCAN_EXCEED_NONE_PTE, - SCAN_EXCEED_SWAP_PTE, - SCAN_EXCEED_SHARED_PTE, - SCAN_PTE_NON_PRESENT, - SCAN_PTE_UFFD, - SCAN_PTE_MAPPED_HUGEPAGE, - SCAN_LACK_REFERENCED_PAGE, - SCAN_PAGE_NULL, - SCAN_SCAN_ABORT, - SCAN_PAGE_COUNT, - SCAN_PAGE_LRU, - SCAN_PAGE_LOCK, - SCAN_PAGE_ANON, - SCAN_PAGE_LAZYFREE, - SCAN_PAGE_COMPOUND, - SCAN_ANY_PROCESS, - SCAN_VMA_NULL, - SCAN_VMA_CHECK, - SCAN_ADDRESS_RANGE, - SCAN_DEL_PAGE_LRU, - SCAN_ALLOC_HUGE_PAGE_FAIL, - SCAN_CGROUP_CHARGE_FAIL, - SCAN_TRUNCATED, - SCAN_PAGE_HAS_PRIVATE, - SCAN_STORE_FAILED, - SCAN_COPY_MC, - SCAN_PAGE_FILLED, - SCAN_PAGE_DIRTY_OR_WRITEBACK, -}; +#include "page_alloc.h" #define CREATE_TRACE_POINTS #include @@ -91,7 +57,6 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait); * * Note that these are only respected if collapse was initiated by khugepaged. */ -#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) unsigned int khugepaged_max_ptes_none __read_mostly; static unsigned int khugepaged_max_ptes_swap __read_mostly; static unsigned int khugepaged_max_ptes_shared __read_mostly; @@ -101,24 +66,6 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS); static struct kmem_cache *mm_slot_cache __ro_after_init; -#define KHUGEPAGED_MIN_MTHP_ORDER 2 - -struct collapse_control { - bool is_khugepaged; - - /* Num pages scanned per node */ - u32 node_load[MAX_NUMNODES]; - - /* Num pages scanned (see khugepaged_pages_to_scan) */ - unsigned int progress; - - /* nodemask for allocation fallback */ - nodemask_t alloc_nmask; - - /* Each bit marks a PTE the scan accepted as a collapse source */ - DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); -}; - /** * struct khugepaged_scan - cursor for scanning * @mm_head: the head of the mm list to scan @@ -267,7 +214,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj, unsigned long max_ptes_none; err = kstrtoul(buf, 10, &max_ptes_none); - if (err || max_ptes_none > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_none > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_none = max_ptes_none; @@ -292,7 +239,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj, unsigned long max_ptes_swap; err = kstrtoul(buf, 10, &max_ptes_swap); - if (err || max_ptes_swap > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_swap > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_swap = max_ptes_swap; @@ -318,7 +265,7 @@ static ssize_t max_ptes_shared_store(struct kobject *kobj, unsigned long max_ptes_shared; err = kstrtoul(buf, 10, &max_ptes_shared); - if (err || max_ptes_shared > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_shared > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_shared = max_ptes_shared; @@ -378,19 +325,19 @@ static unsigned int collapse_max_ptes_none(struct collapse_control *cc, if (is_pmd_order(order)) return max_ptes_none; /* - * for mTHP collapse with the sysctl value set to KHUGEPAGED_MAX_PTES_LIMIT, + * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, * scale the maximum number of PTEs to the order of the collapse. */ - if (max_ptes_none == KHUGEPAGED_MAX_PTES_LIMIT) + if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT) return (1 << order) - 1; /* - * For mTHP collapse of values other than 0 or KHUGEPAGED_MAX_PTES_LIMIT, + * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT, * emit a warning and return 0. */ if (max_ptes_none) pr_warn_once("mTHP collapse does not support max_ptes_none" " values other than 0 or %u, defaulting to 0.\n", - KHUGEPAGED_MAX_PTES_LIMIT); + COLLAPSE_MAX_PTES_LIMIT); return 0; } @@ -476,7 +423,7 @@ int __init khugepaged_init(void) return -ENOMEM; khugepaged_pages_to_scan = HPAGE_PMD_NR * 8; - khugepaged_max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT; + khugepaged_max_ptes_none = COLLAPSE_MAX_PTES_LIMIT; khugepaged_max_ptes_swap = HPAGE_PMD_NR / 8; khugepaged_max_ptes_shared = HPAGE_PMD_NR / 2; @@ -1601,8 +1548,8 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, * any smaller order enabled. When at the smallest order * we must always move to the next offset. */ - if (order > KHUGEPAGED_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + if (order > COLLAPSE_MIN_MTHP_ORDER && + (enabled_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1666,7 +1613,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, * is then checked again in mthp_collapse() for each attempted order. */ if (enabled_orders != BIT(HPAGE_PMD_ORDER)) - max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT; + max_ptes_none = COLLAPSE_MAX_PTES_LIMIT; pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl); if (!pte) { From 446065057621345d12ad9bb468a0b9ae5e88284d Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:38 +0100 Subject: [PATCH 166/213] mm/collapse: state what a collapse may do in the policy Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle: - which VMAs are eligible, and how hard to try for a folio; - how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD; - whether a range has to look used, and whether a MADV_FREE'd page is left alone; - whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying. None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out. Add struct collapse_policy for the caller to fill: khugepaged from its own settings, MADV_COLLAPSE from the fact that a user asked explicitly. Every test becomes a read of a field, and cc->is_khugepaged goes, having no reader left. The PTE limits come as two sets, one for a PMD-sized window and one for anything smaller, so that the helpers pick a set for the order and read it. khugepaged takes no swapped-out or shared PTE into a sub-PMD window, and empty PTEs only when the knob says all or nothing; the warning for a knob value in between moves to where khugepaged fills its policy. The fields only one side reads say which: anon_ or file_. David Hildenbrand asked for both. khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then treated alike. collapse_file() also drops a NULL check on the collapse_control. It has one call site, reached only from collapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead. Link: https://lore.kernel.org/20261014171145.3292736-6-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Tested-by: Baolin Wang Reviewed-by: Lance Yang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/collapse.h | 41 +++++++++++++- mm/khugepaged.c | 147 +++++++++++++++++++++++++----------------------- 2 files changed, 118 insertions(+), 70 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index b115034d9018..dcd117071955 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -45,8 +45,47 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; +/* How many PTEs of a window may be missing, swapped out or shared */ +struct collapse_limits { + /* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */ + unsigned int max_ptes_none; + unsigned int max_ptes_swap; + unsigned int max_ptes_shared; +}; + +/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { + /* Limits for a PMD-sized window */ + struct collapse_limits pmd; + + /* + * Limits for a smaller window. Its max_ptes_none is either 0 or + * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the + * window whatever its order; any other value counts as 0. + */ + struct collapse_limits sub_pmd; + + /* Leave clean lazyfree folios to reclaim rather than collapse them */ + bool anon_skip_lazyfree; + + /* Refuse an anonymous range with no sign of use */ + bool anon_require_referenced; + + /* Map the PMD over a file collapse instead of leaving it to a fault */ + bool file_install_pmd; + + /* Write dirty pages back and retry once instead of refusing them */ + bool file_writeback_dirty; + + /* How hard to try for a destination folio */ + gfp_t gfp; + + /* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ + enum tva_type tva_type; +}; + struct collapse_control { - bool is_khugepaged; + struct collapse_policy policy; /* Num pages scanned per node */ u32 node_load[MAX_NUMNODES]; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a6f59c54bc82..2c6dfc3341ce 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -314,30 +314,17 @@ static bool pte_none_or_zero(pte_t pte) static unsigned int collapse_max_ptes_none(struct collapse_control *cc, struct vm_area_struct *vma, unsigned int order) { - const unsigned int max_ptes_none = khugepaged_max_ptes_none; + unsigned int max_ptes_none; if (vma && userfaultfd_armed(vma)) return 0; - /* for MADV_COLLAPSE, allow any empty/shared zeropage PTEs */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for PMD collapse, respect the user defined maximum */ if (is_pmd_order(order)) - return max_ptes_none; - /* - * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, - * scale the maximum number of PTEs to the order of the collapse. - */ + return cc->policy.pmd.max_ptes_none; + + /* Below PMD order: all but one PTE of the window, or none */ + max_ptes_none = cc->policy.sub_pmd.max_ptes_none; if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT) return (1 << order) - 1; - /* - * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT, - * emit a warning and return 0. - */ - if (max_ptes_none) - pr_warn_once("mTHP collapse does not support max_ptes_none" - " values other than 0 or %u, defaulting to 0.\n", - COLLAPSE_MAX_PTES_LIMIT); return 0; } @@ -353,20 +340,9 @@ static unsigned int collapse_max_ptes_none(struct collapse_control *cc, static unsigned int collapse_max_ptes_shared(struct collapse_control *cc, unsigned int order) { - /* - * For MADV_COLLAPSE, do not restrict the number of PTEs that map shared - * anonymous pages. - */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* - * for mTHP collapse do not allow collapsing anonymous memory pages that - * are shared between processes. - */ - if (!is_pmd_order(order)) - return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_shared; + if (is_pmd_order(order)) + return cc->policy.pmd.max_ptes_shared; + return cc->policy.sub_pmd.max_ptes_shared; } /** @@ -381,17 +357,9 @@ static unsigned int collapse_max_ptes_shared(struct collapse_control *cc, static unsigned int collapse_max_ptes_swap(struct collapse_control *cc, unsigned int order) { - /* - * For MADV_COLLAPSE, do not restrict the number PTEs entries or - * pagecache entries that are non-present. - */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for mTHP collapse do not allow any non-present PTEs or pagecache entries */ - if (!is_pmd_order(order)) - return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_swap; + if (is_pmd_order(order)) + return cc->policy.pmd.max_ptes_swap; + return cc->policy.sub_pmd.max_ptes_swap; } int hugepage_madvise(struct vm_area_struct *vma, @@ -678,7 +646,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result = SCAN_PAGE_LAZYFREE; goto out; @@ -767,12 +735,12 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, if (folio_test_large(folio)) list_add_tail(&folio->lru, compound_pagelist); next: - if (cc->is_khugepaged && + if (cc->policy.anon_require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } - if (unlikely(cc->is_khugepaged && !referenced)) { + if (unlikely(cc->policy.anon_require_referenced && !referenced)) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -938,9 +906,7 @@ static void khugepaged_alloc_sleep(void) remove_wait_queue(&khugepaged_wait, &wait); } -static struct collapse_control khugepaged_collapse_control = { - .is_khugepaged = true, -}; +static struct collapse_control khugepaged_collapse_control; static bool collapse_scan_abort(int nid, struct collapse_control *cc) { @@ -976,6 +942,52 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void) return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT; } +/* khugepaged collapses on its own initiative, so it obeys its own settings */ +static void collapse_policy_khugepaged(struct collapse_policy *p) +{ + p->pmd.max_ptes_none = READ_ONCE(khugepaged_max_ptes_none); + p->pmd.max_ptes_swap = READ_ONCE(khugepaged_max_ptes_swap); + p->pmd.max_ptes_shared = READ_ONCE(khugepaged_max_ptes_shared); + + /* + * A sub-PMD window takes no swapped-out and no shared PTE: reading + * pages back or breaking CoW is not worth it for an mTHP. Empty PTEs + * it takes all or nothing, since anything in between would let one + * collapse feed the next. + */ + p->sub_pmd.max_ptes_none = p->pmd.max_ptes_none; + if (p->sub_pmd.max_ptes_none && + p->sub_pmd.max_ptes_none != COLLAPSE_MAX_PTES_LIMIT) + pr_warn_once("mTHP collapse does not support max_ptes_none values other than 0 or %u, defaulting to 0.\n", + COLLAPSE_MAX_PTES_LIMIT); + p->sub_pmd.max_ptes_swap = 0; + p->sub_pmd.max_ptes_shared = 0; + + p->anon_skip_lazyfree = true; + p->anon_require_referenced = true; + p->file_install_pmd = false; + p->file_writeback_dirty = false; + p->gfp = alloc_hugepage_khugepaged_gfpmask(); + p->tva_type = TVA_KHUGEPAGED; +} + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_madvise(struct collapse_policy *p) +{ + p->pmd.max_ptes_none = HPAGE_PMD_NR; + p->pmd.max_ptes_swap = HPAGE_PMD_NR; + p->pmd.max_ptes_shared = HPAGE_PMD_NR; + /* Never read: MADV_COLLAPSE collapses to PMD order only */ + p->sub_pmd = p->pmd; + + p->anon_skip_lazyfree = false; + p->anon_require_referenced = false; + p->file_install_pmd = true; + p->file_writeback_dirty = true; + p->gfp = GFP_TRANSHUGE; + p->tva_type = TVA_FORCED_COLLAPSE; +} + #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -1013,8 +1025,7 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l struct collapse_control *cc, unsigned int order) { struct vm_area_struct *vma; - enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED : - TVA_FORCED_COLLAPSE; + enum tva_type type = cc->policy.tva_type; if (unlikely(collapse_test_exit_or_disable(mm))) return SCAN_ANY_PROCESS; @@ -1197,8 +1208,7 @@ static enum scan_result __collapse_huge_page_swapin(struct mm_struct *mm, static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, struct collapse_control *cc, unsigned int order) { - gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : - GFP_TRANSHUGE); + gfp_t gfp = cc->policy.gfp; int node = collapse_find_target_node(cc); struct folio *folio; @@ -1581,7 +1591,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER); const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER); unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER); - enum tva_type tva_flags = cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE; + enum tva_type tva_flags = cc->policy.tva_type; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1681,7 +1691,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result = SCAN_PAGE_LAZYFREE; failed_pfn = folio_pfn(folio); @@ -1747,13 +1757,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } - if (cc->is_khugepaged && + if (cc->policy.anon_require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } - if (cc->is_khugepaged && - (!referenced || - (unmapped && referenced < HPAGE_PMD_NR / 2))) { + if (cc->policy.anon_require_referenced && + (!referenced || + (unmapped && referenced < HPAGE_PMD_NR / 2))) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -2608,11 +2618,11 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, xas_unlock_irq(&xas); /* - * Remove pte page tables, so we can re-fault the page as huge. - * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp(). + * Remove pte page tables, so we can re-fault the page as huge. A + * caller that wants the PMD mapped now is told to go and do that. */ retract_page_tables(mapping, start); - if (cc && !cc->is_khugepaged) + if (cc->policy.file_install_pmd) result = SCAN_PTE_MAPPED_HUGEPAGE; folio_unlock(new_folio); @@ -2799,11 +2809,8 @@ static enum scan_result collapse_single_pmd(unsigned long addr, retry: result = collapse_scan_file(mm, addr, file, pgoff, cc); - /* - * For MADV_COLLAPSE, when encountering dirty pages, try to writeback, - * then retry the collapse one time. - */ - if (!cc->is_khugepaged && result == SCAN_PAGE_DIRTY_OR_WRITEBACK && + /* Dirty pages are worth a writeback and one more try, if asked for */ + if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && mapping_can_writeback(file->f_mapping)) { const loff_t lstart = (loff_t)pgoff << PAGE_SHIFT; const loff_t lend = lstart + HPAGE_PMD_SIZE - 1; @@ -2820,7 +2827,7 @@ static enum scan_result collapse_single_pmd(unsigned long addr, result = SCAN_ANY_PROCESS; else result = try_collapse_pte_mapped_thp(mm, addr, - !cc->is_khugepaged); + cc->policy.file_install_pmd); if (result == SCAN_PMD_MAPPED) result = SCAN_SUCCEED; mmap_read_unlock(mm); @@ -2969,6 +2976,8 @@ static void khugepaged_do_scan(struct collapse_control *cc) lru_add_drain_all(); + collapse_policy_khugepaged(&cc->policy); + cc->progress = 0; while (true) { cond_resched(); @@ -3196,7 +3205,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, cc = kmalloc_obj(*cc); if (!cc) return -ENOMEM; - cc->is_khugepaged = false; + collapse_policy_madvise(&cc->policy); cc->progress = 0; lru_add_drain_all(); From d5dd87dbd60b86cafdda2ec2065341625dd3a429 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:39 +0100 Subject: [PATCH 167/213] mm/collapse: drop the collapse_possible() wrapper collapse_possible() only forwards to collapse_possible_orders() and turns its mask into a bool. Its three callers can test the mask themselves. No functional change. Link: https://lore.kernel.org/20261014171145.3292736-7-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/khugepaged.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2c6dfc3341ce..fcb38c465ee6 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -494,17 +494,11 @@ static unsigned long collapse_possible_orders(struct vm_area_struct *vma, return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders); } -static bool collapse_possible(struct vm_area_struct *vma, - vm_flags_t vm_flags, enum tva_type tva_flags) -{ - return collapse_possible_orders(vma, vm_flags, tva_flags); -} - void khugepaged_enter_vma(struct vm_area_struct *vma, vm_flags_t vm_flags) { - if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() - && collapse_possible(vma, vm_flags, TVA_KHUGEPAGED)) + if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() && + collapse_possible_orders(vma, vm_flags, TVA_KHUGEPAGED)) __khugepaged_enter(vma->vm_mm); } @@ -2881,7 +2875,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, cc->progress++; break; } - if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) { + if (!collapse_possible_orders(vma, vma->vm_flags, + TVA_KHUGEPAGED)) { cc->progress++; continue; } @@ -3193,7 +3188,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); - if (!collapse_possible(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) return -EINVAL; hstart = ALIGN(start, HPAGE_PMD_SIZE); From e7c4148ad1517f33ce01d27e47ce94f9c6c52ff7 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:40 +0100 Subject: [PATCH 168/213] mm/collapse: name the per-table scan reset for what it resets collapse_control_init_scan() resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing. Call it collapse_scan_reset(). Preparation for giving a control a real init, run once for a whole series of scans. Two names a word apart would then stand for two different jobs. No functional change. Link: https://lore.kernel.org/20261014171145.3292736-8-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/khugepaged.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index fcb38c465ee6..a6cb9e92f7c9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -532,7 +532,7 @@ void __khugepaged_exit(struct mm_struct *mm) } } -static void collapse_control_init_scan(struct collapse_control *cc) +static void collapse_scan_reset(struct collapse_control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); @@ -1607,7 +1607,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, goto out; } - collapse_control_init_scan(cc); + collapse_scan_reset(cc); enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags); @@ -2681,7 +2681,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, present = 0; swap = 0; - collapse_control_init_scan(cc); + collapse_scan_reset(cc); rcu_read_lock(); xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, folio)) From a2d1c9ea7613f346a8ea215f0419af15b2912cbd Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:41 +0100 Subject: [PATCH 169/213] mm/collapse: call collapse_file() from collapse_single_pmd() collapse_scan_file() reads the page cache to decide whether a table is worth collapsing and, when it is, calls collapse_file() itself. The caller cannot get between the decision and the collapse. Move the collapse_file() call up into collapse_single_pmd(), so the scan stops at the decision. Two things change with it. The writeback retry re-runs collapse_file() alone instead of rescanning first; collapse_file() repeats the scan's checks under the page cache lock anyway. And mm_khugepaged_scan_file fires before the collapse, so for an accepted table its status reads SCAN_SUCCEED; what the collapse made of the table is for mm_khugepaged_collapse_file to report. Preparation for splitting a collapse into a scan under mmap_lock and a run without it. The file scan has to stop where the anonymous one will. Link: https://lore.kernel.org/20261014171145.3292736-9-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Lance Yang Reviewed-by: Baolin Wang Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/khugepaged.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a6cb9e92f7c9..7a6acb5ba863 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2763,13 +2763,9 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, else cc->progress += HPAGE_PMD_NR; - if (result == SCAN_SUCCEED) { - if (present < HPAGE_PMD_NR - max_ptes_none) { - result = SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - } else { - result = collapse_file(mm, addr, file, start, cc); - } + if (result == SCAN_SUCCEED && present < HPAGE_PMD_NR - max_ptes_none) { + result = SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result); @@ -2800,8 +2796,16 @@ static enum scan_result collapse_single_pmd(unsigned long addr, mmap_read_unlock(mm); *lock_dropped = true; -retry: + + /* + * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds + * the PMD folio, and only the PTE table is left to retract. + */ result = collapse_scan_file(mm, addr, file, pgoff, cc); + if (result != SCAN_SUCCEED) + goto put; +retry: + result = collapse_file(mm, addr, file, pgoff, cc); /* Dirty pages are worth a writeback and one more try, if asked for */ if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK && @@ -2813,6 +2817,7 @@ static enum scan_result collapse_single_pmd(unsigned long addr, triggered_wb = true; goto retry; } +put: fput(file); if (result == SCAN_PTE_MAPPED_HUGEPAGE) { From 6d06041c838e2658d4d1cb9bd7bc3efa0587a28a Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Wed, 14 Oct 2026 18:11:42 +0100 Subject: [PATCH 170/213] mm/collapse: separate scanning a PTE table from collapsing it A collapse is two jobs. One reads a PTE table under mmap_lock and decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first. collapse_single_pmd() did both, so the boundary between them was somewhere in the middle of a function. Give each half its own function: - collapse_scan_pmd() scans one table. It picks the side: collapse_scan_anon(), the anonymous scan that used to carry the collapse_scan_pmd() name, or collapse_scan_file(). - collapse_pmd() does the collapse the scan asked for, and is handed what the scan returned. SCAN_SUCCEED means there is something to collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already holds the PMD folio and only the PTE table is left to retract. Anything else is why there is nothing to do. It calls collapse_anon(), formerly mthp_collapse(), or collapse_file(). For now collapse_single_pmd() stays, as the two of them with the mmap_lock drop in between, so its callers see what they saw before. collapse_control_init() sets a control up before its first scan. What the scan found and the collapse needs travels in collapse_control. For an anonymous table that is the orders and the referenced and swapped-out counts. collapse_anon() and collapse_huge_page() now read them from there instead of taking them as arguments. For a file it is the file itself and the offset in it. A file collapse works on the page cache and never sees a VMA, so the scan takes the reference while it still has one and the collapse gives it back. The file scan moves under mmap_lock with the anonymous one, where before the lock was given up first. The lock is now held over the page cache walk, an RCU walk over one table's worth of slots with no PTL, and taken fewer times. collapse_scan_mm_slot() ends its walk whenever the lock was dropped, so a refused file table used to cost khugepaged an unlock, a trip back through khugepaged_do_scan(), a relock and a VMA lookup. Now only a table that goes on to be collapsed does. Link: https://lore.kernel.org/20261014171145.3292736-10-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Andrew Morton Acked-by: Zi Yan Reviewed-by: Lance Yang Acked-by: David Hildenbrand (Arm) Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Usama Arif Cc: Vlastimil Babka Cc: Jann Horn --- mm/collapse.h | 18 ++++++ mm/khugepaged.c | 151 +++++++++++++++++++++++++++++++----------------- 2 files changed, 117 insertions(+), 52 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index dcd117071955..759ec9cd4541 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -98,6 +98,24 @@ struct collapse_control { /* Each bit marks a PTE the scan accepted as a collapse source */ DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); + + /* + * What a scan found and the collapse after it needs. Live only between + * the two, and read by nobody else. + */ + + /* Anonymous only: orders to try, referenced and swap PTE counts */ + unsigned long scan_orders; + int scan_referenced; + int scan_unmapped; + + /* + * File only. A file collapse works on the page cache and never sees a + * VMA, so the scan takes a reference to the file while it still has + * one, and the collapse gives it back. + */ + struct file *scan_file; + pgoff_t scan_pgoff; }; #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 7a6acb5ba863..17e0e0162e90 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -646,7 +646,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, goto out; } - /* See collapse_scan_pmd(). */ + /* See collapse_scan_anon(). */ if (folio_maybe_mapped_shared(folio)) { /* * TODO: Support shared pages without leading to further @@ -1099,7 +1099,7 @@ static enum scan_result check_pmd_still_valid(struct mm_struct *mm, /* * Bring missing pages in from swap, to complete THP collapse. - * Only done if collapse_scan_pmd() believes it is worthwhile. + * Only done if collapse_scan_anon() believes it is worthwhile. * * For mTHP orders the function bails on the first swap entry, because * faulting pages back in during collapse could re-populate PTEs that @@ -1252,8 +1252,8 @@ static pgtable_t alloc_deposit_pte_table(struct mm_struct *mm) * while allocating a THP, as that could trigger direct reclaim/compaction. * Note that the VMA must be rechecked after grabbing the mmap_lock again. */ -static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long start_addr, - int referenced, int unmapped, struct collapse_control *cc, +static enum scan_result collapse_huge_page(struct mm_struct *mm, + unsigned long start_addr, struct collapse_control *cc, unsigned int order) { const unsigned long pmd_addr = start_addr & HPAGE_PMD_MASK; @@ -1300,14 +1300,14 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s goto out_nolock; } - if (unmapped) { + if (cc->scan_unmapped) { /* * __collapse_huge_page_swapin() will return with mmap_lock * released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ result = __collapse_huge_page_swapin(mm, vma, start_addr, pmd, - referenced, order); + cc->scan_referenced, order); if (result != SCAN_SUCCEED) goto out_nolock; } @@ -1460,8 +1460,8 @@ static unsigned int max_order_from_offset(unsigned int offset) } /* - * mthp_collapse() consumes the bitmap that is generated during - * collapse_scan_pmd() to determine what regions and mTHP orders fit best. + * collapse_anon() consumes the bitmap that is generated during + * collapse_scan_anon() to determine what regions and mTHP orders fit best. * * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse * source. We start at the PMD order and check if it is eligible for collapse; @@ -1476,9 +1476,8 @@ static unsigned int max_order_from_offset(unsigned int offset) * If a collapse is permitted, we attempt to collapse the PTE range into a * mTHP. */ -static enum scan_result mthp_collapse(struct mm_struct *mm, - unsigned long address, int referenced, int unmapped, - struct collapse_control *cc, unsigned long enabled_orders) +static enum scan_result collapse_anon(struct mm_struct *mm, + unsigned long address, struct collapse_control *cc) { unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none; enum scan_result last_result = SCAN_FAIL; @@ -1491,7 +1490,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, while (offset < HPAGE_PMD_NR) { nr_ptes = 1UL << order; - if (!test_bit(order, &enabled_orders)) + if (!test_bit(order, &cc->scan_orders)) goto next_order; max_ptes_none = collapse_max_ptes_none(cc, NULL, order); @@ -1499,19 +1498,18 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, offset + nr_ptes); /* - * Swap PTEs accepted during the scan are counted in @unmapped, - * not in cc->eligible_ptes. Account them for the PMD-order - * candidate. + * Swap PTEs accepted during the scan are counted in + * cc->scan_unmapped, not in cc->eligible_ptes. Account them for + * the PMD-order candidate. */ if (is_pmd_order(order)) - nr_eligible_ptes += unmapped; + nr_eligible_ptes += cc->scan_unmapped; if (nr_eligible_ptes >= nr_ptes - max_ptes_none) { enum scan_result ret; collapse_address = address + offset * PAGE_SIZE; - ret = collapse_huge_page(mm, collapse_address, referenced, - unmapped, cc, order); + ret = collapse_huge_page(mm, collapse_address, cc, order); switch (ret) { /* Cases where we continue to next collapse candidate */ @@ -1553,7 +1551,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, * we must always move to the next offset. */ if (order > COLLAPSE_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + (cc->scan_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1578,14 +1576,14 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, return last_result; } -static enum scan_result collapse_scan_pmd(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long start_addr, - bool *lock_dropped, struct collapse_control *cc) +static enum scan_result collapse_scan_anon(struct vm_area_struct *vma, + unsigned long start_addr, struct collapse_control *cc) { const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER); const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER); unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER); enum tva_type tva_flags = cc->policy.tva_type; + struct mm_struct *mm = vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1614,7 +1612,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, /* * If PMD is the only enabled order, enforce max_ptes_none, otherwise * scan all pages to populate the bitmap for mTHP collapse. The bitmap - * is then checked again in mthp_collapse() for each attempted order. + * is then checked again in collapse_anon() for each attempted order. */ if (enabled_orders != BIT(HPAGE_PMD_ORDER)) max_ptes_none = COLLAPSE_MAX_PTES_LIMIT; @@ -1765,12 +1763,9 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { - /* collapse_huge_page() expects the lock to be dropped before calling */ - mmap_read_unlock(mm); - result = mthp_collapse(mm, start_addr, referenced, - unmapped, cc, enabled_orders); - /* mmap_lock was released above, set lock_dropped */ - *lock_dropped = true; + cc->scan_orders = enabled_orders; + cc->scan_referenced = referenced; + cc->scan_unmapped = unmapped; } out: trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced, @@ -2768,41 +2763,67 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } - trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result); + trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, + result); return result; } -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) +static void collapse_control_init(struct collapse_control *cc) +{ + cc->progress = 0; + cc->scan_file = NULL; +} + +static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc) { - struct mm_struct *mm = vma->vm_mm; - bool triggered_wb = false; enum scan_result result; - struct file *file; pgoff_t pgoff; - mmap_assert_locked(mm); + mmap_assert_locked(vma->vm_mm); + /* A scan that found work has to have been followed by collapse_pmd() */ + if (WARN_ON_ONCE(cc->scan_file)) { + fput(cc->scan_file); + cc->scan_file = NULL; + } if (vma_is_anonymous(vma)) - return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); + return collapse_scan_anon(vma, addr, cc); - file = get_file(vma->vm_file); pgoff = linear_page_index(vma, addr); - - mmap_read_unlock(mm); - *lock_dropped = true; - + result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc); /* * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds - * the PMD folio, and only the PTE table is left to retract. + * the PMD folio, and collapse_pmd() still has to retract the PTE table. */ - result = collapse_scan_file(mm, addr, file, pgoff, cc); - if (result != SCAN_SUCCEED) + if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE) + return result; + + /* + * A file collapse works on the page cache and never sees a VMA, so take + * what it needs from this one while it is still here. + */ + cc->scan_file = get_file(vma->vm_file); + cc->scan_pgoff = pgoff; + return result; +} + +static enum scan_result collapse_pmd(struct mm_struct *mm, + unsigned long addr, enum scan_result result, + struct collapse_control *cc) +{ + struct file *file = cc->scan_file; + bool triggered_wb = false; + pgoff_t pgoff; + + if (!file) + return collapse_anon(mm, addr, cc); + + cc->scan_file = NULL; + pgoff = cc->scan_pgoff; + + /* The scan found the PMD folio in place: nothing to collapse */ + if (result == SCAN_PTE_MAPPED_HUGEPAGE) goto put; retry: result = collapse_file(mm, addr, file, pgoff, cc); @@ -2820,6 +2841,10 @@ static enum scan_result collapse_single_pmd(unsigned long addr, put: fput(file); + /* + * A PMD folio is in the page cache, whether the collapse just put it + * there or found it: retract the PTE table, and map the PMD if asked. + */ if (result == SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); if (collapse_test_exit_or_disable(mm)) @@ -2834,6 +2859,28 @@ static enum scan_result collapse_single_pmd(unsigned long addr, return result; } +/* + * Try to collapse a single PMD starting at a PMD aligned addr, and return + * the results. + */ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *lock_dropped, + struct collapse_control *cc) +{ + struct mm_struct *mm = vma->vm_mm; + enum scan_result result; + + result = collapse_scan_pmd(vma, addr, cc); + if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE) + return result; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped = true; + + return collapse_pmd(mm, addr, result, cc); +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2976,9 +3023,9 @@ static void khugepaged_do_scan(struct collapse_control *cc) lru_add_drain_all(); + collapse_control_init(cc); collapse_policy_khugepaged(&cc->policy); - cc->progress = 0; while (true) { cond_resched(); @@ -3205,8 +3252,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, cc = kmalloc_obj(*cc); if (!cc) return -ENOMEM; + collapse_control_init(cc); collapse_policy_madvise(&cc->policy); - cc->progress = 0; lru_add_drain_all(); From 4425c6730022df01c952139df97141201f709ebc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re=20=28SUSE=29?= Date: Wed, 14 Oct 2026 15:46:46 -0300 Subject: [PATCH 171/213] selftests/mm: mremap_test: Validate the preamble region before mapping it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit remap_region() checks that the destination region is free before mapping it, but not the preamble region placed right before it, so mapping the preamble can intermittently fail with EEXIST: # Failed to map dest preamble region: File exists not ok 15 5MB mremap - Source 1MB-aligned, Dest 1MB-aligned with 40MB Preamble This happens when the kernel's heap ASLR (arch_randomize_brk()) lands the process heap inside that unchecked range. Confirmed by correlating brk(NULL) at startup with pass/fail across 60 runs: every failure had brk land inside that range. Fix it by including the preamble in the free-region check. Link: https://lore.kernel.org/20261014-selftests-mm-mremap_preamble-v1-1-2e97f501cb56@marliere.net Signed-off-by: Ricardo B. Marlière (SUSE) Signed-off-by: Andrew Morton Reviewed-by: Sarthak Sharma Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan --- tools/testing/selftests/mm/mremap_test.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/mm/mremap_test.c b/tools/testing/selftests/mm/mremap_test.c index 97abf4713cc5..31b484517bda 100644 --- a/tools/testing/selftests/mm/mremap_test.c +++ b/tools/testing/selftests/mm/mremap_test.c @@ -1006,8 +1006,10 @@ static long long remap_region(struct config c, unsigned int threshold_mb, if (!((unsigned long long) addr & c.dest_alignment)) addr = (void *) ((unsigned long long) addr | c.dest_alignment); - /* Don't destroy existing mappings unless expected to overlap */ - while (!is_remap_region_valid(addr, c.region_size) && !c.overlapping) { + /* Don't destroy existing mappings unless expected to overlap. */ + while (!is_remap_region_valid(addr - c.dest_preamble_size, + c.dest_preamble_size + c.region_size) && + !c.overlapping) { /* Check for unsigned overflow */ tmp_addr = addr + c.dest_alignment; if (tmp_addr < addr) { From 4a0ef6078731cac11cdde5e5f355f2252828eb56 Mon Sep 17 00:00:00 2001 From: Sasha Levin Date: Wed, 14 Oct 2026 16:01:37 -0400 Subject: [PATCH 172/213] selftests/mm: report the configuration check on stderr The mm Makefile includes local_config.mk, which is generated by a rule that announces itself with "CHK [config] local_config.mk" on standard output, which make -s does not suppress. When make has to regenerate the file while it reads the Makefile for emit_tests, for example on a fresh tree or when the selftests are installed with make -B, the message becomes part of the emit_tests output that the selftests install target appends to kselftest-list.txt, so the runlist gains a line that is not a test. Print the message on standard error. With V=1 the message is empty and only the redirection of an empty command remains. Link: https://lore.kernel.org/20261014200137.1364121-1-sashal@kernel.org Fixes: 04cf82a741e0 ("selftests/mm: suppress compiler error in liburing check") Signed-off-by: Sasha Levin Signed-off-by: Andrew Morton Cc: Li Wang Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Shuah Khan --- tools/testing/selftests/mm/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index beacc0f87304..0ea72e997934 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -266,7 +266,7 @@ $(OUTPUT)/migration: LDLIBS += -lnuma $(OUTPUT)/rmap: LDLIBS += -lnuma local_config.mk local_config.h_gen: check_config.sh - $(call msg,CHK,config,$@) + @{ $(patsubst @%,%,$(call msg,CHK,config,$@)) :; } >&2 $(Q)CC="$(CC)" CFLAGS="$(CFLAGS)" ./check_config.sh EXTRA_CLEAN += local_config.mk local_config.h_gen From 3ae91e3079b1504ef162b2884454d5f1421f4eb9 Mon Sep 17 00:00:00 2001 From: Nickolai Zeldovich Date: Wed, 14 Oct 2026 09:13:58 -0400 Subject: [PATCH 173/213] mm/mlock: use a long for the page count in mlock_fixup() mlock_fixup() adjusts mm->locked_vm, an unsigned long, by the number of pages whose VM_LOCKED state it changes, but computes that number in an int: int nr_pages; ... nr_pages = (end - start) >> PAGE_SHIFT; ... mm->locked_vm += nr_pages; When a single VMA range of 2^31 pages (8 TiB with 4 KiB pages) or more is locked or unlocked, the count is truncated to 32 bits and then sign-extended, so locked_vm is changed by the wrong amount, while munmap(), mremap() and the other paths that maintain locked_vm use the full unsigned long page count. Such a range is reachable on 64-bit architectures with a 47-bit or larger user address space (x86_64, arm64 with 48-bit VA, riscv64 Sv48) by a task with CAP_IPC_LOCK or a raised RLIMIT_MEMLOCK, which do_mlock() checks against the full untruncated length. It costs nothing with MAP_NORESERVE and MLOCK_ONFAULT/MCL_ONFAULT, since then no page is populated. On x86_64, after p = mmap(NULL, 8UL << 40, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); mlock2(p, 8UL << 40, MLOCK_ONFAULT); VmLck in /proc/self/status reads 18446744065119617024 kB instead of 8589934592 kB; munmap() of the region then subtracts the true size and leaves VmLck at 18446744056529682432 kB instead of 0. With 16 TiB, locked_vm is not changed at all, VmLck stays 0 kB, and after the task drops CAP_IPC_LOCK or lowers RLIMIT_MEMLOCK, mlock(), mmap(MAP_LOCKED) and the other mlock_future_ok() and account_locked_vm() users enforce RLIMIT_MEMLOCK against that count, so they let the task lock memory beyond its limit (or, in the 8 TiB case, refuse every further lock). The error persists for the life of the mm; unlocking the range in pieces skews it further. The wrong count is per-mm accounting only: it does not change which pages are locked. The int has been there since before git history (the variable was named "pages" until it was renamed in 2.6.28); every other updater of locked_vm uses an unsigned long or long count. Make nr_pages a long, which holds the page count of any user range and its negation, so the existing sign handling is unchanged. Found by formal verification of mm/mlock.c against the kernel binary. Link: https://lore.kernel.org/20261014131358.780240-1-nickolai@csail.mit.edu Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Signed-off-by: Nickolai Zeldovich Signed-off-by: Andrew Morton Reviewed-by: Pedro Falcato Assisted-by: LLM Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Vlastimil Babka Cc: Jann Horn Cc: --- mm/mlock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mlock.c b/mm/mlock.c index 4235a1518fc9..67ae9b18b923 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -457,7 +457,7 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, { const vma_flags_t old_vma_flags = vma->flags; struct mm_struct *mm = vma->vm_mm; - int nr_pages; + long nr_pages; int ret = 0; if (vma_flags_same_pair(&old_vma_flags, new_vma_flags) || From 454143aeb4c6c85ca7d50df36d1a66a64d88283e Mon Sep 17 00:00:00 2001 From: "Cen Zhang (Microsoft)" Date: Mon, 12 Oct 2026 16:15:14 -0400 Subject: [PATCH 174/213] userfaultfd: fix UFFDIO_MOVE livelock on large folio with extra references To move one page out of a large folio, move_pages_ptes() first has to split the folio. __folio_split() refuses if the folio has any reference beyond its page table mappings and the caller's own, and it reports the refusal as -EAGAIN. move_pages() treats returned -EAGAIN as a transient race and keeps retrying the same address, holding the VMA read locks the whole time. This is incorrect since in some cases userspace can hold such a reference indefinitely, such as by registering the page as an io_uring buffer. An unprivileged user can trigger this with three threads: first holds such a reference on a page of a THP, second calls UFFDIO_MOVE on another page of the same THP and spins in the kernel forever holding the VMA read locks, and third calls fork() or mprotect(), which takes mmap_lock for writing and then waits forever for those read locks. From then on every mmap_lock user of that mm hangs, including other processes inspecting it through /proc//, until the process is SIGKILLed. The hung task detector reports the blocked tasks (third one), and with kernel.hung_task_panic=1 the machine panics: INFO: task wedge:95 blocked for more than 122 seconds. task:wedge state:D stack:12528 pid:95 tgid:95 ppid:94 Call Trace: __schedule+0x596/0x1110 schedule+0x36/0xf0 __vma_start_exclude_readers+0x100/0x170 __vma_start_write+0x51/0xd0 __split_vma+0x208/0x3d0 vma_modify+0x6a8/0x840 vma_modify_flags+0xce/0x110 mprotect_fixup+0xbe/0x480 do_mprotect_pkey+0x2d9/0x580 __x64_sys_mprotect+0x1a/0x30 Fix by returning -EBUSY instead of -EAGAIN in move_pages_ptes(), the caller of split_folio(), when the split fails. Thus, move_pages() does not retry -EBUSY and the ioctl drops its locks and returns to userspace instead of spinning. Link: https://lore.kernel.org/20261012201514.97015-1-cenzhang@linux.microsoft.com Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI") Signed-off-by: Cen Zhang (Microsoft) Signed-off-by: Andrew Morton Reported-by: Acked-by: David Hildenbrand (Arm) Assisted-by: LLM Cc: Mike Rapoport Cc: Peter Xu Cc: Suren Baghdasaryan Cc: Andrea Arcangeli Cc: Cc: --- mm/userfaultfd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 405f2d72f204..669f71d485d0 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1635,8 +1635,12 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd pte_unmap(dst_pte); src_pte = dst_pte = NULL; ret = split_folio(src_folio); - if (ret) + if (ret) { + /* do not retry, extra refs may be held forever */ + if (ret == -EAGAIN) + ret = -EBUSY; goto out; + } /* have to reacquire the folio after it got split */ folio_unlock(src_folio); folio_put(src_folio); From c7a467fbcbbc31e57a5077bfc9456d9f29cb1571 Mon Sep 17 00:00:00 2001 From: "Harry Yoo (Meta)" Date: Tue, 13 Oct 2026 20:34:39 +0100 Subject: [PATCH 175/213] mm/percpu: account full object size for percpu memory allocations When memory allocation profiling is enabled, the pages backing percpu memory chunks are attributed to func:pcpu_alloc_pages. However, the accounting of individual percpu memory allocations via alloc_percpu() and friends record only the size of a single object rather than its full size across all CPUs. As a result, it is often not straightforward to identify which kernel code is actually responsible for percpu memory, because the size is under-accounted by a factor of num_possible_cpus(). Account full object size with pcpu_obj_full_size(size) instead, which multiplies the object size by num_possible_cpus() and adds the objcg pointer overhead. This is consistent with what memcg already does for charging percpu allocations. Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and measuring mm_init()'s rss_stat counters. Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo: bytes calls per allocation unpatched 6320 395 16 patched 418176 396 1056 'percpu' field in memory.stat: 422400 (1056 bytes per process) Memory allocation profiling now accounts the same amount of memory as what memcg charges. Link: https://lore.kernel.org/20261013-fix-alloc-percpu-accounting-v1-1-03e788a1e41f@kernel.org Fixes: 60fa4a9e2323 ("mm: percpu: add codetag reference into pcpuobj_ext") Signed-off-by: Harry Yoo (Meta) Signed-off-by: Andrew Morton Reviewed-by: Joshua Hahn Assisted-by: LLM # Verification Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Suren Baghdasaryan Cc: Hao Ge Cc: --- mm/percpu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 3eff382e565a..d7883507dc15 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1696,14 +1696,15 @@ static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, { if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, - current->alloc_tag, size); + current->alloc_tag, pcpu_obj_full_size(size)); } } static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) - alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); + alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + pcpu_obj_full_size(size)); } #else static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, From 246c8cc33438a71c28f9b86baa492d9a0a130316 Mon Sep 17 00:00:00 2001 From: Pedro Falcato Date: Tue, 13 Oct 2026 20:17:19 +0100 Subject: [PATCH 176/213] mm/madvise: atomically clear the young bit on THP MADV_COLD As-is, madvise_cold_or_pageout_pte_range() discards the hardware young bit by invalidating the PMD, doing pmd_mkold() and subsequently writing to the pmd again. However, it does so in a racy manner, looking at a possibly-stale snapshot. It's entirely possible that a write elsewhere set the hardware D bit, which this logic now loses. Fix it by using pmdp_test_and_clear_young(). This is an atomic modification of the PMD entry, and thus safe against any possible racing writes. As a (nice) side-effect, this elides one non-required TLB flush in pmdp_invalidate(). This bug can interact with the swapping system in complicated ways, leading to data loss. For instance, if a folio is marked lazyfree, but was later read from, and then MADV_COLD'd while racing a write to that same THP, dropping the dirty bit may lead to data loss as reclaim/MADV_PAGEOUT drops the "clean" anon folio: T0 T1 ================================================================ MADV_FREE(thp) | folio marked lazyfree | read(thp) | PMD marked young by hardware | | MADV_COLD(thp) write(thp) | if (pmd_young(pmd)) { PMD marked dirty by hardware | set(pmdp, pmd_mkold(pmd)); MADV_PAGEOUT/reclaim | dirty bit may or may not be | set | reclaim/pageout may drop | the folio | Link: https://lore.kernel.org/20261013191719.95502-1-pfalcato@suse.de Fixes: 9c276cc65a58 ("mm: introduce MADV_COLD") Signed-off-by: Pedro Falcato Signed-off-by: Andrew Morton Reported-by: Samarth Narang Closes: https://lore.kernel.org/linux-mm/20261011132227.413217-1-samarth.colleges@gmail.com/ Reviewed-by: Gregory Price (Meta) Acked-by: Zi Yan Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: SJ Park Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Minchan Kim Cc: Michal Hocko Cc: Johannes Weiner Cc: --- mm/madvise.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 8702b082a02d..75b409e97dbf 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -430,10 +430,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, } if (!pageout && pmd_young(orig_pmd)) { - pmdp_invalidate(vma, addr, pmd); - orig_pmd = pmd_mkold(orig_pmd); - - set_pmd_at(mm, addr, pmd, orig_pmd); + pmdp_test_and_clear_young(vma, addr, pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } From 00a51d09912a12232d47bc282b74397908d1ed5c Mon Sep 17 00:00:00 2001 From: Hui Zhu Date: Wed, 14 Oct 2026 17:00:23 +0800 Subject: [PATCH 177/213] mm/vmscan: fix missing NR_ISOLATED counter update in MGLRU reclaim path MGLRU evict_folios() isolates folios from the LRU without updating the NR_ISOLATED_ANON/FILE counters, unlike the legacy shrink_inactive_list() path. This causes compaction's too_many_isolated() check to under-count isolated pages when MGLRU reclaim is active. Add NR_ISOLATED counter updates in evict_folios(): increment after isolate_folios() and decrement after all retry passes complete, using the existing nr_isolated which holds the original isolated count. Link: https://lore.kernel.org/141901355a7d9e4bbca5c4840ae32ea5c65fe3b6.1791968227.git.zhuhui@kylinos.cn Link: https://lore.kernel.org/cover.1791968227.git.zhuhui@kylinos.cn Signed-off-by: Hui Zhu Signed-off-by: Andrew Morton Reviewed-by: Baolin Wang Reviewed-by: Barry Song Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Johannes Weiner Cc: David Hildenbrand Cc: Michal Hocko Cc: Lorenzo Stoakes --- mm/vmscan.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/vmscan.c b/mm/vmscan.c index ac386422bc91..28dbc5137e68 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5081,6 +5081,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &list, &isolated, &type, &type_scanned); nr_isolated = isolated; + if (nr_isolated) + __mod_node_page_state(pgdat, NR_ISOLATED_ANON + type, + nr_isolated); /* Scanning may have emptied the oldest gen, flush it */ if (scanned) @@ -5149,6 +5152,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, goto retry; } + mod_node_page_state(pgdat, NR_ISOLATED_ANON + type, -nr_isolated); + if (nr_isolated > total_reclaimed) mod_lruvec_state(lruvec, PGROTATE_ANON + type, nr_isolated - total_reclaimed); From e387ebe5cd31720b1e0281fd045be4eef36fadb3 Mon Sep 17 00:00:00 2001 From: Weilin Tong Date: Wed, 14 Oct 2026 16:22:18 +0800 Subject: [PATCH 178/213] ipc/shm: fix RSS accounting for hugetlb shared memory Commit a08c7193e4f1 ("mm/filemap: remove hugetlb special casing in filemap.c") changed mapping->nrpages to count base pages for hugetlb folios. However, shm_add_rss_swap() still multiplies it by pages_per_huge_page(), overcounting RSS in both SHM_INFO and /proc/sysvipc/shm. Use mapping->nrpages for RSS accounting across all backing types, and account swap only for non-hugetlb shmem. Use READ_ONCE() for both counters instead of info->lock, which does not protect nrpages and cannot provide a consistent snapshot. Link: https://lore.kernel.org/20261014082218.2236088-1-tongweilin@linux.alibaba.com Fixes: a08c7193e4f1 ("mm/filemap: remove hugetlb special casing in filemap.c") Signed-off-by: Weilin Tong Signed-off-by: Andrew Morton Suggested-by: Muchun Song Acked-by: Muchun Song Cc: Oscar Salvador Cc: David Hildenbrand Cc: Sidhartha Kumar Cc: Lorenzo Stoakes Cc: Dan Carpenter Cc: Baolin Wang Cc: --- ipc/shm.c | 23 ++++------------------- 1 file changed, 4 insertions(+), 19 deletions(-) diff --git a/ipc/shm.c b/ipc/shm.c index bb1a721a3e74..6862f681373f 100644 --- a/ipc/shm.c +++ b/ipc/shm.c @@ -936,26 +936,11 @@ static inline unsigned long copy_shminfo_to_user(void __user *buf, struct shminf static void shm_add_rss_swap(struct shmid_kernel *shp, unsigned long *rss_add, unsigned long *swp_add) { - struct inode *inode; + struct inode *inode = file_inode(shp->shm_file); - inode = file_inode(shp->shm_file); - - if (is_file_hugepages(shp->shm_file)) { - struct address_space *mapping = inode->i_mapping; - struct hstate *h = hstate_file(shp->shm_file); - *rss_add += pages_per_huge_page(h) * mapping->nrpages; - } else { -#ifdef CONFIG_SHMEM - struct shmem_inode_info *info = SHMEM_I(inode); - - spin_lock_irq(&info->lock); - *rss_add += inode->i_mapping->nrpages; - *swp_add += info->swapped; - spin_unlock_irq(&info->lock); -#else - *rss_add += inode->i_mapping->nrpages; -#endif - } + *rss_add += READ_ONCE(inode->i_mapping->nrpages); + if (IS_ENABLED(CONFIG_SHMEM) && !is_file_hugepages(shp->shm_file)) + *swp_add += READ_ONCE(SHMEM_I(inode)->swapped); } /* From 2ca8b38962f239b17f7881a4710c11328ce38ac9 Mon Sep 17 00:00:00 2001 From: Vernon Yang Date: Mon, 12 Oct 2026 10:29:17 +0800 Subject: [PATCH 179/213] mm: vmscan: don't count per-node proactive reclaim as memory pressure When the cgroup has no memory pressure at all, writing to /sys/devices/system/nodeX/reclaim triggers proactive reclaim on NUMA node, causing increase in the writer cgroup's memory PSI. Due to this reclaim is performed in the context of the write(), accounted as memory pressure on the writer, like commit e22c6ed90aa9 ("mm: memcontrol: don't count limit-setting reclaim as memory pressure"). This is unexpected, the similar scenarios (e.g. senpai) will continue to occur, only this time it is for the /sys/devices/system/nodeX/reclaim knob. The Documentation/ABI/stable/sysfs-devices-node documentation also notes that "This interface is equivalent to the memcg variant." This patch unifies the semantics of the memcg and node interfaces: per-node proactive reclaim is no longer counted as memory pressure, and the per-node proactive reclaim interface no longer produces phantom pressure. I ran demo[1] that performs per-node proactive reclaim 10000 times in qemu, writer cgroup memory.pressure as follows: without patch: some avg10=31.53 avg60=13.42 avg300=3.31 total=10602985 full avg10=31.53 avg60=13.42 avg300=3.31 total=10602985 with patch: some avg10=9.59 avg60=3.41 avg300=0.81 total=2686221 full avg10=9.59 avg60=3.41 avg300=0.81 total=2686221 Link: https://lore.kernel.org/20261012022917.1102467-1-vernon2gm@gmail.com Link: https://github.com/vernon2gh/app_and_module/tree/main/reclaim_node_psi [1] Fixes: b980077899ea ("mm: introduce per-node proactive reclaim interface") Signed-off-by: Vernon Yang Signed-off-by: Andrew Morton Reviewed-by: Baolin Wang Reviewed-by: Barry Song Reviewed-by: Ridong Chen Acked-by: Davidlohr Bueso Cc: David Hildenbrand Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Johannes Weiner Cc: Michal Hocko Cc: Lorenzo Stoakes Cc: Roman Gushchin Cc: --- mm/vmscan.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 28dbc5137e68..daa7e37c2236 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -7998,7 +7998,8 @@ static unsigned long __node_reclaim(struct pglist_data *pgdat, sc->gfp_mask); cond_resched(); - psi_memstall_enter(&pflags); + if (!sc->proactive) + psi_memstall_enter(&pflags); delayacct_freepages_start(); fs_reclaim_acquire(sc->gfp_mask); /* @@ -8015,7 +8016,8 @@ static unsigned long __node_reclaim(struct pglist_data *pgdat, memalloc_noreclaim_restore(noreclaim_flag); fs_reclaim_release(sc->gfp_mask); delayacct_freepages_end(); - psi_memstall_leave(&pflags); + if (!sc->proactive) + psi_memstall_leave(&pflags); trace_mm_vmscan_node_reclaim_end(sc->nr_reclaimed, NULL); From 86aca64badd203af6035979991a9be63f2af7b9e Mon Sep 17 00:00:00 2001 From: "Vlastimil Babka (SUSE)" Date: Thu, 15 Oct 2026 18:31:46 +0200 Subject: [PATCH 180/213] MAINTAINERS: exclude DAMON docs from MEMORY MANAGEMENT - MISC MEMORY MANAGEMENT - MISC is a list of misc mm sources and also acts as catch-all for the mm Documentation subdirectories. Documentation for DAMON is well maintained within its own subsystem and also actively updated, so there's no need to Cc all MISC maintainers for them as well. Thus add exclusion entries for them. Link: https://lore.kernel.org/20261015-maint-mm-core-x-doc-damon-v1-1-226decb5cd0d@kernel.org Signed-off-by: Vlastimil Babka (SUSE) Signed-off-by: Andrew Morton Acked-by: Lorenzo Stoakes (ARM) Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Acked-by: SJ Park Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan Cc: Randy Dunlap --- MAINTAINERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 5c38da7090db..6f1f35a017a6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17273,6 +17273,8 @@ F: Documentation/admin-guide/mm/ F: Documentation/core-api/memory-allocation.rst F: Documentation/core-api/mm-api.rst F: Documentation/mm/ +X: Documentation/admin-guide/mm/damon/ +X: Documentation/mm/damon/ F: mm/char-mem.c F: include/linux/cma.h F: include/linux/dmapool.h From 7135a05838e85845e1bbf6caa6596a7e3b791a40 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Fri, 16 Oct 2026 12:34:33 +0100 Subject: [PATCH 181/213] mm/vma: avoid spurious write locks when mm->mm_lock_seq overflows So far the assumption has been that mm->mm_lock_seq overflow is benign, resulting only in readers spuriously falling back to the mmap lock, however it turns out that this was an unsafe assumption. If a VMA can be spuriously considered write locked that means read lock exclusion does not take place, and thus a VMA can end up write-locked with concurrent readers. Since overflow only occurs every 2^31 write lock/unlock cycles, resolve the issue by resetting every VMA's vma->vm_lock_seq each time such an overflow occurs. This is done in vma_end_write_all(), which is called when the mmap write lock is released, i.e. in mmap_write_unlock() and mmap_write_downgrade(). Care must be taken, as resetting the sequence count at overflow means changing the seqcount of VMAs which are already attached and observable. The mm's seqcount is guaranteed to be odd when a write lock is held, and thus any write locked VMA will have an odd seqcount assigned. Therefore resetting to an even value prevents spurious write locks, since these can only occur with the mmap write lock held, at which point the mmap seqcount will be odd. However, vma_start_read() also checks vma->vm_lock_seq == mm->mm_lock_seq, bailing if so (indicating a write lock). This would cause any even reset value chosen to prevent VMA read locks from being taken - avoid this by also checking whether the VMA seqcount is odd before bailing on the read lock. So far, vma->vm_lock_seq has defaulted to UINT_MAX, but this has been unnecessary since commit e5e7fb278e59 ("mm: convert mm_lock_seq to a proper seqcount"), since this is assigned when the VMA is still detached, and cannot be observed. Therefore perform both reset and initialisation of vma->vm_lock_seq by simply setting vma->vm_lock_seq to zero. The reset, performed in __mm_reset_vma_seqcounts(), only needs to WRITE_ONCE() the value. However the read of vma->vm_lock_seq in vma_start_read() needs to have acquire semantics to pair with the smp_wmb() in do_raw_write_seqcount_end() so a reader reading a reset value also observes VMA modifications made under the write lock. Since all VMA write locks were just released and the mmap write lock is still held, there are no concurrent VMA writers. Concurrent VMA readers are unaffected. New VMA readers observe either zero, never a write lock, or the prior seqcount, behaving as before the reset. Since this only occurs every 2^31 write lock/unlock cycles, the time taken to update all VMAs is considered to be acceptable. Also update comments, documentation and VMA userland tests to reflect the change. User facing effects are - dangerous races that could cause memory corruption or NULL pointer derefences (especially on page table traversal) are avoided. Link: https://lore.kernel.org/20261016-fix-vma-lock-overflow-v2-1-9eb778b79b58@kernel.org Fixes: 5e31275cc997 ("mm: add per-VMA lock and helper functions to control it") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reported-by: Daehyeon Ko <4ncienth@gmail.com> Closes: https://lore.kernel.org/all/20261015143735.1630957-1-4ncienth@gmail.com/ Reviewed-by: Jose A. Perez de Azpillaga Tested-by: Jose A. Perez de Azpillaga Cc: Suren Baghdasaryan Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Shakeel Butt Cc: David Hildenbrand Cc: Mike Rapoport Cc: Michal Hocko Cc: Jonathan Corbet Cc: Randy Dunlap Cc: Jann Horn Cc: Pedro Falcato Cc: # 7.3.x --- Documentation/mm/process_addrs.rst | 14 ++++- include/linux/mm_types.h | 12 ++-- include/linux/mmap_lock.h | 24 ++++++-- mm/mmap_lock.c | 89 ++++++++++++++++++++++++++---- tools/testing/vma/include/dup.h | 2 +- tools/testing/vma/shared.c | 8 +-- 6 files changed, 120 insertions(+), 29 deletions(-) diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index 78231995e490..04b649818cc1 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -819,9 +819,17 @@ All this is achieved by the use of per-mm and per-VMA sequence counts, which are used in order to reduce complexity, especially for operations which write-lock multiple VMAs at once. -If the mm sequence count, :c:member:`!mm->mm_lock_seq` is equal to the VMA -sequence count :c:member:`!vma->vm_lock_seq` then the VMA is write-locked. If -they differ, then it is not. +The mm sequence count :c:member:`!mm->mm_lock_seq` is incremented on both mmap +write lock and unlock, so it is odd while the mmap write lock is held. If it is +equal to the VMA sequence count :c:member:`!vma->vm_lock_seq`, and that value +is odd, then the VMA is write-locked. If they differ, then it is not. + +A VMA's sequence count is not updated on unlock, so after 2^31 write lock/unlock +cycles the mm sequence count would wrap and match stale VMA values again. To +prevent this, when the mm sequence count wraps to zero, +:c:func:`!vma_end_write_all` invokes :c:func:`!__mm_reset_vma_seqcounts` to +reset every VMA's sequence count to zero, an even value which can never +indicate a write lock. Each time the mmap write lock is released in :c:func:`!mmap_write_unlock` or :c:func:`!mmap_write_downgrade`, :c:func:`!vma_end_write_all` is invoked which diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0dead656d3e3..1ed35d258d2e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -992,15 +992,19 @@ struct vm_area_struct { * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_FLAG is set + * The one exception is __mm_reset_vma_seqcounts(), which stores zero + * (an even value, which can never indicate a write lock) under the + * mmap write lock alone once the mm seqcount has wrapped, so a reader + * holding only vm_refcnt may observe the value change from odd to zero. * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_BIT is set or vm_refcnt > 1 + * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_BIT is set * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * - * This sequence counter is explicitly allowed to overflow; sequence - * counter reuse can only lead to occasional unnecessary use of the - * slowpath. + * Assigned the (odd) mm seqcount on write lock and not updated on + * unlock, so a stale value would alias the mm seqcount again 2^31 write + * lock/unlock cycles later. The reset described above prevents this. */ unsigned int vm_lock_seq; diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e5553f4a414c..11b3ffef3966 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -154,7 +154,7 @@ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) #endif if (reset_refcnt) refcount_set(&vma->vm_refcnt, 0); - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; } /* @@ -229,8 +229,9 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) /* * Use only while holding mmap read lock which guarantees that vma lock is not - * contended (nobody can concurrently write-lock the vma). vma_start_read() should - * not be used in such cases because it might fail due to mm_lock_seq overflow. + * contended (nobody can concurrently write-lock the vma). vma_start_read() need + * not be used in such cases, as the mmap lock already excludes VMA writers so + * its sequence number checks are redundant. * This functionality is used to obtain vma read lock and drop the mmap read lock. * * VMA can't be detached while we are holding mmap lock, therefore in practice this @@ -252,8 +253,9 @@ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int /* * Use only while holding mmap read lock which guarantees that vma lock is not - * contended (nobody can concurrently write-lock the vma). vma_start_read() should - * not be used in such cases because it might fail due to mm_lock_seq overflow. + * contended (nobody can concurrently write-lock the vma). vma_start_read() need + * not be used in such cases, as the mmap lock already excludes VMA writers so + * its sequence number checks are redundant. * This functionality is used to obtain vma read lock and drop the mmap read lock. * * VMA can't be detached while we are holding mmap lock, therefore in practice this @@ -552,6 +554,8 @@ static inline int __must_check mmap_write_lock_killable(struct mm_struct *mm) return ret; } +void __mm_reset_vma_seqcounts(struct mm_struct *mm); + /* * Drop all currently-held per-VMA locks. * This is called from the mmap_lock implementation directly before releasing @@ -564,6 +568,16 @@ static inline void vma_end_write_all(struct mm_struct *mm) { mmap_assert_write_locked(mm); mm_lock_seqcount_end(mm); + + /* + * VMA seqcounts are not updated on unlock, so once the mm seqcount has + * wrapped, any VMA still carrying a sequence number from before the + * wrap would satisfy __is_vma_write_locked() when that number comes + * round again, and readers would not be excluded. Reset every VMA's + * seqcount so that no stale value survives the wrap. + */ + if (unlikely(!mm->mm_lock_seq.sequence)) + __mm_reset_vma_seqcounts(mm); } static inline void mmap_write_unlock(struct mm_struct *mm) diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c index 2f94ee0fdee2..83b7d3ded0dc 100644 --- a/mm/mmap_lock.c +++ b/mm/mmap_lock.c @@ -194,21 +194,38 @@ void __vma_exclude_readers_for_detach(struct vm_area_struct *vma) WARN_ON_ONCE(!ves.detached); } +/* Do the VMA, mm seqcounts indicate the VMA is write locked? */ +static bool vma_seqcount_is_write_locked(unsigned int vma_seq, unsigned int mm_seq) +{ + /* VMA and mm sequence numbers must match for a write lock. */ + if (vma_seq != mm_seq) + return false; + + /* + * The VMA seqcount is always odd if locked. For an attached VMA an even + * value can only be the zero stored by __mm_reset_vma_seqcounts(), + * never a write lock. + */ + return vma_seq & 1; +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. - * False locked result is possible if mm_lock_seq overflows or if vma gets - * reused and attached to a different mm before we lock it. + * False locked result is possible if vma gets reused and attached to a + * different mm before we lock it. * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got * detached. * * IMPORTANT: RCU lock must be held upon entering the function, but upon error * IT IS RELEASED. The caller must handle this correctly. */ -static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, - struct vm_area_struct *vma) +static struct vm_area_struct *vma_start_read(struct mm_struct *mm, + struct vm_area_struct *vma) { + unsigned int vma_seq; + unsigned int mm_seq; struct mm_struct *other_mm; int oldcnt; @@ -220,7 +237,9 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) { + vma_seq = READ_ONCE(vma->vm_lock_seq); + mm_seq = READ_ONCE(mm->mm_lock_seq.sequence); + if (vma_seqcount_is_write_locked(vma_seq, mm_seq)) { vma = NULL; goto err; } @@ -246,17 +265,26 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, goto err_unstable; /* - * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. - * False unlocked result is impossible because we modify and check + * False unlocked result is impossible because a write lock sets * vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * - * We must use ACQUIRE semantics for the mm_lock_seq so that if we are - * racing with vma_end_write_all(), we only start reading from the VMA - * after it has been unlocked. - * This pairs with RELEASE semantics in vma_end_write_all(). + * ACQUIRE semantics are required for both reads. + * + * __mm_reset_vma_seqcounts() might reset vma->vm_lock_seq to zero + * after an overflow. A reader observing that zero must also observe + * the VMA modifications made under the write lock which just ended, so + * the ACQUIRE pairs with the smp_wmb() in do_raw_write_seqcount_end() + * that precedes the reset stores. + * + * The ACQUIRE semantics for the mm_lock_seq read is required so that, + * if we are racing with vma_end_write_all(), we only start reading from + * the VMA after it has been unlocked. This pairs with the RELEASE + * semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { + vma_seq = smp_load_acquire(&vma->vm_lock_seq); + mm_seq = raw_read_seqcount(&mm->mm_lock_seq); + if (unlikely(vma_seqcount_is_write_locked(vma_seq, mm_seq))) { vma_refcount_put(vma); vma = NULL; goto err; @@ -466,6 +494,43 @@ struct vm_area_struct *lock_next_vma(struct mm_struct *mm, return vma; } +/* + * Called with mmap write lock held after mm->mm_lock_seq just overflowed (and + * therefore zero), with all VMA write locks now released. + * + * The stores below follow the smp_wmb() in do_raw_write_seqcount_end(), so + * a reader that observes a reset value also observes this cycle's VMA + * modifications (ordered by the ACQUIRE on vma->vm_lock_seq in + * vma_start_read()). + */ +void __mm_reset_vma_seqcounts(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + VMA_ITERATOR(vmi, mm, 0); + + mmap_assert_write_locked(mm); + VM_WARN_ON_ONCE(mm->mm_lock_seq.sequence); + + /* + * Failing to reset means stale vma->vm_lock_seq values could result in + * those VMAs becoming spuriously write locked when mm->mm_lock_seq + * happens to match their values. + * + * This is problematic, as it causes the VMA write lock to not exclude + * readers, which can result in concurrent readers while a VMA write + * lock is held. + * + * Avoid this by resetting to an even value (indicates no write lock + * held). + * + * Readers will either observe zero (not write locked) or the previous + * value, so this is safe. + */ + for_each_vma(vmi, vma) + WRITE_ONCE(vma->vm_lock_seq, 0); +} +EXPORT_SYMBOL(__mm_reset_vma_seqcounts); + #ifdef CONFIG_LOCK_MM_AND_FIND_VMA #include diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 718132580fac..f449129ea97a 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1225,7 +1225,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; } /* diff --git a/tools/testing/vma/shared.c b/tools/testing/vma/shared.c index 8c4826499f40..c1e0c2280101 100644 --- a/tools/testing/vma/shared.c +++ b/tools/testing/vma/shared.c @@ -65,7 +65,7 @@ struct vm_area_struct *alloc_and_link_vma(struct mm_struct *mm, * begun. Linking to the tree will have caused this to be incremented, * which means we will get a false positive otherwise. */ - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; return vma; } @@ -97,13 +97,13 @@ int cleanup_mm(struct mm_struct *mm, struct vma_iterator *vmi) bool vma_write_started(struct vm_area_struct *vma) { - int seq = vma->vm_lock_seq; + unsigned int seq = vma->vm_lock_seq; /* We reset after each check. */ - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; /* The vma_start_write() stub simply increments this value. */ - return seq > -1; + return seq > 0; } void __vma_set_dummy_anon_vma(struct vm_area_struct *vma, From 397e087605526e31c067afd5fba85bd999e5738f Mon Sep 17 00:00:00 2001 From: Nimrod Oren Date: Wed, 14 Oct 2026 17:08:59 +0300 Subject: [PATCH 182/213] mm: stop recalculating min_free_kbytes once set by userspace min_free_kbytes is recalculated on memory hotplug and on THP sysfs writes. The new value comes from calculate_min_free_kbytes() or, while THP is enabled, from set_recommended_min_free_kbytes(). A value written to the vm.min_free_kbytes sysctl is replaced whenever the new value is higher, so a lower value does not stick. This matters most when THP is enabled on configurations with large pageblocks, where the new value can be excessive [1]. For example, with default arm64 pageblock sizes, the THP recommendation can reach 5.5 GiB per eligible zone with 64 KiB pages vs 22 MiB with 4 KiB pages. This keeps a significant amount of memory free, reducing what applications can use and triggering reclaim much earlier. Users can lower vm.min_free_kbytes to avoid this, but the lower value only sticks until the next recalculation. Stop recalculating min_free_kbytes once a value has been written to the sysctl. Systems where the sysctl is never written keep the current behavior. Link: https://lore.kernel.org/20261014140859.1006967-1-noren@nvidia.com Link: https://lore.kernel.org/r/20260716173504.760369-1-noren@nvidia.com/ [1] Signed-off-by: Nimrod Oren Signed-off-by: Andrew Morton Suggested-by: Vlastimil Babka (SUSE) Link: https://lore.kernel.org/r/d202dcc3-d7d7-4e98-a9a5-3137cfbfd041@kernel.org/ Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Johannes Weiner Acked-by: Zi Yan Reviewed-by: Lorenzo Stoakes (ARM) Cc: David Hildenbrand Cc: Liam R. Howlett Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jonathan Corbet Cc: Shuah Khan Cc: Randy Dunlap Cc: Baolin Wang Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Kiryl Shutsemau Cc: Brendan Jackman Cc: David Laight --- Documentation/admin-guide/sysctl/vm.rst | 7 +++++++ mm/khugepaged.c | 7 +------ mm/page_alloc.c | 6 +----- 3 files changed, 9 insertions(+), 11 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 5b318d17aa4b..189fc57dac5f 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -539,6 +539,13 @@ watermark[WMARK_MIN] value for each lowmem zone in the system. Each lowmem zone gets a number of reserved free pages based proportionally on its size. +By default, the kernel sets this value based on the amount of lowmem +and recalculates it on memory hotplug. When Transparent Hugepage (THP) +support is enabled through global or per-size controls, the kernel may +also raise it to help keep pageblocks free and reduce fragmentation for +THP allocations. Once a value is written to this file, the kernel no +longer adjusts it automatically. + Some minimal amount of memory is needed to satisfy PF_MEMALLOC allocations; if you set this to lower than 1024KB, your system will become subtly broken, and prone to deadlock under high loads. diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 17e0e0162e90..4b71d010b2af 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -3144,13 +3144,8 @@ void set_recommended_min_free_kbytes(void) (unsigned long) nr_free_buffer_pages() / 20); recommended_min <<= (PAGE_SHIFT-10); - if (recommended_min > min_free_kbytes) { - if (user_min_free_kbytes >= 0) - pr_info_ratelimited("raising min_free_kbytes from %d to %lu to help transparent hugepage allocations\n", - min_free_kbytes, recommended_min); - + if (user_min_free_kbytes < 0 && recommended_min > min_free_kbytes) min_free_kbytes = recommended_min; - } update_wmarks: setup_per_zone_wmarks(); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 914e3c9375f2..f14e2ac21787 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6722,12 +6722,8 @@ void calculate_min_free_kbytes(void) lowmem_kbytes = nr_free_buffer_pages() * (PAGE_SIZE >> 10); new_min_free_kbytes = int_sqrt(lowmem_kbytes * 16); - if (new_min_free_kbytes > user_min_free_kbytes) + if (user_min_free_kbytes < 0) min_free_kbytes = clamp(new_min_free_kbytes, 128, 262144); - else - pr_warn_ratelimited("min_free_kbytes is not updated to %d because user defined value %d is preferred\n", - new_min_free_kbytes, user_min_free_kbytes); - } int __meminit init_per_zone_wmark_min(void) From 60f34fedf3a4c91ed631ed2f80713ab993cde5ee Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Thu, 15 Oct 2026 14:05:27 +0800 Subject: [PATCH 183/213] mm/swap: fix swap_cache_get_folio() returning a folio that left the swap cache swap_cache_get_folio() reads the folio from the swap table and takes a reference with folio_try_get(), without any lock. If the folio is removed from the swap cache and freed in between, and its memory gets reused, folio_try_get() still succeeds and we return a page that has nothing to do with the swap entry: CPU 0 CPU 1 do_swap_page() swap_cache_get_folio() swp_tb = swap_table_get() folio = swp_tb_to_folio(swp_tb) madvise(MADV_DONTNEED) zap_nonpresent_ptes() swap_put_entries_direct() swap_put_entries_cluster() __try_to_reclaim_swap() swap_cache_del_folio() folio_put() <- freed page reused, e.g. as an anon folio of another task folio_try_get(folio) <- succeeds return folio swap_update_readahead() <- may clear PG_readahead on it folio_lock_or_retry() <- locks it folio_matches_swap_entry() fails, unlock and put filemap_get_entry() does the same lockless lookup for the page cache, and after folio_try_get() it rechecks the xarray, because the folio may have been freed and reused in between. swap_cache_get_folio() lost that recheck when the swap cache moved to the swap table. So CPU 0 writes to a folio it doesn't own: it may clear PG_readahead and takes the folio lock of someone else's page, before folio_matches_swap_entry() finds out the folio is not ours. When the race occurs, a swap fault can acquire a reference to a physical page that has been freed and reused for an unrelated purpose, potentially by another process. The fault handler may then clear that page's PG_readahead/PG_reclaim flag, wait on or acquire its folio lock, and potentially attempt to remove it from an unrelated swap cache. These operations can interfere with unrelated workloads and may cause unexpected stalls and impaired reclaim. The subsequent swap-entry validation normally prevents the wrong folio from being mapped into the faulting process. It's documented that caller always double checks the folio so this part is harmless from the aspect of data corruption or crash. Recheck the slot after folio_try_get() and retry if it no longer holds this folio. [akpm@linux-foundation.org: add LLM-generated description of userspace impact] Link: https://lore.kernel.org/20261015060528.302634-1-jiayuan.chen@linux.dev Fixes: 8578e0c00dcf ("mm, swap: use the swap table for the swap cache and switch API") Signed-off-by: Jiayuan Chen Signed-off-by: Andrew Morton Reviewed-by: Kairui Song Cc: Chris Li Cc: Kemeng Shi Cc: Nhat Pham Cc: Baoquan He Cc: Barry Song Cc: Youngjun Park Cc: --- mm/swap_state.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index ebef568cd62e..d9e36c505398 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -92,17 +92,27 @@ void show_swap_cache_info(void) */ struct folio *swap_cache_get_folio(swp_entry_t entry) { + struct swap_cluster_info *ci = __swap_entry_to_cluster(entry); + unsigned int ci_off = swp_cluster_offset(entry); unsigned long swp_tb; struct folio *folio; for (;;) { - swp_tb = swap_table_get(__swap_entry_to_cluster(entry), - swp_cluster_offset(entry)); + swp_tb = swap_table_get(ci, ci_off); if (!swp_tb_is_folio(swp_tb)) return NULL; folio = swp_tb_to_folio(swp_tb); - if (likely(folio_try_get(folio))) + if (!folio_try_get(folio)) + continue; + /* + * The folio may have been freed and its memory reused before + * we got the reference. Make sure it is still in the slot. + */ + swp_tb = swap_table_get(ci, ci_off); + if (likely(swp_tb_is_folio(swp_tb) && + swp_tb_to_folio(swp_tb) == folio)) return folio; + folio_put(folio); } return NULL; From 4b77739fe7893bced405891233511310554871a7 Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Sat, 10 Oct 2026 15:57:50 +0800 Subject: [PATCH 184/213] mm/sparse-vmemmap: drop VMEMMAP_POPULATE_DAX Patch series "mm: Unify device DAX and HugeTLB vmemmap population paths", v4. This series is split out from the earlier, larger series "mm: Generalize HVO for HugeTLB and device DAX" [2]. While the parent series generalizes vmemmap optimization across HugeTLB and device DAX, this subset addresses a single, self-contained step: unifying their vmemmap population paths. After the preceding Device DAX conversion, both HugeTLB and Device DAX describe optimized vmemmap mappings through memory-section metadata and use per-zone shared tail vmemmap pages. The generic code, however, still carries a Device DAX-specific population flag and compound-page population path, along with arguments and helpers needed only by that path. This series first removes VMEMMAP_POPULATE_DAX and moves selection and reference handling for the shared tail page into the common vmemmap population path. It then removes the generic Device DAX-specific compound-page population path and routes section vmemmap population through vmemmap_populate(). The powerpc radix path continues to use its architecture-specific compound-page population implementation for optimizable sections. The remaining patches remove the unused ptpfn argument, open-code vmemmap_populate_address() now that no caller needs its returned PTE, and add a warning for inconsistent zone initialization of shared tail vmemmap pages. This is the fourth smaller step toward the broader HVO generalization. After this series, HugeTLB and Device DAX use the same population model instead of parallel generic paths, while powerpc keeps its architecture-specific implementation. This patch (of 6): VMEMMAP_POPULATE_DAX currently distinguishes DAX vmemmap population in two places: it keeps allocations on the normal path and takes a reference when a backing page is supplied for reuse. After Device DAX switched to the common per-zone shared tail page, both conditions can be determined locally. DAX supplies ptpfn for every shared tail mapping and requests an allocation only for compound head mappings, whose PFNs are not optimizable. Therefore, vmemmap_optimizable_pfn() alone selects the correct allocation path. When ptpfn is supplied, the caller is reusing an existing backing page. Once the slab allocator is available, take a reference for each reused mapping to balance the release performed by vmemmap_free(). Although the buddy allocator is available before slab, no vmemmap population occurs in that interval. Earlier mappings are backed by memblock/reserved memory and do not need page reference accounting. Remove VMEMMAP_POPULATE_DAX and the flags argument from the vmemmap population helpers. Link: https://lore.kernel.org/20261010075755.83339-1-songmuchun@bytedance.com Link: https://lore.kernel.org/20261010075755.83339-2-songmuchun@bytedance.com Link: https://lore.kernel.org/20260930140627.57431-1-songmuchun@bytedance.com/ [1] Link: https://lore.kernel.org/20260513130542.35604-1-songmuchun@bytedance.com/ [2] Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Reviewed-by: Lance Yang Acked-by: Qi Zheng Cc: Madhavan Srinivasan Cc: Mike Rapoport Cc: David Hildenbrand Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Ritesh Harjani (IBM) Cc: Shrikanth Hegde Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko --- mm/sparse-vmemmap.c | 36 +++++++++++++----------------------- 1 file changed, 13 insertions(+), 23 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 098975861523..b23efff80137 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -32,12 +32,6 @@ #include #include -/* - * Flags for vmemmap_populate_range and friends. - */ -/* Vmemmap population for ZONE_DEVICE compound pages */ -#define VMEMMAP_POPULATE_DAX 0x0001 - #include "internal.h" #include "mm_init.h" #include "sparse.h" @@ -241,7 +235,7 @@ struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zon #endif static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, - struct vmem_altmap *altmap, unsigned long flags) + struct vmem_altmap *altmap) { struct zone *zone; struct page *page; @@ -251,7 +245,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, * Device DAX still relies on vmemmap_populate_compound_pages() for * head/first-tail allocation and tail-page reuse. */ - if (!vmemmap_optimizable_pfn(pfn) || flags & VMEMMAP_POPULATE_DAX) + if (!vmemmap_optimizable_pfn(pfn)) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); zone = pfn_to_zone(pfn, node); @@ -263,8 +257,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, } static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn, unsigned long flags) + struct vmem_altmap *altmap, unsigned long ptpfn) { pte_t *pte = pte_offset_kernel(pmd, addr); unsigned long pfn = page_to_pfn((struct page *)addr); @@ -273,7 +266,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in pte_t entry; if (ptpfn == (unsigned long)-1) { - void *p = vmemmap_alloc_pte(pfn, node, altmap, flags); + void *p = vmemmap_alloc_pte(pfn, node, altmap); if (!p) return NULL; @@ -291,7 +284,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in * Use try_get_page() to prevent the shared page refcount * from overflowing. */ - if ((flags & VMEMMAP_POPULATE_DAX) && + if (slab_is_available() && !try_get_page(pfn_to_page(ptpfn))) return NULL; } @@ -355,8 +348,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, struct vmem_altmap *altmap, - unsigned long ptpfn, - unsigned long flags) + unsigned long ptpfn) { pgd_t *pgd; p4d_t *p4d; @@ -376,7 +368,7 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn, flags); + pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn); if (!pte) return NULL; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); @@ -387,15 +379,14 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, static int __meminit vmemmap_populate_range(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap, - unsigned long ptpfn, - unsigned long flags) + unsigned long ptpfn) { unsigned long addr = start; pte_t *pte; for (; addr < end; addr += PAGE_SIZE) { pte = vmemmap_populate_address(addr, node, altmap, - ptpfn, flags); + ptpfn); if (!pte) return -ENOMEM; } @@ -406,7 +397,7 @@ static int __meminit vmemmap_populate_range(unsigned long start, int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_range(start, end, node, altmap, -1, 0); + return vmemmap_populate_range(start, end, node, altmap, -1); } /* @@ -532,7 +523,6 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long end, int node, struct dev_pagemap *pgmap) { - const unsigned long flags = VMEMMAP_POPULATE_DAX; const unsigned int order = pfn_to_section_compound_order(start_pfn); unsigned long size, addr; pte_t *pte; @@ -545,14 +535,14 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, if (reuse_compound_section(start_pfn, pgmap)) return vmemmap_populate_range(start, end, node, NULL, - page_to_pfn(page), flags); + page_to_pfn(page)); size = min(end - start, (1UL << order) * sizeof(struct page)); for (addr = start; addr < end; addr += size) { unsigned long next, last = addr + size; /* Populate the head page vmemmap page */ - pte = vmemmap_populate_address(addr, node, NULL, -1, flags); + pte = vmemmap_populate_address(addr, node, NULL, -1); if (!pte) return -ENOMEM; @@ -562,7 +552,7 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, */ next = addr + PAGE_SIZE; rc = vmemmap_populate_range(next, last, node, NULL, - page_to_pfn(page), flags); + page_to_pfn(page)); if (rc) return -ENOMEM; } From a123cf7ac6cbcb127a24a825e82d9f7d2de84468 Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Sat, 10 Oct 2026 15:57:51 +0800 Subject: [PATCH 185/213] mm/sparse-vmemmap: support device DAX in common vmemmap path The common vmemmap population path cannot yet handle optimized Device DAX mappings on its own. It uses pfn_to_zone() to find the shared tail page, but Device DAX populates its vmemmap at runtime before the ZONE_DEVICE span is initialized. Teach the common path to use device_zone() for runtime optimized vmemmap population while retaining pfn_to_zone() for early boot. This allows the same path to support both early boot mappings and Device DAX. The backing PFN supplied by the Device DAX-specific population path is no longer used, allowing the redundant lookup and population code to be removed later. Link: https://lore.kernel.org/20261010075755.83339-3-songmuchun@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Reviewed-by: Lance Yang Acked-by: Qi Zheng Cc: Madhavan Srinivasan Cc: Mike Rapoport Cc: David Hildenbrand Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Ritesh Harjani (IBM) Cc: Shrikanth Hegde Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko --- mm/sparse-vmemmap.c | 62 +++++++++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 28 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b23efff80137..62327ac15206 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -241,18 +241,43 @@ static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct page *page; const unsigned int order = pfn_to_section_compound_order(pfn); - /* - * Device DAX still relies on vmemmap_populate_compound_pages() for - * head/first-tail allocation and tail-page reuse. - */ if (!vmemmap_optimizable_pfn(pfn)) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - zone = pfn_to_zone(pfn, node); + /* + * Before slab is available, vmemmap optimization is used for early + * system RAM, whose zone can be determined from the PFN. + * + * Once slab is available, only ZONE_DEVICE memory reaches this + * optimized population path. Its zone span has not been initialized + * while its vmemmap is being populated, so pfn_to_zone() cannot be + * used. Obtain ZONE_DEVICE directly from the node instead. + */ + zone = slab_is_available() ? device_zone(node) : pfn_to_zone(pfn, node); page = vmemmap_shared_tail_page(order, zone); if (!page) return NULL; + /* + * During early vmemmap population, the shared tail vmemmap backing + * page is allocated from memblock before its struct page can safely + * participate in page refcounting. Therefore, no reference can be + * held for each shared PTE mapping, and the mappings must be unshared + * before the vmemmap is depopulated. + * + * Once slab is available, the shared backing page is allocated from + * the buddy allocator and can be refcounted. Hold one reference for + * each shared PTE mapping. The architecture vmemmap teardown drops + * the reference through __free_pages() when removing the mapping, + * preventing the backing page from being freed while it is shared. + * + * The backing page may be shared by enough PTE mappings to exhaust + * the positive range of its reference count. Stop populating the + * vmemmap if another reference cannot be acquired. + */ + if (slab_is_available() && !try_get_page(page)) + return NULL; + return page_address(page); } @@ -264,31 +289,12 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in if (pte_none(ptep_get(pte))) { pte_t entry; + void *p = vmemmap_alloc_pte(pfn, node, altmap); - if (ptpfn == (unsigned long)-1) { - void *p = vmemmap_alloc_pte(pfn, node, altmap); + if (!p) + return NULL; - if (!p) - return NULL; - ptpfn = PHYS_PFN(__pa(p)); - } else { - /* - * When a PTE/PMD entry is freed from the init_mm - * there's a free_pages() call to this page allocated - * above. Thus this try_get_page() is paired with the - * put_page_testzero() on the freeing path. - * This can only called by certain ZONE_DEVICE path, - * and through vmemmap_populate_compound_pages() when - * slab is available. - * - * Use try_get_page() to prevent the shared page refcount - * from overflowing. - */ - if (slab_is_available() && - !try_get_page(pfn_to_page(ptpfn))) - return NULL; - } - entry = pfn_pte(ptpfn, PAGE_KERNEL); + entry = pfn_pte(PHYS_PFN(__pa(p)), PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) return NULL; From bd0862382adb3373bf5cd79531fecd41fb4c25e4 Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Sat, 10 Oct 2026 15:57:52 +0800 Subject: [PATCH 186/213] mm/sparse-vmemmap: drop Device DAX-specific population path The common vmemmap path selects the shared page for optimized mappings itself, so Device DAX no longer needs vmemmap_populate_compound_pages() to find a shared tail page and pass its backing PFN through the generic population helpers. Remove the Device DAX-specific population path and let section memmap population always use vmemmap_populate(). The powerpc retains an architecture-specific compound-page implementation, so select it directly from radix__vmemmap_populate() for optimizable sections. Link: https://lore.kernel.org/20261010075755.83339-4-songmuchun@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Reviewed-by: Lance Yang Acked-by: Qi Zheng Cc: Madhavan Srinivasan Cc: Mike Rapoport Cc: David Hildenbrand Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Ritesh Harjani (IBM) Cc: Shrikanth Hegde Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko --- arch/powerpc/mm/book3s64/radix_pgtable.c | 3 + mm/mm_init.c | 2 +- mm/sparse-vmemmap.c | 71 +----------------------- 3 files changed, 5 insertions(+), 71 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index a3332f32ffb2..44868b62b14e 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1126,7 +1126,10 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in pud_t *pud; pmd_t *pmd; pte_t *pte; + unsigned long pfn = page_to_pfn((struct page *)start); + if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn))) + return vmemmap_populate_compound_pages(pfn, start, end, node, NULL); /* * If altmap is present, Make sure we align the start vmemmap addr * to PAGE_SIZE so that we calculate the correct start_pfn in diff --git a/mm/mm_init.c b/mm/mm_init.c index 56bb4567a494..1650d6bc1211 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1046,7 +1046,7 @@ static void zone_device_page_init_from_template(struct page *page, * initialize is a lot smaller that the total amount of struct pages being * mapped. This is a paired / mild layering violation with explicit knowledge * of how the sparse_vmemmap internals handle compound pages in the lack - * of an altmap. See vmemmap_populate_compound_pages(). + * of an altmap. */ static inline unsigned long compound_nr_pages(unsigned long pfn, struct dev_pagemap *pgmap) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 62327ac15206..270b5def58e1 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -503,71 +503,6 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return 0; } -#ifndef vmemmap_populate_compound_pages -/* - * For compound pages bigger than section size (e.g. x86 1G compound - * pages with 2M subsection size) fill the rest of sections as tail - * pages. - * - * Note that memremap_pages() resets @nr_range value and will increment - * it after each range successful onlining. Thus the value or @nr_range - * at section memmap populate corresponds to the in-progress range - * being onlined here. - */ -static bool __meminit reuse_compound_section(unsigned long start_pfn, - struct dev_pagemap *pgmap) -{ - unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); - unsigned long offset = start_pfn - - PHYS_PFN(pgmap->ranges[pgmap->nr_range].start); - - return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; -} - -static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, - unsigned long start, - unsigned long end, int node, - struct dev_pagemap *pgmap) -{ - const unsigned int order = pfn_to_section_compound_order(start_pfn); - unsigned long size, addr; - pte_t *pte; - struct page *page; - int rc; - - page = vmemmap_shared_tail_page(order, device_zone(node)); - if (!page) - return -ENOMEM; - - if (reuse_compound_section(start_pfn, pgmap)) - return vmemmap_populate_range(start, end, node, NULL, - page_to_pfn(page)); - - size = min(end - start, (1UL << order) * sizeof(struct page)); - for (addr = start; addr < end; addr += size) { - unsigned long next, last = addr + size; - - /* Populate the head page vmemmap page */ - pte = vmemmap_populate_address(addr, node, NULL, -1); - if (!pte) - return -ENOMEM; - - /* - * Reuse the shared page for the rest of tail pages - * See layout diagram in Documentation/mm/vmemmap_dedup.rst - */ - next = addr + PAGE_SIZE; - rc = vmemmap_populate_range(next, last, node, NULL, - page_to_pfn(page)); - if (rc) - return -ENOMEM; - } - - return 0; -} - -#endif - struct page * __meminit __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) @@ -580,11 +515,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - if (pgmap && section_vmemmap_optimizable(__pfn_to_section(pfn))) - r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); - else - r = vmemmap_populate(start, end, nid, altmap); - + r = vmemmap_populate(start, end, nid, altmap); if (r < 0) return NULL; From 2815c5d88f2a1255669cfa7ed260e7fd70532188 Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Sat, 10 Oct 2026 15:57:53 +0800 Subject: [PATCH 187/213] mm/sparse-vmemmap: remove the unused ptpfn argument vmemmap_pte_populate() no longer uses ptpfn as an input. Drop the argument to simplify the code. Link: https://lore.kernel.org/20261010075755.83339-5-songmuchun@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Reviewed-by: Lance Yang Acked-by: Qi Zheng Cc: Madhavan Srinivasan Cc: Mike Rapoport Cc: David Hildenbrand Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Ritesh Harjani (IBM) Cc: Shrikanth Hegde Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko --- mm/sparse-vmemmap.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 270b5def58e1..07188cdf00f6 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -282,7 +282,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, } static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap, unsigned long ptpfn) + struct vmem_altmap *altmap) { pte_t *pte = pte_offset_kernel(pmd, addr); unsigned long pfn = page_to_pfn((struct page *)addr); @@ -353,8 +353,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) } static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn) + struct vmem_altmap *altmap) { pgd_t *pgd; p4d_t *p4d; @@ -374,7 +373,7 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn); + pte = vmemmap_pte_populate(pmd, addr, node, altmap); if (!pte) return NULL; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); @@ -384,15 +383,13 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, static int __meminit vmemmap_populate_range(unsigned long start, unsigned long end, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn) + struct vmem_altmap *altmap) { unsigned long addr = start; pte_t *pte; for (; addr < end; addr += PAGE_SIZE) { - pte = vmemmap_populate_address(addr, node, altmap, - ptpfn); + pte = vmemmap_populate_address(addr, node, altmap); if (!pte) return -ENOMEM; } @@ -403,7 +400,7 @@ static int __meminit vmemmap_populate_range(unsigned long start, int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_range(start, end, node, altmap, -1); + return vmemmap_populate_range(start, end, node, altmap); } /* From cc1fd8f4017f68597091b3dc6d8c5eba2a8c3afc Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Sat, 10 Oct 2026 15:57:54 +0800 Subject: [PATCH 188/213] mm/sparse-vmemmap: open-code vmemmap_populate_address() vmemmap_populate_address() no longer has any callers that need the returned PTE. Its only remaining user, vmemmap_populate_range(), only checks whether population succeeded. Open-code vmemmap_populate_address() directly in vmemmap_populate_basepages(), remove the now-redundant range helper, and return -ENOMEM directly on failure. Link: https://lore.kernel.org/20261010075755.83339-6-songmuchun@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Reviewed-by: Lance Yang Acked-by: Qi Zheng Cc: Madhavan Srinivasan Cc: Mike Rapoport Cc: David Hildenbrand Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Ritesh Harjani (IBM) Cc: Shrikanth Hegde Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko --- mm/sparse-vmemmap.c | 54 ++++++++++++++------------------------------- 1 file changed, 17 insertions(+), 37 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 07188cdf00f6..40eba70c0048 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -352,8 +352,8 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap) +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) { pgd_t *pgd; p4d_t *p4d; @@ -361,48 +361,28 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd_t *pmd; pte_t *pte; - pgd = vmemmap_pgd_populate(addr, node); - if (!pgd) - return NULL; - p4d = vmemmap_p4d_populate(pgd, addr, node); - if (!p4d) - return NULL; - pud = vmemmap_pud_populate(p4d, addr, node); - if (!pud) - return NULL; - pmd = vmemmap_pmd_populate(pud, addr, node); - if (!pmd) - return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); - if (!pte) - return NULL; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - -static int __meminit vmemmap_populate_range(unsigned long start, - unsigned long end, int node, - struct vmem_altmap *altmap) -{ - unsigned long addr = start; - pte_t *pte; - - for (; addr < end; addr += PAGE_SIZE) { - pte = vmemmap_populate_address(addr, node, altmap); + for (unsigned long addr = start; addr < end; addr += PAGE_SIZE) { + pgd = vmemmap_pgd_populate(addr, node); + if (!pgd) + return -ENOMEM; + p4d = vmemmap_p4d_populate(pgd, addr, node); + if (!p4d) + return -ENOMEM; + pud = vmemmap_pud_populate(p4d, addr, node); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_populate(pud, addr, node); + if (!pmd) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap); if (!pte) return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); } return 0; } -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) -{ - return vmemmap_populate_range(start, end, node, altmap); -} - /* * Write protect the mirrored tail page structs for HVO. This will be * called from the hugetlb code when gathering and initializing the From 3d670c0dcea42afd021fe15c7438f7a48c63b552 Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Sat, 10 Oct 2026 15:57:55 +0800 Subject: [PATCH 189/213] mm/mm_init: add zone mismatch warning during page init For vmemmap-optimized sections, tail struct pages may be backed by shared vmemmap pages. Those shared pages must carry the same page zone ID as the struct pages initialized for the section. Warn in __init_single_page() if the shared tail page has a different page_zone_id(), which would indicate inconsistent initialization. [muchun.song@linux.dev: document ordering of shared and retained struct page initialization] Link: https://lore.kernel.org/20261010115523.4005071-1-songmuchun@bytedance.com Link: https://lore.kernel.org/20261010075755.83339-7-songmuchun@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Reviewed-by: Lance Yang Acked-by: Qi Zheng Cc: Madhavan Srinivasan Cc: Mike Rapoport Cc: David Hildenbrand Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Ritesh Harjani (IBM) Cc: Shrikanth Hegde Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Suren Baghdasaryan Cc: Michal Hocko --- mm/mm_init.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/mm_init.c b/mm/mm_init.c index 1650d6bc1211..4c17a892924d 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -593,6 +593,15 @@ static void __init find_zone_movable_pfns_for_nodes(void) node_states[N_MEMORY] = saved_node_state; } +static inline void vmemmap_optimization_verify_zone(struct page *page) +{ + const unsigned int order = pfn_to_section_compound_order(page_to_pfn(page)); + + VM_WARN_ON_ONCE(vmemmap_optimizable_order(order) && + page_zone_id(page + VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES) != + page_zone_id(page)); +} + void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { @@ -609,6 +618,11 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, if (!is_highmem_idx(zone)) set_page_address(page, __va(pfn << PAGE_SHIFT)); #endif + /* + * Shared tail struct pages are initialized during vmemmap population, + * before retained struct pages reach here. + */ + vmemmap_optimization_verify_zone(page); } #ifdef CONFIG_NUMA From 1503710b3b0cec02b531978129c6c1eb70a7c490 Mon Sep 17 00:00:00 2001 From: Rik van Riel Date: Fri, 16 Oct 2026 12:23:04 -0400 Subject: [PATCH 190/213] mm/page_alloc: count guard pages as free again when their buddy merges With debug_guardpage_minorder set, expand() turns the unused halves of a split into guard pages, which page_del_and_expand() leaves out of the free counts. Freeing the allocated half credits only that half, and __free_one_page() then merges the guard buddy with clear_page_guard(), which no longer touches the counters. The merged block goes on a free list with the guard pages never counted, so NR_FREE_PAGES falls behind the free lists by every guard that merges. Before commit e0932b6c1f94 ("mm: page_alloc: consolidate free page accounting"), __set_page_guard() and __clear_page_guard() adjusted the counts themselves; that commit dropped both adjustments but kept the subtraction implicit in expand(). Count the guard pages under the merged block's migratetype when the buddy is cleared. account_freepages() skips an isolated block as it does for every other free page, and moving the block off the isolated list counts it then. Booting a 16GB VM with debug_pagealloc=on debug_guardpage_minorder=1, check the difference in free pages reported between /proc/vmstat and /proc/buddyinfo. At three points (idle, after a read and file-creation load, after a second load), check the difference between nr_free_pages and the buddyinfo numbers in the normal zone, as a number of pages: idle after load after second load unpatched -130 79205 99121 patched -16 45 0 The small remaining difference seems to be due to the numbers not being read at exactly the same time, and is also seen without guard pages. Link: https://lore.kernel.org/20261016122304.5b1eba8e@fangorn Fixes: e0932b6c1f94 ("mm: page_alloc: consolidate free page accounting") Signed-off-by: Rik van Riel Signed-off-by: Andrew Morton Reviewed-by: Zi Yan Reviewed-by: Johannes Weiner Reviewed-by: Dev Jain Reviewed-by: Vlastimil Babka (SUSE) Assisted-by: LLM Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Brendan Jackman Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: Baoquan He Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Frank van der Linden Cc: --- mm/page_alloc.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f14e2ac21787..6f0ae24bc9c9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -971,10 +971,12 @@ static inline void __free_one_page(struct page *page, * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page, * merge with it and move up one order. */ - if (page_is_guard(buddy)) + if (page_is_guard(buddy)) { clear_page_guard(zone, buddy, order); - else + account_freepages(zone, 1 << order, migratetype); + } else { __del_page_from_free_list(buddy, zone, order, buddy_mt); + } if (unlikely(buddy_mt != migratetype)) { /* From 8f9039840588dd618c72a726c192126bb7b98228 Mon Sep 17 00:00:00 2001 From: Nickolai Zeldovich Date: Fri, 16 Oct 2026 18:27:21 -0400 Subject: [PATCH 191/213] mm/truncate: fix the zeroing end of pagecache_isize_extended() A process that extends a file by 4 GiB or more past a dirty folio, on a file system whose block size is smaller than the page size (ext4 with 1k blocks), reads back the bytes it wrote through a shared mapping beyond the old end of file instead of zeroes, and a later write into that block carries them to the disk. With the example of commit 52aecaee1c26 ("mm: zero range of eof folio exposed by inode size extension") and the truncate taken 4 GiB further (pwrite 2k at 0, mmap the first page and write 'X' over [2k, 4k), truncate(2) to 4 GiB + 2k), the read of [2k, 4k) returns the 2048 'X' bytes, and a pwrite of 16 bytes at 3k followed by fsync() and a remount leaves 1008 'X' bytes on the disk in [3k+16, 4k). No privilege is needed, and every size extension of 4 GiB or more reaches it: truncate(2)/ftruncate(2) through truncate_setsize(), writes beyond the end of file through generic_write_end(), ext4, iomap, btrfs, exfat, ntfs, afs and orangefs. Affected since v6.13. pagecache_isize_extended() zeroes the part of the dirty folio straddling the old end of file that the size extension exposes, from the old size to the new size or the end of the folio, whichever comes first: end = min_t(unsigned int, to - folio_pos(folio), folio_size(folio)); min_t() converts both operands to unsigned int before comparing, so the loff_t difference to - folio_pos(folio) is reduced modulo 2^32 first. When the file grows by 4 GiB or more past the folio, end is that residue instead of folio_size(folio): below offset nothing is zeroed, above it only [offset, end) is. The bytes beyond the old end of file stay in the dirty folio and, now inside i_size, are file data. Compare in loff_t; the result is at most folio_size(folio) and fits end. Found while formally verifying pagecache_isize_extended() against the kernel binary. Link: https://lore.kernel.org/20261016222721.791785-1-nickolai@csail.mit.edu Fixes: 52aecaee1c26 ("mm: zero range of eof folio exposed by inode size extension") Signed-off-by: Nickolai Zeldovich Signed-off-by: Andrew Morton Reviewed-by: Jan Kara Reviewed-by: Baokun Li Assisted-by: LLM Cc: Matthew Wilcox (Oracle) Cc: Brian Foster Cc: Theodore Ts'o Cc: --- mm/truncate.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 5b1b13cf8ff0..733d2713a875 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -919,8 +919,7 @@ void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to) unsigned int offset, end; offset = from - folio_pos(folio); - end = min_t(unsigned int, to - folio_pos(folio), - folio_size(folio)); + end = min_t(loff_t, to - folio_pos(folio), folio_size(folio)); folio_zero_segment(folio, offset, end); } From d2f7bdfdeac8137e942b32e0badc40e181127d7b Mon Sep 17 00:00:00 2001 From: Nickolai Zeldovich Date: Fri, 16 Oct 2026 10:00:43 -0400 Subject: [PATCH 192/213] resource: report a devm_request_resource() conflict under the lock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit devm_request_resource() calls request_resource_conflict(), which takes resource_lock, tries to insert the new resource and returns a pointer to the conflicting resource after dropping the lock. The error message is then formatted from that pointer: conflict->name and the %pR argument read the name, start, end, flags and desc of a resource that nothing pins any more. The owner of the conflicting resource may call release_resource() and free it the moment the lock is dropped. The conflicting resource normally belongs to another device. With two devices whose drivers claim overlapping regions (two PCI host bridges whose "ranges" windows overlap, two instances of a controller on the same aperture, or a stale claim an administrator is replacing), the following interleaving reads freed memory: CPU 0: driver B probe CPU 1: driver A unbind devm_request_resource(devB, root, resB) request_resource_conflict() write_lock(&resource_lock) __request_resource() -> resA write_unlock(&resource_lock) devres_release_all(devA) devm_resource_release() release_resource(resA) put_device() -> kfree(resA) dev_err(..., resA->name, resA) use after free Both halves are reachable from userspace on any kernel: a write to the "unbind" file of the driver that holds the region and a write to the "bind" file of the driver that wants it. One in-tree path is drivers/pci/bus.c devm_request_pci_bus_resources(), called from host bridge probe for every bridge window; the windows are the embedded struct resource of kzalloc'ed resource_entry objects that pci_free_resource_list() frees when the other bridge is released. In the common case the "resource collision" message is garbled; with the slab object poisoned (CONFIG_SLUB_DEBUG) or reused, formatting conflict->name oopses, and KASAN reports a slab-use-after-free read in devm_request_resource(). All of this happens in a probe that is already failing with -EBUSY. The window is a few hundred instructions wide and exists only when two devices actually collide, so the race has not been observed outside of analysis. Do the same here: take resource_lock, insert with __request_resource() and print the diagnostic while the lock still keeps the conflicting resource in the tree, then drop the lock. The return values and the devres handling are unchanged. The change was built for x86_64 and RISC-V; the resource KUnit tests pass on both. Found by formal verification of kernel/resource.c against the kernel binary. Link: https://lore.kernel.org/20261016140043.790048-1-nickolai@csail.mit.edu Fixes: 8d38821cbcf5 ("resources: Add device-managed request/release_resource()") Signed-off-by: Nickolai Zeldovich Signed-off-by: Andrew Morton Acked-by: Tejun Heo Reviewed-by: Bradley Morgan Assisted-by: LLM Cc: Bjorn Helgaas Cc: Ilpo Järvinen Cc: Thierry Reding Cc: --- kernel/resource.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/kernel/resource.c b/kernel/resource.c index e60539a55541..08bd7579cf40 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -1695,10 +1695,15 @@ int devm_request_resource(struct device *dev, struct resource *root, *ptr = new; - conflict = request_resource_conflict(root, new); - if (conflict) { + /* Report a conflict while the lock still keeps it in the tree. */ + write_lock(&resource_lock); + conflict = __request_resource(root, new); + if (conflict) dev_err(dev, "resource collision: %pR conflicts with %s %pR\n", new, conflict->name, conflict); + write_unlock(&resource_lock); + + if (conflict) { devres_free(ptr); return -EBUSY; } From c82dba3183ac66ea282e817b12edf735235fc4ac Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Fri, 16 Oct 2026 21:13:34 +0800 Subject: [PATCH 193/213] mm: warn on unaligned compound zone device ranges A compound page of order N must start at a PFN aligned to 2^N pages. Therefore, a range used to initialize compound pages must start on a compound-page boundary and contain an integral number of compound pages. Two recent Device DAX bugs were caused by ranges that violated this invariant, but neither was detected by memmap_init_zone_device(). Warn if the adjusted zone-device range violates this invariant so future violations are reported when the compound pages are initialized. Link: https://lore.kernel.org/20261016131334.3419393-1-songmuchun@bytedance.com Link: https://lore.kernel.org/r/20261016053944.2579541-1-songmuchun@bytedance.com Link: https://lore.kernel.org/r/20261016054914.2601510-1-songmuchun@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Assisted-by: LLM Cc: Mike Rapoport Cc: Vishal Verma Cc: Dave Jiang Cc: Alison Schofield Cc: Joao Martins Cc: Ritesh Harjani (IBM) --- mm/mm_init.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/mm_init.c b/mm/mm_init.c index 4c17a892924d..01ced0553ce9 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1142,6 +1142,8 @@ void __ref memmap_init_zone_device(struct zone *zone, if (!nr_pages) return; + VM_WARN_ON(!IS_ALIGNED(start_pfn | nr_pages, pfns_per_compound)); + /* * Seed the reusable head-page template from the first real struct * page. The normal page-init and refcount helpers must operate on From 8730e622bfff3b9357ce537eb73f8649207c1ddb Mon Sep 17 00:00:00 2001 From: Qiliang Yuan Date: Fri, 16 Oct 2026 20:54:48 +0800 Subject: [PATCH 194/213] selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers With -cgroup-v2, charge_reserved_hugetlb.sh and hugetlb_reparenting_test.sh mount a cgroup2 hierarchy on a temporary directory when none is mounted and enable the hugetlb controller (hugetlb_reparenting_test.sh also enables memory) in its cgroup.subtree_control. On a system that still binds these controllers to cgroup v1, they are not available in the new hierarchy. Writing to cgroup.subtree_control fails with ENOENT, "set -e" aborts the script, the test is reported as failed, and the temporary cgroup2 mount is left behind. Check cgroup.controllers first, and skip the test when a controller is missing, after unmounting the hierarchy and removing the directory the script created. With hugetlb and memory bound to cgroup v1, per script: before after result fail skip cgroup2 mounts left behind 1 0 Link: https://lore.kernel.org/20261016-bug-selftests-mm-hugetlb-cgroup2-skip-v1-1-7c78883a67b4@gmail.com Fixes: 29750f71a9b4 ("hugetlb_cgroup: add hugetlb_cgroup reservation tests") Signed-off-by: Qiliang Yuan Signed-off-by: Andrew Morton Reviewed-by: Joshua Hahn Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Shuah Khan Cc: Mina Almasry Cc: Muchun Song Cc: Oscar Salvador --- tools/testing/selftests/mm/charge_reserved_hugetlb.sh | 8 ++++++++ tools/testing/selftests/mm/hugetlb_reparenting_test.sh | 10 ++++++++++ 2 files changed, 18 insertions(+) diff --git a/tools/testing/selftests/mm/charge_reserved_hugetlb.sh b/tools/testing/selftests/mm/charge_reserved_hugetlb.sh index a1cfd3a349db..65065b1377b2 100755 --- a/tools/testing/selftests/mm/charge_reserved_hugetlb.sh +++ b/tools/testing/selftests/mm/charge_reserved_hugetlb.sh @@ -39,6 +39,14 @@ if [[ $cgroup2 ]]; then mount -t cgroup2 none $cgroup_path do_umount=1 fi + if ! grep -qw hugetlb $cgroup_path/cgroup.controllers; then + echo "hugetlb controller not available in cgroup v2. Skipping..." + if [[ $do_umount ]]; then + umount $cgroup_path + rmdir $cgroup_path + fi + exit $ksft_skip + fi echo "+hugetlb" >$cgroup_path/cgroup.subtree_control else cgroup_path=$(mount -t cgroup | grep ",hugetlb" | awk '{print $3}') diff --git a/tools/testing/selftests/mm/hugetlb_reparenting_test.sh b/tools/testing/selftests/mm/hugetlb_reparenting_test.sh index 95f517c3bd16..811912ae7d97 100755 --- a/tools/testing/selftests/mm/hugetlb_reparenting_test.sh +++ b/tools/testing/selftests/mm/hugetlb_reparenting_test.sh @@ -29,6 +29,16 @@ if [[ $cgroup2 ]]; then mount -t cgroup2 none $CGROUP_ROOT do_umount=1 fi + for ctrl in hugetlb memory; do + if ! grep -qw $ctrl $CGROUP_ROOT/cgroup.controllers; then + echo "$ctrl controller not available in cgroup v2. Skipping..." + if [[ $do_umount ]]; then + umount $CGROUP_ROOT + rmdir $CGROUP_ROOT + fi + exit $ksft_skip + fi + done echo "+hugetlb +memory" >$CGROUP_ROOT/cgroup.subtree_control else CGROUP_ROOT=$(mount -t cgroup | grep ",hugetlb" | awk '{print $3}') From 1a2993227e8a27fc4cdea305bbdae82d290bc885 Mon Sep 17 00:00:00 2001 From: Kyle Zeng Date: Thu, 15 Oct 2026 18:15:49 -0700 Subject: [PATCH 195/213] mm/page_vma_mapped: continue past nonmatching huge PMDs A PTE-mapped large folio can straddle PMD boundaries after an unaligned mremap(). If the first part is discarded and replaced with an unrelated huge PMD, page_vma_mapped_walk() stops instead of looking for the rest of the folio in subsequent PMDs. This also affects migration entries. A walk that stops at an unrelated huge PMD can leave migration PTEs behind when migration or an anonymous folio split restores the mappings. Those entries do not hold mapping references and can outlive the source folio. This issue can lead to memory corruption and I have a reproducer for it. Skip the current PMD rather than ending the walk on a flag or PFN mismatch. Do the same if the PMD became nonpresent before the locked recheck. Release the PMD lock, clear the per-PMD state and advance with step_forward(), so the existing range limit controls termination. This covers present, migration and device-private huge PMDs while preserving the matching-PMD and split-to-PTE paths. Link: https://lore.kernel.org/20261016011549.61146-1-kylebot@openai.com Fixes: ace71a19cec5 ("mm: introduce page_vma_mapped_walk()") Signed-off-by: Kyle Zeng Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Assisted-by: LLM Cc: Lorenzo Stoakes Cc: Rik van Riel Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Harry Yoo Cc: Jann Horn Cc: Lance Yang Cc: --- mm/page_vma_mapped.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 8408aee7571b..e6b6170c2f4f 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -265,28 +265,28 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) softleaf_t entry; if (!(pvmw->flags & PVMW_MIGRATION)) - return not_found(pvmw); + goto next_pmd; entry = softleaf_from_pmd(pmde); if (!check_pmd(softleaf_to_pfn(entry), pvmw)) - return not_found(pvmw); + goto next_pmd; return true; } else if (pmd_is_device_private_entry(pmde)) { softleaf_t entry; if (pvmw->flags & PVMW_MIGRATION) - return not_found(pvmw); + goto next_pmd; entry = softleaf_from_pmd(pmde); if (!check_pmd(softleaf_to_pfn(entry), pvmw)) - return not_found(pvmw); + goto next_pmd; return true; } else if (!pmd_present(pmde)) { - return not_found(pvmw); + goto next_pmd; } if (likely(pmd_trans_huge(pmde))) { if (pvmw->flags & PVMW_MIGRATION) - return not_found(pvmw); + goto next_pmd; if (!check_pmd(pmd_pfn(pmde), pvmw)) - return not_found(pvmw); + goto next_pmd; return true; } /* THP/device-private pmd was split under us: handle on pte level */ @@ -299,8 +299,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) (pvmw->nr_pages >= HPAGE_PMD_NR)) sync_with_folio_pmd_zap(mm, pvmw->pmd); - step_forward(pvmw, PMD_SIZE); - continue; + goto next_pmd; } if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) @@ -343,6 +342,17 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) pvmw->ptl = ptl; } goto this_pte; +next_pmd: + /* + * A large folio can be PTE-mapped across PMDs, so a + * nonmatching PMD does not end the walk. + */ + if (pvmw->ptl) { + spin_unlock(pvmw->ptl); + pvmw->ptl = NULL; + } + pvmw->pmd = NULL; + step_forward(pvmw, PMD_SIZE); } while (pvmw->address < end); return false; From b428f1966821cd285957fb41a621141b22622a7d Mon Sep 17 00:00:00 2001 From: Nickolai Zeldovich Date: Fri, 16 Oct 2026 23:06:20 -0400 Subject: [PATCH 196/213] mm: bdi: reset min_ratio after the sysfs attributes are gone A write to /sys/class/bdi//min_ratio (or min_ratio_fine or min_bytes) that races with the removal of the device leaves the global sum of minimum dirty shares charged for a bdi that no longer exists. Until the next reboot, that share of the dirty threshold is withheld from every other bdi (__wb_calc_thresh() scales by 100 % minus the sum), and a later min_ratio store on any bdi that would push the sum to 100 % fails with EINVAL although every bdi reads 0. Root privilege is needed for the store; the removal is any del_gendisk() (a disk or loop device going away) or the unmount of a filesystem with its own bdi. On an x86_64 defconfig kernel (v7.3-rc7), 2000 rounds of LOOP_CTL_ADD/LOOP_CTL_REMOVE on one loop device with a second thread writing 50 to its min_ratio leak the share on the first round the store lands in the window; afterwards `echo 99 > min_ratio` on another loop device fails with EINVAL. With the fix the same run keeps the sum at 0 and the store succeeds. Affected since v5.16, when the reset in bdi_unregister() was added. bdi_unregister() gives the bdi's share back with bdi_set_min_ratio(bdi, 0) and only then calls device_unregister(), which removes the sysfs attributes. A min_ratio_store() that gets its kernfs active reference before the removal runs to completion after the reset: __bdi_set_min_ratio() takes bdi_lock, sees bdi->min_ratio == 0, adds the new value to bdi_min_ratio and stores it in the bdi, which bdi_put() then frees. Nothing gives that share back. Move the reset after device_unregister(): once the attributes are gone no store can run, and bdi_set_min_ratio() needs nothing from the device. Found while formally verifying bdi_unregister() against the kernel binary. Link: https://lore.kernel.org/20261017030620.793108-1-nickolai@csail.mit.edu Fixes: 3c376dfafbf7 ("mm: bdi: initialize bdi_min_ratio when bdi is unregistered") Signed-off-by: Nickolai Zeldovich Signed-off-by: Andrew Morton Reviewed-by: Jan Kara Assisted-by: LLM Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Peter Zijlstra Cc: Manjong Lee --- mm/backing-dev.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/backing-dev.c b/mm/backing-dev.c index 18e999053bae..fad1495d3b20 100644 --- a/mm/backing-dev.c +++ b/mm/backing-dev.c @@ -1162,19 +1162,20 @@ void bdi_unregister(struct backing_dev_info *bdi) wb_shutdown(&bdi->wb); cgwb_bdi_unregister(bdi); - /* - * If this BDI's min ratio has been set, use bdi_set_min_ratio() to - * update the global bdi_min_ratio. - */ - if (bdi->min_ratio) - bdi_set_min_ratio(bdi, 0); - if (bdi->dev) { bdi_debug_unregister(bdi); device_unregister(bdi->dev); bdi->dev = NULL; } + /* + * If this BDI's min ratio has been set, use bdi_set_min_ratio() to + * update the global bdi_min_ratio. After device_unregister(): the + * sysfs min_ratio stores can no longer run and set it again. + */ + if (bdi->min_ratio) + bdi_set_min_ratio(bdi, 0); + if (bdi->owner) { put_device(bdi->owner); bdi->owner = NULL; From 1caede5a55b7f0731a437aeb7157ce0f560b049f Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Mon, 19 Oct 2026 14:34:26 +0300 Subject: [PATCH 197/213] MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP Split out files related to SPARSE and SPARSE_VMEMMAP management from MEMORY MANAGEMENT - CORE into a dedicated entry. Also include architecture specific sparsemem.h headers, pfn_to_page() and page_to_pfn() conversions and Documentation/mm/memory-model.rst. Muchun, who did a lot of work cleaning up SPARSE_VMEMMAP and extending vmemmap optimization, volunteered to help with the maintenance. David and I will continue to look after that part of the mm subsystem. Link: https://lore.kernel.org/20261019-memmap-maint-v1-1-80850871d0cc@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Acked-by: Harry Yoo (Meta) Acked-by: Muchun Song Acked-by: SJ Park --- MAINTAINERS | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index 6f1f35a017a6..f93d7ed7a14c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17162,9 +17162,6 @@ F: mm/mmzone.c F: mm/pagewalk.c F: mm/pgtable-generic.c F: mm/ptdump.c -F: mm/sparse-vmemmap.c -F: mm/sparse.c -F: mm/sparse.h F: mm/util.c F: mm/vmpressure.c F: mm/vmstat.c @@ -17212,6 +17209,21 @@ F: include/trace/events/ksm.h F: mm/ksm.c F: mm/mm_slot.h +MEMORY MANAGEMENT - MEMORY MAP +M: Muchun Song +M: Mike Rapoport +M: David Hildenbrand +L: linux-mm@kvack.org +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/mm/core.git memmap +F: Documentation/mm/memory-model.rst +F: arch/*/include/asm/sparsemem.h +F: include/asm-generic/memory_model.h +F: mm/sparse-vmemmap.c +F: mm/sparse.c +F: mm/sparse.h +K: (?i:vmemmap) + MEMORY MANAGEMENT - MGLRU (MULTI-GEN LRU) M: Andrew Morton R: Kairui Song From 34e753f8d927069785f6b675f5769da5da4ef78e Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Mon, 19 Oct 2026 15:28:27 +0100 Subject: [PATCH 198/213] mm/mremap: populate calculated fields in vrm state correctly Commit d5c8aec0542e ("mm/mremap: initial refactor of move_vma()") introduced the vma_remap_struct helper to pass state sanely through mremap() logic. Part of this state is vrm->delta - equal to the absolute difference between vrm->old_len and vrm->new_len (the old and new lengths of the VMA, respectively). This value is used to correctly charge accounted VMAs. When shrinking a VMA, mremap_to() sets vrm->old_len equal to vrm->new_len having performed the shrink of the VMA first, prior to any potential move (e.g. MREMAP_FIXED | MREMAP_MAYMOVE specified). However it does not recalculate vrm->delta, which results in incorrectly overcharging the VMA when both moving and shrinking a VMA. The charge leaks to Committed_AS globally, meaning a mischievous user can cause allocations to fail across the system in strict overcommit mode. Resolve the issue by abstracting the update of vrm->delta to vrm_populate_calculated() and invoking this from mremap_to() when vrm->old_len is changed. Additionally, update vrm->remap_type whose value depends on vrm->delta. The field is not read again after the shrink, but it is the logically correct thing to do and protects against future changes which break this. Link: https://lore.kernel.org/20261019-mremap-fix-shrink-charge-v1-1-a3ce209e5360@kernel.org Fixes: d5c8aec0542e ("mm/mremap: initial refactor of move_vma()") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reported-by: Nickolai Zeldovich Closes: https://lore.kernel.org/all/20261017233527.797191-1-nickolai@csail.mit.edu/ Reviewed-by: Xueyuan Chen Reviewed-by: Pedro Falcato Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: --- mm/mremap.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 6970783fc445..24827e13e617 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -887,12 +887,6 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) return pmc_progress(pmc); } -/* Set vrm->delta to the difference in VMA size specified by user. */ -static void vrm_set_delta(struct vma_remap_struct *vrm) -{ - vrm->delta = abs_diff(vrm->old_len, vrm->new_len); -} - /* Determine what kind of remap this is - shrink, expand or no resize at all. */ static enum mremap_type vrm_remap_type(struct vma_remap_struct *vrm) { @@ -905,6 +899,13 @@ static enum mremap_type vrm_remap_type(struct vma_remap_struct *vrm) return MREMAP_EXPAND; } +/* Populate calculated fields. */ +static void vrm_populate_calculated(struct vma_remap_struct *vrm) +{ + vrm->delta = abs_diff(vrm->old_len, vrm->new_len); + vrm->remap_type = vrm_remap_type(vrm); +} + /* * When moving a VMA to vrm->new_adr, does this result in the new and old VMAs * overlapping? @@ -1523,6 +1524,7 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm) /* Set up for the move now shrink has been executed. */ vrm->old_len = vrm->new_len; + vrm_populate_calculated(vrm); } /* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ @@ -1795,8 +1797,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm) if (vma_is_hugetlb(vma) && !align_hugetlb(vrm)) return -EINVAL; - vrm_set_delta(vrm); - vrm->remap_type = vrm_remap_type(vrm); + vrm_populate_calculated(vrm); /* For convenience, we set new_addr even if VMA won't move. */ if (!vrm_implies_new_addr(vrm)) vrm->new_addr = addr; From 65e722a74df992719676ead85a9e93fb2b5a0983 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Mon, 19 Oct 2026 16:57:48 +0100 Subject: [PATCH 199/213] mm/mremap: correctly uncharge accountable VMAs in error path The mremap error path is a fragile and unwieldy beast - it reverses the page table move then updates vma_remap_struct state to swap VMA, length and address between old and new VMAs in copy_vma_and_data(). If the mremap() operation didn't specify MREMAP_DONTUNMAP then its caller, move_vma(), invokes unmap_source_vma(), which unmaps the new VMA rather than the old one. For accountable VMAs when MREMAP_DONTUNMAP is not specified, unmap_source_vma() unsets VMA_ACCOUNT_BIT during this unmap in order that only the delta of a VMA expansion is charged (performed by vrm_calc_charge() earlier). Therefore nothing uncharges the expanded VMA size, and there is a permanent leak in global Committed_AS. Resolve the issue by uncharging when MREMAP_DONTUNMAP is not specified. Link: https://lore.kernel.org/20261019-other-mremap-charge-leak-v1-1-d94477fe6545@kernel.org Fixes: fdbef6149135 ("mm/mremap: don't account pages in vma_to_resize()") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Brian Geffon Cc: Dmitry Safonov <0x7f454c46@gmail.com> Cc: --- mm/mremap.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/mremap.c b/mm/mremap.c index 24827e13e617..0d06d7f37f19 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -1339,6 +1339,20 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, pmc_revert.need_rmap_locks = true; move_page_tables(&pmc_revert); + /* + * When copy_vma_and_data() returns, the new VMA will be + * unmapped rather than the old one. + * + * When an accountable move is being performed, this unmapping + * will not uncharge any expansion, so do that now. + * + * A move is accountable if the VMA is accountable and + * MREMAP_DONTUNMAP is not specified, so only do so if the + * latter is true. + */ + if (!(vrm->flags & MREMAP_DONTUNMAP)) + vrm_uncharge(vrm); + vrm->vma = new_vma; vrm->old_len = vrm->new_len; vrm->addr = vrm->new_addr; From d0a12d80d2f7e9f4ed2030b6f04c8a897956bf84 Mon Sep 17 00:00:00 2001 From: "Gregory Price (Meta)" Date: Mon, 19 Oct 2026 14:50:02 -0400 Subject: [PATCH 200/213] mm/vmscan: reclaim every zone in per-node proactive reclaim user_proactive_reclaim() limits per-node reclaim to gfp_zone(GFP_KERNEL), which is ZONE_NORMAL. Proactive reclaim serves no allocation, but that limit makes shrink_node() skip every ZONE_MOVABLE folio, so writing to /sys/devices/system/node/nodeN/reclaim never reclaims a node's movable memory. A movable-only node, such as dax/kmem or hotplugged memory onlined movable, reclaims nothing and fails with -EAGAIN. To reproduce, fill a movable-only node with clean page cache and write 200M to its nodeN/reclaim: the write fails with -EAGAIN and the page cache stays. The same write to a ZONE_NORMAL node frees 200MB. Scan every zone, as memory.reclaim does. Link: https://lore.kernel.org/20261019185002.3428594-1-gourry@gourry.net Fixes: b980077899ea ("mm: introduce per-node proactive reclaim interface") Signed-off-by: Gregory Price (Meta) Signed-off-by: Andrew Morton Reviewed-by: Barry Song Reviewed-by: Johannes Weiner Reviewed-by: Harry Yoo (Meta) Acked-by: Shakeel Butt Reviewed-by: Yosry Ahmed Reviewed-by: SJ Park Reviewed-by: Balbir Singh Acked-by: Davidlohr Bueso Acked-by: Michal Hocko Reviewed-by: Xueyuan Chen Acked-by: Lorenzo Stoakes (ARM) Acked-by: Vishal Moola (Fractile) Cc: Kairui Song Cc: Qi Zheng Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Baolin Wang Cc: David Hildenbrand Cc: Michal Hocko Cc: Roman Gushchin Cc: --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index daa7e37c2236..fc823fe3bd1f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -8185,7 +8185,7 @@ int user_proactive_reclaim(char *buf, } else { struct scan_control sc = { .gfp_mask = current_gfp_context(gfp_mask), - .reclaim_idx = gfp_zone(gfp_mask), + .reclaim_idx = MAX_NR_ZONES - 1, .proactive_swappiness = swappiness == -1 ? NULL : &swappiness, .priority = DEF_PRIORITY, .may_writepage = 1, From d4ea410ce4b8aa46a316476ea2624a357bf31fa9 Mon Sep 17 00:00:00 2001 From: "Gregory Price (Meta)" Date: Sat, 17 Oct 2026 22:33:49 -0400 Subject: [PATCH 201/213] mm/mempolicy: apply bind policies against the scoped allocation zone QEMU fails to start a guest with a VFIO device when guest RAM comes from a memory backend bound to a movable-only node: vfio_container_dma_map(0x55b1b1317800, 0x0, 0x10000000, 0x7f39a2200000) = -12 (Cannot allocate memory) VFIO pins guest RAM when the device is realized, and the pin faults the pages in. io_uring fixed buffers fail the same way. Faulting the range in before pinning lets the same pin succeed, since resident pages are migrated off the node instead. A long-term pin faults under memalloc_pin_save(). PF_MEMALLOC_PIN makes current_gfp_context() clear __GFP_MOVABLE, but policy_nodemask() tests apply_policy_zone() against the caller's gfp. The bind keeps its movable-only nodemask, and the allocation finds no usable zone: page allocation failure: order:0, mode:0x140dc2(GFP_HIGHUSER|__GFP_ZERO|__GFP_COMP), nodemask=1,cpuset=/,mems_allowed=0-1 Call Trace: warn_alloc+0x136/0x150 __alloc_frozen_pages_noprof+0xed8/0x1720 alloc_pages_mpol+0xaa/0x170 vma_alloc_folio_noprof+0x65/0xd0 __handle_mm_fault+0x137b/0x1ac0 handle_mm_fault+0xa0/0x270 __get_user_pages+0x250/0xfd0 __gup_longterm_locked+0x116/0x870 gup_fast_fallback+0xcfc/0xdf0 pin_user_pages_fast+0x47/0x60 io_pin_pages+0x9a/0x110 Base page faults fail with -ENOMEM. hugetlb faults fail with -EFAULT. Check the policy zone against current_gfp_context(gfp), so a pin-scoped fault falls back off the movable-only nodemask, like any other non-movable allocation under the same bind. Tested in qemu with a dax/kmem movable-only node 1. io_uring and VFIO pins of unfaulted base pages and 2MB hugetlb pages now succeed and land on node 0 (correct). QEMU now starts with the same backend binding. Link: https://lore.kernel.org/20261018023349.1697434-1-gourry@gourry.net Fixes: 8e3560d963d2 ("mm: honor PF_MEMALLOC_PIN for all movable pages") Signed-off-by: Gregory Price (Meta) Signed-off-by: Andrew Morton Reviewed-by: Joshua Hahn Cc: David Hildenbrand Cc: Zi Yan Cc: Rakie Kim Cc: Ying Huang Cc: Pasha Tatashin Cc: Michal Hocko Cc: --- mm/mempolicy.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 70298fded1b4..0884c629670d 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2302,7 +2302,7 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, break; case MPOL_BIND: /* Restrict to nodemask (but not on lower zones) */ - if (apply_policy_zone(pol, gfp_zone(gfp)) && + if (apply_policy_zone(pol, gfp_zone(current_gfp_context(gfp))) && cpuset_nodemask_valid_mems_allowed(&pol->nodes)) nodemask = &pol->nodes; if (pol->home_node != NUMA_NO_NODE) From 4fb975471717c953f912776bb60301add1ffae11 Mon Sep 17 00:00:00 2001 From: Ngoc Quy Date: Wed, 14 Oct 2026 16:55:40 +0700 Subject: [PATCH 202/213] fs/proc/task_mmu: pin mm_users in do_pagemap_scan() The PAGEMAP_SCAN ioctl takes mmap_read_lock() on an mm obtained from /proc//pagemap without holding a reference to mm_users. proc_mem_open() deliberately grabs only mm_count, so a consumer that touches the address space has to call mmget_not_zero() first. Every other consumer in this file does -- m_start(), do_procmap_query(), show_smaps_rollup() and pagemap_read() -- but do_pagemap_scan() does not. Without that reference the target can reach exit_mmap() concurrently, and exit_mmap() calls unmap_vmas() while holding mmap_lock only for reading, which does not exclude the scanner. For ordinary VMAs this is harmless, as free_pgtables() and remove_vma() run afterwards under mmap_write_lock. A shared hugetlb VMA is different: __hugetlb_zap_end() frees the VMA's hugetlb_vma_lock inside the read-locked phase. walk_hugetlb_range() can therefore load vma->vm_private_data, have the lock freed underneath it, and then down_read() a freed rw_semaphore. The load happens before the freeing side clears the pointer, so this is a use-after-free and not a NULL dereference. Any unprivileged user can reach it with no huge pages configured, as hugetlb_reserve_pages() allocates the lock before the VM_NORESERVE early return. Pin mm_users across the scan, the way the other consumers do. mmput() from the exiting task can then no longer reach exit_mmap() while a walk is in progress. [akpm@linux-foundation.org: remove unneeded & undesirable test for !mm] Link: https://lore.kernel.org/20261014095540.3261469-1-ngocquy0700@gmail.com Fixes: 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs") Signed-off-by: Ngoc Quy Signed-off-by: Andrew Morton Assisted-by: Claude:claude-opus-5 Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Muhammad Usama Anjum Cc: --- fs/proc/task_mmu.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 44147ba0b899..7452b695fdae 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -3120,15 +3120,18 @@ static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) size_t n_ranges_out = 0; int ret; + if (!mmget_not_zero(mm)) + return -ESRCH; + ret = pagemap_scan_get_args(&p.arg, uarg); if (ret) - return ret; + goto out_mmput; p.masks_of_interest = p.arg.category_mask | p.arg.category_anyof_mask | p.arg.return_mask; ret = pagemap_scan_init_bounce_buffer(&p); if (ret) - return ret; + goto out_mmput; for (walk_start = p.arg.start; walk_start < p.arg.end; walk_start = p.arg.walk_end) { @@ -3183,6 +3186,8 @@ static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) ret = -EFAULT; kfree(p.vec_buf); +out_mmput: + mmput(mm); return ret; } From bbeec2d8324d7aac438bb12dd2d22b184e0a8d43 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Mon, 19 Oct 2026 17:41:18 -0700 Subject: [PATCH 203/213] mm/damon/core: filter out damos address filter-split failed region When an address type DAMOS filter is configured, DAMOS tries to split the region for the filter range, and apply DAMOS action to the filter-in region. The split operation (damon_split_region_at()) could fail from allocation failure under extreme memory pressure. In the case, DAMOS action could be applied more or less than expected, ignoring the address filter. The issue only rarely happens under extreme memory pressure. No crash or leak happens. Occasionally not applying the action to a few regions shouldn't be a real problem. But applying the action to a region that the user has excluded via address filter could be a surprise depending on the use case. It is only theoretical. No such real case is found yet. Filter out the entire region if the split is failed. Update damos_filter_match() signature to receive the error from split. Also update kunit tests for the updated signature. The issue was discovered [1] by sashiko. Link: https://lore.kernel.org/20261020004119.106690-1-sj@kernel.org Link: https://lore.kernel.org/20260801203121.117449-1-sj@kernel.org [1] Fixes: ab9bda001b68 ("mm/damon/core: introduce address range type damos filter") Signed-off-by: SJ Park Signed-off-by: Andrew Morton Cc: # 6.6.x --- mm/damon/core.c | 16 ++++++++++++---- mm/damon/tests/core-kunit.h | 11 ++++++----- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/mm/damon/core.c b/mm/damon/core.c index b63e60ef8990..479169dc9ac7 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -2551,13 +2551,14 @@ static void damos_update_stat(struct damos *s, static bool damos_filter_match(struct damon_ctx *ctx, struct damon_target *t, struct damon_region *r, struct damos_filter *filter, - unsigned long min_region_sz) + unsigned long min_region_sz, int *err) { bool matched = false; struct damon_target *ti; int target_idx = 0; unsigned long start, end, wsum; + *err = 0; switch (filter->type) { case DAMOS_FILTER_TYPE_TARGET: damon_for_each_target(ti, ctx) { @@ -2583,12 +2584,13 @@ static bool damos_filter_match(struct damon_ctx *ctx, struct damon_target *t, } /* start before the range and overlap */ if (r->ar.start < start) { - damon_split_region_at(t, r, start - r->ar.start); + *err = damon_split_region_at(t, r, + start - r->ar.start); matched = false; break; } /* start inside the range */ - damon_split_region_at(t, r, end - r->ar.start); + *err = damon_split_region_at(t, r, end - r->ar.start); matched = true; break; case DAMOS_FILTER_TYPE_PROBE_HITS_WSUM: @@ -2607,10 +2609,16 @@ static bool damos_core_filter_out(struct damon_ctx *ctx, struct damon_target *t, struct damon_region *r, struct damos *s) { struct damos_filter *filter; + bool matching; + int err; s->core_filters_allowed = false; damos_for_each_core_filter(filter, s) { - if (damos_filter_match(ctx, t, r, filter, ctx->min_region_sz)) { + matching = damos_filter_match(ctx, t, r, filter, + ctx->min_region_sz, &err); + if (err) + return true; + if (matching) { if (filter->allow) s->core_filters_allowed = true; return !filter->allow; diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index 3fbb4e4e36fa..43e2f950f8ff 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -1783,6 +1783,7 @@ static void damos_test_filter_out(struct kunit *test) struct damon_target *t; struct damon_region *r, *r2; struct damos_filter *f; + int err; f = damos_new_filter(DAMOS_FILTER_TYPE_ADDR, true, false); if (!f) @@ -1803,27 +1804,27 @@ static void damos_test_filter_out(struct kunit *test) damon_add_region(r, t); /* region in the range */ - KUNIT_EXPECT_TRUE(test, damos_filter_match(NULL, t, r, f, 1)); + KUNIT_EXPECT_TRUE(test, damos_filter_match(NULL, t, r, f, 1, &err)); KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 1); /* region before the range */ r->ar.start = 1; r->ar.end = 2; KUNIT_EXPECT_FALSE(test, - damos_filter_match(NULL, t, r, f, 1)); + damos_filter_match(NULL, t, r, f, 1, &err)); KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 1); /* region after the range */ r->ar.start = 6; r->ar.end = 8; KUNIT_EXPECT_FALSE(test, - damos_filter_match(NULL, t, r, f, 1)); + damos_filter_match(NULL, t, r, f, 1, &err)); KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 1); /* region started before the range */ r->ar.start = 1; r->ar.end = 4; - KUNIT_EXPECT_FALSE(test, damos_filter_match(NULL, t, r, f, 1)); + KUNIT_EXPECT_FALSE(test, damos_filter_match(NULL, t, r, f, 1, &err)); /* filter should have split the region */ KUNIT_EXPECT_EQ(test, r->ar.start, 1); KUNIT_EXPECT_EQ(test, r->ar.end, 2); @@ -1839,7 +1840,7 @@ static void damos_test_filter_out(struct kunit *test) r->ar.start = 2; r->ar.end = 8; KUNIT_EXPECT_TRUE(test, - damos_filter_match(NULL, t, r, f, 1)); + damos_filter_match(NULL, t, r, f, 1, &err)); /* filter should have split the region */ KUNIT_EXPECT_EQ(test, r->ar.start, 2); KUNIT_EXPECT_EQ(test, r->ar.end, 6); From bdda2887cbf3937c2d0f56589c2124c4167324e1 Mon Sep 17 00:00:00 2001 From: Yunhui Cui Date: Sat, 23 May 2026 12:20:52 +0800 Subject: [PATCH 204/213] riscv: mm: exclude invalid THP PMDs from page table check RISC-V THP splitting uses a temporary invalid PMD state where pmd_mkinvalid() clears _PAGE_PRESENT and _PAGE_PROT_NONE but leaves _PAGE_LEAF set so the MM code can still recognize the PMD as a THP split in-progress entry. That temporary state no longer describes a user-accessible mapping, but page_table_check currently treats it as one because the RISC-V PMD user-accessibility test only checks whether the PMD is a leaf and has user permissions. As a result, when a PMD-sized anonymous THP is split during a COW fault, page_table_check can account the invalid intermediate PMD as a live PMD mapping, and then account the replacement PTE mappings again when the split installs the PTE table. This leaves stale PMD accounting behind and later triggers page_table_check failures such as a non-zero anon_map_count when the folio is freed. Fix this by tightening pmd_user_accessible_page() so PMD page-table-check accounting only considers leaf PMDs that still carry either _PAGE_PRESENT or _PAGE_PROT_NONE. This preserves the THP split semantics required by the MM code while preventing page_table_check from treating invalid split PMDs as live user mappings. With CONFIG_PAGE_TABLE_CHECK=y and CONFIG_PAGE_TABLE_CHECK_ENFORCED=y, tools/testing/selftests/mm/cow completes successfully on RISC-V after this change. Link: https://lore.kernel.org/20260523042052.35476-1-cuiyunhui@bytedance.com Fixes: 3fee229a8eb9 ("riscv/mm: enable ARCH_SUPPORTS_PAGE_TABLE_CHECK") Signed-off-by: Yunhui Cui Signed-off-by: Andrew Morton Cc: Albert Ou Cc: Palmer Dabbelt Cc: Pasha Tatashin Cc: Paul Walmsley Cc: tongtiangen Cc: Alexandre Ghiti Cc: David Hildenbrand (Arm) Cc: Vivian Wang Cc: Yaxing Guo Cc: --- arch/riscv/include/asm/pgtable.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 4c8fc6845503..d48f90140841 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -989,7 +989,14 @@ static inline bool pte_user_accessible_page(struct mm_struct *mm, unsigned long static inline bool pmd_user_accessible_page(struct mm_struct *mm, unsigned long addr, pmd_t pmd) { - return pmd_leaf(pmd) && pmd_user(pmd); + /* + * page_table_check() must ignore THP split invalidation entries created by + * pmd_mkinvalid(). These retain _PAGE_LEAF so pmd_present()/pmd_leaf() stay + * true during the split, but they no longer describe a user-accessible + * mapping once both _PAGE_PRESENT and _PAGE_PROT_NONE are cleared. + */ + return (pmd_val(pmd) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) && + (pmd_val(pmd) & _PAGE_LEAF) && pmd_user(pmd); } static inline bool pud_user_accessible_page(struct mm_struct *mm, unsigned long addr, pud_t pud) From faaf8b406c5178fc54662c91e7ce687c0c4cb4be Mon Sep 17 00:00:00 2001 From: Konstantin Khorenko Date: Mon, 19 Oct 2026 18:11:44 +0200 Subject: [PATCH 205/213] mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache shrinker_info_unit holds BITS_PER_LONG atomic_long_t of deferred work plus a one-word bitmap, 520 bytes on 64-bit. kzalloc_node() takes it from kmalloc-1k, so almost half of every unit is padding. Allocate the units from a cache of their own, sized to the unit. The cache is not cache-line aligned: the counters of a unit already share cache lines between shrinkers, and aligning the unit would only keep neighbouring units off each other's boundary line, at the cost of part of the saving. Link: https://lore.kernel.org/20261019-vstor-144715-shrinker_info_unit-kmem_cache-v3-1-0fb061e09bf4@virtuozzo.com Signed-off-by: Konstantin Khorenko Signed-off-by: Andrew Morton Suggested-by: Andrew Morton Reviewed-by: Harry Yoo (Meta) Reviewed-by: Qi Zheng Acked-by: Muchun Song Assisted-by: LLM Cc: Dave Chinner Cc: Roman Gushchin Cc: Johannes Weiner Cc: Michal Hocko Cc: Shakeel Butt --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 2 ++ mm/shrinker.c | 11 +++++++++-- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 74110a324f9e..69e1bcc67c58 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1717,6 +1717,7 @@ static inline u64 mem_cgroup_get_socket_pressure(const struct mem_cgroup *memcg) } #endif +void shrinker_unit_cache_init(void); int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index aad0498a7bd6..e7d7db5718f9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5984,6 +5984,8 @@ int __init mem_cgroup_init(void) memcg_pn_cachep = KMEM_CACHE(mem_cgroup_per_node, SLAB_PANIC | SLAB_HWCACHE_ALIGN); + shrinker_unit_cache_init(); + return 0; } diff --git a/mm/shrinker.c b/mm/shrinker.c index 7ec2a9704f6f..4fb424e721bc 100644 --- a/mm/shrinker.c +++ b/mm/shrinker.c @@ -12,6 +12,12 @@ DEFINE_MUTEX(shrinker_mutex); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +static struct kmem_cache *shrinker_unit_cachep; + +void __init shrinker_unit_cache_init(void) +{ + shrinker_unit_cachep = KMEM_CACHE(shrinker_info_unit, SLAB_PANIC); +} static inline int shrinker_unit_size(int nr_items) { @@ -33,7 +39,7 @@ static inline void shrinker_unit_free(struct shrinker_info *info, int start) if (!unit[i]) break; - kfree(unit[i]); + kmem_cache_free(shrinker_unit_cachep, unit[i]); unit[i] = NULL; } } @@ -47,7 +53,8 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new, int i; for (i = start; i < nr; i++) { - unit = kzalloc_node(sizeof(*unit), GFP_KERNEL, nid); + unit = kmem_cache_alloc_node(shrinker_unit_cachep, + GFP_KERNEL | __GFP_ZERO, nid); if (!unit) { shrinker_unit_free(new, start); return -ENOMEM; From 5695abc4e742b947f586bfb6a080a80ad9957e5d Mon Sep 17 00:00:00 2001 From: SJ Park Date: Mon, 19 Oct 2026 19:39:56 -0700 Subject: [PATCH 206/213] mm/damon/core: remove unreachable damos charge_{target,addr}_from damos_quota->charge_{target,addr}_from saves the target and region the scheme's quota limit is reached. damos_skip_charged_region() skips the target and regions before the point. The target that was assigned to charge_target_from could be dynamically removed from the context by the runtime parameters update (aka commit) feature. In this case, damos_skip_charged_region() cannot find if the point is passed or not, so the scheme is effectively disabled. Similarly, the user could also remove all the regions of the target using the runtime parameters update feature. It would be uncommon to do that, but it is a supported feature. Because the skip pointer detection logic (damos_skip_charged_region()) works for each region of targets, this result in the logic is entirely skipped and the scheme is effectively disabled again. The user impact is that the DAMOS scheme could suddenly stop working under certain uses of the runtime parameters update feature. It could surprise users. No crash or leak happens though. Users could also simply work around by re-installing the scheme. Catch the corner cases in damos_adjust_quota() and reset the pointers. Link: https://lore.kernel.org/20261020023958.108276-1-sj@kernel.org Fixes: da87878010e5 ("mm/damon/sysfs: support online inputs update") Signed-off-by: SJ Park Signed-off-by: Andrew Morton Closes: https://sashiko.dev/#/bug/linux-cd0c55fc-0cc8-45d8-9637-3334811c49b8 Cc: # 5.19.x --- mm/damon/core.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/damon/core.c b/mm/damon/core.c index 479169dc9ac7..eb1fcdabc007 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -3383,6 +3383,26 @@ static void damos_trace_esz(struct damon_ctx *c, struct damos *s, trace_damos_esz(cidx, sidx, quota->esz); } +static void damos_fix_stale_charge_pointers(struct damon_ctx *c, + struct damos *s) +{ + struct damos_quota *quota; + struct damon_target *t; + + quota = &s->quota; + if (!quota->charge_target_from) + return; + damon_for_each_target(t, c) { + if (t == quota->charge_target_from) { + if (damon_nr_regions(t) > 0) + return; + break; + } + } + quota->charge_target_from = NULL; + quota->charge_addr_from = 0; +} + static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) { struct damos_quota *quota = &s->quota; @@ -3418,6 +3438,8 @@ static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) damos_trace_esz(c, s, quota); } + damos_fix_stale_charge_pointers(c, s); + if (damos_quota_is_full(quota, c->min_region_sz)) return; if (!c->ops.get_scheme_score) From c9ed13789061af00d9239f9fab8e246adbed2076 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Google)" Date: Mon, 19 Oct 2026 23:31:28 -0400 Subject: [PATCH 207/213] tools/testing/shared: catch a failed posix_memalign() posix_memalign() returns zero or a positive error number and never sets errno, so the checks for a negative return in kmem_cache_alloc_lru() and kmem_cache_alloc_bulk() is not possible. A failed aligned allocation carries on with the pointer unset, into the constructor or the memset(). Test for any non-zero return. Link: https://lore.kernel.org/20261020033128.1934349-1-liam@infradead.org Fixes: 8801c35c3672 ("tools: fix -Wunused-result in linux.c") Signed-off-by: Liam R. Howlett (Google) Signed-off-by: Andrew Morton Reviewed-by: Andrew Morton Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Vishal Moola (Fractile) Reviewed-by: SJ Park Cc: Matthew Wilcox (Oracle) Cc: Vlastimil Babka --- tools/testing/shared/linux.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c index e0a0693df08f..a37d2f86d325 100644 --- a/tools/testing/shared/linux.c +++ b/tools/testing/shared/linux.c @@ -83,7 +83,7 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, } else { pthread_mutex_unlock(&cachep->lock); if (cachep->align) { - if (posix_memalign(&p, cachep->align, cachep->size) < 0) + if (posix_memalign(&p, cachep->align, cachep->size)) return NULL; } else { p = malloc(cachep->size); @@ -191,7 +191,7 @@ bool kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size, if (cachep->align) { if (posix_memalign(&p[i], cachep->align, - cachep->size) < 0) + cachep->size)) break; } else { p[i] = malloc(cachep->size); From 68cb098eadda190783f8f8bdf9fe764df59f7d32 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 20 Oct 2026 15:55:30 +0100 Subject: [PATCH 208/213] mm/vma: avoid broken uffd merge on mremap Userfaultfd often causes unexpected interactions within mm, and does so at the intersection of uffd, mremap() and VMA merging. When an mremap of a VMA occurs, copy_vma_and_data() calls mremap_userfaultfd_prep() to notify uffd of this. However, mremap_userfaultfd_prep() simply drops the UFFD context for the VMA if the user did not register it for remap, i.e. didn't specify the UFFD_FEATURE_EVENT_REMAP flag. This is OK for the VMA itself, as upon page table move, move_ptes() clears UFFD page table flags, using vma_has_uffd_without_event_remap() to determine whether to do so. However, this doesn't account for merging with adjacent VMAs, whose page tables won't have received this treatment, and thus latent UFFD markers will be left around. Worse, it'll break UFFD for any VMAs it is merged with. This is very surprising behaviour for a user, who will then simply not receive UFFD events for VMAs they ought to. Resolve the issue by explicitly checking for this case via vma_has_uffd_without_event_remap() in vma_merge_copied_range(), the merge function explicitly designated for mremap() operations. Also update the userland VMA tests accordingly. Link: https://lore.kernel.org/20261020-uffd-remap-fix-v1-1-0f34fc181f0a@kernel.org Fixes: 3cfd22be0ad6 ("userfaultfd: clear flag if remap event not enabled") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Andrew Morton Reported-by: shaos Closes: https://lore.kernel.org/all/20261020110748.959870-1-shawnq1@163.com/ Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: Gregory Price (Meta) Reviewed-by: Xueyuan Chen Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Jann Horn Cc: Pedro Falcato Cc: Mike Rapoport Cc: Peter Xu Cc: David Hildenbrand Cc: --- mm/vma.c | 5 +++++ tools/testing/vma/include/stubs.h | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/mm/vma.c b/mm/vma.c index de5ebc57e7a5..1be613044b4a 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -1211,6 +1211,11 @@ static struct vm_area_struct *vma_merge_copied_range(struct vma_merge_struct *vm vmg->copied_from = vmg->middle; vmg->middle = NULL; + + /* If not registered for remap, uffd context will be cleared. */ + if (vma_has_uffd_without_event_remap(vmg->copied_from)) + return NULL; + return vma_merge_new_range(vmg); } diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h index be8f5161f92a..f786a585caf0 100644 --- a/tools/testing/vma/include/stubs.h +++ b/tools/testing/vma/include/stubs.h @@ -386,6 +386,11 @@ static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, return true; } +static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma) +{ + return false; +} + static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, struct anon_vma_name *anon_name2) { From baed735a956f2a7badb5436bef6810303034f920 Mon Sep 17 00:00:00 2001 From: Dimitris Charisis Date: Mon, 5 Oct 2026 12:47:27 +0000 Subject: [PATCH 209/213] maple_tree: fix maple_range_64 crashes in RCU mode Patch series "maple_tree: fix maple_range_64 crashes in RCU mode", v3. The patch series fixes two RCU destroy path issues on maple_range_64 nodes. Both arising from the fact that a maple_range_64 node repurposes the last slot to hold metadata when the node is not full. Patch 1 addresses the issues, and patch 2 adds a test used to catch them. Currently there is no in-tree user of maple_range_64 nodes in RCU mode. All in-tree users of MT_FLAGS_USE_RCU also set MT_FLAGS_ALLOC_RANGE which makes the internal nodes maple_arange_64. Below is the minimal reproducer used during debugging: static DEFINE_MTREE(tree); static int __init maple_fix_init(void) { unsigned long i, j; for (i = 10; i < 4000; i++) { mt_init_flags(&tree, MT_FLAGS_USE_RCU); for (j = 0; j < i; j++) { mtree_insert_range(&tree, j * 10, j * 10 + 9, xa_mk_value(j), GFP_KERNEL); } mtree_destroy(&tree); rcu_barrier(); } return 0; } The splat for the first issue addressed by the first patch is shown below. mt_clear_meta() runs after the slots have been rewritten with raw pointers by mte_dead_leaves(), so it cannot recognize a valid last pointer because the type information is stripped. On my 64-bit LE machine, it clears bytes 248 and 249 of the maple_range_64 which are the two least significant bytes of the last pointer. The lowest byte is already zero, since nodes are 256-byte aligned, but zeroing byte 249 corrupts the pointer. KASAN catches it as a double-free, because that corrupted pointer happens to land on a node that was freed previously. The splat appears at 226 insertions in the reproducer. This is when the tree has a full root node on a 64-bit build. ================================================================== BUG: KASAN: double-free in mt_free_walk+0x138/0x3a0 Free of addr ffff0000f3d40000 by task swapper/2/0 [...] Call trace: [...] kmem_cache_free_bulk+0x5b8/0xc18 mt_free_walk+0x138/0x3a0 rcu_core+0x5d0/0x1638 rcu_core_si+0x18/0x30 [...] Allocated by task 1218: [...] kmem_cache_alloc_from_sheaf_noprof+0xa8/0x2f0 dst_setup+0x210/0x630 mas_wr_split+0x568/0x26e0 mas_wr_store_entry+0x98c/0x1de0 mas_insert.isra.0+0x3d0/0x5d8 mtree_insert_range+0xec/0x1c8 maple_fix_init+0x94/0xff8 [maple_fix_obj] [...] Freed by task 0: [...] __kasan_slab_free+0x88/0xb8 __rcu_free_sheaf_prepare+0x94/0x360 rcu_free_sheaf+0x30/0x138 rcu_core+0x5d0/0x1638 rcu_core_si+0x18/0x30 [...] The buggy address belongs to the object at ffff0000f3d40000 which belongs to the cache maple_node of size 256 The buggy address is located 0 bytes inside of 256-byte region [ffff0000f3d40000, ffff0000f3d40100) [...] Memory state around the buggy address: ffff0000f3d3ff00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ffff0000f3d3ff80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc >ffff0000f3d40000: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff0000f3d40080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff0000f3d40100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ================================================================== The second issue fixed in the first patch is in the mt_free_walk(). mt_free_walk() descends by checking if the corresponding slot[offset] entry is non-zero. In a maple_range_64 node that has MAPLE_RANGE64_SLOTS-1 valid pointers to child nodes, all slots will be non-zero, since the last slot holds the metadata. Thus, mte_dead_walk() dereferences the metadata as if it were a node and hits the null-ptr-deref as shown in the splat below. After fixing the first issue, the reproducer reaches this case at 3166 insertions. The affected node is the root on a tree with height 3 and it has 15 valid child nodes on a 64-bit machine. The first fix does not introduce this bug, but it exposes it for the root node because it no longer clears the root metadata. On an unmodified kernel, the same bug exists and is reachable when such a maple_range_64 node appears below the root, where mt_clear_meta() is not called. Second KASAN splat: Unable to handle kernel paging request at virtual address dfff800000000006 KASAN: null-ptr-deref in range [0x0000000000000030-0x0000000000000037] [...] Call trace: mte_dead_walk+0x78/0x140 (P) mt_free_walk+0x1f8/0x3a0 rcu_core+0x5d0/0x1638 rcu_core_si+0x18/0x30 [..] Both splats are from an arm64 QEMU guest. The test_maple_tree and the userspace suite pass after each patch in the series. This patch (of 2): This commit fixes two issues in maple_range_64 nodes under RCU mode. A maple_range_64 node repurposes its last slot to hold metadata whenever the node is not full. Both issues arise from mishandling that last slot. More specifically the two fixes are: 1. Remove mt_clear_meta() function. mt_clear_meta() decides whether the last slot of a maple_range_64 node holds a child pointer or a struct maple_metadata with the check if (unlikely((mte_to_node(next) && mte_node_type(next)))) return; /* no metadata, could be node */ The check expects the pointer to be encoded. But the only callsite of mt_clear_meta() is mt_destroy_walk() on the RCU destroy path, and by the time it runs, mte_dead_leaves() has overwritten every slot on a full node with a raw pointer, stripping the type information. Thus, the check above never returns early for a full node as it should. It falls through and then: meta->gap = 0; meta->end = 0; zeroes two bytes of a valid child pointer. Later, mt_free_walk() dereferences the corrupted pointer. Fix this by removing mt_clear_meta() along with its only callsite. mt_clear_meta() is only called for the root of each sub-tree destroyed under RCU. Descendant nodes retain their metadata until they are freed. RCU readers may use the metadata while traversing a node, but do not use cleared metadata to detect that a node has been removed. They detect a dead node via ma_dead_node(). 2. Fix invalid memory access in mt_free_walk(). mt_free_walk() descends to the left-most unvisited "parent-of-a-leaf" node by checking the condition: if ((offset < mt_slots[type]) && rcu_dereference_protected(slots[offset], lock_is_held(&rcu_callback_map))) slots = mte_dead_walk(&enode, offset); On a maple_range_64 node with MAPLE_RANGE64_SLOTS-1 children *all* slots are non-NULL. The first MAPLE_RANGE64_SLOTS-1 hold valid pointers to child nodes, and the last slot contains metadata. The above check therefore passes for all offsets, and mte_dead_walk() dereferences the metadata as if it were a node. To trigger this, a node at least two levels above the leaves has to have exactly MAPLE_RANGE64_SLOTS-1 valid pointers to other nodes. maple_arange_64 nodes cannot hit this since they store the metadata in a separate field. Fix this by bounding the descent with slot_len which holds the number of children of a dead node. Link: https://lore.kernel.org/20261005-fix-maple-tree-range64-rcu-v3-0-ebe7fb278e20@cslab.ece.ntua.gr Link: https://lore.kernel.org/20261005-fix-maple-tree-range64-rcu-v3-1-ebe7fb278e20@cslab.ece.ntua.gr Fixes: 54a611b60590 ("Maple Tree: add new data structure") Fixes: 2e5b4921f8ef ("maple_tree: fix freeing of nodes in rcu mode") Signed-off-by: Dimitris Charisis Signed-off-by: Andrew Morton Reviewed-by: Liam R. Howlett (Google) Reviewed-by: Florian Schmaus Cc: Alice Ryhl Cc: Andrew Ballance Cc: Suren Baghdasaryan Cc: Matthew Wilcox (Oracle) --- lib/maple_tree.c | 43 +------------------------------------------ 1 file changed, 1 insertion(+), 42 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 1aba6cced713..0b0036c9f848 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -763,43 +763,6 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } -/* - * mt_clear_meta() - clear the metadata information of a node, if it exists - * @mt: The maple tree - * @mn: The maple node - * @type: The maple node type - */ -static inline void mt_clear_meta(struct maple_tree *mt, struct maple_node *mn, - enum maple_type type) -{ - struct maple_metadata *meta; - unsigned long *pivots; - void __rcu **slots; - void *next; - - switch (type) { - case maple_range_64: - pivots = mn->mr64.pivot; - if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { - slots = mn->mr64.slot; - next = mt_slot_locked(mt, slots, - MAPLE_RANGE64_SLOTS - 1); - if (unlikely((mte_to_node(next) && - mte_node_type(next)))) - return; /* no metadata, could be node */ - } - fallthrough; - case maple_arange_64: - meta = ma_meta(mn, type); - break; - default: - return; - } - - meta->gap = 0; - meta->end = 0; -} - /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -4789,9 +4752,7 @@ static void mt_free_walk(struct rcu_head *head) type = mte_node_type(enode); slots = ma_slots(mte_to_node(enode), type); - if ((offset < mt_slots[type]) && - rcu_dereference_protected(slots[offset], - lock_is_held(&rcu_callback_map))) + if (offset < mte_to_node(enode)->slot_len) slots = mte_dead_walk(&enode, offset); node = mte_to_node(enode); } while ((node != start) || (node->slot_len < offset)); @@ -4885,8 +4846,6 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt, free_leaf: if (free) kfree(node); - else - mt_clear_meta(mt, node, node->type); } /* From 0fd271230891ad741b8a2f9ea814d584032afac9 Mon Sep 17 00:00:00 2001 From: Dimitris Charisis Date: Mon, 5 Oct 2026 12:47:28 +0000 Subject: [PATCH 210/213] test_maple_tree: test maple_range_64 node in RCU mode This commit adds a test that builds two maple_range_64 trees in RCU mode. The first tree has a root that is a full maple_range_64 node and ensures that the destruction under RCU does not misinterpret the last slot as metadata. 226 insertions produce such a root on a 64-bit build. The second tree is a height 3 tree whose root holds 15 child pointers and metadata in its last slot, so every slot is non-zero and ensures that the destruction under RCU does not descend into the metadata slot. 3166 insertions produce such a tree on a 64-bit build. Link: https://lore.kernel.org/20261005-fix-maple-tree-range64-rcu-v3-2-ebe7fb278e20@cslab.ece.ntua.gr Signed-off-by: Dimitris Charisis Signed-off-by: Andrew Morton Reviewed-by: Liam R. Howlett (Google) Cc: Alice Ryhl Cc: Andrew Ballance Cc: Suren Baghdasaryan Cc: Matthew Wilcox (Oracle) Cc: Florian Schmaus --- lib/test_maple_tree.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/lib/test_maple_tree.c b/lib/test_maple_tree.c index b9367c61e8b5..fb40e193a3cb 100644 --- a/lib/test_maple_tree.c +++ b/lib/test_maple_tree.c @@ -3710,6 +3710,36 @@ static noinline void __init alloc_cyclic_testing(struct maple_tree *mt) MT_BUG_ON(mt, ret != 1); } +static noinline void __init check_range64_in_rcu(struct maple_tree *mt) +{ + unsigned long i; + unsigned long nr_entries = 226; /* Build a full maple_range_64 root node */ + + MT_BUG_ON(mt, !mtree_empty(mt)); + mt_init_flags(mt, MT_FLAGS_USE_RCU); + + for (i = 0; i < nr_entries; i++) { + MT_BUG_ON(mt, mtree_test_insert_range(mt, i*10, i*10 + 9, + xa_mk_value(i))); + } + + mtree_destroy(mt); + rcu_barrier(); + + nr_entries = 3166; /* Height 3. Root has 15 children + metadata */ + + MT_BUG_ON(mt, !mtree_empty(mt)); + mt_init_flags(mt, MT_FLAGS_USE_RCU); + + for (i = 0; i < nr_entries; i++) { + MT_BUG_ON(mt, mtree_test_insert_range(mt, i*10, i*10 + 9, + xa_mk_value(i))); + } + + mtree_destroy(mt); + rcu_barrier(); +} + static DEFINE_MTREE(tree); static int __init maple_tree_seed(void) { @@ -3999,6 +4029,8 @@ static int __init maple_tree_seed(void) alloc_cyclic_testing(&tree); mtree_destroy(&tree); + check_range64_in_rcu(&tree); + #if defined(BENCH) skip: From 30a397468e10076969c8df128d09986d718d50b4 Mon Sep 17 00:00:00 2001 From: Joshua Hahn Date: Tue, 20 Oct 2026 18:00:50 -0700 Subject: [PATCH 211/213] mm/memcontrol: publish null pointer before dropping css reference struct memcg_stock_pcp hold references to cgroups that it tracks to make sure that the memcg never gets freed with unaccounted charges still living in the stock. When the cached stock goes to 0 for a memcg, its reference is dropped and the cached memcg pointer is set to NULL. If that was the last reference, the css_put would free the associated cgroup before publishing the NULL pointer, which can be racy on PREEMPT_RT. To summarize Shakeel Butt's analysis in [2], CPU 1 CPU 2 local_trylock() takes a per-CPU spinlock, which is an rcu read lock along with migrate_disable drain_stock css_put drops the last reference release work is queued kworker runs css_release_work_fn() call_rcu() starts grace period G which waits for the refill task. drain_all_stock is_memcg_drain_needed() enters rcu_read_lock(), but grace period G does not wait since it started after G did stock->cached[i] is cleared and unlocks. Grace period G ends, old is freed mem_cgroup_is_descendant() reads old->css.cgroup which has just been freed. To close this gap on PREEMPT_RT, switch the order of the referene drop and the publishing so that we publish the NULL pointer before dropping the reference. Note that this can only happen from drain_stock who can hold the last reference, but not on consume_stock since another reference will always be held. But fix this in both sites to maintain consistency. This is based on a report from Sashiko, who found this pre-existing issue in my page_counter_stock series [1]. Link: https://lore.kernel.org/20261021010051.1529261-1-joshua.hahnjy@gmail.com Link: https://sashiko.dev/#/patchset/20260928192349.3432886-1-joshua.hahnjy%40gmail.com [1] Link: https://lore.kernel.org/all/atfMngq7nyZpt4u-@linux.dev/ [2] Fixes: e1a366be5cb4 ("mm: memcontrol: switch to rcu protection in drain_all_stock()") Signed-off-by: Joshua Hahn Signed-off-by: Andrew Morton Acked-by: Shakeel Butt Acked-by: Muchun Song Cc: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin --- mm/memcontrol.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e7d7db5718f9..a05d9a8baa1b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2172,8 +2172,8 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) stock_pages -= nr_pages; WRITE_ONCE(stock->nr_pages[i], stock_pages); if (!stock_pages) { - css_put(&memcg->css); WRITE_ONCE(stock->cached[i], NULL); + css_put(&memcg->css); } ret = true; } @@ -2209,8 +2209,8 @@ static void drain_stock(struct memcg_stock_pcp *stock, int i) WRITE_ONCE(stock->nr_pages[i], 0); } - css_put(&old->css); WRITE_ONCE(stock->cached[i], NULL); + css_put(&old->css); } static void drain_stock_fully(struct memcg_stock_pcp *stock) From 51542d55bcd59d428896168fd5b99b0d32440765 Mon Sep 17 00:00:00 2001 From: Shakeel Butt Date: Wed, 21 Oct 2026 13:48:18 -0700 Subject: [PATCH 212/213] mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root page_counter_calculate_protection() runs without locks. Several reclaimers can update the same counter's emin and elow at the same time, and readers such as mem_cgroup_below_min(), mem_cgroup_protection() and the dmem eviction code load them with READ_ONCE(). Commit b3a7822e5e75 ("mm, memcg: prevent mem_cgroup_protected store tearing") made the main path store emin and elow with WRITE_ONCE(), but it missed the early return for a counter whose parent is the reclaim root. Those two stores are the only plain accesses to emin and elow left. With CONFIG_KCSAN_STRICT=y, a few tasks writing to the same cgroup's memory.reclaim trigger reports like this within seconds: BUG: KCSAN: data-race in page_counter_calculate_protection / shrink_node read (marked) to 0xff1f91b146d40e40 of 8 bytes by task 163 on cpu 4: shrink_node+0x57a/0x1210 do_try_to_free_pages+0x194/0x930 try_to_free_mem_cgroup_pages+0x204/0x3f0 user_proactive_reclaim+0x28a/0x4d0 memory_reclaim+0x35/0x50 ... write to 0xff1f91b146d40e40 of 8 bytes by task 167 on cpu 6: page_counter_calculate_protection+0x1fa/0x360 mem_cgroup_calculate_protection+0x4e/0x70 shrink_node+0x53e/0x1210 ... The default KCSAN config does not report it, because it treats aligned word-sized plain writes as atomic. Use WRITE_ONCE() for these two stores as well. There is no functional change. Readers already accept stale values, and compilers emit a single word-sized store either way: with GCC 14 the x86-64 object code is the same before and after this change. Link: https://lore.kernel.org/20261021204818.3431099-1-shakeel.butt@linux.dev Fixes: b3a7822e5e75 ("mm, memcg: prevent mem_cgroup_protected store tearing") Signed-off-by: Shakeel Butt Signed-off-by: Andrew Morton Reported-by: Sashiko Reviewed-by: Gregory Price (Meta) Reviewed-by: SJ Park Acked-by: Muchun Song Reviewed-by: Anshuman Khandual Cc: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin Cc: Chris Down --- mm/page_counter.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/page_counter.c b/mm/page_counter.c index 9167ffd1380c..5293460831f8 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -472,8 +472,8 @@ void page_counter_calculate_protection(struct page_counter *root, return; if (parent == root) { - counter->emin = READ_ONCE(counter->min); - counter->elow = READ_ONCE(counter->low); + WRITE_ONCE(counter->emin, READ_ONCE(counter->min)); + WRITE_ONCE(counter->elow, READ_ONCE(counter->low)); return; } From 183895979c441e52dbd651b388553b09eaa35cfa Mon Sep 17 00:00:00 2001 From: Siddharth Chintamaneni Date: Sat, 17 Oct 2026 05:09:27 +0000 Subject: [PATCH 213/213] mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline When CONFIG_MEMCG=y, lruvec_pgdat() returns lruvec->pgdat, which for the per-node pgdat->__lruvec is left NULL by lruvec_init() and only filled in lazily by mem_cgroup_lruvec(). Normally that happens early, e.g. on the first folio_add_lru(). With cgroup_disable=memory, lru_gen_shrink_node() passes &pgdat->__lruvec directly to shrink_one(). If direct reclaim runs before anything has called mem_cgroup_lruvec() for the node, which can happen during early boot under memory pressure (e.g. a kdump kernel), get_swappiness() ends up dereferencing a NULL pgdat: Unable to handle kernel NULL pointer dereference at virtual address 0000000000005040 pc : get_swappiness+0x68/0xb0 lr : try_to_shrink_lruvec+0x54/0x2c0 Call trace: get_swappiness+0x68/0xb0 (P) try_to_shrink_lruvec+0x54/0x2c0 shrink_one+0x8c/0x240 shrink_node+0xc6c/0xe08 do_try_to_free_pages+0xdc/0x570 try_to_free_pages+0x124/0x278 __alloc_frozen_pages_noprof+0x6e8/0xf78 alloc_pages_mpol+0xc4/0x1c0 alloc_frozen_pages_noprof+0x54/0xe8 alloc_pages_noprof+0x1c/0x40 atomic_pool_expand+0x98/0x300 __dma_atomic_pool_init+0x58/0xd8 dma_atomic_pool_init+0x154/0x170 do_one_initcall+0x64/0x350 kernel_init_freeable+0x214/0x430 kernel_init+0x2c/0x208 ret_from_fork+0x10/0x20 The classic LRU path and MGLRU before per-node lru_gen_folio lists obtained the lruvec through mem_cgroup_lruvec(), which sets lruvec->pgdat. The above was hit on an arm64 kdump kernel (6.18-based) booted with cgroup_disable=memory nr_cpus=4. It also reproduces on mainline on an arm64 VM with CONFIG_ARM64_64K_PAGES=y, CONFIG_MEMCG=y, CONFIG_LRU_GEN_ENABLED=y and: mem=160M nr_cpus=4 coherent_pool=512M cgroup_disable=memory These options only serve to force direct reclaim early in boot; with 160M the boot eventually ends in OOM regardless. On mainline the first LRU reclaim happens while unpacking the initramfs, and faults the same way: Unable to handle kernel NULL pointer dereference at virtual address 0000000000005040 pc : get_swappiness+0xd4/0x150 Call trace: get_swappiness+0xd4/0x150 (P) try_to_shrink_lruvec+0x44/0x2b8 shrink_one+0x80/0x1a0 shrink_node+0xd98/0xea8 do_try_to_free_pages+0xf4/0x4b0 try_to_free_pages+0x110/0x360 __alloc_frozen_pages_noprof+0x8b0/0x1718 ... __filemap_get_folio_mpol+0x2ac/0x548 simple_write_begin+0x44/0xf8 page_symlink+0x178/0x198 ... do_populate_rootfs+0x13c/0x200 Link: https://lore.kernel.org/20261017050928.128582-1-sidchintamaneni@gmail.com Fixes: e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists") Signed-off-by: Siddharth Chintamaneni Signed-off-by: Hardik Garg Signed-off-by: Andrew Morton Reviewed-by: Barry Song Reviewed-by: Baolin Wang Assisted-by: LLM Cc: Johannes Weiner Cc: David Hildenbrand Cc: Michal Hocko Cc: Qi Zheng Cc: Shakeel Butt Cc: Lorenzo Stoakes Cc: Kairui Song Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Baoquan He Cc: Yu Zhao Cc: Cc: Cc: # 6.3+ --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index fc823fe3bd1f..9959ca158ce8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5454,7 +5454,7 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * sc->nr_reclaimed = 0; if (mem_cgroup_disabled()) - shrink_one(&pgdat->__lruvec, sc); + shrink_one(mem_cgroup_lruvec(NULL, pgdat), sc); else shrink_many(pgdat, sc);