From 68b2aa37afa390f08de13bc0bac61a9ded3318bc Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Thu, 29 Oct 2026 11:43:22 +0100 Subject: [PATCH 01/61] mm/mempolicy: charge shared policy nodes Any local user can bypass memory.max with mbind() on a tmpfs file, and allocate unbounded kernel memory. mbind() sets a NUMA memory policy for a range of a mapping. If the mapping is of a shmem and friends, the kernel also stores the policy on the inode, in a tree with one entry per range of the file. Each mbind() on a single page of a mapping can add one shared policy node (struct sp_node) and one struct mempolicy to that tree. They outlive the mapping and stay around until the inode is destroyed. The tree is not bounded by the file size, so the file can be empty. Memory that user space can allocate must be charged to a memory cgroup. A slab cache does that for all of its objects if it is created with SLAB_ACCOUNT. numa_policy_init() creates the slab caches for both structs without SLAB_ACCOUNT, so they are not charged to any cgroup. Reproduced with mbind() on single pages of an empty tmpfs file, from a cgroup with memory.max=8M: the global OOM killer killed tasks owned by root, and the machine panicked. Fix this by creating both caches with SLAB_ACCOUNT. This change implies that mbind() can now fail when running into memcg limits. Further, the policy_cache also serves task and VMA policies, so they are similarly accounted now. Note that per-file limits are not really helpful, as the user could simply use many files; having limits per user is tricky as well. Fix generated by an LLM and submitted by Chris Mason in private. Fixes: 5d097056c9a0 ("kmemcg: account certain kmem allocations to memcg") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Acked-by: Pedro Falcato Acked-by: Kiryl Shutsemau (Meta) Acked-by: Zi Yan Reviewed-by: Baolin Wang Tested-by: Gregory Price (Meta) [ david: smaller description tweaks ] Reviewed-by: Balbir Singh Link: https://patch.msgid.link/20261029-fixes_shmem-v4-1-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/mempolicy.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 0884c629670d..4da1cb8777f6 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -3372,11 +3372,11 @@ void __init numa_policy_init(void) policy_cache = kmem_cache_create("numa_policy", sizeof(struct mempolicy), - 0, SLAB_PANIC, NULL); + 0, SLAB_PANIC | SLAB_ACCOUNT, NULL); sn_cache = kmem_cache_create("shared_policy_node", sizeof(struct sp_node), - 0, SLAB_PANIC, NULL); + 0, SLAB_PANIC | SLAB_ACCOUNT, NULL); for_each_node(nid) { preferred_node_policy[nid] = (struct mempolicy) { From 968d73463059156e1b89bfa6b189acd7d27197b2 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 29 Oct 2026 11:43:23 +0100 Subject: [PATCH 02/61] tmpfs: keep the quota soft limits of an unused id When the last reference to a dquot goes away, shmem_release_dquot() stores its limits in the rb tree of the mount, unless shmem_is_empty_dquot() says that there is nothing worth storing. That function looks at the usage and at the two hard limits only. So if the administrator sets nothing but soft limits for an id that owns no file yet, the entry is thrown away as soon as the dquot that Q_SETQUOTA used is released, and the id never gets to see a limit. To fix it, compare the soft limits as well: an entry can be empty only if both soft limits are zero as well, which is exactly what shmem_acquire_dquot() sets for a new entry. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: de4c0e7ca8b5 ("shmem: Add default quota limit mount options") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik Reviewed-by: Jan Kara Tested-by: Gregory Price (Meta) [ david: slightly adjusted description ] Link: https://patch.msgid.link/20261029-fixes_shmem-v4-2-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem_quota.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/shmem_quota.c b/mm/shmem_quota.c index a3fa4bd93e75..43d3b4fd6cb1 100644 --- a/mm/shmem_quota.c +++ b/mm/shmem_quota.c @@ -256,6 +256,8 @@ static bool shmem_is_empty_dquot(struct dquot *dquot) if (test_bit(DQ_FAKE_B, &dquot->dq_flags) || (dquot->dq_dqb.dqb_curspace == 0 && dquot->dq_dqb.dqb_curinodes == 0 && + dquot->dq_dqb.dqb_bsoftlimit == 0 && + dquot->dq_dqb.dqb_isoftlimit == 0 && dquot->dq_dqb.dqb_bhardlimit == bhardlimit && dquot->dq_dqb.dqb_ihardlimit == ihardlimit)) return true; From d5d861ee303da01e59fdf8e8b2f7096c76e7efd5 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 29 Oct 2026 11:43:24 +0100 Subject: [PATCH 03/61] tmpfs: preserve explicitly disabled quota limits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An administrator’s decision to disable a user’s or group’s quota limits can be silently lost when tmpfs restores the mount defaults after releasing and reacquiring the quota record. This can cause later file creation or writes to fail with EDQUOT, even though the administrator has explicitly disabled quota limits for the user or group. do_set_dqblk() sets DQ_FAKE_B when all block and inode limits are zero. This means quota enforcement is disabled for that user or group. shmem_is_empty_dquot() treats the flag as a reason to discard the entry from the mount's quota tree. The next shmem_acquire_dquot() finds no saved entry and recreates it with the mount's default hard limits. If the cached dquot is reused, DQ_FAKE_B remains set: Q_GETQUOTA reports the restored defaults, but enforcement is still disabled. Once that dquot has been reclaimed, a new dquot enforces the restored defaults. Remove the DQ_FAKE_B test. Only discard entries with zero usage, zero soft limits and hard limits equal to the mount defaults. Explicitly disabled limits are then preserved whenever the mount defaults are nonzero. Explicitly disabled limits now retain a quota-tree entry, just like other limits that differ from the mount defaults. Rename shmem_is_empty_dquot() to shmem_is_default_dquot() to describe the check for default values. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: de4c0e7ca8b5 ("shmem: Add default quota limit mount options") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik Reviewed-by: Jan Kara Tested-by: Gregory Price (Meta) [ david: code cleanups; rewrite description ] Link: https://patch.msgid.link/20261029-fixes_shmem-v4-3-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem_quota.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/mm/shmem_quota.c b/mm/shmem_quota.c index 43d3b4fd6cb1..d191ebda3f00 100644 --- a/mm/shmem_quota.c +++ b/mm/shmem_quota.c @@ -239,7 +239,7 @@ static int shmem_acquire_dquot(struct dquot *dquot) return ret; } -static bool shmem_is_empty_dquot(struct dquot *dquot) +static bool shmem_is_default_dquot(struct dquot *dquot) { struct shmem_sb_info *sbinfo = dquot->dq_sb->s_fs_info; qsize_t bhardlimit; @@ -253,21 +253,21 @@ static bool shmem_is_empty_dquot(struct dquot *dquot) ihardlimit = sbinfo->qlimits.grpquota_ihardlimit; } - if (test_bit(DQ_FAKE_B, &dquot->dq_flags) || - (dquot->dq_dqb.dqb_curspace == 0 && - dquot->dq_dqb.dqb_curinodes == 0 && - dquot->dq_dqb.dqb_bsoftlimit == 0 && - dquot->dq_dqb.dqb_isoftlimit == 0 && - dquot->dq_dqb.dqb_bhardlimit == bhardlimit && - dquot->dq_dqb.dqb_ihardlimit == ihardlimit)) - return true; - - return false; + /* + * Defaults are zero usage and soft limits, and the mount's hard limits. + * DQ_FAKE_B only means that *all* limits are zero, which need not match + * the mount defaults. + */ + return dquot->dq_dqb.dqb_curspace == 0 && + dquot->dq_dqb.dqb_curinodes == 0 && + dquot->dq_dqb.dqb_bsoftlimit == 0 && + dquot->dq_dqb.dqb_isoftlimit == 0 && + dquot->dq_dqb.dqb_bhardlimit == bhardlimit && + dquot->dq_dqb.dqb_ihardlimit == ihardlimit; } /* - * Store limits from dquot in the tree unless it's fake. If it is fake - * remove the id from the tree since there is no useful information in - * there. + * Remove the ID from the tree if it has no usage and its limits match + * the mount defaults. Otherwise, save its limits for the next acquire. */ static int shmem_release_dquot(struct dquot *dquot) { @@ -302,7 +302,7 @@ static int shmem_release_dquot(struct dquot *dquot) return -ENOENT; found: - if (shmem_is_empty_dquot(dquot)) { + if (shmem_is_default_dquot(dquot)) { /* Remove entry from the tree */ rb_erase(&entry->node, info->dqi_priv); kfree(entry); From 0237c47ca17005ea41b510c6116f0257dbf84eab Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Thu, 29 Oct 2026 11:43:25 +0100 Subject: [PATCH 04/61] mm/shmem: honour F_SEAL_EXEC when an ACL is set The owner of a memfd sealed with F_SEAL_EXEC can still change its execute bits by setting a POSIX access ACL. With CONFIG_TMPFS_POSIX_ACL, this lets an unprivileged owner make a non-executable memfd executable, bypassing vm.memfd_noexec=2. simple_set_acl() updates i_mode directly, bypassing the seal check in shmem_setattr(). Add shmem_set_acl() to check the resulting mode. Share the F_SEAL_EXEC check with shmem_setattr() using memfd_check_seals_mode() in include/linux/memfd.h. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 6fd7353829ca ("mm/memfd: add F_SEAL_EXEC") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Acked-by: Pedro Falcato Acked-by: Kiryl Shutsemau (Meta) Acked-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Pratyush Yadav (Google) Tested-by: Gregory Price (Meta) [ david: use common helper; simplify description ] Link: https://patch.msgid.link/20261029-fixes_shmem-v4-4-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- include/linux/memfd.h | 9 +++++++++ mm/shmem.c | 35 ++++++++++++++++++++++++++++++----- 2 files changed, 39 insertions(+), 5 deletions(-) diff --git a/include/linux/memfd.h b/include/linux/memfd.h index c159e40e3f34..272479317cf1 100644 --- a/include/linux/memfd.h +++ b/include/linux/memfd.h @@ -2,10 +2,19 @@ #ifndef __LINUX_MEMFD_H #define __LINUX_MEMFD_H +#include #include #define MEMFD_ANON_NAME "[memfd]" +static inline int memfd_check_seals_mode(unsigned int seals, umode_t old_mode, + umode_t new_mode) +{ + if ((seals & F_SEAL_EXEC) && ((old_mode ^ new_mode) & 0111)) + return -EPERM; + return 0; +} + #ifdef CONFIG_MEMFD_CREATE extern long memfd_fcntl(struct file *file, unsigned int cmd, unsigned int arg); struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx); diff --git a/mm/shmem.c b/mm/shmem.c index af21a89abb0f..6df0237832c0 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -77,6 +77,7 @@ static struct vfsmount *shm_mnt __ro_after_init; #include #include #include +#include #include #include #include @@ -1546,10 +1547,11 @@ static int shmem_setattr(const struct mnt_idmap *idmap, if (error) return error; - if ((info->seals & F_SEAL_EXEC) && (attr->ia_valid & ATTR_MODE)) { - if ((inode->i_mode ^ attr->ia_mode) & 0111) { - return -EPERM; - } + if (attr->ia_valid & ATTR_MODE) { + error = memfd_check_seals_mode(info->seals, inode->i_mode, + attr->ia_mode); + if (error) + return error; } if (S_ISREG(inode->i_mode) && (attr->ia_valid & ATTR_SIZE)) { @@ -5404,12 +5406,35 @@ static const struct file_operations shmem_file_operations = { #endif }; +#ifdef CONFIG_TMPFS_POSIX_ACL +static int shmem_set_acl(const struct mnt_idmap *idmap, struct dentry *dentry, + struct posix_acl *acl, int type) +{ + struct inode *inode = d_inode(dentry); + umode_t mode = inode->i_mode; + int error; + + if (type == ACL_TYPE_ACCESS) { + error = posix_acl_equiv_mode(acl, &mode); + if (error < 0) + return error; + error = memfd_check_seals_mode(SHMEM_I(inode)->seals, + inode->i_mode, mode); + if (error) + return error; + } + return simple_set_acl(idmap, dentry, acl, type); +} +#else +#define shmem_set_acl simple_set_acl +#endif + static const struct inode_operations shmem_inode_operations = { .getattr = shmem_getattr, .setattr = shmem_setattr, #ifdef CONFIG_TMPFS_XATTR .listxattr = shmem_listxattr, - .set_acl = simple_set_acl, + .set_acl = shmem_set_acl, .fileattr_get = shmem_fileattr_get, .fileattr_set = shmem_fileattr_set, #endif From b7f570e7cea761ec7eb4a3a9557a3a082d14e88c Mon Sep 17 00:00:00 2001 From: Eva Kurchatova Date: Thu, 29 Oct 2026 11:43:26 +0100 Subject: [PATCH 05/61] hugetlbfs: enforce F_SEAL_EXEC in setattr F_SEAL_EXEC is enforced by shmem_setattr(), but hugetlbfs_setattr() does not check it. As a result, fchmod() can change the execute bits of a sealed hugetlbfs memfd. Use memfd_check_seals_mode() in hugetlbfs_setattr() to reject changes to execute bits when F_SEAL_EXEC is set. This also fixes the memfd_test hugetlbfs SEAL-EXEC subtest, where fchmod() incorrectly succeeded after applying F_SEAL_EXEC. Fixes: 6fd7353829ca ("mm/memfd: add F_SEAL_EXEC") Cc: stable@vger.kernel.org Signed-off-by: Eva Kurchatova Link: https://patch.msgid.link/20260629180247.2343229-1-eva.kurchatova@virtuozzo.com Acked-by: Pedro Falcato Acked-by: Kiryl Shutsemau (Meta) Acked-by: Zi Yan Tested-by: Gregory Price (Meta) Reviewed-by: Muchun Song [ david: use common helper; fixup Fixes:; check only regular files ] Acked-by: Oscar Salvador Link: https://patch.msgid.link/20261029-fixes_shmem-v4-5-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- fs/hugetlbfs/inode.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 01c4d6f44ebe..02e0a65d596d 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -840,6 +841,13 @@ static int hugetlbfs_setattr(const struct mnt_idmap *idmap, if (error) return error; + if (S_ISREG(inode->i_mode) && (ia_valid & ATTR_MODE)) { + error = memfd_check_seals_mode(info->seals, inode->i_mode, + attr->ia_mode); + if (error) + return error; + } + trace_hugetlbfs_setattr(inode, dentry, attr); if (ia_valid & ATTR_SIZE) { From 8f59173b5ca358bf4d1017b44da3cbe6faa80daf Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Thu, 29 Oct 2026 11:43:27 +0100 Subject: [PATCH 06/61] mm/shmem: hold off eviction in shmem_writeout() Once shmem_writeout() has moved the folio from the page cache to the swap cache, the folio lock no longer keeps the inode alive: truncation and eviction find a swap entry, free it and do not wait for anybody. That did not matter as long as swap_writeout() was the last thing the function did. But since it puts the folio back when swap_writeout() returns AOP_WRITEPAGE_ACTIVATE, it uses the mapping and the inode afterwards: reclaim last close of an unlinked file shmem_writeout() shmem_delete_from_page_cache() swap_writeout() zswap_store() fails shmem_evict_inode() writeback is disabled shmem_truncate_range() return AOP_WRITEPAGE_ACTIVATE shmem_free_swap() destroy_inode() ... RCU grace period, freed shmem_add_to_page_cache(mapping) xas_lock_irq() <- use after free swap_writeout() behaves that way if zswap is enabled, cannot take the folio and the memcg or one of its ancestors has memory.zswap.writeback set to 0. shmem_unuse() already added infrastructure to deal with that. Reuse info->stop_eviction to protect the inode during writeout. Increment it before removing the folio from the page cache, and decrement it after swap_writeout() and any attempted reinsertion and inode accounting. In shmem_evict_inode(), wait for the counter to reach zero even if swapoff has already removed the inode from the swap list. Add some helpers to make the code easier to digest. There is a known remaining issue with two swapoffs racing, and the empty list check not being able to detect that race. That will be addressed separately later. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 6344a6d9ce13 ("mm/shmem: writeout free swap if swap_writeout() reactivates") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik Acked-by: Pedro Falcato Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Nhat Pham Reviewed-by: Baolin Wang Tested-by: Baolin Wang [ david: code cleanups; improve description ] Tested-by: Gregory Price (Meta) Link: https://patch.msgid.link/20261029-fixes_shmem-v4-6-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 36 ++++++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 6df0237832c0..c44d2dd71c89 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1616,6 +1616,17 @@ static int shmem_setattr(const struct mnt_idmap *idmap, return error; } +static void shmem_disallow_eviction(struct shmem_inode_info *info) +{ + atomic_inc(&info->stop_eviction); +} + +static void shmem_allow_eviction(struct shmem_inode_info *info) +{ + if (atomic_dec_and_test(&info->stop_eviction)) + wake_up_var(&info->stop_eviction); +} + static void shmem_evict_inode(struct inode *inode) { struct shmem_inode_info *info = SHMEM_I(inode); @@ -1628,16 +1639,21 @@ static void shmem_evict_inode(struct inode *inode) mapping_set_exiting(inode->i_mapping); shmem_truncate_range(inode, 0, (loff_t)-1); shmem_unused_huge_del(inode); - while (!list_empty(&info->swaplist)) { - /* Wait while shmem_unuse() is scanning this inode... */ + /* + * Writeout may still use the inode after swapoff has removed it + * from the swaplist, so always wait at least once. + */ + do { wait_var_event(&info->stop_eviction, !atomic_read(&info->stop_eviction)); + if (list_empty(&info->swaplist)) + break; spin_lock(&shmem_swaplist_lock); /* ...but beware of the race if we peeked too early */ if (!atomic_read(&info->stop_eviction)) list_del_init(&info->swaplist); spin_unlock(&shmem_swaplist_lock); - } + } while (!list_empty(&info->swaplist)); } simple_xattrs_free(&sbinfo->xa_cache, &info->xattrs, sbinfo->max_inodes ? &freed : NULL); @@ -1776,15 +1792,14 @@ int shmem_unuse(unsigned int type) * remove placeholder inode from swaplist, nor let it be freed * (igrab() would protect from unlink, but not from unmount). */ - atomic_inc(&info->stop_eviction); + shmem_disallow_eviction(info); spin_unlock(&shmem_swaplist_lock); error = shmem_unuse_inode(&info->vfs_inode, type); cond_resched(); spin_lock(&shmem_swaplist_lock); - if (atomic_dec_and_test(&info->stop_eviction)) - wake_up_var(&info->stop_eviction); + shmem_allow_eviction(info); if (error) break; if (list_empty(&info->swaplist)) @@ -1915,6 +1930,13 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, spin_unlock(&shmem_swaplist_lock); } + /* + * Once the folio has left the page cache, its lock no longer + * keeps the inode from being evicted and freed: hold eviction + * off, in case the folio has to be put back below. + */ + shmem_disallow_eviction(info); + folio_dup_swap(folio, NULL); shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap)); @@ -1922,6 +1944,7 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, error = swap_writeout(ctx, folio); if (error != AOP_WRITEPAGE_ACTIVATE) { /* folio has been unlocked */ + shmem_allow_eviction(info); return error; } @@ -1938,6 +1961,7 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, shmem_recalc_inode(inode, 0, -nr_pages); folio_put_swap(folio, NULL); } + shmem_allow_eviction(info); /* * The swap_cache_del_folio() below could be left for From a842a1f947da8a7020e72f885ffbf2ce6a93685f Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Thu, 29 Oct 2026 11:43:28 +0100 Subject: [PATCH 07/61] mm/shmem: stop retrying folios that cannot be split A fault or collapse can insert a large folio crossing EOF after shmem_undo_range() has handled the boundary folios. If splitting that folio fails, truncate_inode_partial_folio() retains it when it is dirty. The final scan then repeatedly finds it because it restarts from an unchanged start. Truncation keeps the inode lock and ignores fatal signals while looping. A concurrent unlink() can then block other directory operations while holding the directory lock and waiting for the file's lock. Advance start past the retained folio, whose truncated part has already been zeroed. Do not move start backwards if the failed split has shrunk the folio. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 55ac8bbe358b ("mm/shmem: fix race in shmem_undo_range w/THP") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Tested-by: Gregory Price (Meta) [ david: split fixes, simplify descriptions, adjust comments ] Reviewed-by: Zi Yan Reviewed-by: Balbir Singh Link: https://patch.msgid.link/20261029-fixes_shmem-v4-7-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/shmem.c b/mm/shmem.c index c44d2dd71c89..af49d6bf6382 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1481,6 +1481,15 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, index = start; break; } + } else if (folio->mapping) { + /* + * Splitting failed, but the part in the + * range is already zeroed. The folio + * was not dropped, so skip it when + * restarting the scan to avoid retrying + * it forever. + */ + start = max(start, folio_next_index(folio)); } } folio_unlock(folio); From 0c800435394a172f33bf4df4a5a04e187546cc9b Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Thu, 29 Oct 2026 11:43:29 +0100 Subject: [PATCH 08/61] mm/shmem: preserve data in clean folios when splitting fails MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A shared writable mapping can have dirty PTEs while its shmem folio is still clean. A read fault can install a writable PTE, and a later store sets the PTE's dirty bit without marking the folio dirty. After shmem_undo_range() has handled the boundary folios, a concurrent read fault into a hole below the new EOF can allocate a large folio. For example, on a tmpfs mount with huge=always, the fault's offset must be within the file, but the allocated folio can extend beyond i_size. The final scan then finds this folio and calls truncate_inode_partial_folio() without marking it dirty. Splitting normally collects the PTE dirty bits when unmapping the folio. An extra reference, for example from vmsplice(), can make it fail before that step. The fallback unmapping on split failure excludes shmem, so the helper still sees a clean folio and drops it, losing data outside the range. CPU0: ftruncate(fd, 3M) CPU1: MAP_SHARED, tmpfs huge=always ───────────────────────────────────── ────────────────────────────────── i_size = 3M shmem_undo_range(3M, -1) first pass: hole, nothing to do boundary folio at 3M: hole, none read fault at 2M+x (below EOF) PMD folio [2M,4M), crosses EOF writable PTE, folio clean store to 2M+x PTE dirty, folio still clean vmsplice(2M+x) extra folio reference whole_folios: finds [2M,4M) folio truncate_inode_partial_folio() zero [3M,4M) folio_split() unmap_folio(): extra ref → -EAGAIN, PTE dirty bit never reaches the folio folio clean → truncate_inode_folio(whole) [2M,3M) dropped read 2M+x → refault → zeroes Mark large folios dirty before partial truncation in the final scan, as the boundary handling already does. Exclude failed-fallocate rollback, which must discard the folios it added. Fix generated by an LLM and sent by Chris Mason in private. Fixes: b9a8a4195c7d ("truncate,shmem: Handle truncates that split large folios") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan Tested-by: Gregory Price (Meta) [ david: simplify descriptions, adjust comments ] Link: https://patch.msgid.link/20261029-fixes_shmem-v4-8-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/shmem.c b/mm/shmem.c index af49d6bf6382..11c3e6b1aa76 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1464,6 +1464,15 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, VM_BUG_ON_FOLIO(folio_test_writeback(folio), folio); + /* + * A shared mapping can have dirty PTEs while + * the folio is still clean. Mark the folio + * dirty so a failed split cannot discard data + * outside the range. + */ + if (!unfalloc && folio_test_large(folio)) + folio_mark_dirty(folio); + if (!folio_test_large(folio)) { truncate_inode_folio(mapping, folio); } else if (truncate_inode_partial_folio(folio, From 23f2a880c0b9518fcb7e7e2ad54a29c0a2c4fba0 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 29 Oct 2026 11:43:30 +0100 Subject: [PATCH 09/61] mm/shmem: discard whole new large folios on fallocate rollback When undoing a failed fallocate(), shmem_undo_range() must discard each newly allocated folio. Its second scan instead passes large folios to truncate_inode_partial_folio(), which does not necessarily discard the entire large folio, but instead tries splitting it to discard only the requested range. For example, with huge=always, a failed allocation starting at offset 4 KiB can leave page 0 allocated in an otherwise empty file. Discard the whole folio when unfalloc is set. Note that we enter this path only for folios that are not uptodate, and can consequently be dropped without losing any data. Fixes: 55ac8bbe358b ("mm/shmem: fix race in shmem_undo_range w/THP") Cc: stable@vger.kernel.org Reviewed-by: Zi Yan Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Tested-by: Gregory Price (Meta) Link: https://patch.msgid.link/20261029-fixes_shmem-v4-9-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/shmem.c b/mm/shmem.c index 11c3e6b1aa76..151704b2091d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1473,7 +1473,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (!unfalloc && folio_test_large(folio)) folio_mark_dirty(folio); - if (!folio_test_large(folio)) { + if (unfalloc || !folio_test_large(folio)) { truncate_inode_folio(mapping, folio); } else if (truncate_inode_partial_folio(folio, lstart, lend, NULL, NULL)) { From f0d67b3d0a464847337d5063766d55e16041323e Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 29 Oct 2026 11:43:31 +0100 Subject: [PATCH 10/61] mm/shmem: rescan after nonuniform folio splitting The final scan in shmem_undo_range() restarts after a successful split only if the original folio is no longer large. A nonuniform split can leave that folio large, so the scan skips the newly created folios. Hole punching then leaves pages within the hole allocated even though they could be discarded. For example, a fault below the hole can allocate a 2 MiB folio after boundary handling. Punching a 1 MiB hole at offset 64 KiB splits the folio successfully but leaves the original folio at order 4. The scan continues past all 512 pages, leaving 2 MiB allocated instead of 1 MiB. The hole reads as zero, but the pages and their block charges remain. Fixes: 7460b470a131 ("mm/truncate: use folio_split() in truncate operation") Cc: stable@vger.kernel.org Reviewed-by: Baolin Wang Acked-by: Kiryl Shutsemau (Meta) Tested-by: Gregory Price (Meta) Reviewed-by: Zi Yan Link: https://patch.msgid.link/20261029-fixes_shmem-v4-10-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 151704b2091d..e96dc3beac95 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1478,18 +1478,12 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, } else if (truncate_inode_partial_folio(folio, lstart, lend, NULL, NULL)) { /* - * If we split a page, reset the loop so - * that we pick up the new sub pages. - * Otherwise the THP was entirely - * dropped or the target range was - * zeroed, so just continue the loop as - * is. + * The folio was split. Retry the scan + * to find the split folios. */ - if (!folio_test_large(folio)) { - folio_unlock(folio); - index = start; - break; - } + folio_unlock(folio); + index = start; + break; } else if (folio->mapping) { /* * Splitting failed, but the part in the From fa213b08f37f9dff78ce240a87579dc264fc7a5d Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 29 Oct 2026 11:43:32 +0100 Subject: [PATCH 11/61] mm/shmem: fix truncate hang in large swap entry Truncating a tmpfs file or memfd to a page-aligned size inside a large swap entry can loop indefinitely in shmem_undo_range(). The task holds the inode lock and cannot be killed. A large swap entry represents a whole swapped-out folio. For example, truncate a file whose pages 0-7 share one swap entry to four pages: [ 0 1 2 3 | 4 5 6 7 ] keep | discard ^ new EOF, start = 4 i_size has already been updated when boundary handling asks shmem_get_partial_folio() for page 4. Since it is swapped out, the helper calls shmem_get_folio() with SGP_READ. That rejects page 4 with -EINVAL because its starting offset equals i_size. Nothing is read back. The final scan cannot free the entry without losing pages 0-3. It skips it, restarts from start = 4, and finds it again indefinitely. An unaligned truncate avoids this particular failure because the requested page starts below i_size. Folios already in the page-cache cannot trigger it as the direct lookup will return them beyond EOF. Fix it by letting shmem_get_partial_folio() call shmem_swapin_folio() directly for swap entries. This bypasses the read path's i_size check, allowing the existing boundary handling to zero and split the folio. If swap-in returns a smaller folio, it also splits the swap entry so the final scan can free the entries beyond EOF. Retry the lookup if the entry changes during swap-in. Handling swap-in allocation failures remains broken and is unchanged by this commit. Even when shmem_swapin_folio() correctly returns -ENOMEM instead of -EEXIST, shmem_get_partial_folio() returns NULL and shmem_undo_range() skips boundary handling. If the large swap entry remains intact, truncation can still loop in the final scan, while hole punching can return without zeroing the boundary. This may read an entire large folio back from swap while holding i_rwsem, only to discard part of it. Avoiding that I/O would require splitting both the swap entry and any large folio still in the swap cache. That, however, is much bigger work. Fixes: 8a1968bd997f ("mm/shmem, swap: fix race of truncate and swap entry split") Cc: stable@vger.kernel.org Reported-by: Chris Mason Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Tested-by: Gregory Price (Meta) Link: https://patch.msgid.link/20261029-fixes_shmem-v4-11-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index e96dc3beac95..e1b7cf4b5280 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1298,12 +1298,10 @@ void shmem_unlock_mapping(struct address_space *mapping) static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index) { + const gfp_t gfp = mapping_gfp_mask(inode->i_mapping); struct folio *folio; - /* - * At first avoid shmem_get_folio(,,,SGP_READ): that fails - * beyond i_size, and reports fallocated folios as holes. - */ +retry: folio = filemap_get_entry(inode->i_mapping, index); if (!folio) return folio; @@ -1314,13 +1312,14 @@ static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index) /* The folio has been swapped out */ folio_unlock(folio); folio_put(folio); + goto retry; } /* - * But read a folio back from swap if any of it is within i_size - * (although in some cases this is just a waste of time). + * Truncation can look up a large swap entry at or beyond i_size. + * Swap it in directly to bypass the size check in shmem_get_folio(). */ - folio = NULL; - shmem_get_folio(inode, index, 0, &folio, SGP_READ); + if (shmem_swapin_folio(inode, index, &folio, SGP_READ, gfp, NULL, NULL) == -EEXIST) + goto retry; return folio; } From f4ce23b1feef16d435aa1188b6c2a637a5373ed5 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 29 Oct 2026 11:43:33 +0100 Subject: [PATCH 12/61] mm/shmem: preserve allocation errors when swapping in large entries shmem_swapin_folio() adjusts swap to the requested slot within a large swap entry before allocating a folio. If allocation or charging fails, the error path compares this adjusted value with the original entry still stored in the XArray in shmem_confirm_swap(). Unless the index is at the start of the entry, this turns -ENOMEM into -EEXIST even though nothing changed. The caller retries -EEXIST, so persistent allocation failures can leave the task spinning instead of returning the error. Restore swap to index_entry on allocation failure so the existing error path checks the original entry. This preserves the allocation error if the entry is unchanged and still retries if it changed concurrently. Fixes: 91ab656ece13 ("mm/shmem, swap: tidy up swap entry splitting") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Tested-by: Gregory Price (Meta) Link: https://patch.msgid.link/20261029-fixes_shmem-v4-12-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/shmem.c b/mm/shmem.c index e1b7cf4b5280..02de6326ef3b 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2552,6 +2552,8 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, else error = -ENOMEM; folio = NULL; + /* Make sure to validate the original value. */ + swap = index_entry; goto failed; } if (fault_type) { From 5f1ec2cf92a7d655a8eb39711987822f9b1f9d67 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 29 Oct 2026 11:43:34 +0100 Subject: [PATCH 13/61] mm/shmem: fix truncate loop after boundary swapin failure If swapping in a large entry at the new EOF fails, shmem_undo_range() leaves the entry in place. Its final scan skips the entry, then restarts from the same start index and finds it again. Truncation can loop forever while holding i_rwsem. Advance start past the skipped entry so that a restarted scan can finish, as we already do for folios that cannot be split. Note that this does not fix the missing zeroing when boundary swapin fails, it merely prevents the endless loop. Fixes: 8a1968bd997f ("mm/shmem, swap: fix race of truncate and swap entry split") Cc: stable@vger.kernel.org Acked-by: Kiryl Shutsemau (Meta) Tested-by: Gregory Price (Meta) Reviewed-by: Baolin Wang Reviewed-by: Nhat Pham Link: https://patch.msgid.link/20261029-fixes_shmem-v4-13-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 02de6326ef3b..eb2e6e50630c 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1434,14 +1434,16 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, order = shmem_confirm_swap(mapping, indices[i], radix_to_swp_entry(folio)); /* - * If found a large swap entry cross the end or start - * border, skip it as the truncate_inode_partial_folio - * above should have at least zerod its content once. + * Skip boundary entries when restarting + * the scan to avoid endless loops after + * swapin failure. */ if (order > 0) { base = round_down(base, 1 << order); - if (base < start || base + (1 << order) > end) + if (base < start || base + (1 << order) > end) { + start = max(start, base + (1 << order)); continue; + } } /* Swap was replaced by page or extended, retry */ index = base; From 66ce68a249f4ac56cd6968aa6d8ded20fb4b7d09 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 29 Oct 2026 11:43:35 +0100 Subject: [PATCH 14/61] mm/shmem: fix accounting of swapin-error markers A failed swap read replaces the swap entry with a swapin-error marker in the page-cache and subtracts its pages from "alloced" and "swapped"; however, it leaves their block charge in place. Removing the marker during truncate then subtracts from "swapped" again. This can underflow the counters or make swapped reach zero while other pages remain swapped out, causing swapoff to loop without finding them. When installing the marker, subtract only from "swapped" and let shmem_recalc_inode() reduce "alloced" and release the block charge. Make shmem_free_swap() report successful removal separately from the number of swap pages freed, so removing a swapin-error marker does not subtract from "swapped" again. It's a bit confusing, but replacing a swapped out page by a swapin-error marker in shmem_set_folio_swapin_error() behaves as follows: Initially: alloced=1, swapped=1, nrpages=0 Calling shmem_recalc_inode(inode, 0, -1) -> swapped = 0 -> freed = 1 - 0 - 0 = 1 -> alloced -= freed -> alloced=0, swapped=0, block charge released Fixes: 6cec2b95dadf ("mm/shmem: fix infinite loop when swap in shmem error at swapoff time") Cc: stable@vger.kernel.org Tested-by: Gregory Price (Meta) Reviewed-by: Baolin Wang Link: https://patch.msgid.link/20261029-fixes_shmem-v4-14-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index eb2e6e50630c..897f19ed0fe3 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1177,12 +1177,11 @@ static void shmem_delete_from_page_cache(struct folio *folio, void *radswap) } /* - * Remove swap entry from page cache, free the swap and its page cache. Returns - * the number of pages being freed. 0 means entry not found in XArray (0 pages - * being freed). + * Remove a swap entry or swapin-error marker wholly within [index, end]. + * Return whether it was removed, counting freed swap pages in nr_swaps_freed. */ -static long shmem_free_swap(struct address_space *mapping, - pgoff_t index, pgoff_t end, void *radswap) +static bool shmem_free_swap(struct address_space *mapping, + pgoff_t index, pgoff_t end, void *radswap, long *nr_swaps_freed) { XA_STATE(xas, &mapping->i_pages, index); const softleaf_t swp = radix_to_swp_entry(radswap); @@ -1203,8 +1202,10 @@ static long shmem_free_swap(struct address_space *mapping, xas_unlock_irq(&xas); /* A swapin-error marker holds no swap slot, so just drop it. */ - if (nr_pages && softleaf_is_swap(swp)) + if (nr_pages && softleaf_is_swap(swp)) { swap_put_entries_direct(swp, nr_pages); + *nr_swaps_freed += nr_pages; + } return nr_pages; } @@ -1358,8 +1359,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (xa_is_value(folio)) { if (unfalloc) continue; - nr_swaps_freed += shmem_free_swap(mapping, indices[i], - end - 1, folio); + shmem_free_swap(mapping, indices[i], end - 1, + folio, &nr_swaps_freed); continue; } @@ -1422,13 +1423,11 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (xa_is_value(folio)) { int order; - long swaps_freed; if (unfalloc) continue; - swaps_freed = shmem_free_swap(mapping, indices[i], - end - 1, folio); - if (!swaps_freed) { + if (!shmem_free_swap(mapping, indices[i], end - 1, + folio, &nr_swaps_freed)) { pgoff_t base = indices[i]; order = shmem_confirm_swap(mapping, indices[i], @@ -1449,7 +1448,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, index = base; break; } - nr_swaps_freed += swaps_freed; continue; } @@ -2409,11 +2407,10 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index, folio_put_swap(folio, NULL); swap_cache_del_folio(folio); /* - * Don't treat swapin error folio as alloced. Otherwise inode->i_blocks - * won't be 0 when inode is released and thus trigger WARN_ON(i_blocks) - * in shmem_evict_inode(). + * Let shmem_recalc_inode() reduce alloced and release the block charge + * for the freed swap pages. */ - shmem_recalc_inode(inode, -nr_pages, -nr_pages); + shmem_recalc_inode(inode, 0, -nr_pages); } static int shmem_split_large_entry(struct inode *inode, pgoff_t index, From 59511d630a6942356ac4b6984797998a8f3a9017 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 29 Oct 2026 11:43:36 +0100 Subject: [PATCH 15/61] mm/shmem: fix truncation of files containing only swapin-error markers It might not be very common, but we can end up with shmem files that are composed only of swapin-error/poison markers. Such markers are not counted in info->alloced. Consequently, shmem_setattr() can skip shmem_truncate_range(), leaving the markers behind. Truncating the file to zero and growing it again will result in reads failing with -EIO instead of returning zeroes. Check whether the mapping contains entries, in addition to info->alloced, before skipping the truncation. Keep the info->alloced check so that shmem_recalc_inode(), called at the end of shmem_undo_range(), can fix up accounting after reclaim has emptied the mapping. Fixes: 6cec2b95dadf ("mm/shmem: fix infinite loop when swap in shmem error at swapoff time") Cc: stable@vger.kernel.org Tested-by: Gregory Price (Meta) Reviewed-by: Baolin Wang Link: https://patch.msgid.link/20261029-fixes_shmem-v4-15-392fc715d8f6@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/shmem.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/shmem.c b/mm/shmem.c index 897f19ed0fe3..923957e5e6a2 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1591,7 +1591,8 @@ static int shmem_setattr(const struct mnt_idmap *idmap, if (oldsize > holebegin) unmap_mapping_range(inode->i_mapping, holebegin, 0, 1); - if (info->alloced) + /* Swapin-error markers are not counted in alloced. */ + if (info->alloced || !mapping_empty(inode->i_mapping)) shmem_truncate_range(inode, newsize, (loff_t)-1); /* unmap again to remove racily COWed private pages */ From 65a9bf08a3e4d6e58df211ed900c16bafb79b267 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:30 +0100 Subject: [PATCH 16/61] mm/mglru: don't use stack pointers when walking PUD and PMD tables MGLRU passes stack copies of page-table entries in walk_pud_range() and walk_pmd_range() when walking to the next level. These copies can become stale when khugepaged or MADV_DONTNEED (CONFIG_PT_RECLAIM) removes a PTE table concurrently. This pattern dates back to commit bd74fdaea146 ("mm: multi-gen LRU: support page table walks"). Likely, earlier locking masked the latent bug. Later changes exposed it: commit 1d65b771bc08 ("mm/khugepaged: retract_page_tables() without mmap or vma lock") allowed concurrent retraction, and commit cf4a65539c13 ("memcg-v1: no need for memcg locking for MGLRU") dropped the walker's outer RCU protection. The fix is simple, so just fix the original commit so we don't have to speculate. To fix it, pass the actual PMD/PUD pointers. This restores the RCU-protected lookup and validation against the live PMD. Note that the PUD handling is required as well to handle folded page tables correctly in all cases. Fix generated by an LLM and sent by Josef Bacik in private. [ david: handle walk_pud_range() as well, clarify+simplify description, adjust Fixes: ] Fixes: bd74fdaea146 ("mm: multi-gen LRU: support page table walks") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik Reviewed-by: Rik van Riel Reviewed-by: Qi Zheng Reviewed-by: Dev Jain Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Kairui Song Tested-by: Muhammad Usama Anjum Reviewed-by: Harry Yoo (Meta) Reviewed-by: Barry Song Reviewed-by: Baolin Wang Reviewed-by: Pedro Falcato Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: SJ Park Link: https://patch.msgid.link/20261031-fixes_core-v3-1-d6437171ecc3@kernel.org Reviewed-by: Baoquan He Signed-off-by: David Hildenbrand (Arm) --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9959ca158ce8..6a1af9b92349 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3911,7 +3911,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_NONLEAF_FOUND]++; - if (!walk_pte_range(&val, addr, next, args)) + if (!walk_pte_range(pmd + i, addr, next, args)) continue; walk->mm_stats[MM_NONLEAF_ADDED]++; @@ -3947,7 +3947,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, if (!pud_present(val) || WARN_ON_ONCE(pud_leaf(val))) continue; - walk_pmd_range(&val, addr, next, args); + walk_pmd_range(pud + i, addr, next, args); if (need_resched() || walk->batched >= MAX_LRU_BATCH) { end = (addr | ~PUD_MASK) + 1; From ef694e2d210111cc08c52fd25da9ab130170bfa3 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:31 +0100 Subject: [PATCH 17/61] fork: take mmap_lock before copying mm_struct dup_mm() copies the parent's mm_struct before dup_mmap() takes the mmap_lock. A concurrent brk() can leave the child with a stale mm->brk and newer heap mappings, causing later brk() requests to fail. Other operations (e.g., PR_SET_MM) on the parent's mm_struct can similarly cause inconsistencies. As a simple and robust fix, grab the parent's mmap_lock earlier. This way we don't need to update individual mm fields in dup_mmap() and keep missing ones. Stop updating the 4 existing members (total_vm, data_vm, exec_vm, stack_vm), as that is now no longer required. Keep releasing the parent's mmap_lock in dup_mmap(), before userfaultfd completion, to avoid moving the userfaultfd handling into the caller. This should get cleaned up later. Move uprobe_start_dup_mmap() to preserve the lock ordering. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Rik van Riel Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Barry Song Reviewed-by: Pedro Falcato Reviewed-by: Harry Yoo (Meta) Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: SJ Park Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-2-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- kernel/fork.c | 20 ++++++++++++++------ mm/mmap.c | 8 +------- mm/nommu.c | 2 +- 3 files changed, 16 insertions(+), 14 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 8b108ed37152..5efceb8d58a7 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1533,12 +1533,19 @@ static struct mm_struct *dup_mm(struct task_struct *tsk, if (!mm) goto fail_nomem; + uprobe_start_dup_mmap(); + if (mmap_write_lock_killable(oldmm)) { + free_mm(mm); + goto fail_uprobe; + } + memcpy(mm, oldmm, sizeof(*mm)); - if (!mm_init(mm, tsk)) - goto fail_nomem; + if (!mm_init(mm, tsk)) { + mmap_write_unlock(oldmm); + goto fail_uprobe; + } - uprobe_start_dup_mmap(); err = dup_mmap(mm, oldmm); if (err) goto free_pt; @@ -1557,9 +1564,10 @@ static struct mm_struct *dup_mm(struct task_struct *tsk, mm->binfmt = NULL; mm_init_owner(mm, NULL); mmput(mm); - if (err) - uprobe_end_dup_mmap(); - + if (!err) + return NULL; +fail_uprobe: + uprobe_end_dup_mmap(); fail_nomem: return NULL; } diff --git a/mm/mmap.c b/mm/mmap.c index 148ba01c0373..7618e72766a5 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1713,8 +1713,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) LIST_HEAD(uf); VMA_ITERATOR(vmi, mm, 0); - if (mmap_write_lock_killable(oldmm)) - return -EINTR; + mmap_assert_write_locked(oldmm); flush_cache_dup_mm(oldmm); uprobe_dup_mmap(oldmm, mm); /* @@ -1725,11 +1724,6 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) /* No ordering required: file already has been exposed. */ dup_mm_exe_file(mm, oldmm); - mm->total_vm = oldmm->total_vm; - mm->data_vm = oldmm->data_vm; - mm->exec_vm = oldmm->exec_vm; - mm->stack_vm = oldmm->stack_vm; - /* Use __mt_dup() to efficiently build an identical maple tree. */ retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL); if (unlikely(retval)) diff --git a/mm/nommu.c b/mm/nommu.c index 91ab65afda70..6c6f95329674 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1887,7 +1887,7 @@ subsys_initcall(init_admin_reserve); int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) { - mmap_write_lock(oldmm); + mmap_assert_write_locked(oldmm); dup_mm_exe_file(mm, oldmm); mmap_write_unlock(oldmm); return 0; From d3710111839d11339e414b1bb3fea3d05ab83f6e Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:32 +0100 Subject: [PATCH 18/61] mm/shmem: make shmem_file() check the file's inode shmem_file() only checks if file->f_mapping is a shmem mapping. This mapping can belong to another file, however, as with a Coda file backed by a tmpfs container after mmap(). So it does not guarantee that the file inode is an actual shmem inode. Seal handling and memfd preservation use this check before accessing shmem inode state, so they can access an inode of the wrong type. Let's make shmem_file() check the inode's file operations instead, to identify shmem inodes reliably. This will identify regular shmem inodes, including memfds and ordinary tmpfs files, independently of wrapper file operations. Use shmem_mapping() in collapse_file() instead to not change the behavior of collapse_file(): it operates on the page cache and uses mapping->host for shmem inode operations. Focusing on the file does not seem relevant. Fixes: c49fcfcda8b5 ("mm/shmem: update file sealing comments and file checking") Fixes: b3749f174d68 ("mm: memfd_luo: allow preserving memfd") Cc: stable@vger.kernel.org Reported-by: Chris Mason Suggested-by: Pratyush Yadav Reviewed-by: Pratyush Yadav (Google) Link: https://patch.msgid.link/20261031-fixes_core-v3-3-d6437171ecc3@kernel.org Reviewed-by: Baolin Wang Signed-off-by: David Hildenbrand (Arm) --- include/linux/shmem_fs.h | 15 ++++++--------- mm/khugepaged.c | 2 +- mm/shmem.c | 5 +++++ 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index a7c7a96a7cbf..f89b4b151133 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -117,8 +117,14 @@ extern unsigned long shmem_get_unmapped_area(struct file *, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); extern int shmem_lock(struct file *file, int lock, struct ucounts *ucounts); #ifdef CONFIG_SHMEM +bool shmem_file(struct file *file); bool shmem_mapping(const struct address_space *mapping); #else +static inline bool shmem_file(struct file *file) +{ + return false; +} + static inline bool shmem_mapping(const struct address_space *mapping) { return false; @@ -193,15 +199,6 @@ static inline struct page *shmem_read_mapping_page( mapping_gfp_mask(mapping)); } -static inline bool shmem_file(struct file *file) -{ - if (!IS_ENABLED(CONFIG_SHMEM)) - return false; - if (!file || !file->f_mapping) - return false; - return shmem_mapping(file->f_mapping); -} - /* Must be called with inode lock taken exclusive. */ static inline void shmem_freeze(struct inode *inode, bool freeze) { diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4b71d010b2af..3a6f4b7b2845 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2230,6 +2230,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, struct file *file, pgoff_t start, struct collapse_control *cc) { struct address_space *mapping = file->f_mapping; + const bool is_shmem = shmem_mapping(mapping); struct page *dst; struct folio *folio, *tmp, *new_folio; unsigned long new_pfn = -1; @@ -2238,7 +2239,6 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); enum scan_result result = SCAN_SUCCEED; int nr_none = 0; - bool is_shmem = shmem_file(file); /* * MADV_COLLAPSE ignores shmem huge config, so do not check shmem diff --git a/mm/shmem.c b/mm/shmem.c index 923957e5e6a2..e64704ac7222 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -276,6 +276,11 @@ static const struct vm_operations_struct shmem_vm_ops; static const struct vm_operations_struct shmem_anon_vm_ops; static struct file_system_type shmem_fs_type; +bool shmem_file(struct file *file) +{ + return file && file_inode(file)->i_fop == &shmem_file_operations; +} + bool shmem_mapping(const struct address_space *mapping) { return mapping->a_ops == &shmem_aops; From 5426cd43a4d28f1041c9ca45955121b921f29548 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:33 +0100 Subject: [PATCH 19/61] mm/memfd: make f_pos accesses atomic Commit 9c225f2655e3 ("vfs: atomic f_pos accesses as per POSIX") made read(), write() and lseek() on a shared open file description of a regular file serialize on ->f_pos_lock. The flag that asks for it, FMODE_ATOMIC_POS, is set in do_dentry_open(). A memfd is a regular file too, but its struct file comes from alloc_file_pseudo() and never passes through do_dentry_open(), so it does not get the flag: CPU0 CPU1 write(memfd, "a", 1) write(memfd, "b", 1) pos = file->f_pos (= 0) pos = file->f_pos (= 0) vfs_write(..., &pos) vfs_write(..., &pos) file->f_pos = pos (= 1) file->f_pos = pos (= 1) One of the two bytes is overwritten. Two threads writing 100000 bytes each, one at a time, to a memfd end up with a file that is far shorter than 200000 bytes, while the same program on a tmpfs file never loses a byte. Set the flag when the memfd is created. As for any other regular file the lock is only taken if the file is reachable in more than one way, and nothing a memfd does in read or write waits for another operation on the same file. This cannot be done for all alloc_file_pseudo() files: anon inode files such as eventfd are S_IFREG as well, and a blocked read holding ->f_pos_lock would lock out the write that is supposed to wake it. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 9183df25fe7b ("shm: add memfd_create() syscall") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik [ david: minor description tweaks ] Reviewed-by: Rik van Riel Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Signed-off-by: Gregory Price (Meta) Reviewed-by: Mike Rapoport (Microsoft) Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Pedro Falcato Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: Pasha Tatashin Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-4-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/memfd.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/memfd.c b/mm/memfd.c index 0f6fff004f5e..0e3ff3eebfa8 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -482,6 +482,8 @@ struct file *memfd_alloc_file(const char *name, unsigned int flags) } file->f_mode |= FMODE_LSEEK | FMODE_PREAD | FMODE_PWRITE; + /* a regular file like any other: POSIX.1-2008/SUSv4 Section XSI 2.9.7 */ + file->f_mode |= FMODE_ATOMIC_POS; file->f_flags |= O_LARGEFILE; if (flags & MFD_NOEXEC_SEAL) { From 9cf632e139737314afa889ca18122bc3fddb9120 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:34 +0100 Subject: [PATCH 20/61] mm/truncate: look again if a folio left the mapping invalidate_inode_pages2_range() skips a folio which no longer belongs to the mapping by the time it has got the folio lock, on the assumption that somebody else has removed it from the page cache. But the folio may have been replaced, and then there is a new folio with the same data at that index. For most cases, like folio migration, that is impossible. However, collapse_file() only checks the refcount of the old folios, under the i_pages lock, and relies on their folio locks from there on. That works for users which repeat the lookup when they find folio->mapping changed. The following scenario is, however, possible: CPU0 pwrite(O_DIRECT) CPU1 collapse_file() kiocb_invalidate_pages() read the folio (old data), lock it refcount check the write completes copy to new_folio kiocb_invalidate_post_direct_write() find_get_entries() -> old folio xas_store(new_folio) folio->mapping = NULL folio_unlock(folio) folio_lock() folio->mapping != mapping -> continue In that scenario, the collapsed folio with the data from before the write stays in the page cache after the write has returned. This used to be impossible, as khugepaged only collapsed files that nobody had open for writing. Restart the lookup at that index, like shmem_undo_range() does. The same problem seems to exist in truncate_inode_pages_range() as well when truncate_inode_folio() fails, so handle it there as well, but slightly differently: simply check once more through find_get_entries(). Fixes: 1b6444331beb ("mm/khugepaged: enable clean pagecache folio collapse for writable files") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Rik van Riel Tested-by: Muhammad Usama Anjum Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: Zi Yan Link: https://patch.msgid.link/20261031-fixes_core-v3-5-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/truncate.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 733d2713a875..9e9bf029635c 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -475,8 +475,11 @@ void truncate_inode_pages_range(struct address_space *mapping, } } + if (start >= end) + return; + index = start; - while (index < end) { + while (true) { cond_resched(); if (!find_get_entries(mapping, &index, end - 1, &fbatch, indices)) { @@ -499,6 +502,7 @@ void truncate_inode_pages_range(struct address_space *mapping, folio_lock(folio); VM_BUG_ON_FOLIO(!folio_contains(folio, indices[i]), folio); folio_wait_writeback(folio); + /* Do not handle errors, we'll do another lookup */ truncate_inode_folio(mapping, folio); folio_unlock(folio); } @@ -760,7 +764,12 @@ int invalidate_inode_pages2_range(struct address_space *mapping, folio_lock(folio); if (unlikely(folio->mapping != mapping)) { folio_unlock(folio); - continue; + /* + * Folio might have been replaced: look up + * again. + */ + index = indices[i]; + break; } VM_BUG_ON_FOLIO(!folio_contains(folio, indices[i]), folio); folio_wait_writeback(folio); From dc24e273244e9d712257d5a6c20a32c7f5ca84e6 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:35 +0100 Subject: [PATCH 21/61] mm/filemap: read mapping->a_ops->free_folio under i_lock memory_failure() -> ... -> filemap_remove_folio() can read mapping->a_ops after the inode has been freed. Once the folio is deleted from the pagecache it no longer pins the mapping, so after i_lock is dropped e.g., a concurrent final iput() can evict and free the inode before filemap_free_folio() looks up mapping->a_ops->free_folio. Most callers hold a reference on the inode, but not all. Documentation/filesystems/vfs.rst explicitly states that ->free_folio() must not assume the original address_space mapping still exists. Fix it by loading ->free_folio while i_lock is still held, similar to what folio_unmap_invalidate() does. Inline filemap_free_folio() into its remaining caller such that delete_from_page_cache_batch() also just loads the callback before dropping the i_lock. The bug was found by code inspection and has not been reproduced. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 59c66c5f8c4f ("mm: factor out page cache page freeing into a separate function") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik [ david: get rid of filemap_free_folio() entirely; improve patch description ] Reviewed-by: Rik van Riel Tested-by: Muhammad Usama Anjum Reviewed-by: Zi Yan Link: https://patch.msgid.link/20261031-fixes_core-v3-6-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/filemap.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index b74bc1e5015c..906b013c11eb 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -227,18 +227,6 @@ void __filemap_remove_folio(struct folio *folio, void *shadow) page_cache_delete(mapping, folio, shadow); } -static void filemap_free_folio(const struct address_space *mapping, - struct folio *folio) -{ - void (*free_folio)(struct folio *); - - free_folio = mapping->a_ops->free_folio; - if (free_folio) - free_folio(folio); - - folio_put_refs(folio, folio_nr_pages(folio)); -} - /** * filemap_remove_folio - Remove folio from page cache. * @folio: The folio. @@ -250,6 +238,7 @@ static void filemap_free_folio(const struct address_space *mapping, void filemap_remove_folio(struct folio *folio) { struct address_space *mapping = folio->mapping; + void (*free_folio)(struct folio *folio); BUG_ON(!folio_test_locked(folio)); spin_lock(&mapping->host->i_lock); @@ -258,9 +247,12 @@ void filemap_remove_folio(struct folio *folio) xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_lru_list_add(mapping->host); + free_folio = mapping->a_ops->free_folio; spin_unlock(&mapping->host->i_lock); - filemap_free_folio(mapping, folio); + if (free_folio) + free_folio(folio); + folio_put_refs(folio, folio_nr_pages(folio)); } /* @@ -320,6 +312,7 @@ static void page_cache_delete_batch(struct address_space *mapping, void delete_from_page_cache_batch(struct address_space *mapping, struct folio_batch *fbatch) { + void (*free_folio)(struct folio *folio); int i; if (!folio_batch_count(fbatch)) @@ -337,10 +330,16 @@ void delete_from_page_cache_batch(struct address_space *mapping, xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_lru_list_add(mapping->host); + free_folio = mapping->a_ops->free_folio; spin_unlock(&mapping->host->i_lock); - for (i = 0; i < folio_batch_count(fbatch); i++) - filemap_free_folio(mapping, fbatch->folios[i]); + for (i = 0; i < folio_batch_count(fbatch); i++) { + struct folio *folio = fbatch->folios[i]; + + if (free_folio) + free_folio(folio); + folio_put_refs(folio, folio_nr_pages(folio)); + } } int filemap_check_errors(struct address_space *mapping) From 8ccb6ded6a47b59b4907ed4f8c293cf8122de889 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:36 +0100 Subject: [PATCH 22/61] mm/filemap: don't lose a folio lock wakeup to a signal An exclusive waiter in folio_wait_bit_common() can leave the wait loop because of a fatal signal, then be woken by folio_unlock() before finish_wait(). If the waker did not hand over the lock, the waiter returns -EINTR without trying to acquire it. folio_wake_bit() stops after waking the first exclusive waiter, so this loses the wakeup and can leave other waiters asleep indefinitely on an unlocked folio. After finish_wait(), let a waiter woken without a lock handoff try once more to acquire the lock. If successful, return success so the caller's eventual unlock wakes the next waiter. Otherwise, the current lock owner will do so. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 5ef64cc8987a ("mm: allow a controlled amount of unfairness in the page lock") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik [ david: clarify and simplify comment + description ] Reviewed-by: Rik van Riel Acked-by: Pedro Falcato Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-7-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/filemap.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 906b013c11eb..6b5e1d866a9e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1373,10 +1373,22 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, * return value based on that state without races. * * Also note that WQ_FLAG_WOKEN is sufficient for a non-exclusive - * waiter, but an exclusive one requires WQ_FLAG_DONE. + * waiter, but an exclusive one must also acquire the bit. */ - if (behavior == EXCLUSIVE) - return wait->flags & WQ_FLAG_DONE ? 0 : -EINTR; + if (behavior == EXCLUSIVE) { + if (wait->flags & WQ_FLAG_DONE) + return 0; + /* + * We might have just been woken up and the waker will not + * wake up anybody else. If we ignore that now, the remaining + * waiters may not get woken up: take the bit now if it is + * still free. + */ + if ((wait->flags & WQ_FLAG_WOKEN) && + !test_and_set_bit(bit_nr, folio_flags(folio, 0))) + return 0; + return -EINTR; + } return wait->flags & WQ_FLAG_WOKEN ? 0 : -EINTR; } From 97156a6c355c21f7122421915f05461bd11699a4 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:37 +0100 Subject: [PATCH 23/61] mm/filemap: finish fault cleanup before dropping the fault lock On a retried fault, a fatal signal interrupting the folio lock wait makes lock_folio_maybe_drop_mmap() drop the fault lock without pinning the file. However, filemap_fault() may still need to release mapping->invalidate_lock. A concurrent munmap() can release the last file reference and free the mapping before that unlock. Defer dropping the fault lock to filemap_fault()'s retry cleanup, after releasing invalidate_lock. If the fault lock was already dropped for I/O, release the existing file reference instead. Keep the fault lock held for FAULT_FLAG_RETRY_NOWAIT as before. Fixes: 730633f0b7f9 ("mm: Protect operations adding pages to page cache with invalidate_lock") Cc: stable@vger.kernel.org Reported-by: Chris Mason Reviewed-by: Thadeu Lima de Souza Cascardo Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-8-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/filemap.c | 37 +++++++++++++------------------------ 1 file changed, 13 insertions(+), 24 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6b5e1d866a9e..b04be64a5069 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3308,11 +3308,11 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start, * @folio - the folio to lock. * @fpin - the pointer to the file we may pin (or is already pinned). * - * This works similar to lock_folio_or_retry in that it can drop the - * mmap_lock. It differs in that it actually returns the folio locked - * if it returns 1 and 0 if it couldn't lock the folio. If we did have - * to drop the mmap_lock then fpin will point to the pinned file and - * needs to be fput()'ed at a later point. + * Returns 1 with the folio locked, or 0 if locking failed. + * + * On both entry and return, a non-NULL *fpin means the fault lock has + * already been dropped and the caller must eventually fput() the file. + * Otherwise the fault lock is still held, including on failure. */ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, struct file **fpin) @@ -3329,21 +3329,9 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, return 0; *fpin = maybe_unlock_mmap_for_io(vmf, *fpin); - if (vmf->flags & FAULT_FLAG_KILLABLE) { - if (__folio_lock_killable(folio)) { - /* - * We didn't have the right flags to drop the - * fault lock, but all fault_handlers only check - * for fatal signals if we return VM_FAULT_RETRY, - * so we need to drop the fault lock here and - * return 0 if we don't have a fpin. - */ - if (*fpin == NULL) - release_fault_lock(vmf); - return 0; - } - } else - __folio_lock(folio); + if (vmf->flags & FAULT_FLAG_KILLABLE) + return !__folio_lock_killable(folio); + __folio_lock(folio); return 1; } @@ -3572,7 +3560,7 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) * vma->vm_mm->mmap_lock must be held on entry. * * If our return value has VM_FAULT_RETRY set, it's because the mmap_lock - * may be dropped before doing I/O or by lock_folio_maybe_drop_mmap(). + * may be dropped before doing I/O or when aborting an interrupted wait. * * If our return value does not have VM_FAULT_RETRY set, the mmap_lock * has not been released. @@ -3728,9 +3716,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) out_retry: /* - * We dropped the mmap_lock, we need to return to the fault handler to - * re-find the vma and come back and find our hopefully still populated - * page. + * With fpin set, the fault lock was already dropped. Otherwise it is + * still held. Release invalidate_lock before dropping either protection. */ if (!IS_ERR(folio)) folio_put(folio); @@ -3738,6 +3725,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) filemap_invalidate_unlock_shared(mapping); if (fpin) fput(fpin); + else if (!(vmf->flags & FAULT_FLAG_RETRY_NOWAIT)) + release_fault_lock(vmf); return ret | VM_FAULT_RETRY; } EXPORT_SYMBOL(filemap_fault); From f3c493d2f42b5f44a5c134d8d7ea7f960caf5e2c Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:38 +0100 Subject: [PATCH 24/61] mm/rmap: handle device-exclusive entries in folio_referenced_one() page_vma_mapped_walk() can return non-present device-exclusive entries, but folio_referenced_one() passes them to PTE young-bit accessors that require present PTEs. We must neither call clear_flush_young_ptes_notify() nor folio_pte_batch() with non-present entries. For example, clearing the young bit on non-present PTEs can corrupt the entry: on arm64 PTE_AF overlaps the swap type, so clearing it turns a device-exclusive entry into an ordinary swap entry. folio_pte_batch() will VM_WARN_ON_FOLIO. Apply the same approach as the previous fixes to e.g., damon_ptep_mkold(): treat these entries as old from the CPU's perspective and obtain their young state through the MMU notifier. Skip PTE batching and MGLRU look-around for non-present entries, and don't modify non-present PTEs. Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Reviewed-by: Lorenzo Stoakes (ARM) Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Harry Yoo (Meta) Reviewed-by: Barry Song Reviewed-by: Dev Jain Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-9-d6437171ecc3@kernel.org Reviewed-by: Baolin Wang Signed-off-by: David Hildenbrand (Arm) --- mm/rmap.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1131b76bbbc2..0faef46806e5 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -924,6 +924,7 @@ static bool folio_referenced_one(struct folio *folio, DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); int ptes = 0, referenced = 0; unsigned int nr; + pte_t pteval; while (page_vma_mapped_walk(&pvmw)) { address = pvmw.address; @@ -968,10 +969,12 @@ static bool folio_referenced_one(struct folio *folio, return false; } - if (pvmw.pte && folio_test_large(folio)) { + if (pvmw.pte) + pteval = ptep_get(pvmw.pte); + + if (pvmw.pte && pte_present(pteval) && folio_test_large(folio)) { const unsigned long end_addr = pmd_addr_end(address, vma->vm_end); const unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT; - pte_t pteval = ptep_get(pvmw.pte); nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr); } @@ -981,11 +984,23 @@ static bool folio_referenced_one(struct folio *folio, * are. —they could be on active/inactive lists or on MGLRU. So the * simplest approach is to disable this look-around optimization. */ - if (lru_gen_enabled() && !lru_gen_switching() && pvmw.pte) { + if (lru_gen_enabled() && !lru_gen_switching() && pvmw.pte && + pte_present(pteval)) { if (lru_gen_look_around(&pvmw, nr)) referenced++; } else if (pvmw.pte) { - if (clear_flush_young_ptes_notify(vma, address, pvmw.pte, nr)) + bool young = false; + + /* + * PFN swap PTEs, such as device-exclusive ones, that + * actually map pages are "old" from a CPU perspective. + * The MMU notifier takes care of any device aspects. + */ + if (pte_present(pteval)) + young = clear_flush_young_ptes(vma, address, pvmw.pte, nr); + young |= mmu_notifier_clear_flush_young(vma->vm_mm, address, + address + nr * PAGE_SIZE); + if (young) referenced++; } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { if (pmdp_clear_flush_young_notify(vma, address, From f09171b67ea06a7fe425db33e8b74a92de1426b9 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:39 +0100 Subject: [PATCH 25/61] mm/huge_memory: fix list corruption when deferred_split_scan() races with deferred_split_folio() Since the deferred split queue became a list_lru, the isolate callback moves the folios it took a reference on to an on-stack list, linked through folio->_deferred_list, and deferred_split_scan() takes them off that list with list_del_init() without holding the list_lru lock. deferred_split_folio() does not know about the on-stack list. It only holds the list_lru lock when it looks at the same list_head, and it can run for such a folio: a THP is queued when it is mapped, as a candidate for the underused check, and is queued "again" when a part of it gets unmapped: CPU0 deferred_split_scan() CPU1 deferred_split_folio() list_del_init(&folio->_deferred_list) next->prev = prev prev->next = next entry->next = entry list_lru_lock_irqsave() folio_set_partially_mapped() __list_lru_add() list_empty(item) -> true list_add_tail(item, &l->list) item->next = &l->list item->prev = tail entry->prev = entry The folio now sits on the list_lru with ->prev pointing to itself. The next removal leaves the neighbours pointing to it, and they still do so after the folio has been freed. With CONFIG_DEBUG_LIST: list_del corruption. prev->next should be ffd4000000270090, but was ffd4000000278090. (prev=ffd4000000270090) Fix it by grabbing the lock around list_del_init() so deferred_split_folio() cannot race with it. Fixes: fafaeceb89a5 ("mm: switch deferred split shrinker to list_lru") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Johannes Weiner Reviewed-by: Mani Tofigh Tested-by: Mani Tofigh Reviewed-by: Lance Yang Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-10-d6437171ecc3@kernel.org Reviewed-by: Baolin Wang Signed-off-by: David Hildenbrand (Arm) --- mm/huge_memory.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bd44d58e8684..4c0194af9016 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4522,8 +4522,7 @@ int split_folio_to_list(struct folio *folio, struct list_head *list) * * It is unsafe to call folio_unqueue_deferred_split() until folio refcount is * zero: because even when the list_lru lock is held, a non-empty - * _deferred_list might be in use on deferred_split_scan()'s unlocked - * on-stack list. + * _deferred_list might be in use on deferred_split_scan()'s on-stack list. * * The list_lru sublist is determined by folio's memcg: it is therefore * important to unqueue deferred split before changing folio memcg. @@ -4665,10 +4664,22 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, deferred_split_isolate, &dispose); list_for_each_entry_safe(folio, next, &dispose, _deferred_list) { + struct mem_cgroup *memcg; + struct list_lru_one *lru; bool did_split = false; bool underused = false; + /* + * Sync with deferred_split_folio(), which can requeue the + * folio once list_empty() is true, and sets PG_partially_mapped. + */ + rcu_read_lock(); + memcg = folio_memcg(folio); + lru = list_lru_lock_irq(&deferred_split_lru, folio_nid(folio), + &memcg); list_del_init(&folio->_deferred_list); + list_lru_unlock_irq(lru); + rcu_read_unlock(); if (!folio_test_partially_mapped(folio)) { /* From d90bdc02e36ec08d8248ebc6cdc52e12043eb28c Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:40 +0100 Subject: [PATCH 26/61] mm/huge_memory: properly handle refcounted PUD mappings on fork copy_huge_pud() copies device-DAX entries without taking a folio reference or updating mapcounts and the child's RSS. For example, MADV_GUARD_INSTALL sets VM_MAYBE_GUARD on a device-DAX VMA, forcing copy_page_range() to traverse the whole VMA on fork. Any remaining huge PUDs are then copied without accounting for the new mappings. Unmapping them in the child decrements accounting that was never added. Use vm_normal_page_pud() to identify refcounted mappings. Take a folio reference and add the child's PUD rmap and RSS accounting before installing the copied entry. Pass src and dst VMAs consistently instead of src and dst MM. Move the TODO. Fixes: aed877c2b4257 ("device/dax: properly refcount device dax pages when mapping") Cc: stable@vger.kernel.org Reviewed-by: Zi Yan Reviewed-by: Pedro Falcato Reviewed-by: Lance Yang Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-11-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- include/linux/huge_mm.h | 5 ++--- mm/huge_memory.c | 27 +++++++++++++++++++-------- mm/memory.c | 4 +--- 3 files changed, 22 insertions(+), 14 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c745f7ad2298..fac05120ab77 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -12,9 +12,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); bool huge_pmd_set_accessed(struct vm_fault *vmf); -int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, - pud_t *dst_pud, pud_t *src_pud, unsigned long addr, - struct vm_area_struct *vma); +int copy_huge_pud(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, + pud_t *dst_pud, pud_t *src_pud, unsigned long addr); #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4c0194af9016..283c4bc8b927 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2072,11 +2072,13 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pud(vma, addr, pud); } -int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, - pud_t *dst_pud, pud_t *src_pud, unsigned long addr, - struct vm_area_struct *vma) +int copy_huge_pud(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, + pud_t *dst_pud, pud_t *src_pud, unsigned long addr) { + struct mm_struct *dst_mm = dst_vma->vm_mm; + struct mm_struct *src_mm = src_vma->vm_mm; spinlock_t *dst_ptl, *src_ptl; + struct page *page; pud_t pud; int ret; @@ -2089,11 +2091,20 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pud_trans_huge(pud))) goto out_unlock; - /* - * TODO: once we support anonymous pages, use - * folio_try_dup_anon_rmap_*() and split if duplicating fails. - */ - if (vma_is_cow_mapping(vma) && pud_write(pud)) { + page = vm_normal_page_pud(src_vma, addr, pud); + if (page) { + struct folio *folio = page_folio(page); + + folio_get(folio); + /* + * TODO: once we support anonymous pages, use + * folio_try_dup_anon_rmap_*() and split if duplicating fails. + */ + folio_add_file_rmap_pud(folio, page, dst_vma); + add_mm_counter(dst_mm, mm_counter_file(folio), HPAGE_PUD_NR); + } + + if (vma_is_cow_mapping(src_vma) && pud_write(pud)) { pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } diff --git a/mm/memory.c b/mm/memory.c index 0fb676dd2bd6..5392bcb1d24d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1463,7 +1463,6 @@ copy_pud_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, unsigned long end) { struct mm_struct *dst_mm = dst_vma->vm_mm; - struct mm_struct *src_mm = src_vma->vm_mm; pud_t *src_pud, *dst_pud; unsigned long next; @@ -1477,8 +1476,7 @@ copy_pud_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, int err; VM_BUG_ON_VMA(next-addr != HPAGE_PUD_SIZE, src_vma); - err = copy_huge_pud(dst_mm, src_mm, - dst_pud, src_pud, addr, src_vma); + err = copy_huge_pud(dst_vma, src_vma, dst_pud, src_pud, addr); if (err == -ENOMEM) return -ENOMEM; if (!err) From e6a07d29b0a60f3a4cfe77fe34eeb4b3fb38d6c4 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:39:42 +0100 Subject: [PATCH 27/61] mm/huge_memory: fix truncate racing with PMD zapping zap_huge_pmd_folio() removes the rmap before calling folio_mark_dirty(). If this was the last mapping, truncation can skip unmapping the folio and proceed without waiting for the PMD lock, racing with dirtying. ifs_set_range_dirty() can oops if MADV_DONTNEED races with ftruncate() on XFS, when we get the folio set dirty after it was found to not be mapped anymore. Move folio_mark_dirty() and folio_mark_accessed() before rmap removal, matching the PTE zap path. This keeps the folio mapped while updating its state, so truncation must wait for the page table lock. Fix generated by an LLM and sent by Chris Mason in private. Fixes: fe6cf984939d ("mm/huge_memory: transfer the pmd dirty bit to the folio on zap") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason [ david: simplify comment and description ] Reviewed-by: Gregory Price (Meta) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-13-d6437171ecc3@kernel.org Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: David Hildenbrand (Arm) --- mm/huge_memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 283c4bc8b927..ce52962db16b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2495,10 +2495,6 @@ static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct *vma, { const bool is_device_private = folio_is_device_private(folio); - /* Present and device private folios are rmappable. */ - if (is_present || is_device_private) - folio_remove_rmap_pmd(folio, &folio->page, vma); - if (folio_test_anon(folio)) { add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR); } else { @@ -2512,6 +2508,13 @@ static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct *vma, folio_mark_accessed(folio); } + /* + * Present and device private folios are rmappable. Remove the + * rmap after transferring young+dirty to the folio. + */ + if (is_present || is_device_private) + folio_remove_rmap_pmd(folio, &folio->page, vma); + /* Device private folios are pinned. */ if (is_device_private) folio_put(folio); From 63a940541358d09cc9f02d40b71eb8d3917a21c0 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:39:43 +0100 Subject: [PATCH 28/61] mm/huge_memory: fix zapping migration entries for device-private THPs A PMD migration entry can refer to a device-private folio. zap_huge_pmd_folio() checks the folio type instead of the entry type, so it treats such a migration entry as a device-private entry. Installing the migration entry already removed the reverse mapping and dropped the mapping reference. Zapping it must not repeat those operations. Concurrent MADV_DONTNEED during migration back to system memory can trigger this. Check the PMD entry type using pmd_is_device_private_entry(). Fix generated by an LLM and sent by Chris Mason in private. Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason [ david: simplify description ] Reviewed-by: Gregory Price (Meta) Link: https://patch.msgid.link/20261031-fixes_core-v3-14-d6437171ecc3@kernel.org Reviewed-by: Zi Yan Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- mm/huge_memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ce52962db16b..75b09a06edb2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2493,7 +2493,7 @@ static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd) static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct *vma, pmd_t pmdval, struct folio *folio, bool is_present) { - const bool is_device_private = folio_is_device_private(folio); + const bool is_device_private = pmd_is_device_private_entry(pmdval); if (folio_test_anon(folio)) { add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR); From aa72cf34d90da7019591ab67533a8ece8b41da2d Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:44 +0100 Subject: [PATCH 29/61] mm/khugepaged: always recheck the pmd in try_collapse_pte_mapped_thp() try_collapse_pte_mapped_thp() drops the PTE lock after it has cleared the page table, and takes the PMD lock and the PTE lock to remove it. In between, MADV_DONTNEED with CONFIG_PT_RECLAIM can reclaim the page table. The pmd_same() check is supposed to handle this, but it is only done with CONFIG_SPLIT_PTE_PTLOCKS. Fix it by performing the check unconditionally. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 6375e95f381e ("mm: pgtable: reclaim empty PTE page in madvise(MADV_DONTNEED)") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik [ david: adjust description ] Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-15-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Reviewed-by: Baolin Wang Signed-off-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 3a6f4b7b2845..81496887d827 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2013,12 +2013,11 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); - if (ptl != pml) { + if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); - if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { - flush_tlb_mm(mm); - goto unlock; - } + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + flush_tlb_mm(mm); + goto unlock; } } pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); From db38b902307e687219ec765ef51a6ef459746c87 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:45 +0100 Subject: [PATCH 30/61] mm/khugepaged: fix PMD table lock leak in try_collapse_pte_mapped_thp() For a private mapping with userfaultfd, try_collapse_pte_mapped_thp() takes the PMD table lock before it maps the page table for the second time. If that (pte_offset_map_rw_nolock) fails, because the page table vanished in the meantime due to concurrent page table reclaim, it leaves through the abort label, where the PMD table lock is unlocked if it differs from PTE table lock. In case the PMD table lock and the PTE table lock match, which is the case without CONFIG_SPLIT_PTE_PTLOCKS after pte_offset_map_rw_nolock() failed, we miss to unlock the PMD table. To fix it, forget the PTE table lock when we unmap+unlock the PTE table. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 6375e95f381e ("mm: pgtable: reclaim empty PTE page in madvise(MADV_DONTNEED)") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik [ david: move the ptl = NULL handling; rewrite description, drop wrong comments ] Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-16-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Reviewed-by: Baolin Wang Signed-off-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 81496887d827..34ebb37c862c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1904,7 +1904,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); - if (!start_pte) /* mmap_lock + page lock should prevent this */ + if (!start_pte) goto drop_folio; /* step 1: check all mapped PTEs are to the right huge page */ @@ -1935,6 +1935,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign } pte_unmap_unlock(start_pte, ptl); + ptl = NULL; mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, haddr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); @@ -1952,7 +1953,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign pml = pmd_lock(mm, pmd); start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); - if (!start_pte) /* mmap_lock + page lock should prevent this */ + if (!start_pte) goto abort; if (!pml) spin_lock(ptl); From fe2817e3d5ea347214342c9a5cd8bcbff20c0574 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:39:46 +0100 Subject: [PATCH 31/61] mm/khugepaged: preserve dirty bits when collapsing PTE-mapped THPs try_collapse_pte_mapped_thp() clears PTEs without transferring their dirty bits to the folio. A shmem read fault can map a clean folio writable, so a later store only dirties the PTE. Losing that dirty bit allows reclaim to discard modified data. The assumption that regular files are read-only is also obsolete since commit 1b6444331bebb ("mm/khugepaged: enable clean pagecache folio collapse for writable files"). In any case, transferring PTE dirty bits to the folio when removing mappings (and while the folio is still mapped) is the right thing to do. Use get_and_clear_ptes() to collect the dirty bits and mark the folio dirty before unmapping it. Update the comment. This was reproduced with shmem using MADV_COLLAPSE followed by MADV_PAGEOUT. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP") Fixes: 1b6444331bebb ("mm/khugepaged: enable clean pagecache folio collapse for writable files") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason [ david: adjust comment; simplify description; cover file-thp as well ] Reviewed-by: Pedro Falcato Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-17-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Reviewed-by: Baolin Wang Signed-off-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 34ebb37c862c..ce794d9760a8 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1995,9 +1995,10 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign /* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. - * PTE dirty? Shmem page is already dirty; file is read-only. */ - clear_ptes(mm, addr, pte, nr_batch_ptes); + ptent = get_and_clear_ptes(mm, addr, pte, nr_batch_ptes); + if (pte_dirty(ptent)) + folio_mark_dirty(folio); folio_remove_rmap_ptes(folio, page, nr_batch_ptes, vma); nr_mapped_ptes += nr_batch_ptes; } From 79cb51d538abc459f74e046d27797d36b133ce0e Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:48 +0100 Subject: [PATCH 32/61] mm/gup: recheck the PUD under PTL in follow_pud_mask() follow_pud_mask() reads the PUD without a lock. If it is a leaf, it takes the lock and calls follow_huge_pud(), which reads the entry again, but only tests whether it is present. A leaf can be zapped concurrently and a page fault can fill the empty entry with a page table. This currently only applies when device-dax makes use of PUD entries, and the fault after e.g., MADV_DONTNEED would find THPs to be disabled. That requires a sequence like 1) Fault in the PUD 2) Disable THP 3) Have GUP race against concurrent MADV_DONTNEED + refault Which is rather unlikely to happen in practice, which is why we never received a report about such a problem. Before commit 1b1676180246 ("mm/gup: handle huge pud for follow_pud_mask()"), we performed that check in follow_devmap_pud() by testing for pud_devmap(). Let's fix it by just structuring follow_pud_mask() similar to follow_pmd_mask(), minus protnone stuff that does not apply and minus FOLL_SPLIT_PMD handling. That makes it much easier to spot any such inconsistencies. Perform the pud_bad() check early in follow_pmd_mask() and just forward the PUD value. We can drop the pud_present() check in follow_huge_pud(), just like we have in follow_huge_pmd(). Even though we are performing more changes than strictly necessary, the fix should be easy to backport. Fixes: 1b1676180246 ("mm/gup: handle huge pud for follow_pud_mask()") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-19-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/gup.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7d43ce717b24..f809361227e7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -658,9 +658,6 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, assert_spin_locked(pud_lockptr(mm, pudp)); - if (!pud_present(pud)) - return NULL; - if ((flags & FOLL_WRITE) && !can_follow_write_pud(pud, pfn_to_page(pfn), vma, flags)) return NULL; @@ -896,15 +893,17 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, } static struct page *follow_pmd_mask(struct vm_area_struct *vma, - unsigned long address, pud_t *pudp, - unsigned int flags, - unsigned long *page_mask) + unsigned long address, pud_t *pudp, pud_t pud, + unsigned int flags, unsigned long *page_mask) { pmd_t *pmd, pmdval; spinlock_t *ptl; struct page *page; struct mm_struct *mm = vma->vm_mm; + if (unlikely(pud_bad(pud))) + return no_page_table(vma, flags, address); + pmd = pmd_offset(pudp, address); pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) @@ -953,18 +952,23 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = pudp_get(pudp); if (!pud_present(pud)) return no_page_table(vma, flags, address); - if (pud_leaf(pud)) { - ptl = pud_lock(mm, pudp); - page = follow_huge_pud(vma, address, pudp, flags, page_mask); + if (!pud_leaf(pud)) + return follow_pmd_mask(vma, address, pudp, pud, flags, page_mask); + + ptl = pud_lock(mm, pudp); + pud = *pudp; + if (unlikely(!pud_present(pud))) { spin_unlock(ptl); - if (page) - return page; return no_page_table(vma, flags, address); } - if (unlikely(pud_bad(pud))) - return no_page_table(vma, flags, address); + if (unlikely(!pud_leaf(pud))) { + spin_unlock(ptl); + return follow_pmd_mask(vma, address, pudp, pud, flags, page_mask); + } - return follow_pmd_mask(vma, address, pudp, flags, page_mask); + page = follow_huge_pud(vma, address, pudp, flags, page_mask); + spin_unlock(ptl); + return page; } static struct page *follow_p4d_mask(struct vm_area_struct *vma, From c83ee2fa5a94edb2e04102b9dd4f1ee0eeb2b932 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:39:49 +0100 Subject: [PATCH 33/61] mm/gup: fix clearing folios in memfd_pin_folios() memfd_pin_folios() pins folios obtained directly from the page cache without checking whether they are uptodate. However, shmem fallocate() can leave newly allocated folios uncleared and not uptodate. Callers such as udmabuf can then expose uninitialized memory to userspace. Clear non-uptodate folios and mark them uptodate under the folio lock before returning them to the caller. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 89c1905d9c14 ("mm/gup: introduce memfd_pin_folios() for pinning memfd folios") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason [ david: simplify description and comment ] Reviewed-by: Pratyush Yadav (Google) Reviewed-by: Samiullah Khawaja Tested-by: Samiullah Khawaja Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-20-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- mm/gup.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index f809361227e7..cc4fa18783ce 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3527,6 +3527,20 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, goto err; } + /* + * fallocate() can leave folios in the + * page cache that were never cleared. + */ + if (!folio_test_uptodate(folio)) { + folio_lock(folio); + if (!folio_test_uptodate(folio)) { + folio_zero_range(folio, 0, folio_size(folio)); + flush_dcache_folio(folio); + folio_mark_uptodate(folio); + } + folio_unlock(folio); + } + if (nr_folios == 0) *offset = offset_in_folio(folio, start); From 1dfe4b276c23189394afb42d114ee8a54cc3aff2 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:39:50 +0100 Subject: [PATCH 34/61] mm/gup: fix out-of-bounds write for nr_pages == 0 in GUP-fast With nr_pages == 0, GUP-fast still enters its page table walk and can write past the caller's page array. The old end <= start check caught empty ranges, but was replaced with an overflow check. Users such as OrangeFS can pass a zero page count derived from a userspace-supplied length. So let's return zero for nr_pages == 0 before starting the walk, just like GUP-slow. Fix generated by an LLM and sent by Chris Mason in private. Fixes: c28b1fc70390 ("mm/gup: reorganize internal_get_user_pages_fast()") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason [ david: simplify description ] Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-21-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/gup.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index cc4fa18783ce..1c2ac7c66c30 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3215,6 +3215,9 @@ static int gup_fast_fallback(unsigned long start, unsigned long nr_pages, FOLL_PCI_P2PDMA | FOLL_HONOR_NUMA_FAULT))) return -EINVAL; + if (unlikely(!nr_pages)) + return 0; + if (gup_flags & FOLL_PIN) mm_set_has_pinned_flag(current->mm); From 21a3888d3d2f87b7f15ccea1f0befd57346fe899 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:51 +0100 Subject: [PATCH 35/61] mm/gup: check for overflow when converting nr_pages to bytes in GUP-fast The overflow check in gup_fast_fallback() only covers the addition to start. The preceding nr_pages << PAGE_SHIFT can already have truncated the length on 32bit. Users such as RapidIO's rio_dma_transfer() can derive such page counts from a 64-bit userspace length. Let's just check_shl_overflow() to reject any such overflow with -EOVERFLOW early. Fixes: c28b1fc70390 ("mm/gup: reorganize internal_get_user_pages_fast()") Cc: stable@vger.kernel.org Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Reviewed-by: Barry Song Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-22-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1c2ac7c66c30..ba4907b35232 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3225,8 +3225,8 @@ static int gup_fast_fallback(unsigned long start, unsigned long nr_pages, might_lock_read(¤t->mm->mmap_lock); start = untagged_addr(start) & PAGE_MASK; - len = nr_pages << PAGE_SHIFT; - if (check_add_overflow(start, len, &end)) + if (check_shl_overflow(nr_pages, PAGE_SHIFT, &len) || + check_add_overflow(start, len, &end)) return -EOVERFLOW; if (end > TASK_SIZE_MAX) return -EFAULT; From 35f7ecd55ce7e51cb6d5ef49e6dd51f3c80f61e4 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:52 +0100 Subject: [PATCH 36/61] mm/userfaultfd: write-lock the VMA before waking waiters when unregistering userfaultfd_unregister() wakes pending missing faults before clearing the VMA's userfaultfd registration. With per-VMA fault locking, holding mmap_lock for write does not prevent another fault from queuing after the wakeup but before userfaultfd_clear_vma() write-locks the VMA. That waiter can then remain asleep indefinitely. Call vma_start_write() before waking waiters. Existing fault handlers queue themselves before dropping the VMA read lock, and new readers are excluded until mmap_write_unlock(). Fixes: 29a22b9e08d7 ("mm: handle userfaults under VMA lock") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Suren Baghdasaryan Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-23-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/userfaultfd.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 928ad95eeb2b..3ca04b582329 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -3991,6 +3991,9 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, start = vma->vm_start; vma_end = min(end, vma->vm_end); + /* Prevent new userfaults from being queued after the wakeup. */ + vma_start_write(vma); + if (userfaultfd_missing(vma)) { /* * Wake any concurrent pending userfault while From 3afb19d68cc5e7851fdb1069c8094af97bd80417 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:53 +0100 Subject: [PATCH 37/61] mm/userfaultfd: wake all waiters on unregister UFFDIO_UNREGISTER only wakes pending faults if VM_UFFD_MISSING is set. Threads waiting on write-protect, minor or RWP faults can therefore remain asleep after unregistering a range that does not track missing faults. Wake pending faults regardless of the registration mode so they can retry without an explicit UFFDIO_WAKE or closing the userfaultfd. This matches the existing behavior for missing faults, including removing unread fault messages for the woken waiters. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik [ david: rebase; simplify description ] Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Lorenzo Stoakes (ARM) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-24-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/userfaultfd.c | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 3ca04b582329..be630674fda9 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -3977,6 +3977,8 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, ret = 0; for_each_vma_range(vmi, vma, end) { + struct userfaultfd_wake_range range; + cond_resched(); /* VMA not registered with userfaultfd. */ @@ -3994,18 +3996,14 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, /* Prevent new userfaults from being queued after the wakeup. */ vma_start_write(vma); - if (userfaultfd_missing(vma)) { - /* - * Wake any concurrent pending userfault while - * we unregister, so they will not hang - * permanently and it avoids userland to call - * UFFDIO_WAKE explicitly. - */ - struct userfaultfd_wake_range range; - range.start = start; - range.len = vma_end - start; - wake_userfault(vma->vm_userfaultfd_ctx.ctx, &range); - } + /* + * Wake any concurrent pending userfault while we unregister, + * so they will not hang permanently and it avoids userland to + * call UFFDIO_WAKE explicitly. + */ + range.start = start; + range.len = vma_end - start; + wake_userfault(ctx, &range); vma = userfaultfd_clear_vma(&vmi, prev, vma, start, vma_end); From 4eee9c389b072e161549d46dcb1ae10b0222717e Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:54 +0100 Subject: [PATCH 38/61] mm/userfaultfd: wake waiters when registration modes are removed UFFDIO_REGISTER can replace MISSING or MINOR tracking without waking faults that are already queued. For example, switching an anonymous range from MISSING to WP leaves a missing-fault waiter asleep until userspace explicitly wakes it. When a registration change removes a UFFD mode, wake waiters in the affected interval after updating the VMA. The VMA is write-locked at this point, so no new faults can queue under the old modes and woken threads cannot retry until mmap_write_unlock(). Waiters for retained modes in the same interval may also wake and refault. The hang reproduces for MISSING-to-WP and MINOR-to-MISSING transitions, with both read and unread fault events. Fixes: 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl") Cc: stable@vger.kernel.org Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Signed-off-by: Gregory Price (Meta) Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Lorenzo Stoakes (ARM) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-25-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/userfaultfd.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index be630674fda9..03f1b15bfe63 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -2270,6 +2270,9 @@ static struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi, return ret; } +static void wake_userfault_range(struct userfaultfd_ctx *ctx, + unsigned long start, unsigned long end); + /* Assumes mmap write lock taken, and mm_struct pinned. */ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx, struct vm_area_struct *vma, @@ -2287,6 +2290,8 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx, prev = vma; for_each_vma_range(vmi, vma, end) { + const bool any_mode_removed = vma->vm_flags & __VM_UFFD_FLAGS & ~vm_flags; + cond_resched(); VM_WARN_ON_ONCE(!vma_can_userfault(vma, vm_flags, wp_async)); @@ -2332,6 +2337,10 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx, */ userfaultfd_set_ctx(vma, ctx, vm_flags); + /* Retry faults for removed modes while the VMA is write-locked. */ + if (any_mode_removed) + wake_userfault_range(ctx, start, vma_end); + if (vma_is_hugetlb(vma) && uffd_disable_huge_pmd_share(vma)) hugetlb_unshare_all_pmds(vma); @@ -3648,6 +3657,17 @@ static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx, __wake_userfault(ctx, range); } +static void wake_userfault_range(struct userfaultfd_ctx *ctx, + unsigned long start, unsigned long end) +{ + struct userfaultfd_wake_range range = { + .start = start, + .len = end - start, + }; + + wake_userfault(ctx, &range); +} + static __always_inline int validate_unaligned_range( struct mm_struct *mm, __u64 start, __u64 len) { From 05f64f211a25f7d9771d4fde109aad97174a1cf3 Mon Sep 17 00:00:00 2001 From: Gregory Price Date: Sat, 31 Oct 2026 10:39:55 +0100 Subject: [PATCH 39/61] mm/userfaultfd: compare exact registration modes before skipping Changing registration from MISSING|WP to WP returns success but leaves MISSING enabled. The no-op check sees that WP is already set and skips the update, so later accesses to missing pages still wait for userspace. Skip the update only when the current and requested UFFD modes match exactly. Fixes: 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl") Cc: stable@vger.kernel.org Signed-off-by: Gregory Price (Meta) [ david: apply fixup from Lorenzo ] Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-26-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/userfaultfd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 03f1b15bfe63..dac696bda19f 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -2291,6 +2291,7 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx, for_each_vma_range(vmi, vma, end) { const bool any_mode_removed = vma->vm_flags & __VM_UFFD_FLAGS & ~vm_flags; + const vma_flags_t curr_vma_flags = vma_flags_and_mask(&vma->flags, __VMA_UFFD_FLAGS); cond_resched(); @@ -2304,7 +2305,7 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx, * userfaultfd and with the right tracking mode too. */ if (vma->vm_userfaultfd_ctx.ctx == ctx && - vma_test_all_mask(vma, vma_flags)) + vma_flags_same_pair(&curr_vma_flags, &vma_flags)) goto skip; /* From 64ae7959a1df8af8eec504e31df2ec9670424a4e Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:56 +0100 Subject: [PATCH 40/61] mm/mmu_gather: fix flushing ranges with 4 GiB or larger __tlb_adjust_range() takes the size of the range which has to be flushed as an unsigned int. Its callers, tlb_flush_{pte,pmd,pud,p4d}_range(), get and pass an unsigned long. Consequently, a range that is 4 GiB or larger gets truncated, possibly to 0. On arm64 with hugetlb, a 64k kernel can have 16 GiB hugetlb folios mapped contiguously at the PMD level and flush them when unmapping, and a 16k kernel can flush 64 GiB PUDs through tlb_unshare_pmd_ptdesc(). So at least these two scenarios are affected. Fix it by flushing the proper range instead of the truncated one by properly using "unsigned long" for range_size. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: b528e4b6405b ("mm/hugetlb: add tlb_remove_hugetlb_entry for handling hugetlb pages") Fixes: 8ce720d5bd91 ("mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik Acked-by: Lorenzo Stoakes (ARM) [ david: use two-tab indent, simplify description ] Reviewed-by: Qi Zheng Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Reviewed-by: Dev Jain Reviewed-by: Muchun Song Reviewed-by: Pedro Falcato Reviewed-by: SJ Park Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-27-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- include/asm-generic/tlb.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index fbe1114e9a44..3f4a180b48f3 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -350,8 +350,7 @@ struct mmu_gather { void tlb_flush_mmu(struct mmu_gather *tlb); static inline void __tlb_adjust_range(struct mmu_gather *tlb, - unsigned long address, - unsigned int range_size) + unsigned long address, unsigned long range_size) { tlb->start = min(tlb->start, address); tlb->end = max(tlb->end, address + range_size); From 6d65925777ecae9d7a571a327e41c63f3940a0c7 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:39:57 +0100 Subject: [PATCH 41/61] mm/mmu_notifier: finish non-blocking interval invalidation on error When an ordinary MMU notifier rejects a non-blocking invalidation, the interval invalidation started beforehand is left unfinished: the caller skips mmu_notifier_invalidate_range_end(). Subsequent interval notifier operations can then wait forever. The only in-tree trigger is nouveau SVM, which uses both notifier types and rejects non-blocking invalidations. Reaping via the OOM reaper or process_mrelease() can therefore leave the GPU fault worker stuck, holding a reference that prevents the dying process's address space and memory from being freed. End the interval invalidation when an ordinary notifier fails a non-blocking invalidation, like mn_itree_invalidate() already does when an interval notifier fails. Make that handling clearer in the documentation of mmu_notifier_invalidate_range_start_nonblock(). Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 99cb252f5e68 ("mm/mmu_notifier: add an interval tree notifier") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik [ david: simplify description, update function doc ] Acked-by: Lorenzo Stoakes (ARM) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-28-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- include/linux/mmu_notifier.h | 3 ++- mm/mmu_notifier.c | 17 +++++++++++++---- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index a11a44eef521..5db78e5761c5 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -483,7 +483,8 @@ mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) /* * This version of mmu_notifier_invalidate_range_start() avoids blocking, but it * can return an error if a notifier can't proceed without blocking, in which - * case you're not allowed to modify PTEs in the specified range. + * case you're not allowed to modify PTEs in the specified range or call + * mmu_notifier_invalidate_range_end(). * * This is mainly intended for OOM handling. */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index df69ba6e797f..2e20f88e162d 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -569,16 +569,25 @@ int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { struct mmu_notifier_subscriptions *subscriptions = range->mm->notifier_subscriptions; - int ret; + const bool has_itree = subscriptions->has_itree; + int ret = 0; - if (subscriptions->has_itree) { + if (has_itree) { ret = mn_itree_invalidate(subscriptions, range); if (ret) return ret; } if (!hlist_empty(&subscriptions->list)) - return mn_hlist_invalidate_range_start(subscriptions, range); - return 0; + ret = mn_hlist_invalidate_range_start(subscriptions, range); + + /* + * On error, the non-blocking caller is not allowed to call + * invalidate_range_end(), as in mn_itree_invalidate(). + */ + if (ret && has_itree && !mmu_notifier_range_blockable(range)) + mn_itree_inv_end(subscriptions); + + return ret; } static void From 5ad9e4b0f1104f5dbfb41125c34bb30ea6dc7a94 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:39:58 +0100 Subject: [PATCH 42/61] mm/mmu_notifier: set ->mm only after mmgrab() Radeon uses a non-NULL notifier->mm to decide whether to call mmu_interval_notifier_remove() when freeing a buffer object. __mmu_interval_notifier_insert() sets ->mm before validating the range, so a wrapping userptr range can leave ->mm set after insertion fails. Cleanup then tries to remove an unregistered notifier for which no mm reference was acquired. Set ->mm only after mmgrab(), once insertion can no longer fail. While we could fix radeon's code instead, setting ->mm on success only makes the code more robust. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 3506ff69c3ec ("drm/radeon: use mmu_interval_notifier_insert") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason [ david: simplify description ] Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Reviewed-by: Lorenzo Stoakes (ARM) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-29-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/mmu_notifier.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 2e20f88e162d..a7bae5de5fed 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -947,7 +947,6 @@ static int __mmu_interval_notifier_insert( struct mmu_notifier_subscriptions *subscriptions, unsigned long start, unsigned long length, const struct mmu_interval_notifier_ops *ops) { - interval_sub->mm = mm; interval_sub->ops = ops; RB_CLEAR_NODE(&interval_sub->interval_tree.rb); interval_sub->interval_tree.start = start; @@ -964,8 +963,12 @@ static int __mmu_interval_notifier_insert( if (WARN_ON(atomic_read(&mm->mm_users) <= 0)) return -EINVAL; - /* pairs with mmdrop in mmu_interval_notifier_remove() */ + /* + * Pairs with mmdrop in mmu_interval_notifier_remove(). Drivers take a + * non-NULL ->mm for "registered", so set it only with the reference. + */ mmgrab(mm); + interval_sub->mm = mm; /* * If some invalidate_range_start/end region is going on in parallel From b4417d08034a7b4a3083e0e2d95429d4542524f2 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:39:59 +0100 Subject: [PATCH 43/61] mm/mremap: use pudp_huge_get_and_clear() when moving huge PUDs move_huge_pud() reads the old PUD and clears it in separate operations. Hardware accessed or dirty updates between those operations can be lost when the saved entry is installed at the new address. The raw pud_clear() also bypasses page_table_check_pud_clear(), while set_pud_at() accounts for the destination mapping. This leaves the checker counts unbalanced for mappings tracked by page table checking. To fix it, simply use pudp_huge_get_and_clear(). Fixes: 7d846db7d0e2 ("mm/mremap: convert huge PUD move to separate helper") Cc: stable@vger.kernel.org Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Pedro Falcato Reviewed-by: Dev Jain Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-30-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- mm/mremap.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 0d06d7f37f19..16e736c72d9c 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -509,8 +509,7 @@ static bool move_huge_pud(struct pagetable_move_control *pmc, spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); /* Clear the pud */ - pud = *old_pud; - pud_clear(old_pud); + pud = pudp_huge_get_and_clear(mm, pmc->old_addr, old_pud); VM_BUG_ON(!pud_none(*new_pud)); From 402882943dfa8bab5c2bcc66189f9fb2828f248e Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:40:00 +0100 Subject: [PATCH 44/61] mm/mremap: recheck PUDs under the page table lock Once get_old_pud() is fixed to accept PUD leaves, unmap_mapping_range() can clear a leaf between the lookup and the move. Holding mmap_lock in write mode does not prevent this. If the zap happens before move_page_tables() tests the entry, an empty PUD can reach move_normal_pud(), where pud_populate() turns it into a present page table entry. If it happens after the test, move_huge_pud() can instead try to move an empty entry, triggering CONFIG_DEBUG_VM warnings on powerpc. Recheck the source under the page table lock in both helpers. Treat an empty source in move_normal_pud() as a completed move, and use __pud_trans_huge_lock() in move_huge_pud(). In the future we should likely have the code lookup a PUD/PMD lockless once, instead of multiple times, because one of the problematic bits here is the: pud_trans_huge(*old_pud) Fixes: 0881ace292b6 ("mm/mremap: use pmd/pud_poplulate to update page table entries") Cc: stable@vger.kernel.org Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Pedro Falcato Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-31-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- mm/mremap.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 16e736c72d9c..91f0865f6aee 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -461,14 +461,18 @@ static bool move_normal_pud(struct pagetable_move_control *pmc, if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); - /* Clear the pud */ pud = *old_pud; + if (unlikely(pud_none(pud))) + goto out_unlock; + + /* Clear the pud */ pud_clear(old_pud); VM_BUG_ON(!pud_none(*new_pud)); pud_populate(mm, new_pud, pud_pgtable(pud)); flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + PUD_SIZE); +out_unlock: if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); @@ -503,7 +507,9 @@ static bool move_huge_pud(struct pagetable_move_control *pmc, * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pud_lock(mm, old_pud); + old_ptl = __pud_trans_huge_lock(old_pud, vma); + if (unlikely(!old_ptl)) + return false; new_ptl = pud_lockptr(mm, new_pud); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); From 440c8779757f09f290770b1d361dc1516a9d0b3e Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:40:01 +0100 Subject: [PATCH 45/61] mm/mremap: fix moving PUD leaves pud_none_or_clear_bad() only really works on page tables. PUD leaves confuse the check and make us clear the page table entry and print a warning. Since commit aed877c2b4257 ("device/dax: properly refcount device dax pages when mapping"), this also leaks device-DAX folio references, mapcounts and RSS accounting, resulting in "Bad rss-counter state" warnings on exit. Allow PUD leaves in get_old_pud(). If a PUD leaf cannot be moved as a whole or the move fails, split it. This unmaps it with proper accounting instead of treating the leaf entry as a PMD table. Keep using pud_trans_huge() for now, in the future this code should be cleaned up to just check for PUD leaves. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik Reviewed-by: Lorenzo Stoakes (ARM) [ david: fallthrough after splitting ] Reviewed-by: Pedro Falcato Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-32-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- mm/mremap.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 91f0865f6aee..1237718848f5 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -87,7 +87,7 @@ static pud_t *get_old_pud(struct mm_struct *mm, unsigned long addr) return NULL; pud = pud_offset(p4d, addr); - if (pud_none_or_clear_bad(pud)) + if (!pud_trans_huge(*pud) && pud_none_or_clear_bad(pud)) return NULL; return pud; @@ -847,11 +847,10 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) if (!new_pud) break; if (pud_trans_huge(*old_pud)) { - if (extent == HPAGE_PUD_SIZE) { - move_pgt_entry(pmc, HPAGE_PUD, old_pud, new_pud); - /* We ignore and continue on error? */ + if (extent == HPAGE_PUD_SIZE && + move_pgt_entry(pmc, HPAGE_PUD, old_pud, new_pud)) continue; - } + split_huge_pud(pmc->old, old_pud, pmc->old_addr); } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) && extent == PUD_SIZE) { if (move_pgt_entry(pmc, NORMAL_PUD, old_pud, new_pud)) continue; From 22d043c1da48d6c7274f9dd900714b6d58b7b9c6 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:40:03 +0100 Subject: [PATCH 46/61] mm/madvise: fix data loss from MADV_FREE on a KSM page MADV_FREE can clear the dirty flag of a singly mapped KSM folio. The folio remains in the stable tree and can gain new mappings without being dirtied again. Once marked lazyfree, reclaim can discard data that those mappings still need. KSM always marks a folio dirty when converting it to a KSM folio in try_to_merge_one_page(). Marking a folio lazyfree clears PG_swapbacked but leaves PG_dirty unchanged. Preserving PG_dirty makes reclaim restore PG_swapbacked instead of discarding the folio. This also protects folios that were already lazyfree when KSM converted them. Skip KSM folios before clearing their dirty flag, checking under the folio lock to serialize with KSM conversion. Fixes: 854e9ed09ded ("mm: support madvise(MADV_FREE)") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Xu Xin Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-34-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/madvise.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/madvise.c b/mm/madvise.c index 75b409e97dbf..a16e3cb256b7 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -756,6 +756,15 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { if (!folio_trylock(folio)) continue; + /* + * KSM folios can gain new mappings at any time, so we must + * not clear their dirty flag. + */ + if (folio_test_ksm(folio)) { + folio_unlock(folio); + continue; + } + /* * If we have a large folio at this point, we know it is * fully mapped so if its mapcount is the same as its From 8db6921cc07b74197a3866d62ed262b68aeadcf3 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:40:04 +0100 Subject: [PATCH 47/61] mm/memory: reject remote COW faults on PFN mappings When handling a COW fault without a normal source page (PFN mappings), __wp_page_copy_user() copies from the faulting user address. For remote faults, this accesses the wrong address space, resulting in wrong data getting copied. For example, user_events updates an enable bit in the registering process's memory. If that memory needs to be faulted in, user_event_enabler_fault_fixup() calls fixup_user_fault() from a work item. Placing the enable bit in a private PFN mapping can trigger this remote COW path. Reject these faults with VM_FAULT_SIGBUS when the target mm differs from current->mm: remote faults. Ordinary (non-PFN) mappings are not affected. The user-address copy dates back to commit 6aab341e0a28 ("mm: re-architect the VM_UNPAGED logic"); likely, actual ways to get remote faults in there on COW PFN mappings appeared afterwards. But the fix is simple and straight forward. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 6aab341e0a28 ("mm: re-architect the VM_UNPAGED logic") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) [ david: simplify comment and description ] Reviewed-by: Pedro Falcato Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-35-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 5392bcb1d24d..ae8d567bc6ba 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3761,6 +3761,13 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, return 0; } + /* + * __copy_from_user_inatomic() only works as expected for local faults. + * Reject remote faults. + */ + if (unlikely(mm != current->mm)) + return -EFAULT; + /* * If the source page was a PFN mapping, we don't have * a "struct page" for it. We do a best-effort copy by @@ -4086,6 +4093,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) folio_put(old_folio); delayacct_wpcopy_end(); + if (err == -EFAULT) + return VM_FAULT_SIGBUS; return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0; } kmsan_copy_page_meta(&new_folio->page, vmf->page); From 7a81bc23dade8082187be6d9cc36e4be4b7207e2 Mon Sep 17 00:00:00 2001 From: Josef Bacik Date: Sat, 31 Oct 2026 10:40:05 +0100 Subject: [PATCH 48/61] mm: fix unmap_mapping_range() beyond 4 GiB on 32-bit Commit 9eab0421fa94 ("mm: fix unmap_mapping_range high bits shift bug") made holebegin and holelen unsigned before they are shifted, by casting them to pgoff_t. That is an unsigned long: on a 32-bit kernel the cast cuts off the upper half of the two loff_t values, so a hole which begins at 4 GiB or beyond is taken for one in the first 4 GiB of the file. truncate_pagecache() passes the new size of the file and asks for private COWed pages to be unmapped as well. If a file of 6 GiB is truncated to 5 GiB, everything from 1 GiB on is unmapped, and what a process has written to a private mapping of the range between 1 GiB and 5 GiB is lost: it reads the contents of the file again. Where the range has an end, the wrong range is unmapped instead of the right one. The page cache copes with that, as it unmaps every folio that it removes. The callers which only have unmap_mapping_range() to get rid of user mappings do not, for example drm_vma_node_unmap(), whose fake offsets reach beyond 4 GiB on 32-bit, and revoke_iomem() with physical addresses of more than 32 bits. Cast to u64 instead. Nothing changes for 64-bit kernels. Fix generated by an LLM and sent by Josef Bacik in private. Fixes: 9eab0421fa94 ("mm: fix unmap_mapping_range high bits shift bug") Cc: stable@vger.kernel.org Signed-off-by: Josef Bacik Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Pedro Falcato Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-36-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ae8d567bc6ba..732b8f926de6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4649,8 +4649,8 @@ EXPORT_SYMBOL_GPL(unmap_mapping_pages); void unmap_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen, int even_cows) { - pgoff_t hba = (pgoff_t)(holebegin) >> PAGE_SHIFT; - pgoff_t hlen = ((pgoff_t)(holelen) + PAGE_SIZE - 1) >> PAGE_SHIFT; + pgoff_t hba = (u64)holebegin >> PAGE_SHIFT; + pgoff_t hlen = ((u64)holelen + PAGE_SIZE - 1) >> PAGE_SHIFT; /* Check for overflow. */ if (sizeof(holelen) > sizeof(hlen)) { From 9f02ca8c15a74bc9e4e8e05ec089f63cb87cbf75 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:40:06 +0100 Subject: [PATCH 49/61] mm: fix unkillable fault loop on MAP_DROPPABLE memory handle_mm_fault() suppresses VM_FAULT_OOM for VM_DROPPABLE mappings. For user-mode faults, pending signals are processed before retrying the access. Kernel-mode faults on user buffers instead return directly to the faulting instruction. With persistent allocation failures (esp. memcg limits), a syscall, such as write(), can therefore loop while not dropping locks. Only suppress VM_FAULT_OOM when FAULT_FLAG_USER is set, allowing kernel-mode faults and GUP to fail and signal allocation errors. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 9651fcedf7b9 ("mm: add MAP_DROPPABLE for designating always lazily freeable mappings") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) [ david: simplify description ] Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-37-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/memory.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 732b8f926de6..77d4bbf444bc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6957,8 +6957,12 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, lru_gen_exit_fault(); - /* If the mapping is droppable, then errors due to OOM aren't fatal. */ - if (is_droppable) + /* + * If the mapping is droppable, then errors due to OOM aren't fatal: the + * task returns to user mode, handles its signals and tries again. A + * fault from kernel mode has no such return, so it must see the error. + */ + if (is_droppable && (flags & FAULT_FLAG_USER)) ret &= ~VM_FAULT_OOM; if (flags & FAULT_FLAG_USER) { From d03df42ddae196f20e9cfb775704e60269eb6b44 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:40:07 +0100 Subject: [PATCH 50/61] mm: fix deadlock of mremap() against UFFDIO_MOVE With per-VMA locks, UFFDIO_MOVE can run concurrently with mremap() on different VMAs in the same mm. Those VMAs can share page tables, so both operations can acquire the same pair of page-table locks. UFFDIO_MOVE orders the locks by address, while mremap() takes the source lock first. These conflicting orders can deadlock; mremap() holding mmap_lock for writing no longer helps. Move double_pt_lock() and double_pt_unlock() to mm/internal.h and use them for all page-table moves in mremap(). Preserve the source entry checks after acquiring both locks. Remove the comments claiming that exclusive mmap_lock makes lock ordering irrelevant. In move_ptes(), make sure to just recheck source and destination PMD after taking the locks and drop the overly-long comment. Fix generated by an LLM and sent by Chris Mason in private. [ david: use new helpers consistently; simplify description and comments; move_ptes() fixup ] Fixes: 867a43a34ff8 ("userfaultfd: use per-vma locks in userfaultfd operations") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-38-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- include/linux/userfaultfd_k.h | 2 - mm/huge_memory.c | 19 +++----- mm/hugetlb.c | 14 ++---- mm/internal.h | 28 +++++++++++ mm/mremap.c | 87 ++++++++++++++--------------------- mm/userfaultfd.c | 27 ----------- 6 files changed, 72 insertions(+), 105 deletions(-) diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index a14b8a9ffb7b..565e98f342dc 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -153,8 +153,6 @@ extern int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start, unsigned long len, bool enable_rwp); /* move_pages */ -void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2); -void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2); int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 75b09a06edb2..f10a199548d8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2646,15 +2646,11 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, return false; } - /* - * We don't have to worry about the ordering of src and dst - * ptlocks because exclusive mmap_lock prevents deadlock. - */ - old_ptl = __pmd_trans_huge_lock(old_pmd, vma); - if (old_ptl) { - new_ptl = pmd_lockptr(mm, new_pmd); - if (new_ptl != old_ptl) - spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + old_ptl = pmd_lockptr(mm, old_pmd); + new_ptl = pmd_lockptr(mm, new_pmd); + + double_pt_lock(old_ptl, new_ptl); + if (likely(pmd_is_huge(*old_pmd))) { pmd = pmdp_huge_get_and_clear(mm, old_addr, old_pmd); if (pmd_present(pmd)) force_flush = true; @@ -2683,11 +2679,10 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, set_pmd_at(mm, new_addr, new_pmd, pmd); if (force_flush) flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE); - if (new_ptl != old_ptl) - spin_unlock(new_ptl); - spin_unlock(old_ptl); + double_pt_unlock(old_ptl, new_ptl); return true; } + double_pt_unlock(old_ptl, new_ptl); return false; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index da5d04edf7c1..84141527fe73 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5153,15 +5153,9 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, spinlock_t *src_ptl, *dst_ptl; pte_t pte; - dst_ptl = huge_pte_lock(h, mm, dst_pte); + dst_ptl = huge_pte_lockptr(h, mm, dst_pte); src_ptl = huge_pte_lockptr(h, mm, src_pte); - - /* - * We don't have to worry about the ordering of src and dst ptlocks - * because exclusive mmap_lock (or the i_mmap_lock) prevents deadlock. - */ - if (src_ptl != dst_ptl) - spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + double_pt_lock(dst_ptl, src_ptl); pte = huge_ptep_get_and_clear(mm, old_addr, src_pte, sz); @@ -5190,9 +5184,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, set_huge_pte_at(mm, new_addr, dst_pte, pte, sz); } - if (src_ptl != dst_ptl) - spin_unlock(src_ptl); - spin_unlock(dst_ptl); + double_pt_unlock(dst_ptl, src_ptl); } int move_hugetlb_page_tables(struct vm_area_struct *vma, diff --git a/mm/internal.h b/mm/internal.h index 566cbd2dce48..824c2919a37f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -624,6 +624,34 @@ static inline void force_page_cache_readahead(struct address_space *mapping, force_page_cache_ra(&ractl, nr_to_read); } +/* + * Take two page table locks at the same level in address order such that + * we cannot deadlock. + */ +static inline void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2) + __acquires(ptl1) + __acquires(ptl2) +{ + if (ptl1 > ptl2) + swap(ptl1, ptl2); + spin_lock(ptl1); + if (ptl1 != ptl2) + spin_lock_nested(ptl2, SINGLE_DEPTH_NESTING); + else + __acquire(ptl2); +} + +static inline void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2) + __releases(ptl1) + __releases(ptl2) +{ + spin_unlock(ptl1); + if (ptl1 != ptl2) + spin_unlock(ptl2); + else + __release(ptl2); +} + unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, diff --git a/mm/mremap.c b/mm/mremap.c index 1237718848f5..70b7264aa6bf 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -202,7 +202,7 @@ static int move_ptes(struct pagetable_move_control *pmc, struct mm_struct *mm = vma->vm_mm; pte_t *old_ptep, *new_ptep; pte_t old_pte, pte; - pmd_t dummy_pmdval; + pmd_t old_pmdval, new_pmdval; spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long old_addr = pmc->old_addr; @@ -234,31 +234,30 @@ static int move_ptes(struct pagetable_move_control *pmc, if (pmc->need_rmap_locks) take_rmap_locks(vma); - /* - * We don't have to worry about the ordering of src and dst - * pte locks because exclusive mmap_lock prevents deadlock. - */ - old_ptep = pte_offset_map_lock(mm, old_pmd, old_addr, &old_ptl); + old_ptep = pte_offset_map_rw_nolock(mm, old_pmd, old_addr, &old_pmdval, + &old_ptl); if (!old_ptep) { err = -EAGAIN; goto out; } - /* - * Now new_pte is none, so collapse_scan_file() path can not find - * this by traversing file->f_mapping, so there is no concurrency with - * retract_page_tables(). In addition, we already hold the exclusive - * mmap_lock, so this new_pte page is stable, so there is no need to get - * pmdval and do pmd_same() check. - */ - new_ptep = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, - &new_ptl); + new_ptep = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &new_pmdval, + &new_ptl); if (!new_ptep) { - pte_unmap_unlock(old_ptep, old_ptl); + pte_unmap(old_ptep); err = -EAGAIN; goto out; } - if (new_ptl != old_ptl) - spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + double_pt_lock(old_ptl, new_ptl); + if (unlikely(!pmd_same(old_pmdval, pmdp_get_lockless(old_pmd)) || + !pmd_same(new_pmdval, pmdp_get_lockless(new_pmd)))) { + double_pt_unlock(old_ptl, new_ptl); + pte_unmap(new_ptep); + pte_unmap(old_ptep); + err = -EAGAIN; + goto out; + } + flush_tlb_batched_pending(vma->vm_mm); lazy_mmu_mode_enable(); @@ -317,10 +316,9 @@ static int move_ptes(struct pagetable_move_control *pmc, lazy_mmu_mode_disable(); if (force_flush) flush_tlb_range(vma, old_end - len, old_end); - if (new_ptl != old_ptl) - spin_unlock(new_ptl); + double_pt_unlock(old_ptl, new_ptl); pte_unmap(new_ptep - 1); - pte_unmap_unlock(old_ptep - 1, old_ptl); + pte_unmap(old_ptep - 1); out: if (pmc->need_rmap_locks) drop_rmap_locks(vma); @@ -395,14 +393,9 @@ static bool move_normal_pmd(struct pagetable_move_control *pmc, if (WARN_ON_ONCE(!pmd_none(*new_pmd))) return false; - /* - * We don't have to worry about the ordering of src and dst - * ptlocks because exclusive mmap_lock prevents deadlock. - */ - old_ptl = pmd_lock(mm, old_pmd); + old_ptl = pmd_lockptr(mm, old_pmd); new_ptl = pmd_lockptr(mm, new_pmd); - if (new_ptl != old_ptl) - spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + double_pt_lock(old_ptl, new_ptl); pmd = *old_pmd; @@ -418,9 +411,7 @@ static bool move_normal_pmd(struct pagetable_move_control *pmc, pmd_populate(mm, new_pmd, pmd_pgtable(pmd)); flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + PMD_SIZE); out_unlock: - if (new_ptl != old_ptl) - spin_unlock(new_ptl); - spin_unlock(old_ptl); + double_pt_unlock(old_ptl, new_ptl); return res; } @@ -452,14 +443,10 @@ static bool move_normal_pud(struct pagetable_move_control *pmc, if (WARN_ON_ONCE(!pud_none(*new_pud))) return false; - /* - * We don't have to worry about the ordering of src and dst - * ptlocks because exclusive mmap_lock prevents deadlock. - */ - old_ptl = pud_lock(mm, old_pud); + old_ptl = pud_lockptr(mm, old_pud); new_ptl = pud_lockptr(mm, new_pud); - if (new_ptl != old_ptl) - spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + double_pt_lock(old_ptl, new_ptl); pud = *old_pud; if (unlikely(pud_none(pud))) @@ -473,9 +460,7 @@ static bool move_normal_pud(struct pagetable_move_control *pmc, pud_populate(mm, new_pud, pud_pgtable(pud)); flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + PUD_SIZE); out_unlock: - if (new_ptl != old_ptl) - spin_unlock(new_ptl); - spin_unlock(old_ptl); + double_pt_unlock(old_ptl, new_ptl); return true; } @@ -503,16 +488,14 @@ static bool move_huge_pud(struct pagetable_move_control *pmc, if (WARN_ON_ONCE(!pud_none(*new_pud))) return false; - /* - * We don't have to worry about the ordering of src and dst - * ptlocks because exclusive mmap_lock prevents deadlock. - */ - old_ptl = __pud_trans_huge_lock(old_pud, vma); - if (unlikely(!old_ptl)) - return false; + old_ptl = pud_lockptr(mm, old_pud); new_ptl = pud_lockptr(mm, new_pud); - if (new_ptl != old_ptl) - spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + double_pt_lock(old_ptl, new_ptl); + if (unlikely(!pud_trans_huge(*old_pud))) { + double_pt_unlock(old_ptl, new_ptl); + return false; + } /* Clear the pud */ pud = pudp_huge_get_and_clear(mm, pmc->old_addr, old_pud); @@ -523,9 +506,7 @@ static bool move_huge_pud(struct pagetable_move_control *pmc, /* mark soft_ditry when we add pud level soft dirty support */ set_pud_at(mm, pmc->new_addr, new_pud, pud); flush_pud_tlb_range(vma, pmc->old_addr, pmc->old_addr + HPAGE_PUD_SIZE); - if (new_ptl != old_ptl) - spin_unlock(new_ptl); - spin_unlock(old_ptl); + double_pt_unlock(old_ptl, new_ptl); return true; } diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index dac696bda19f..169f681f9849 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1206,33 +1206,6 @@ int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start, return 0; } -void double_pt_lock(spinlock_t *ptl1, - spinlock_t *ptl2) - __acquires(ptl1) - __acquires(ptl2) -{ - if (ptl1 > ptl2) - swap(ptl1, ptl2); - /* lock in virtual address order to avoid lock inversion */ - spin_lock(ptl1); - if (ptl1 != ptl2) - spin_lock_nested(ptl2, SINGLE_DEPTH_NESTING); - else - __acquire(ptl2); -} - -void double_pt_unlock(spinlock_t *ptl1, - spinlock_t *ptl2) - __releases(ptl1) - __releases(ptl2) -{ - spin_unlock(ptl1); - if (ptl1 != ptl2) - spin_unlock(ptl2); - else - __release(ptl2); -} - static inline bool is_pte_pages_stable(pte_t *dst_pte, pte_t *src_pte, pte_t orig_dst_pte, pte_t orig_src_pte, pmd_t *dst_pmd, pmd_t dst_pmdval) From 890e118e95bfc2aae4432ec2a95192d6f9193211 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:40:08 +0100 Subject: [PATCH 51/61] mm/ksm: reset the unstable trees of all nodes Each full scan resets ksm_nr_node_ids unstable tree roots. Enabling merge_across_nodes resets this count to 1, even if per-node roots were previously allocated. So when toggling "merge_across_nodes" 0 -> 1 -> 0, a scan that starts while the setting is 1 and continues after switching back to 0 can encounter stale unstable tree roots, resulting in UAF. Only root is allowed to change "merge_across_nodes", and the documented sequence is: "merge_across_nodes setting can be changed only when there are no ksm shared pages in the system: set run 2 to unmerge pages first, then to 1 after changing merge_across_nodes, to remerge according to the new setting." Let's fix it by tracking in ksm_nr_node_ids the actual number of allocated tree roots. remove_all_stable_nodes() and ksm_check_stable_tree() will now scan more tree roots (skipping the empty ones), which is harmless. Fixes: ef53d16cded7 ("ksm: allocate roots when needed") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Pasha Tatashin Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-39-d6437171ecc3@kernel.org Reviewed-by: Xu Xin Signed-off-by: David Hildenbrand (Arm) --- mm/ksm.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index dbfb8f98b189..9c51925825cb 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -475,6 +475,7 @@ static void advisor_stop_scan(void) #ifdef CONFIG_NUMA /* Zeroed when merging across nodes is not allowed */ static unsigned int ksm_merge_across_nodes = 1; +/* Number of allocated stable and unstable tree roots. */ static int ksm_nr_node_ids = 1; #else #define ksm_merge_across_nodes 1U @@ -3636,14 +3637,13 @@ static ssize_t merge_across_nodes_store(struct kobject *kobj, else { root_stable_tree = buf; root_unstable_tree = buf + nr_node_ids; + ksm_nr_node_ids = nr_node_ids; /* Stable tree is empty but not the unstable */ root_unstable_tree[0] = one_unstable_tree[0]; } } - if (!err) { + if (!err) ksm_merge_across_nodes = knob; - ksm_nr_node_ids = knob ? 1 : nr_node_ids; - } } mutex_unlock(&ksm_thread_mutex); From 4c2dec1dce368ad9e9b1814307cb5ef3832fa07e Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:40:09 +0100 Subject: [PATCH 52/61] mm/ksm: fix use of a freed anon_vma at swap-in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A folio read from a swap slot previously shared by KSM can remain in the swap cache after the process that first mapped it exits. Its anon_vma can then be freed, so ksm_might_need_to_copy() must not dereference it to compare roots. Doing so can result in some false-positives, resulting in the reverse mapping missing the new folio mapping, preventing migration and reclaim. SLAB_TYPESAFE_BY_RCU allows individual anon_vma objects to be reused immediately after freeing. A stale pointer can then refer to another anon_vma whose root matches, even though the target VMA is not linked to that anon_vma. Move the anonymous rmap compatibility checks into folio_anon_rmap_compatible(). Keep the page-index check and compare the folio's anon_vma pointer against the VMA's own anon_vma and root first. If neither matches, bite the bullet and walk the VMA's anon_vma_chain. The VMA keeps these anon_vmas alive, and finding a match guarantees that reverse mapping can find the new mapping. Lorenzo Stoakes says in a private discussion: "This isn't quite equivalent to the existing check. Every anon_vma in an anon_vma_chain shares the same root, but if you split a VMA then fork both of them like this: VMA A / \ VMA B VMA C \ / root <- anon_vma 1 ----^ ^---- / parent \ anon_vma 2 anon_vma 3 | | VMA D VMA E VMA's D and E both get their own new anon_vma's such that: VMA D: anon_vma 2 -> anon_vma 1 -> root VMA E: anon_vma 3 -> anon_vma 1 -> root And if you found a folio referencing anon_vma 2, the existing check: anon_vma->root == vma->anon-vma->root Would be true for VMA's B, C, D & E. However after your change it'd be true for VMA D only, as it's the only one with anon_vma 2 in its anon_vma_chain. Whether or not that could actually happen in practice... 🙂 I don't know. [...] after your change it'd be true for VMA D only, as it's the only one with anon_vma 2 in its anon_vma_chain. [...] I would [...] say that you are making the test stricter but more correct. " If this affects performance when KSM was never even involved, we could just track whether KSM ever merged anything and just skip these checks. There isn't really a lot we can do otherwise: for mapped folios we might be able to use folio_get_anon_vma(), but for unmapped folios there really is no easy way to see if the anon_vma is still valid. Document the (confusing) lifetime requirements of folio_anon_vma(). Fixes: ba6f0ff3981e ("ksm: fix ksm swapin time optimization") Cc: stable@vger.kernel.org Reported-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Xu Xin Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-40-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/ksm.c | 36 +++++++++++++++++++++++++++++++----- mm/util.c | 13 +++++++++++++ 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 9c51925825cb..e050b1bdf810 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -3137,11 +3137,40 @@ void __ksm_exit(struct mm_struct *mm) trace_ksm_exit(mm); } +static bool folio_anon_rmap_compatible(const struct folio *folio, + struct vm_area_struct *vma, unsigned long addr) +{ + const struct anon_vma *anon_vma = folio_anon_vma(folio); + struct anon_vma_chain *avc; + + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_large(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_ksm(folio), folio); + vma_assert_stabilised(vma); + + if (!anon_vma) + return true; + if (folio->index != linear_anon_page_index(vma, addr)) + return false; + + /* + * Careful: anon_vma might have been freed, so we cannot safely + * dereference it. Add a fast path before we walk the anon_vma chain. + */ + if (anon_vma == vma->anon_vma || anon_vma == vma->anon_vma->root) + return true; + + list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) { + if (avc->anon_vma == anon_vma) + return true; + } + return false; +} + struct folio *ksm_might_need_to_copy(struct folio *folio, struct vm_area_struct *vma, unsigned long addr) { struct page *page = folio_page(folio, 0); - struct anon_vma *anon_vma = folio_anon_vma(folio); struct folio *new_folio; if (folio_test_large(folio)) @@ -3151,10 +3180,7 @@ struct folio *ksm_might_need_to_copy(struct folio *folio, if (folio_stable_node(folio) && !(ksm_run & KSM_RUN_UNMERGE)) return folio; /* no need to copy it */ - } else if (!anon_vma) { - return folio; /* no need to copy it */ - } else if (folio->index == linear_anon_page_index(vma, addr) && - anon_vma->root == vma->anon_vma->root) { + } else if (folio_anon_rmap_compatible(folio, vma, addr)) { return folio; /* still no need to copy it */ } if (PageHWPoison(page)) diff --git a/mm/util.c b/mm/util.c index 8acfe2148ba5..80f6c714844f 100644 --- a/mm/util.c +++ b/mm/util.c @@ -697,6 +697,19 @@ void *vcalloc_noprof(size_t n, size_t size) } EXPORT_SYMBOL(vcalloc_noprof); +/** + * folio_anon_vma - Get a folio's raw anon_vma pointer. + * @folio: The folio. + * + * The returned pointer may be stale for an unmapped folio. The caller + * must ensure the anon_vma stays alive before dereferencing it. A folio + * reference or lock alone is insufficient; use folio_get_anon_vma() when + * the anon_vma's lifetime is not otherwise protected. Note that + * folio_get_anon_vma() is only useful for folios that are still mapped. + * + * Return: The raw anon_vma pointer for an anonymous, non-KSM folio, or + * %NULL otherwise. + */ struct anon_vma *folio_anon_vma(const struct folio *folio) { unsigned long mapping = (unsigned long)folio->mapping; From 7f2c000ffb1047b476cbf267f741272c389d3e31 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:40:10 +0100 Subject: [PATCH 53/61] mm/ksm: check large swap-cache folios before mapping them KSM folios are always small, but their shared swap slots can be read back as part of a large folio in the swap cache. ksm_might_need_to_copy() skips large folios, so another process can map a page with an incompatible anon_vma or page index. Reverse mapping then cannot find the new mapping, preventing migration or reclaim. Apply the same compatibility checks to large folios as to small ones. Pass the page corresponding to the swap entry and copy only that page when necessary. In do_swap_page()'s copy path, decrement only that slot's swap count, because the source swap-cache folio can now be large. Fix generated by an LLM and sent by Chris Mason in private. Fixes: f1879e8a0c60 ("mm, swap: never bypass the swap cache even for SWP_SYNCHRONOUS_IO") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-41-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- include/linux/ksm.h | 6 +++--- mm/ksm.c | 22 ++++++++++++---------- mm/memory.c | 5 +++-- mm/swapfile.c | 3 ++- 4 files changed, 20 insertions(+), 16 deletions(-) diff --git a/include/linux/ksm.h b/include/linux/ksm.h index d39d0d5483a2..a0d358a1feb6 100644 --- a/include/linux/ksm.h +++ b/include/linux/ksm.h @@ -91,8 +91,8 @@ static inline void ksm_exit(struct mm_struct *mm) * We'd like to make this conditional on vma->vm_flags & VM_MERGEABLE, * but what if the vma was unmerged while the page was swapped out? */ -struct folio *ksm_might_need_to_copy(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr); +struct folio *ksm_might_need_to_copy(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long addr); void rmap_walk_ksm(struct folio *folio, struct rmap_walk_control *rwc); void folio_migrate_ksm(struct folio *newfolio, struct folio *folio); @@ -145,7 +145,7 @@ static inline int ksm_madvise(struct vm_area_struct *vma, unsigned long start, } static inline struct folio *ksm_might_need_to_copy(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr) + struct page *page, struct vm_area_struct *vma, unsigned long addr) { return folio; } diff --git a/mm/ksm.c b/mm/ksm.c index e050b1bdf810..e95f6b672360 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -3138,19 +3138,20 @@ void __ksm_exit(struct mm_struct *mm) } static bool folio_anon_rmap_compatible(const struct folio *folio, - struct vm_area_struct *vma, unsigned long addr) + const struct page *page, struct vm_area_struct *vma, + unsigned long addr) { const struct anon_vma *anon_vma = folio_anon_vma(folio); struct anon_vma_chain *avc; VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); - VM_WARN_ON_ONCE_FOLIO(folio_test_large(folio), folio); + VM_WARN_ON_ONCE_FOLIO(page_folio(page) != folio, folio); VM_WARN_ON_ONCE_FOLIO(folio_test_ksm(folio), folio); vma_assert_stabilised(vma); if (!anon_vma) return true; - if (folio->index != linear_anon_page_index(vma, addr)) + if (page_pgoff(folio, page) != linear_anon_page_index(vma, addr)) return false; /* @@ -3167,20 +3168,21 @@ static bool folio_anon_rmap_compatible(const struct folio *folio, return false; } -struct folio *ksm_might_need_to_copy(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr) +struct folio *ksm_might_need_to_copy(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long addr) { - struct page *page = folio_page(folio, 0); struct folio *new_folio; - if (folio_test_large(folio)) - return folio; - + /* + * A KSM folio is small, but the slot that it left behind can be swapped + * in as a part of a large folio by one of the processes that shared it. + * So test, and copy, the page of the slot. + */ if (folio_test_ksm(folio)) { if (folio_stable_node(folio) && !(ksm_run & KSM_RUN_UNMERGE)) return folio; /* no need to copy it */ - } else if (folio_anon_rmap_compatible(folio, vma, addr)) { + } else if (folio_anon_rmap_compatible(folio, page, vma, addr)) { return folio; /* still no need to copy it */ } if (PageHWPoison(page)) diff --git a/mm/memory.c b/mm/memory.c index 77d4bbf444bc..91836d217d25 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5096,7 +5096,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * folio->index of non-ksm folios would be nonlinear inside the * anon VMA -- the ksm flag is lost on actual swapout. */ - folio = ksm_might_need_to_copy(folio, vma, vmf->address); + folio = ksm_might_need_to_copy(folio, page, vma, vmf->address); if (unlikely(!folio)) { ret = VM_FAULT_OOM; folio = swapcache; @@ -5278,7 +5278,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(folio != swapcache)) { folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE); folio_add_lru_vma(folio, vma); - folio_put_swap(swapcache, NULL); + folio_put_swap(swapcache, + folio_file_page(swapcache, swp_offset(entry))); } else if (!folio_test_anon(folio)) { /* * We currently only expect !anon folios that are fully diff --git a/mm/swapfile.c b/mm/swapfile.c index 254ce86fa923..71ef9e2efb3a 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2514,7 +2514,8 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, return 0; swapcache = folio; - folio = ksm_might_need_to_copy(folio, vma, addr); + folio = ksm_might_need_to_copy(folio, + folio_file_page(folio, swp_offset(entry)), vma, addr); if (unlikely(!folio)) return -ENOMEM; else if (unlikely(folio == ERR_PTR(-EHWPOISON))) { From 0193e83368022a308d19ee0293f55a4f5d548e40 Mon Sep 17 00:00:00 2001 From: Chris Mason Date: Sat, 31 Oct 2026 10:40:11 +0100 Subject: [PATCH 54/61] mm/vma: call ->may_split() before a partial merge A driver can use the ->may_split() hook to forbid a VMA boundary at certain addresses. A device-dax device is a character device that gives raw access to a range of memory through mmap(). device-dax refuses any address that is not a multiple of the device's alignment through vm_ops->may_split. vma_merge_existing_range() can move boundaries, but does not call vm_ops->may_split. If a range at the edge of a VMA gets the attributes of the adjacent VMA, the function moves only that range over to the neighbour: one VMA shrinks and the other grows. This is a partial merge, and vma_modify() tries it before any split. Take a device-dax VMA that lies next to another VMA of the same file. The user calls mprotect() on the one page of the first VMA that touches the second VMA, and asks for the protection that the second VMA already has. The result is a partial merge. It puts the boundary inside a 1 GiB mapping, which is a single leaf entry at the PUD level. As part of the merge, the kernel looks for a huge PMD to split at the new boundary. That code does not expect a PUD leaf: vma_merge_existing_range() commit_merge() vma_adjust_trans_huge() split_huge_pmd_address() mm_find_pmd() treats the PUD leaf as a pointer to a PMD table, so the "table" is the device memory that the leaf maps __split_huge_pmd() reads a PMD that the user wrote to the device memory folio_put() on the folio that this PMD names Bad. While the code must at some point be taught to split PUD entries as well, the issue here is that we move VMA boundaries such that they are incompatible with the driver. Fix this by calling ->may_split() on the address that a partial merge would move the boundary to, and by abandoning the merge if the driver refuses. vma_modify() then tries the split, which fails with -EINVAL. Factor out existing split handling into check_split() and reuse it. Fix generated by an LLM and sent by Chris Mason in private. Fixes: 38607c62b34b ("fs/dax: properly refcount fs dax pages") Cc: stable@vger.kernel.org Signed-off-by: Chris Mason Reviewed-by: Lorenzo Stoakes (ARM) [ david: improve description; cleanup the code ] Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) Signed-off-by: Gregory Price (Meta) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-42-d6437171ecc3@kernel.org Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- mm/vma.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/mm/vma.c b/mm/vma.c index 1be613044b4a..acdf00346cda 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -516,6 +516,15 @@ void unmap_region(struct unmap_desc *unmap) tlb_finish_mmu(&tlb); } +static int check_split(struct vm_area_struct *vma, unsigned long addr) +{ + if (!vma->vm_ops) + return 0; + if (!vma->vm_ops->may_split) + return 0; + return vma->vm_ops->may_split(vma, addr); +} + /* * __split_vma() bypasses sysctl_max_map_count checking. We use this where it * has already been checked or doesn't make sense to fail. @@ -532,11 +541,9 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); - if (vma->vm_ops && vma->vm_ops->may_split) { - err = vma->vm_ops->may_split(vma, addr); - if (err) - return err; - } + err = check_split(vma, addr); + if (err) + return err; new = vm_area_dup(vma); if (!new) @@ -948,6 +955,10 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( if (vmg->__remove_middle && !can_merge_remove_vma(middle)) return NULL; + /* If middle isn't removed then it's resized, i.e. splits. */ + if (!vmg->__remove_middle && check_split(middle, left_side ? end : start)) + return NULL; + /* * If we merge both VMAs, then next is also deleted. This implies * merge_will_delete_vma also. From 23e570927421d569fdc510d5f3f29542a43eeda5 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:40:12 +0100 Subject: [PATCH 55/61] mm/huge_memory: split huge PUDs when adjusting VMA boundaries PFN mappings such as VFIO PCI BAR mappings can use PUD leaves while allowing VMA splits at page granularity. However, vma_adjust_trans_huge() only handles PMDs. After faulting a writable PUD-sized VFIO BAR mapping, changing its first page to read-only with mprotect() makes the PMD walk treat the PUD leaf as a PMD table. Applying MADV_DONTFORK to the first PMD-sized chunk leaves the PUD spanning two VMAs: the PMD-aligned boundary skips splitting. Walk the page tables once at each adjusted boundary and split a huge PUD or PMD leaf as needed. Handle PUD leaves even when the boundary is PMD-aligned. Fixes: f9e54c3a2f5b7 ("vfio/pci: implement huge_fault support") Cc: stable@vger.kernel.org Tested-by: Muhammad Usama Anjum Reviewed-by: Kiryl Shutsemau (Meta) Link: https://patch.msgid.link/20261031-fixes_core-v3-43-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/huge_memory.c | 70 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f10a199548d8..7de94775a359 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3504,16 +3504,66 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, __split_huge_pmd(vma, pmd, address, freeze); } -static inline void split_huge_pmd_if_needed(struct vm_area_struct *vma, unsigned long address) +static bool pud_split_required(const struct vm_area_struct *vma, + unsigned long address) { + if (!IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)) + return false; + if (IS_ALIGNED(address, HPAGE_PUD_SIZE)) + return false; /* - * If the new address isn't hpage aligned and it could previously - * contain an hugepage: check if we need to split an huge pmd. + * Only selected VMAs support mapping PUDs for now. Lift the + * restriction once this changes. */ - if (!IS_ALIGNED(address, HPAGE_PMD_SIZE) && - range_in_vma(vma, ALIGN_DOWN(address, HPAGE_PMD_SIZE), - ALIGN(address, HPAGE_PMD_SIZE))) - split_huge_pmd_address(vma, address, false); + if (!vma_is_dax(vma) && !vma_test_any(vma, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT)) + return false; + return range_in_vma(vma, ALIGN_DOWN(address, HPAGE_PUD_SIZE), + ALIGN(address, HPAGE_PUD_SIZE)); +} + +static bool pmd_split_required(const struct vm_area_struct *vma, + unsigned long address) +{ + if (IS_ALIGNED(address, HPAGE_PMD_SIZE)) + return false; + + return range_in_vma(vma, ALIGN_DOWN(address, HPAGE_PMD_SIZE), + ALIGN(address, HPAGE_PMD_SIZE)); +} + +static void split_huge_leaf(struct vm_area_struct *vma, + unsigned long address) +{ + const bool split_pud = pud_split_required(vma, address); + const bool split_pmd = pmd_split_required(vma, address); + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + /* + * mmap, VMA and rmap write locks protect the walk; the split helpers + * take the page table locks. + */ + if (!split_pud && !split_pmd) + return; + + pgd = pgd_offset(vma->vm_mm, address); + if (!pgd_present(*pgd)) + return; + + p4d = p4d_offset(pgd, address); + if (!p4d_present(*p4d)) + return; + + pud = pud_offset(p4d, address); + if (pud_leaf(*pud)) + __split_huge_pud(vma, pud, address); + if (!split_pmd || !pud_present(*pud)) + return; + + pmd = pmd_offset(pud, address); + __split_huge_pmd(vma, pmd, address, false); } void vma_adjust_trans_huge(struct vm_area_struct *vma, @@ -3522,14 +3572,14 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, struct vm_area_struct *next) { /* Check if we need to split start first. */ - split_huge_pmd_if_needed(vma, start); + split_huge_leaf(vma, start); /* Check if we need to split end next. */ - split_huge_pmd_if_needed(vma, end); + split_huge_leaf(vma, end); /* If we're incrementing next->vm_start, we might need to split it. */ if (next) - split_huge_pmd_if_needed(next, end); + split_huge_leaf(next, end); } /* From b1ee87802111156e27557e82acc31fa7dfc76e2a Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:40:13 +0100 Subject: [PATCH 56/61] mm/vma: don't leak the charge for stack growth acct_stack_growth() charges stack expansion even without VM_ACCOUNT. This can happen for read-only or MAP_NORESERVE mappings created with MAP_GROWSDOWN. Unmapping these VMAs does not release the charge, leaking Committed_AS. Only check and charge growth for VM_ACCOUNT VMAs, matching mremap(). Unaccounted growth skips the overcommit check while remaining subject to the other stack-growth limits. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Pedro Falcato Reviewed-by: Harry Yoo (Meta) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-44-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/vma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/vma.c b/mm/vma.c index acdf00346cda..695d0be85ac8 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -3391,7 +3391,8 @@ static int acct_stack_growth(struct vm_area_struct *vma, * Overcommit.. This must be the final test, as it will * update security statistics. */ - if (security_vm_enough_memory_mm(mm, grow)) + if (vma_test(vma, VMA_ACCOUNT_BIT) && + security_vm_enough_memory_mm(mm, grow)) return -ENOMEM; return 0; From e8abc61d8a549a91c8fa639f68be2ef13d5fa63b Mon Sep 17 00:00:00 2001 From: Rik van Riel Date: Sat, 31 Oct 2026 10:40:14 +0100 Subject: [PATCH 57/61] mm/cma: don't release CMA pages still in use When a driver calls dma_free_contiguous() before quiescing DMA, the page might still have a reference from the device. put_page_testzero() there returns false, WARN fires, but the code proceeds to free_contig_frozen_range() putting a live page onto buddy and clearing the bitmap. A subsequent allocation can hand the same PFN to a new owner while the original holder still references it. A concurrent put_page() that drops the last reference between the testzero loop and free_contig_frozen_range() can double-queue the page via page->lru, corrupting buddy lists. Avoid the corruption by not freeing a CMA region if any of the pages inside are still in use. Make it explicit in the warning that the driver allowed a leak. While at it, use page_ref_freeze() to make the semantics clearer and switch to pr_warn_ratelimited(). Fixes: 9bda131c6093 ("mm: cma: add cma_alloc_frozen{_compound}()") Cc: stable@vger.kernel.org Signed-off-by: Rik van Riel Link: https://patch.msgid.link/20260811132045.5f114772@fangorn [ david: use page_ref_freeze() + pr_warn_ratelimited() ] Acked-by: Lorenzo Stoakes (ARM) Acked-by: Muchun Song Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-45-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/cma.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index a10ea37a261d..871f95c9ae9f 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -1020,7 +1020,7 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned long count) { struct cma_memrange *cmr; - unsigned long ret = 0; + unsigned long leaked = 0; unsigned long i, pfn; cmr = find_cma_memrange(cma, pages, count); @@ -1029,9 +1029,13 @@ bool cma_release(struct cma *cma, const struct page *pages, pfn = page_to_pfn(pages); for (i = 0; i < count; i++, pfn++) - ret += !put_page_testzero(pfn_to_page(pfn)); + leaked += !page_ref_freeze(pfn_to_page(pfn), 1); - WARN(ret, "%lu pages are still in use!\n", ret); + if (leaked) { + pr_warn_ratelimited("%lu pages are still in use, not freeing CMA region!\n", + leaked); + return true; + } __cma_release_frozen(cma, cmr, pages, count); From 500624f28eca441e35f978979ccfb373202fefd3 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Sat, 31 Oct 2026 10:40:15 +0100 Subject: [PATCH 58/61] mm/memory-tiers: unregister empty tiers outside memory_tier_lock When offlining a node's last memory empties its tier, clear_node_memory_tier() calls device_unregister() while holding memory_tier_lock. A concurrent sysfs read can hold a kernfs active reference and block in nodelist_show() on that mutex. device_unregister() then waits in kernfs_drain() for the read to finish and release its active reference, while the reader waits for the offlining task to release the mutex. Return the node's former tier from clear_node_memory_tier(), making the helper more similar to set_node_memory_tier(), which also returns a memory_tier pointer. Let the caller unlink an empty tier under memory_tier_lock before rebuilding demotion targets, then unregister its device after dropping the lock. Fixes: 9832fb87834e ("mm/demotion: expose memory tier details via sysfs") Cc: stable@vger.kernel.org Reported-by: Josef Bacik Reviewed-by: Gregory Price (Meta) Tested-by: Muhammad Usama Anjum Link: https://patch.msgid.link/20261031-fixes_core-v3-46-d6437171ecc3@kernel.org Signed-off-by: David Hildenbrand (Arm) --- mm/memory-tiers.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 25e121851b58..8eda313cf619 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -572,21 +572,14 @@ static struct memory_tier *set_node_memory_tier(int node) return memtier; } -static void destroy_memory_tier(struct memory_tier *memtier) +static struct memory_tier *clear_node_memory_tier(int node) { - list_del(&memtier->list); - device_unregister(&memtier->dev); -} - -static bool clear_node_memory_tier(int node) -{ - bool cleared = false; pg_data_t *pgdat; struct memory_tier *memtier; pgdat = NODE_DATA(node); if (!pgdat) - return false; + return NULL; /* * Make sure that anybody looking at NODE_DATA who finds @@ -604,14 +597,10 @@ static bool clear_node_memory_tier(int node) synchronize_rcu(); memtype = node_memory_types[node].memtype; node_clear(node, memtype->nodes); - if (nodes_empty(memtype->nodes)) { + if (nodes_empty(memtype->nodes)) list_del_init(&memtype->tier_sibling); - if (list_empty(&memtier->memory_types)) - destroy_memory_tier(memtier); - } - cleared = true; } - return cleared; + return memtier; } static void release_memtype(struct kref *kref) @@ -884,13 +873,21 @@ static int __meminit memtier_hotplug_callback(struct notifier_block *self, { struct memory_tier *memtier; struct node_notify *nn = _arg; + bool empty = false; switch (action) { case NODE_REMOVED_LAST_MEMORY: mutex_lock(&memory_tier_lock); - if (clear_node_memory_tier(nn->nid)) + memtier = clear_node_memory_tier(nn->nid); + if (memtier) { + empty = list_empty(&memtier->memory_types); + if (empty) + list_del(&memtier->list); establish_demotion_targets(); + } mutex_unlock(&memory_tier_lock); + if (empty) + device_unregister(&memtier->dev); break; case NODE_ADDED_FIRST_MEMORY: mutex_lock(&memory_tier_lock); From 466d6844c8f19fba16b668b49861bdb8d7ded282 Mon Sep 17 00:00:00 2001 From: "Ritesh Harjani (IBM)" Date: Fri, 30 Oct 2026 05:37:36 +0530 Subject: [PATCH 59/61] mm/huge_memory: set the mmu_gather page size in zap_huge_pud() Unmapping a device-dax mapping with 1G alignment on powerpc radix kernel built with CONFIG_DEBUG_VM triggers this warning on every munmap: WARNING: mm/mmu_gather.c:176 at __tlb_remove_folio_pages_size+0x38/0x230 NIP [c00000000063d508] __tlb_remove_folio_pages_size+0x38/0x230 LR [c0000000006fb688] zap_huge_pud+0x1b8/0x2d0 Call Trace: zap_huge_pud+0x170/0x2d0 (unreliable) __zap_vma_range+0x1604/0x1e20 unmap_vmas+0xa0/0x1f0 unmap_region+0xb0/0x130 vms_complete_munmap_vmas+0x120/0x310 do_vmi_align_munmap+0x1ec/0x240 __vm_munmap+0x148/0x220 sys_munmap+0x28/0x40 To trigger this - it is enough to mmap() /dev/dax0.0, touch the first byte so that the whole 1G is mapped by a single PUD and then munmap() it. PowerPC needs MMU_GATHER_PAGE_SIZE in which mmu_gather needs to record the page size for which tlb flush is needed. zap_huge_pmd(), zap_pte_range(), change_huge_pud() etc all do this already. However, since commit: dbe54153296d ("mm/huge_memory: add vmf_insert_folio_pud()") a device-dax PUD maps a real folio and zap_huge_pud() now hands it to tlb_remove_page_size(tlb, page, HPAGE_PUD_SIZE). At that point tlb->page_size is still whatever the previous user left behind, which is 0 for the fresh gather set up by munmap() and so the check fires. Fix this in the same way like how other similar paths do. Fixes: dbe54153296d0 ("mm/huge_memory: add vmf_insert_folio_pud()") Cc: stable@vger.kernel.org Reported-by: Pavithra Signed-off-by: Ritesh Harjani (IBM) Acked-by: Zi Yan Acked-by: Muchun Song Tested-by: Pavithra Prakash Link: https://patch.msgid.link/8b853e5f9f16145859407b28af654b0f79e12642.1793318297.git.ritesh.list@gmail.com Signed-off-by: David Hildenbrand (Arm) --- mm/huge_memory.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7de94775a359..eb913acb03b7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3049,6 +3049,8 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, spinlock_t *ptl; pud_t orig_pud; + tlb_change_page_size(tlb, HPAGE_PUD_SIZE); + ptl = __pud_trans_huge_lock(pud, vma); if (!ptl) return 0; From eadce52a9ef657d561b3e76c9b5f178bd99735f5 Mon Sep 17 00:00:00 2001 From: Lance Yang Date: Mon, 26 Oct 2026 23:20:13 +0800 Subject: [PATCH 60/61] khugepaged: fix readahead lock inversion in collapse_file() collapse_file() can call page_cache_sync_readahead() while still holding source folios locked. If truncate holds mapping->invalidate_lock exclusively and waits for one of those folios, readahead waits for invalidate_lock and neither side can make progress. We need to release those folio locks before starting readahead. On a page cache miss, roll back first, populate the collapse range, and retry once. Doing readahead after rollback avoids an extra pass when all folios are already present. If another miss occurs on retry, give up rather than keep retrying while truncate or reclaim removes folios. Keep page_cache_sync_readahead() and wait for read I/O with filemap_lock_folio(), releasing each folio before moving to the next. The existing uptodate check during collapse still handles read errors. Shmem uses shmem_get_folio() for swapin and fallocated folios without taking mapping->invalidate_lock, so it is unaffected. Reported-by: syzbot+16bf7cd0ebeb1de93aa5@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=16bf7cd0ebeb1de93aa5 Fixes: 730633f0b7f9 ("mm: Protect operations adding pages to page cache with invalidate_lock") Cc: stable@vger.kernel.org Signed-off-by: Lance Yang Reviewed-by: Pedro Falcato Acked-by: David Hildenbrand (Arm) Acked-by: Zi Yan Reviewed-by: Baolin Wang Link: https://patch.msgid.link/20261026152013.10511-1-lance.yang@linux.dev Signed-off-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 44 +++++++++++++++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index ce794d9760a8..6d5a44cf3a00 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2240,6 +2240,8 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); enum scan_result result = SCAN_SUCCEED; int nr_none = 0; + bool do_readahead = false; + bool readahead_done = false; /* * MADV_COLLAPSE ignores shmem huge config, so do not check shmem @@ -2257,6 +2259,10 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, mapping_set_update(&xas, mapping); +retry: + result = SCAN_SUCCEED; + do_readahead = false; + xas_set_order(&xas, start, HPAGE_PMD_ORDER); __folio_set_locked(new_folio); if (is_shmem) __folio_set_swapbacked(new_folio); @@ -2321,17 +2327,9 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, } } else { /* !is_shmem */ if (!folio || xa_is_value(folio)) { - xas_unlock_irq(&xas); - page_cache_sync_readahead(mapping, &file->f_ra, - file, index, - end - index); - /* drain lru cache to help folio_isolate_lru() */ - lru_add_drain(); - folio = filemap_lock_folio(mapping, index); - if (IS_ERR(folio)) { - result = SCAN_FAIL; - goto xa_unlocked; - } + do_readahead = !readahead_done; + result = SCAN_FAIL; + goto xa_locked; } else if (folio_test_dirty(folio)) { /* * This page is dirty because it hasn't @@ -2654,6 +2652,30 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, new_folio->mapping = NULL; folio_unlock(new_folio); + + /* Readahead may take invalidate_lock; source folios are unlocked. */ + if (do_readahead) { + for (index = start; index < end;) { + folio = filemap_lock_folio(mapping, index); + if (IS_ERR(folio)) { + page_cache_sync_readahead(mapping, &file->f_ra, + file, index, end - index); + folio = filemap_lock_folio(mapping, index); + if (IS_ERR(folio)) { + result = SCAN_FAIL; + goto out_put; + } + } + index = folio_next_index(folio); + folio_unlock(folio); + folio_put(folio); + } + /* Drain the local LRU batch before retrying isolation. */ + lru_add_drain(); + readahead_done = true; + goto retry; + } +out_put: folio_put(new_folio); out: VM_BUG_ON(!list_empty(&pagelist)); From 71555ae8c7572fd99158013ca2e29187c5d2508d Mon Sep 17 00:00:00 2001 From: Samiullah Khawaja Date: Sat, 31 Oct 2026 20:35:40 +0000 Subject: [PATCH 61/61] mm/gup: check folio->mapping under folio lock in memfd_pin_folios() memfd_pin_folios() pins folios from the page cache without taking the folio lock. khugepaged or MADV_COLLAPSE can therefore replace a folio after checking its refcount while memfd_pin_folios() still pins the old folio. The caller then receives a folio which is no longer part of the memfd. For udmabuf and iommufd, this can cause data corruption: DMA accesses the detached folio rather than the live memfd contents. DMA reads can return stale data, while DMA writes update the detached copy and are lost from the live memfd. The detached folio remains pinned, so these accesses do not constitute a use-after-free or target another application's memory. For memfd live update, the stale contents can reappear in the memfd after handover. Lock the folio and recheck folio->mapping before pinning it, like other page cache users do. If the folio was removed, retry the folio lookup from the index that was removed. Fixes: 89c1905d9c14 ("mm/gup: introduce memfd_pin_folios() for pinning memfd folios") Reported-by: Andy Lutomirski Closes: https://lore.kernel.org/all/CALCETrV5eqg6d9=+BQhCpRwBNe3aCQiut8vxPBs8b7PToqjaHg@mail.gmail.com/ Cc: stable@vger.kernel.org Signed-off-by: Samiullah Khawaja Reviewed-by: Andy Lutomirski Link: https://patch.msgid.link/20261031203540.2282507-1-skhawaja@google.com Signed-off-by: David Hildenbrand (Arm) --- mm/gup.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index ba4907b35232..2b405cce2ca5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3471,7 +3471,7 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, { unsigned int flags, nr_folios, nr_found; unsigned int i, pgshift = PAGE_SHIFT; - pgoff_t start_idx, end_idx; + pgoff_t start_idx, end_idx, retry_idx; struct folio *folio = NULL; struct folio_batch fbatch; struct hstate *h; @@ -3512,6 +3512,7 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, * reason, we try to allocate them and add them to the * page cache. */ + retry_idx = start_idx; nr_found = filemap_get_folios_contig(memfd->f_mapping, &start_idx, end_idx, @@ -3524,7 +3525,21 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, for (i = 0; i < nr_found; i++) { folio = fbatch.folios[i]; + /* + * The folio might have been removed from the + * page cache (e.g., truncated or collapsed) + * after the lookup. Recheck under the folio + * lock. + */ + folio_lock(folio); + if (unlikely(folio->mapping != memfd->f_mapping)) { + folio_unlock(folio); + start_idx = retry_idx; + break; + } + if (try_grab_folio(folio, 1, FOLL_PIN)) { + folio_unlock(folio); folio_batch_release(&fbatch); ret = -EINVAL; goto err; @@ -3535,14 +3550,12 @@ long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, * page cache that were never cleared. */ if (!folio_test_uptodate(folio)) { - folio_lock(folio); - if (!folio_test_uptodate(folio)) { - folio_zero_range(folio, 0, folio_size(folio)); - flush_dcache_folio(folio); - folio_mark_uptodate(folio); - } - folio_unlock(folio); + folio_zero_range(folio, 0, folio_size(folio)); + flush_dcache_folio(folio); + folio_mark_uptodate(folio); } + retry_idx = folio_next_index(folio); + folio_unlock(folio); if (nr_folios == 0) *offset = offset_in_folio(folio, start);