github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-09-12 04:23:03 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

mm: provide vma_[flags_]is_cow_mapping() and remove is_cow_mapping()

Browse Source 
All remaining callers of is_cow_mapping() are invoking it in the form of
is_cow_mapping(vma->vm_flags) or an indirected version of this.

Therefore, provide a helper - vma_is_cow_mapping() to directly test the
VMA.

Additionally provide a new helper vma_flags_is_cow_mapping() which
performs the check using the new vma_flags_t type, and share this logic
between vma_is_cow_mapping() and vma_desc_is_cow_mapping().

With these changes, no callers of is_cow_mapping() remain, so remove it.

Also update the userland VMA tests to reflect the change.

No functional change intended.

[akpm@linux-foundation.org: fix kerneldoc comment typo, per Lorenzo]
  Link: https://lore.kernel.org/aob1goSSPH6sTN9y@gremlin
Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-2-c21581c0c3c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Deucher <alexander.deucher@amd.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Boris Brezillon <boris.brezillon@collabora.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christan König <christian.koenig@amd.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Gregory Price (Meta) <gourry@gourry.net>
Cc: Harry Yoo <harry@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Huang Ray <Ray.Huang@amd.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Liviu Dudau <liviu.dudau@arm.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Namhyung kim <namhyung@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
This commit is contained in:
Lorenzo Stoakes (ARM) 2026-08-13 18:32:19 +01:00 committed by Andrew Morton
parent 0bd14001eb
commit 51943a18ad
16 changed files with 107 additions and 38 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
arch/s390/mm/gmap_helpers.c
View File

@ -200,7 +200,7 @@ static int find_zeropage_pte_entry(pte_t *pte, unsigned long addr,
* currently only works in COW mappings, which is also where
* mm_forbids_zeropage() is checked.
*/
if (!is_cow_mapping(walk->vma->vm_flags))
if (!vma_is_cow_mapping(walk->vma))
return -EFAULT;
*found_addr = addr;

4
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
View File

@ -377,9 +377,9 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str
/* Workaround for Thunk bug creating PROT_NONE,MAP_PRIVATE mappings
* for debugger access to invisible VRAM. Should have used MAP_SHARED
* instead. Clearing VM_MAYWRITE prevents the mapping from ever
* becoming writable and makes is_cow_mapping(vm_flags) false.
* becoming writable and makes vma_is_cow_mapping(vma) false.
*/
if (is_cow_mapping(vma->vm_flags) &&
if (vma_is_cow_mapping(vma) &&
!(vma->vm_flags & VM_ACCESS_FLAGS))
vm_flags_clear(vma, VM_MAYWRITE);

2
drivers/gpu/drm/drm_gem_shmem_helper.c
View File

@ -753,7 +753,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct
return ret;
}
if (is_cow_mapping(vma->vm_flags))
if (vma_is_cow_mapping(vma))
return -EINVAL;
dma_resv_lock(shmem->base.resv, NULL);

2
drivers/gpu/drm/panthor/panthor_gem.c
View File

@ -761,7 +761,7 @@ static int panthor_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *v
return ret;
}
if (is_cow_mapping(vma->vm_flags))
if (vma_is_cow_mapping(vma))
return -EINVAL;
if (!refcount_inc_not_zero(&bo->cmap.mmap_count)) {

2
drivers/gpu/drm/ttm/ttm_bo_vm.c
View File

@ -489,7 +489,7 @@ static const struct vm_operations_struct ttm_bo_vm_ops = {
int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo)
{
/* Enforce no COW since would have really strange behavior with it. */
if (is_cow_mapping(vma->vm_flags))
if (vma_is_cow_mapping(vma))
return -EINVAL;
drm_gem_object_get(&bo->base);

2
drivers/gpu/drm/xe/xe_device.c
View File

@ -330,7 +330,7 @@ static int xe_pci_barrier_mmap(struct file *filp,
if (vma->vm_end - vma->vm_start > SZ_4K)
return -EINVAL;
if (is_cow_mapping(vma->vm_flags))
if (vma_is_cow_mapping(vma))
return -EINVAL;
if (vma->vm_flags & (VM_READ | VM_EXEC))

2
fs/proc/task_mmu.c
View File

@ -1693,7 +1693,7 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr,
if (!pte_write(pte))
return false;
if (!is_cow_mapping(vma->vm_flags))
if (!vma_is_cow_mapping(vma))
return false;
if (likely(!mm_flags_test(MMF_HAS_PINNED, vma->vm_mm)))
return false;

75
include/linux/mm.h
View File

@ -2271,19 +2271,78 @@ void unpin_user_pages(struct page **pages, unsigned long npages);
void unpin_user_folio(struct folio *folio, unsigned long npages);
void unpin_folios(struct folio **folios, unsigned long nfolios);
static inline bool is_cow_mapping(vm_flags_t flags)
/**
* vma_flags_is_cow_mapping() - Do these VMA flags imply a CoW mapping?
* @flags: The VMA flags to check.
*
* Mappings which could be CoW'd (subject to Copy-On-Write faults) are
* described as CoW mappings.
*
* All mappings backed by anonymous folios (all anonymous mappings and most
* MAP_PRIVATE-file backed ranges) are CoW mappings.
*
* All other mappings (including all MAP_SHARED mappings) are non-CoW.
*
* The criteria are !VMA_SHARED_BIT, VMA_MAYWRITE_BIT.
*
* VMA_MAYWRITE_BIT is checked instead of VMA_WRITE_BIT to account for both
* future mprotect() calls which can render a read-only mapping writable, and
* GUP with FOLL_FORCE (e.g. ptrace) which can CoW a read-only mapping.
*
* - No anonymous mapping can ever clear VMA_MAYWRITE_BIT.
*
* - Writes to anonymous mappings do not immediately result in CoW faults but
* may do so after the process is forked or if a read is followed by a
* write.
*
* - Writes to MAP_PRIVATE file-backed mappings result in CoW faults and may
* do so again after fork.
*
* - MAP_SHARED mappings of a file opened read-only are transformed into
* VMA_MAYSHARE_BIT, !VMA_SHARED_BIT, !VMA_MAYWRITE_BIT mappings, so remain
* non-CoW.
*
* - Drivers may clear VMA_MAYWRITE_BIT but do so at mmap() time and cannot
* mark themselves anonymous. Having cleared this flag it is not valid for
* them to leave the VMA_WRITE_BIT flag set.
*
* As a consequence, the anonymous reverse mapping only tracks CoW mappings.
*
* Returns: true if the flags indicate a CoW mapping, otherwise false.
*/
static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags)
{
return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
}
static inline bool vma_desc_is_cow_mapping(struct vm_area_desc *desc)
{
const vma_flags_t *flags = &desc->vma_flags;
return vma_flags_test(flags, VMA_MAYWRITE_BIT) &&
!vma_flags_test(flags, VMA_SHARED_BIT);
}
/**
* vma_is_cow_mapping() - Is this VMA a CoW mapping?
* @vma: The VMA to check.
*
* See vma_flags_is_cow_mapping() for details.
*
* Returns: true if the VMA is a CoW mapping, otherwise false.
*/
static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma)
{
return vma_flags_is_cow_mapping(&vma->flags);
}
/**
* vma_desc_is_cow_mapping() - Is this VMA descriptor a CoW mapping?
* @desc: The VMA descriptor to check.
*
* See vma_flags_is_cow_mapping() for details.
*
* Returns: true if the VMA descriptor describes a CoW mapping, otherwise
* false.
*/
static inline bool vma_desc_is_cow_mapping(struct vm_area_desc *desc)
{
return vma_flags_is_cow_mapping(&desc->vma_flags);
}
#ifndef CONFIG_MMU
static inline bool is_nommu_shared_mapping(vm_flags_t flags)
{

2
kernel/events/uprobes.c
View File

@ -513,7 +513,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
uprobe = container_of(auprobe, struct uprobe, arch);
if (WARN_ON_ONCE(!is_cow_mapping(vma->vm_flags)))
if (WARN_ON_ONCE(!vma_is_cow_mapping(vma)))
return -EINVAL;
/*

2
mm/gup.c
View File

@ -1236,7 +1236,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
* Anon pages in shared mappings are surprising: now
* just reject it.
*/
if (!is_cow_mapping(vm_flags))
if (!vma_is_cow_mapping(vma))
return -EFAULT;
}
} else if (!(vm_flags & VM_READ)) {

8
mm/huge_memory.c
View File

@ -1681,7 +1681,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn,
BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
(VM_PFNMAP|VM_MIXEDMAP));
BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma));
pfnmap_setup_cachemode_pfn(pfn, &pgprot);
@ -1789,7 +1789,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn,
BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
(VM_PFNMAP|VM_MIXEDMAP));
BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma));
pfnmap_setup_cachemode_pfn(pfn, &pgprot);
@ -1931,7 +1931,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
* applied special bit, or we made the PRIVATE mapping be
* able to wrongly write to the backend MMIO.
*/
VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd));
VM_WARN_ON_ONCE(vma_is_cow_mapping(src_vma) && pmd_write(pmd));
goto set_pmd;
}
@ -2052,7 +2052,7 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,
* TODO: once we support anonymous pages, use
* folio_try_dup_anon_rmap_*() and split if duplicating fails.
*/
if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) {
if (vma_is_cow_mapping(vma) && pud_write(pud)) {
pudp_set_wrprotect(src_mm, addr, src_pud);
pud = pud_wrprotect(pud);
}

2
mm/hugetlb.c
View File

@ -4898,7 +4898,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
pte_t *src_pte, *dst_pte, entry;
struct folio *pte_folio;
unsigned long addr;
bool cow = is_cow_mapping(src_vma->vm_flags);
bool cow = vma_is_cow_mapping(src_vma);
struct hstate *h = hstate_vma(src_vma);
unsigned long sz = huge_page_size(h);
unsigned long npages = pages_per_huge_page(h);

2
mm/internal.h
View File

@ -1353,7 +1353,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
* ... because we only care about writable private ("COW")
* mappings where we have to break COW early.
*/
return is_cow_mapping(vma->vm_flags);
return vma_is_cow_mapping(vma);
}
/* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */

25
mm/memory.c
View File

@ -780,7 +780,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma,
/* Only CoW'ed anon folios are "normal". */
if (pfn == index)
return NULL;
if (!is_cow_mapping(vma->vm_flags))
if (!vma_is_cow_mapping(vma))
return NULL;
}
}
@ -1002,7 +1002,6 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *dst_vma,
struct vm_area_struct *src_vma, unsigned long addr, int *rss)
{
vm_flags_t vm_flags = dst_vma->vm_flags;
pte_t orig_pte = ptep_get(src_pte);
softleaf_t entry = softleaf_from_pte(orig_pte);
pte_t pte = orig_pte;
@ -1026,7 +1025,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
rss[mm_counter(folio)]++;
if (!softleaf_is_migration_read(entry) &&
is_cow_mapping(vm_flags)) {
vma_is_cow_mapping(dst_vma)) {
/*
* COW mappings require pages in both parent and child
* to be set to read. A previously exclusive entry is
@ -1067,7 +1066,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
* save and restore device driver state).
*/
if (softleaf_is_device_private_write(entry) &&
is_cow_mapping(vm_flags)) {
vma_is_cow_mapping(dst_vma)) {
entry = make_readable_device_private_entry(
swp_offset(entry));
pte = swp_entry_to_pte(entry);
@ -1082,7 +1081,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
* exclusive entries currently only support private writable
* (ie. COW) mappings.
*/
VM_BUG_ON(!is_cow_mapping(src_vma->vm_flags));
VM_BUG_ON(!vma_is_cow_mapping(src_vma));
if (try_restore_exclusive_pte(src_vma, addr, src_pte, orig_pte))
return -EBUSY;
return -ENOENT;
@ -1181,7 +1180,7 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
}
/* If it's a COW mapping, write protect it both processes. */
if (is_cow_mapping(src_vma->vm_flags) && writable) {
if (vma_is_cow_mapping(src_vma) && writable) {
wrprotect_ptes(src_mm, addr, src_pte, nr);
pte = pte_wrprotect(pte);
}
@ -1602,9 +1601,9 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
* We need to invalidate the secondary MMU mappings only when
* there could be a permission downgrade on the ptes of the
* parent mm. And a permission downgrade will only happen if
* is_cow_mapping() returns true.
* vma_is_cow_mapping() returns true.
*/
is_cow = is_cow_mapping(src_vma->vm_flags);
is_cow = vma_is_cow_mapping(src_vma);
if (is_cow) {
mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE,
@ -2437,7 +2436,7 @@ static bool vm_mixed_zeropage_allowed(struct vm_area_struct *vma)
if (mm_forbids_zeropage(vma->vm_mm))
return false;
/* zeropages in COW mappings are common and unproblematic. */
if (is_cow_mapping(vma->vm_flags))
if (vma_is_cow_mapping(vma))
return true;
/* Mappings that do not allow for writable PTEs are unproblematic. */
if (!(vma->vm_flags & (VM_WRITE | VM_MAYWRITE)))
@ -2888,7 +2887,7 @@ vm_fault_t vmf_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr,
BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
(VM_PFNMAP|VM_MIXEDMAP));
BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma));
BUG_ON((vma->vm_flags & VM_MIXEDMAP) && pfn_valid(pfn));
if (addr < vma->vm_start || addr >= vma->vm_end)
@ -3300,7 +3299,7 @@ static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma,
unsigned long size)
{
const unsigned long end = addr + PAGE_ALIGN(size);
const bool is_cow = is_cow_mapping(vma->vm_flags);
const bool is_cow = vma_is_cow_mapping(vma);
int err;
err = get_remap_pgoff(is_cow, addr, end, vma->vm_start, vma->vm_end,
@ -6800,7 +6799,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma,
* FAULT_FLAG_UNSHARE only applies to COW mappings. Let's
* just treat it like an ordinary read-fault otherwise.
*/
if (!is_cow_mapping(vma->vm_flags))
if (!vma_is_cow_mapping(vma))
*flags &= ~FAULT_FLAG_UNSHARE;
} else if (*flags & FAULT_FLAG_WRITE) {
/* Write faults on read-only mappings are impossible ... */
@ -6808,7 +6807,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma,
return VM_FAULT_SIGSEGV;
/* ... and FOLL_FORCE only applies to COW mappings. */
if (WARN_ON_ONCE(!(vma->vm_flags & VM_WRITE) &&
!is_cow_mapping(vma->vm_flags)))
!vma_is_cow_mapping(vma)))
return VM_FAULT_SIGSEGV;
}
#ifdef CONFIG_PER_VMA_LOCK

2
mm/mempolicy.c
View File

@ -844,7 +844,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
return false;
/* Also skip shared copy-on-write folios */
if (is_cow_mapping(vma->vm_flags) && folio_maybe_mapped_shared(folio))
if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
return false;
/* Folios are pinned and can't be migrated */

11
tools/testing/vma/include/dup.h
View File

@ -1162,6 +1162,17 @@ static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma)
return is_shared_maywrite(&vma->flags);
}
static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags)
{
return vma_flags_test(flags, VMA_MAYWRITE_BIT) &&
!vma_flags_test(flags, VMA_SHARED_BIT);
}
static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma)
{
return vma_flags_is_cow_mapping(&vma->flags);
}
static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi)
{
/*