mirror of
https://github.com/torvalds/linux.git
synced 2026-11-07 06:17:02 +01:00
v7.4-rc1
662 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Wen Jiang
|
bf033d0436 |
mm/vmalloc: bail out early on invalid page_shift in __vmap_pages_range_noflush()
__vmap_pages_range_noflush() currently only warns when page_shift is smaller than PAGE_SHIFT, then continues with the invalid shift. Return -EINVAL instead so the bogus argument is caught immediately. Link: https://lore.kernel.org/20261004071639.119857-1-jiangwenxiaomi@gmail.com Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Suggested-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Vishal Moola (Fractile) <vishal.moola@gmail.com> Cc: Dev Jain <dev.jain@arm.com> |
||
|
Dima Koziuk
|
101fea09a8 |
mm: kmsan: fix iounmap metadata teardown
kmsan_vmalloc_to_page_or_null() rejects addresses outside the regular
vmalloc and module ranges, including the shadow and origin addresses
passed by its callers. As a result, iounmap does not free the metadata
backing pages. Also, the first iteration of the teardown loop unmaps the
entire metadata range, preventing subsequent iterations from finding their
pages even if the address lookup is fixed.
Factor the page-table walk out of vmalloc_to_page() into
__vmalloc_to_page(), keeping the address check in the public wrapper. Use
the unchecked helper in kmsan_vmalloc_to_page_or_null() and check for NULL
before converting the returned page to a PFN. Mark __vmalloc_to_page() as
__always_inline to avoid an additional function call in vmalloc_to_page().
Move metadata teardown into kmsan_iounmap_pages(). Free the backing
blocks while their mappings are still available, then unmap each metadata
range once and flush its TLB entries.
Link: https://lore.kernel.org/20261002200508.546-1-dmytrokoziuk68@gmail.com
Fixes:
|
||
|
Ye Liu
|
4b52da0120 |
mm/vmalloc: fix vmalloc_dump_obj cross-zone VA lookup
vmalloc_dump_obj() searches only one vmap node (addr_to_node(addr)), but a vmalloc allocation may span multiple vmap zones. The VA is stored in only one node's rb-tree (addr_to_node(va_start)), so an object pointer in a different zone than va_start maps to a different node and the search misses. This affects any allocation larger than vmap_zone_size (64 KiB) on multi-CPU systems. Extract find_vmap_area_lock() from find_vmap_area() to share the cross-node iteration logic. The helper supports both spin_lock and spin_trylock, the latter for atomic dump contexts (OOM, KASAN, RCU). Link: https://lore.kernel.org/20260928-vmalloc_dump_obj-v4-2-6f288a431edc@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Paul Walmsley <pjw@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> |
||
|
Ye Liu
|
f4399c9927 |
mm/vmalloc: fix vmalloc_dump_obj address alignment for last-page lookups
Patch series "mm/vmalloc: fix vmalloc_dump_obj VA lookup", v4.
vmalloc_dump_obj() has two bugs that cause it to miss vmalloc
allocations.
1. PAGE_ALIGN() rounds up, pushing last-page addresses to va_end
and outside the VA lookup range.
2. The function searches only one vmap node, but allocations
larger than 64 KiB may span multiple vmap zones whose VA
is stored in a different node's rb-tree.
Patch 1 fixes the alignment, patch 2 fixes the cross-zone search.
This patch (of 2):
vmalloc_dump_obj() uses PAGE_ALIGN() to normalize the input address before
looking it up in the per-node busy tree. PAGE_ALIGN() rounds up, which
can push an address in the last page of a vmalloc allocation to va_end --
outside the [va_start, va_end) range that __find_vmap_area() searches.
This causes the lookup to miss the VA and return false, degrading
diagnostic output in OOM dumps and KASAN reports to the less informative
"vmalloc memory" fallback.
The upward alignment can also change the addr_to_node() mapping when the
page boundary crosses a vmap zone boundary, causing the search to hit the
wrong node entirely.
Use PAGE_ALIGN_DOWN() instead, which rounds down to the page containing
the address. This keeps the address within the VA range and preserves the
correct node mapping.
Link: https://lore.kernel.org/20260928-vmalloc_dump_obj-v4-0-6f288a431edc@linux.dev
Link: https://lore.kernel.org/20260928-vmalloc_dump_obj-v4-1-6f288a431edc@linux.dev
Signed-off-by: Ye Liu <liuye@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
|
||
|
Palla Raghunath
|
499baaff38 |
mm/vmalloc: do not warn on -ENOMEM from va_clip() in pcpu_get_vm_areas()
When pcpu_get_vm_areas() has to split a free vmap_area in the middle (NE_FIT_TYPE), va_clip() needs an extra vmap_area object. It takes the per-cpu ne_fit_preload_node if one is there, and otherwise falls back to kmem_cache_alloc(GFP_NOWAIT), which may fail and return -ENOMEM. pcpu_get_vm_areas() never preloads, and a single call can do more than one such split: on a NUMA system it places one area per node group, so the first split consumes the preloaded object and the next one depends on the GFP_NOWAIT allocation. That failure is expected and already handled: the recovery path returns the areas clipped so far to the free tree, purges lazily freed areas and retries. But the error is checked with WARN_ON_ONCE(), so a transient allocation failure under memory pressure or fault injection triggers a kernel warning, and a panic with panic_on_warn. syzbot hit this on a two-node VM while creating a per-cpu BPF array map. Keep the WARN_ON_ONCE() for errors other than -ENOMEM, which do indicate a bug, and take the recovery path either way. This matches what commit |
||
|
Zhijian Han
|
c2d4ccfbba |
mm: fix typos in various comments
Fix spelling errors found by codespell in several mm source files: - "aray" -> "array" in include/linux/mm.h - "indiciate" -> "indicate" in include/linux/mm_types.h - "aggresive" -> "aggressive" in include/linux/mm_types.h - "progated" -> "propagated" in mm/huge_memory.c - "Pressumably" -> "Presumably" in mm/hugetlb.c - "fime" -> "time" in mm/hugetlb.c - "farest" -> "farthest" in mm/list_lru.c - "trivally" -> "trivially" in mm/madvise.c - "atmost" -> "at most" in mm/memcontrol.c - "splited" -> "split" in mm/memory-failure.c - "appliable" -> "applicable" in mm/memory.c - "incase" -> "in case" in mm/page_io.c - "contigous" -> "contiguous" in mm/vmalloc.c - "probablity" -> "probability" in mm/kfence/core.c - "possesss" -> "possesses" in mm/util.c No functional changes. Link: https://lore.kernel.org/20260922031843.2857104-1-hanzhijian1991@gmail.com Signed-off-by: Zhijian Han <hanzhijian1991@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Sebastian Andrzej Siewior
|
25aa09142e |
mm/vmalloc: use %p for pointer formatting
Commit |
||
|
Ye Liu
|
3b23842264 |
mm/vmalloc: extract show_busy_info from vmalloc_info_show
Extract the busy vmap area iteration into show_busy_info, mirroring the existing show_purge_info pattern. Link: https://lore.kernel.org/20260915-vmalloc_study-v2-3-cc4dfe635e22@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> |
||
|
Ye Liu
|
c10ed0f231 |
mm/vmalloc: extract vmap_insert_free_area helper
The allocation and insertion of a free vmap_area is duplicated between the loop body and the tail of vmap_init_free_space. Factor it into a small helper so the main function only deals with computing the free gaps between busy regions. Link: https://lore.kernel.org/20260915-vmalloc_study-v2-2-cc4dfe635e22@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Dev Jain <dev.jain@arm.com> |
||
|
Ye Liu
|
ebb07f8a8b |
mm/vmalloc: group xa_init with vbq field initializations
Patch series "mm/vmalloc: minor cleanups", v2. Small cleanup series for mm/vmalloc.c, no functional changes: - Group xa_init() with the other vmap_block_queue field initializations in vmalloc_init(), instead of after the unrelated vfree_deferred setup. - Extract vmap_insert_free_area() helper to deduplicate the allocate- and-insert pattern that appeared both inside the loop body and after the loop in vmap_init_free_space(). - Extract show_busy_info() from vmalloc_info_show(), mirroring the existing show_purge_info() pattern, so the top-level show function only orchestrates the two data sources. This patch (of 3): Move xa_init() next to the other vbq field initializations instead of after the unrelated vfree_deferred setup. Link: https://lore.kernel.org/20260915-vmalloc_study-v2-0-cc4dfe635e22@linux.dev Link: https://lore.kernel.org/20260915-vmalloc_study-v2-1-cc4dfe635e22@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> |
||
|
Mike Rapoport (Microsoft)
|
569db54d8b |
mm/execmem: make sure ROX cache always contains multiples of PMD_SIZE
The ROX cache relies on its chunks being PMD mapped. For a PMD mapped chunk, set_memory_rox() updates the direct map alias one PMD at a time and the large mappings there survive. When execmem refills the cache, it rounds up the requested allocation size to PMD_SIZE and tries to allocate that with vmalloc(VM_ALLOW_HUGE_VMAP). If that allocation fails, execmem falls back to vmalloc() of the original size. There are two issues with this approach: * If huge pages are not available, __vmalloc_node_range() silently falls back to base pages. The area execmem gets is virtually contiguous, but it is backed by 512 scattered base pages. Permission updates on such areas split large mappings in the direct map that contain those base pages, up to 512 PMD splits in the worst case. * execmem's own fallback adds a small base page mapped area to the cache. This adds the overhead of cache management to these allocations with no benefit of reducing fragmentation either in the vmalloc/modules address space or in the direct map. Worse, these areas are never freed from the cache, because execmem_cache_clean() releases only chunks that are a multiple of PMD_SIZE and aligned to PMD_SIZE, exactly to minimize the number of base page mappings. Add VM_REQUIRE_HUGE_VMAP option to vmalloc that fails if the allocation of huge pages fails or if such an allocation is not possible because huge page allocations in vmalloc were disabled or the architecture does not support them. Use this option when populating the ROX cache. If vmalloc(VM_REQUIRE_HUGE_VMAP) fails or vmalloc of huge pages is unavailable, handle the memory allocation outside the ROX cache with plain vmalloc(). Since the fallback allocation has to return ROX memory, add an execmem_alloc_rox() helper and use it for both populating the ROX cache and dealing with a fallback allocation in a ROX execmem_range. With that, the cache only ever contains PMD aligned chunks sized as a multiple of PMD_SIZE, and the PMD checks in execmem_cache_clean() become a VM_WARN_ON_ONCE() to ensure that the PMD mapping invariant does not change. Link: https://lore.kernel.org/20260903-execmem-rox-cache-pmd-v1-v1-3-11beb2a3d249@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Assisted-by: copilot:claude-opus-5 Cc: Benjamin Tissoires <bentiss@kernel.org> Cc: Jiri Kosina <jikos@kernel.org> Cc: Luis Chamberalin <mcgrof@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> |
||
|
Mike Rapoport (Microsoft)
|
86dc0280b8 |
mm/vmalloc: make set_area_direct_map HUGE_VMAP friendly
set_area_direct_map() always updates direct map alias permissions in
single page increments.
For HUGE_VMAP areas it's suboptimal. Not only the loop in
set_area_direct_map() needlessly has more iterations (e.g times 512 on
x86), but it also causes fragmentation of the direct map that could be
avoided for the HUGE_VMAP areas populated with large pages.
All pages in an area are always of the same order: either same-order large
pages when VM_ALLOW_HUGE_VMAP is set and all huge pages were successfully
allocated, or order-0 page when VM_ALLOW_HUGE_VMAP is cleared or when huge
pages allocation fails and fallback path is taken.
Instead of updating the direct map permissions for every order-0 page in
an area, use the area's page_order as the loop increment and update the
large pages in one call to set_direct_map_{invalid,default}_noflush().
Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-4-949b64a9f755@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Will Deacon <will@kernel.org>
|
||
|
Mike Rapoport (Microsoft)
|
1f73dfdeb9 |
mm/vmalloc: constify vm parameter of get_vm_area_page_order()
get_vm_area_page_order() and vm_area_page_order() do not need to modify struct vm_struct passed to them. Constify the parameter. Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-3-949b64a9f755@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Dev Jain <dev.jain@arm.com> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Len Brown <lenb@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: "Rafael J. Wysocki" <rafael@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> |
||
|
Mike Rapoport (Microsoft)
|
40702f168a |
mm/vmalloc: set area's page_order after allocation succeeds
__vmalloc_area_node() calls set_vm_area_page_order() to set area's page_order before actually allocating pages to populate the area. If allocation of large pages in HUGE_VMAP case fails midway, this leaves the area with elevated page_order throughout the cleanup path. There is no actual issue with this because the only place that currently relies on area->page_order on the cleanup path is the loop calculating the direct map alias range in vm_reset_perms() and it anyway skips unpopulated pages. But having set_vm_area_page_order() in the middle of __vmalloc_area_node() makes things very obscure, hard to reason about and error prone against future changes of the cleanup path. Move the call to set_vm_area_page_order() just before the successful return from __vmalloc_area_node() where page order is guaranteed. While on it, initialize local page_order variable with its declaration. Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-2-949b64a9f755@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Dev Jain <dev.jain@arm.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Hildenbrand <david@kernel.org> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Len Brown <lenb@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: "Rafael J. Wysocki" <rafael@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> Cc: Kevin Brodsky <kevin.brodsky@arm.com> |
||
|
Mike Rapoport (Microsoft)
|
be7dacb9c1 |
set_memory: add number of pages parameter to set_direct_map APIs
Patch series "arch, mm/execmem: resolve confusion about
set_direct_map_valid_noflush()", v3.
Recent discussion about implementation of execmem's ROX caches on arm64
revealed a confusion about how set_direct_map_valid_noflush() implemented
on different architectures.
On arm64 it sets or clears the PTE_VALID bit marking a PTE as present or
not present.
On other architectures it's a range version of
set_direct_map_invalid_noflush() and set_direct_map_default_noflush()
Unlike arm64::set_direct_map_valid_noflush(),
set_direct_map_default_noflush() not only marks PTE as present, but also
sets its default protection mode.
Other than that, initial design of execmem ROX caches didn't rely on
restoration of large mappings that's now available on x86, but completely
removed the memory allocated for the ROX cache from the direct map to
ensure that large mappings are not split. This precluded usage of
VM_FLUSH_RESET_PERMS for the ROX cache allocations and required execmem to
implement manipulation of the direct map alias.
Current implementation of ROX caches does not remove the direct map alias
but simply calls set_memory_rox() that updates the permissions in both
vmalloc address space and the direct map and relies on
collapse_large_pages() in x86 CPA to keep large mappings.
This allow using VM_FLUSH_RESET_PERMS for execmem ROX cache allocations
with small adjustments to set_direct_map APIs and vmalloc::reset_perms()
behaviour: adding number of pages parameter to set_direct_map APIs and
making resetting of the direct map permissions in vmalloc VMAP_HUGE
friendly.
Implement these adjustments, make execmem always use VM_FLUSH_RESET_PERMS
and revert set_direct_map_valid_noflush() changes.
This patch (of 6):
When set_direct_map APIs were introduced by the commit
|
||
|
Uladzislau Rezki (Sony)
|
b9183788a2 |
mm/vmalloc: do not warn on -ENOMEM from va_alloc()
Since vmalloc() accepts non-blocking GFP flags, allocation requests may fail when callers pass restrictive GFP masks. va_clip() may return -ENOMEM when its GFP_NOWAIT fallback allocation fails during NE_FIT_TYPE splitting. This is an expected failure, so va_alloc() should return the error without triggering a kernel splat. Link: https://lore.kernel.org/20260802104627.63892-1-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reported-by: syzbot+61c997e6be1d9bb300ba@syzkaller.appspotmail.com Closes: https://lore.kernel.org/6a6d3cbd.6ce73036.24301b.000e.GAE@google.com Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Artem Lytkin
|
272b0d84b1 |
mm/vmalloc: make vm_struct.nr_pages an unsigned long
vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte counts from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the type of its promoted left operand, so those are 32-bit arithmetic and wrap at 4 GiB of bytes, which is 2^20 pages. Every site depends on a cast being remembered; vmap() has one, two recent commits did not. vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and /proc/kcore returns it as zeros while reporting a successful read, which drgn, crash or gdb cannot tell from real memory, and the vrealloc() grow-in-place check declines a request that would have fit. Widen the field so the class of bug goes away instead of one site at a time. Everything feeding or consuming it widens too: vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages and old_nr_pages, the index range of vm_area_free_pages(), and three page indexes that were plain int. Five casts go. Two prints needed fixing as well, %u in vmalloc_dump_obj() and %d for the unsigned field in vmalloc_info_show(). No bug report behind this, I found it reading the code. The 4 GiB wrap needs only a machine with over 4 GiB of memory. Neither larger threshold is a practical concern: 2^32 pages, where the field itself truncates, is 16 TiB and beyond what hardware can populate, and 2^31, where the plain int indexes break, is 8 TiB and larger than anything in the tree asks for. The int *nr cursor in the mapping path is unchanged and is separate work. Users outside mm/vmalloc.c need no change either. Those handing the count to a narrower parameter cannot drive it near 2^31, and kho_preserve_vmalloc() stores it into a 32-bit ABI field that still receives the same low bits; above 2^32 pages the truncation just moves out of vm_struct into that store. sizeof(struct vm_struct) on x86-64 stays 72 bytes with CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled, both inside the kmalloc-96 bucket it already comes from. Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com Fixes: |
||
|
Andrew Morton
|
4b47e81e56 |
Merge branch 'mm-hotfixes-stable' into mm-stable to pick up
already-upstream changes to memcontrol.c, needed by "memcg: move mem_cgroup_swappiness and vm_swappiness to mm/swap.h". |
||
|
Lorenzo Stoakes (ARM)
|
9d3277b2c0 |
arm64: remove redundant concurrent ptdump UAF mitigation
This partially reverts commit |
||
|
Lorenzo Stoakes (ARM)
|
26444eb714 |
mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF
Patch series "mm: fix UAF caused by race between ptdump and vmap pgtable freeing", v6. Kernel page table walkers fall into two broad categories - those ranges where no exclusion is required via walk_kernel_page_table_range_lockless() and those where exclusion is required via walk_kernel_page_table_range() or walk_page_range_debug(). The former category is used only by arm64 arch code operating on ranges it both wholly owns and does not concurrently write. The latter category consists of kernel page table walkers operating on ranges that are wholly owned (but which need exclusion against concurrent writers). The lock used for exclusion is the mmap lock, and for kernel ranges this is the mmap lock on init_mm. ptdump is a special case being both the only user of walk_page_range_debug(), and the only case in which it walks ranges it does not own. This presents a problem, as page tables may be freed under ptdump. And indeed there is a use-after-free bug in the kernel as a result, which this series addresses. vmap promotes page tables to huge leaf entries where possible, freeing the lower page table when it does. It does this with no meaningful locks held against concurrent ptdump walks. As a result, use-after-free can currently occur. This series addresses the issue by having the vmap huge promotion logic acquire the mmap read lock while both setting the huge page table entry and freeing the prior leaf page table. The ptdump code already acquires the mmap write lock, so by doing so we ensure that the ptdump walker only ever observes either the huge page table entry or the existing page table entry, and nothing is freed underneath it. A mitigation for this issue was already applied for arm64 in commit |
||
|
Mike Rapoport (Microsoft)
|
ef79e0f5e3 |
mm: split out vmalloc declarations from internal.h
mm/internal.h becomes more and more bloated. Move declarations related to vmalloc to a new mm/vmalloc.h header. No functional changes. Link: https://lore.kernel.org/20260709-internal-h-v2-3-695631425968@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pratyush Yadav <pratyush@kernel.org> Acked-by: SJ Park <sj@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Alexander Potapenko <glider@google.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Marco Elver <elver@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Feng Tang
|
923a89a91a |
mm/vmalloc: add alignment info in warning print as possible failure reason
When running 'fix_align_alloc_test' case of test_vmalloc module with
command:
insmod ./test_vmalloc.ko run_test_mask=64
It will fail, which is the expected result, as the case increment
the alignment parameter gradually to 64bit limit. And the dmesg has
warning msg:
"vmalloc_test/0: vmalloc error: size 4096, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0"
It doesn't give the alignment info, which is the real reason for the
failure (not the 'size').
Add alignment info to the warning print to give the necessary hint
for possible failure reason, and the message will be:
"vmalloc_test/0: vmalloc error: size 4096, align 0x800000000000, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0"
Link: https://lore.kernel.org/20260702112610.21589-1-feng.tang@linux.alibaba.com
Signed-off-by: Feng Tang <feng.tang@linux.alibaba.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Markus Elfring
|
24e9b62985 |
mm/vmalloc: use more common error handling code in pcpu_get_vm_areas()
Use an existing label once more so that a bit of exception handling can be better reused at the end of this function implementation. This issue was detected by using the Coccinelle software. Link: https://lore.kernel.org/453375c4-c3ca-4e6f-8880-0e6ff3c74ee3@web.de Signed-off-by: Markus Elfring <elfring@users.sourceforge.net> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Alexander Potapenko <glider@google.com> Cc: Daniel Axtens <dja@axtens.net> Cc: Dmitry Vyukov <dvyukov@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kaitao Cheng
|
dec068de6d |
mm/vmalloc: honor GFP constraints in pcpu_get_vm_areas()
Patch series "mm/percpu: Fix possible NOFS/NOIO reclaim recursion", v4. Commit |
||
|
Shivam Kalra
|
5ea8ec74c5 |
mm/vmalloc: free unused pages on vrealloc() shrink
When vrealloc() shrinks an allocation and the new size crosses a page boundary, unmap and free the tail pages that are no longer needed. This reclaims physical memory that was previously wasted for the lifetime of the allocation. The heuristic is simple: always free when at least one full page becomes unused. Huge page allocations (page_order > 0) are skipped, as partial freeing would require splitting. Allocations with VM_FLUSH_RESET_PERMS are also skipped, as their direct-map permissions must be reset before pages are returned to the page allocator, which is handled by vm_reset_perms() during vfree(). Additionally, allocations with VM_USERMAP are skipped because remap_vmalloc_range_partial() validates mapping requests against the unchanged vm->size; freeing tail pages would cause vmalloc_to_page() to return NULL for the unmapped range. To protect concurrent readers, the shrink path uses Node lock to synchronize before freeing the pages. Finally, we notify kmemleak of the reduced allocation size using kmemleak_free_part() to prevent the kmemleak scanner from faulting on the newly unmapped virtual addresses. The virtual address reservation (vm->size / vmap_area) is intentionally kept unchanged, preserving the address for potential future grow-in-place support. Link: https://lore.kernel.org/20260519-vmalloc-shrink-v14-4-70b96ee3e9c9@zohomail.in Signed-off-by: Shivam Kalra <shivamkalra98@zohomail.in> Suggested-by: Danilo Krummrich <dakr@kernel.org> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Alice Ryhl <aliceryhl@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Shivam Kalra
|
0bca238046 |
mm/vmalloc: use physical page count in vread_iter() for VM_ALLOC areas
For VM_ALLOC areas in vread_iter(), derive the vm area size from vm->nr_pages rather than get_vm_area_size(). Only VM_ALLOC areas are subject to vrealloc() shrinking, which frees pages without reducing the virtual reservation size. Switch to using vm->nr_pages for VM_ALLOC areas so the reader remains correct once shrink support is added. Other mapping types (vmap, ioremap) do not initialize nr_pages and will continue using get_vm_area_size(). [shivamkalra98@zohomail.in: add an nr_pages check] Link: https://lore.kernel.org/aff47da5-4fd5-481d-be18-e1eb99639490@zohomail.in Link: https://lore.kernel.org/20260519-vmalloc-shrink-v14-3-70b96ee3e9c9@zohomail.in Signed-off-by: Shivam Kalra <shivamkalra98@zohomail.in> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Alice Ryhl <aliceryhl@google.com> Cc: Danilo Krummrich <dakr@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Shivam Kalra
|
d57ac904ff |
mm/vmalloc: use physical page count for vrealloc() grow-in-place check
Update the grow-in-place check in vrealloc() to compare the requested size against the actual physical page count (vm->nr_pages) rather than the virtual area size (alloced_size, derived from get_vm_area_size()). Currently both values are equivalent, but the upcoming vrealloc() shrink functionality will free pages without reducing the virtual reservation size. After such a shrink, the old alloced_size-based comparison would incorrectly allow a grow-in-place operation to succeed and attempt to access freed pages. Switch to vm->nr_pages now so the check remains correct once shrink support is added. Link: https://lore.kernel.org/20260519-vmalloc-shrink-v14-2-70b96ee3e9c9@zohomail.in Signed-off-by: Shivam Kalra <shivamkalra98@zohomail.in> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Alice Ryhl <aliceryhl@google.com> Cc: Danilo Krummrich <dakr@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Shivam Kalra
|
4f1839e225 |
mm/vmalloc: extract vm_area_free_pages() helper from vfree()
Patch series "mm/vmalloc: free unused pages on vrealloc() shrink", v14. This series implements the TODO in vrealloc() to unmap and free unused pages when shrinking across a page boundary. Problem: When vrealloc() shrinks an allocation, it updates bookkeeping (requested_size, KASAN shadow) but does not free the underlying physical pages. This wastes memory for the lifetime of the allocation. Solution: - Patch 1: Extracts a vm_area_free_pages(vm, start_idx, end_idx) helper from vfree() that frees a range of pages with memcg and nr_vmalloc_pages accounting. Freed page pointers are set to NULL to prevent stale references. - Patch 2: Update the grow-in-place check in vrealloc() to compare the requested size against the actual physical page count (vm->nr_pages) rather than the virtual area sizes. This is a prerequisite for shrinking. - Patch 3: For VM_ALLOC areas in vread_iter(), derive the vm area size from vm->nr_pages rather than get_vm_area_size(), which would overestimate the mapped range after a shrink. Other mapping types (vmap, ioremap) don't set nr_pages and keep using get_vm_area_size(). - Patch 4: Uses the helper to free tail pages when vrealloc() shrinks across a page boundary. - Patch 5: Adds a vrealloc test case to lib/test_vmalloc that exercises grow-realloc, shrink-across-boundary, shrink-within-page, and grow-in-place paths. The virtual address reservation is kept intact to preserve the range for potential future grow-in-place support. A concrete user is the Rust binder driver's KVVec::shrink_to [1], which performs explicit vrealloc() shrinks for memory reclamation. This patch (of 5): Extract page freeing and NR_VMALLOC stat accounting from vfree() into a reusable vm_area_free_pages() helper. The helper operates on a range [start_idx, end_idx) of pages from a vm_struct, making it suitable for both full free (vfree) and partial free (upcoming vrealloc shrink). Freed page pointers in vm->pages[] are set to NULL to prevent stale references when the vm_struct outlives the free (as in vrealloc shrink). Link: https://lore.kernel.org/20260519-vmalloc-shrink-v14-0-70b96ee3e9c9@zohomail.in Link: https://lore.kernel.org/20260519-vmalloc-shrink-v14-1-70b96ee3e9c9@zohomail.in Link: https://lore.kernel.org/all/20260216-binder-shrink-vec-v3-v6-0-ece8e8593e53@zohomail.in/ [1] Signed-off-by: Shivam Kalra <shivamkalra98@zohomail.in> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Alice Ryhl <aliceryhl@google.com> Cc: Danilo Krummrich <dakr@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Muhammad Usama Anjum
|
b56ca146a2 |
vmalloc: add __GFP_SKIP_KASAN support
Patch series "kasan: hw_tags: Disable tagging for stack and page-tables", v4. Stacks and page tables are always accessed with the match-all tag, so assigning a new random tag every time at allocation and setting invalid tag at deallocation time, just adds overhead without improving the detection. With __GFP_SKIP_KASAN the page keeps its poison tag and KASAN_TAG_KERNEL (match-all tag) is stored in the page flags while keeping the poison tag in the hardware. The benefit of it is that 256 tag setting instruction per 4 kB page aren't needed at allocation and deallocation time. Thus match-all pointers still work, while non-match tags (other than poison tag) still fault. __GFP_SKIP_KASAN only skips for KASAN_HW_TAGS mode, so coverage is unchanged. Benchmark: The benchmark has two modes. In thread mode, the child process forks and creates N threads. In pgtable mode, the parent maps and faults a specified memory size and then forks repeatedly with children exiting immediately. Thread benchmark: 2000 iterations, 2000 threads: 2.575 s → 2.229 s (~13.4% faster) The pgtable samples: - 2048 MB, 2000 iters 19.08 s → 17.62 s (~7.6% faster) This patch (of 3): For allocations that will be accessed only with match-all pointers (e.g., kernel stacks), setting tags is wasted work. If the caller already set __GFP_SKIP_KASAN, skip tag setting of vmalloc pages. Before this patch, __GFP_SKIP_KASAN wasn't being used with vmalloc APIs. So it wasn't being checked. Now its being checked and acted upon. Other KASAN modes are unchanged because __GFP_SKIP_KASAN is ignored for them in the page allocator, and in vmalloc too we ignore this flag for them. This is a preparatory patch for optimizing kernel stack allocations. Link: https://lore.kernel.org/20260429102704.680174-1-dev.jain@arm.com Link: https://lore.kernel.org/20260429102704.680174-2-dev.jain@arm.com Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com> Co-developed-by: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> Co-developed-by: Dev Jain <dev.jain@arm.com> Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Ben Segall <bsegall@google.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Kees Cook <kees@kernel.org> Cc: K Prateek Nayak <kprateek.nayak@amd.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Valentin Schneider <vschneid@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Ryan Roberts
|
60ced5818f |
vmalloc: optimize vfree with free_pages_bulk()
Whenever vmalloc allocates high order pages (e.g. for a huge mapping) it must immediately split_page() to order-0 so that it remains compatible with users that want to access the underlying struct page. Commit |
||
|
Uladzislau Rezki (Sony)
|
04aa71da5f |
mm/vmalloc: do not trigger BUG() on BH disabled context
__get_vm_area_node() currently triggers a BUG() if in_interrupt() returns
true. However, in_interrupt() also reports true when BH are disabled.
The bridge code can call rhashtable_lookup_insert_fast() with bottom
halves disabled:
__vlan_add()
-> br_fdb_add_local()
spin_lock_bh(&br->hash_lock); <-- Disable BH
-> fdb_add_local()
-> fdb_create()
-> rhashtable_lookup_insert_fast()
-> kvmalloc()
-> vmalloc()
-> __get_vm_area_node()
-> BUG_ON(in_interrupt())
spin_unlock_bh(&br->hash_lock)
this triggers the BUG() despite the caller not being in NMI or
hard IRQ context.
Replace the in_interrupt() check with in_nmi() || in_hardirq().
Link: https://lore.kernel.org/20260515153009.2296191-1-urezki@gmail.com
Fixes:
|
||
|
Marco Elver
|
82d1f01292 |
vmalloc: fix buffer overflow in vrealloc_node_align()
Commit |
||
|
Linus Torvalds
|
c1f49dea2b |
7 hotfixes. 6 are cc:stable and all are for MM. Please see the
individual changelogs for details. -----BEGIN PGP SIGNATURE----- iHUEABYIAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCaeSA8wAKCRDdBJ7gKXxA jlKAAP4z9SQH3J71BVZK0pLi/9a/ISEZyJ9zIpsNi1sQ3shO/gD7BoUTQkepX5dk dv1YBhroYz981dgIIV3kXpIAEYnoeQc= =Os+z -----END PGP SIGNATURE----- Merge tag 'mm-hotfixes-stable-2026-04-19-00-14' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Pull MM fixes from Andrew Morton: "7 hotfixes. 6 are cc:stable and all are for MM. Please see the individual changelogs for details" * tag 'mm-hotfixes-stable-2026-04-19-00-14' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: mm/damon/core: disallow non-power of two min_region_sz on damon_start() mm/vmalloc: take vmap_purge_lock in shrinker mm: call ->free_folio() directly in folio_unmap_invalidate() mm: blk-cgroup: fix use-after-free in cgwb_release_workfn() mm/zone_device: do not touch device folio after calling ->folio_free() mm/damon/core: disallow time-quota setting zero esz mm/mempolicy: fix weighted interleave auto sysfs name |
||
|
Uladzislau Rezki (Sony)
|
ec05f51f1e |
mm/vmalloc: take vmap_purge_lock in shrinker
decay_va_pool_node() can be invoked concurrently from two paths:
__purge_vmap_area_lazy() when pools are being purged, and the shrinker via
vmap_node_shrink_scan().
However, decay_va_pool_node() is not safe to run concurrently, and the
shrinker path currently lacks serialization, leading to races and possible
leaks.
Protect decay_va_pool_node() by taking vmap_purge_lock in the shrinker
path to ensure serialization with purge users.
Link: https://lore.kernel.org/20260413192646.14683-1-urezki@gmail.com
Fixes:
|
||
|
Kexin Sun
|
3cb0dc0d0e |
mm: vmalloc: update outdated comment for renamed vread()
The function vread() was renamed to vread_iter() in commit
|
||
|
Michal Hocko
|
3caedb3b99 |
vmalloc: support __GFP_RETRY_MAYFAIL and __GFP_NORETRY
__GFP_RETRY_MAYFAIL and __GFP_NORETRY haven't been supported so far because their semantic (i.e. to not trigger OOM killer) is not possible with the existing vmalloc page table allocation which is allowing for the OOM killer. Example: __vmalloc(size, GFP_KERNEL | __GFP_RETRY_MAYFAIL); <snip> vmalloc_test/55 invoked oom-killer: gfp_mask=0x40dc0( GFP_KERNEL|__GFP_ZERO|__GFP_COMP), order=0, oom_score_adj=0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 slab_reclaimable:700 slab_unreclaimable:33708 mapped:0 shmem:0 pagetables:5174 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:850 free_pcp:319 free_cma:0 CPU: 4 UID: 0 PID: 639 Comm: vmalloc_test/55 ... Hardware name: QEMU Standard PC (i440FX + PIIX, ... Call Trace: <TASK> dump_stack_lvl+0x5d/0x80 dump_header+0x43/0x1b3 out_of_memory.cold+0x8/0x78 __alloc_pages_slowpath.constprop.0+0xef5/0x1130 __alloc_frozen_pages_noprof+0x312/0x330 alloc_pages_mpol+0x7d/0x160 alloc_pages_noprof+0x50/0xa0 __pte_alloc_kernel+0x1e/0x1f0 ... <snip> There are usecases for these modifiers when a large allocation request should rather fail than trigger OOM killer which wouldn't be able to handle the situation anyway [1]. While we cannot change existing page table allocation code easily we can piggy back on scoped NOWAIT allocation for them that we already have in place. The rationale is that the bulk of the consumed memory is sitting in pages backing the vmalloc allocation. Page tables are only participating a tiny fraction. Moreover page tables for virtually allocated areas are never reclaimed so the longer the system runs to less likely they are. It makes sense to allow an approximation of __GFP_RETRY_MAYFAIL and __GFP_NORETRY even if the page table allocation part is much weaker. This doesn't break the failure mode while it allows for the no OOM semantic. [1] https://lore.kernel.org/all/32bd9bed-a939-69c4-696d-f7f9a5fe31d8@redhat.com/T/#u Link: https://lkml.kernel.org/r/20260302114740.2668450-2-urezki@gmail.com Signed-off-by: Michal Hocko <mhocko@suse.com> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Tested-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Baoquan He <bhe@redhat.com> Cc: Mikulas Patocka <mpatocka@redhat.com> Cc: Vishal Moola (Oracle) <vishal.moola@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Uladzislau Rezki (Sony)
|
0edd78cd4d |
mm/vmalloc: fix incorrect size reporting on allocation failure
When __vmalloc_area_node() fails to allocate pages, the failure message may report an incorrect allocation size, for example: vmalloc error: size 0, failed to allocate pages, ... This happens because the warning prints area->nr_pages * PAGE_SIZE. At this point, area->nr_pages may be zero or partly populated thus it is not valid. Report the originally requested allocation size instead by using nr_small_pages * PAGE_SIZE, which reflects the actual number of pages being requested by user. Link: https://lkml.kernel.org/r/20260302114740.2668450-1-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Mikulas Patocka <mpatocka@redhat.com> Reviewed-by: Vishal Moola (Oracle) <vishal.moola@gmail.com> Cc: Baoquan He <bhe@redhat.com> Cc: Michal Hocko <mhocko@suse.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Pasha Tatashin
|
ec10636539 |
mm/vmalloc: export clear_vm_uninitialized_flag()
Patch series "Fix KASAN support for KHO restored vmalloc regions". When KHO restores a vmalloc area, it maps existing physical pages into a newly allocated virtual memory area. However, because these areas were not properly unpoisoned, KASAN would treat any access to the restored region as out-of-bounds, as seen in the following trace: BUG: KASAN: vmalloc-out-of-bounds in kho_test_restore_data.isra.0+0x17b/0x2cd Read of size 8 at addr ffffc90000025000 by task swapper/0/1 [...] Call Trace: [...] kasan_report+0xe8/0x120 kho_test_restore_data.isra.0+0x17b/0x2cd kho_test_init+0x15a/0x1f0 do_one_initcall+0xd5/0x4b0 The fix involves deferring KASAN's default poisoning by using the VM_UNINITIALIZED flag during allocation, manually unpoisoning the memory once it is correctly mapped, and then clearing the uninitialized flag using a newly exported helper. This patch (of 2): Make clear_vm_uninitialized_flag() available to other parts of the kernel that need to manage vmalloc areas manually, such as KHO for restoring vmallocs. Link: https://lkml.kernel.org/r/20260225220223.1695350-1-pasha.tatashin@soleen.com Link: https://lkml.kernel.org/r/20260225223857.1714801-2-pasha.tatashin@soleen.com Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com> Acked-by: Pratyush Yadav (Google) <pratyush@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Liam Howlett <liam.howlett@oracle.com> Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Johannes Weiner
|
c466412c73 |
mm: memcontrol: switch to native NR_VMALLOC vmstat counter
Eliminates the custom memcg counter and results in a single, consolidated accounting call in vmalloc code. Link: https://lkml.kernel.org/r/20260223160147.3792777-2-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Roman Gushchin <roman.gushchin@linux.dev> Reviewed-by: Vishal Moola (Oracle) <vishal.moola@gmail.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Johannes Weiner
|
b9ec0ed907 |
mm: vmalloc: streamline vmalloc memory accounting
Use a vmstat counter instead of a custom, open-coded atomic. This has the added benefit of making the data available per-node, and prepares for cleaning up the memcg accounting as well. Link: https://lkml.kernel.org/r/20260223160147.3792777-1-hannes@cmpxchg.org Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Roman Gushchin <roman.gushchin@linux.dev> Reviewed-by: Vishal Moola (Oracle) <vishal.moola@gmail.com> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Linus Torvalds
|
bf4afc53b7 |
Convert 'alloc_obj' family to use the new default GFP_KERNEL argument
This was done entirely with mindless brute force, using
git grep -l '\<k[vmz]*alloc_objs*(.*, GFP_KERNEL)' |
xargs sed -i 's/\(alloc_objs*(.*\), GFP_KERNEL)/\1)/'
to convert the new alloc_obj() users that had a simple GFP_KERNEL
argument to just drop that argument.
Note that due to the extreme simplicity of the scripting, any slightly
more complex cases spread over multiple lines would not be triggered:
they definitely exist, but this covers the vast bulk of the cases, and
the resulting diff is also then easier to check automatically.
For the same reason the 'flex' versions will be done as a separate
conversion.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
|
||
|
Kees Cook
|
69050f8d6d |
treewide: Replace kmalloc with kmalloc_obj for non-scalar types
This is the result of running the Coccinelle script from scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to avoid scalar types (which need careful case-by-case checking), and instead replace kmalloc-family calls that allocate struct or union object instances: Single allocations: kmalloc(sizeof(TYPE), ...) are replaced with: kmalloc_obj(TYPE, ...) Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...) are replaced with: kmalloc_objs(TYPE, COUNT, ...) Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...) are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...) (where TYPE may also be *VAR) The resulting allocations no longer return "void *", instead returning "TYPE *". Signed-off-by: Kees Cook <kees@kernel.org> |
||
|
Andrew Morton
|
f84b65b045 |
Merge branch 'mm-hotfixes-stable' into mm-stable to pick up "mm/shmem,
swap: fix race of truncate and swap entry split", needed for merging "mm, swap: cleanup swap entry management workflow". |
||
|
Alice Ryhl
|
d607690750 |
vmalloc: export vrealloc_node_align_noprof
This symbol is used from the Nova driver, so it needs to be exported to avoid a build failure when building Nova as a module. ERROR: modpost: "vrealloc_node_align_noprof" [drivers/gpu/nova-core/nova_core.ko] undefined! ERROR: modpost: "vrealloc_node_align_noprof" [samples/rust/rust_dma.ko] undefined! This error is only triggered if helpers are inlined into Rust. Otherwise, Nova will call the exported symbol rust_helper_vrealloc_node_align() instead. There is no Fixes: tag as that feature is still WIP. I used non-GPL EXPORT_SYMBOL to match the rest of the file, but let me know if I should use EXPORT_SYMBOL_GPL. Link: https://lkml.kernel.org/r/20260107-export-vrealloc_node_align_noprof-v1-1-a581bec13054@google.com Signed-off-by: Alice Ryhl <aliceryhl@google.com> Reviewed-by: Danilo Krummrich <dakr@kernel.org> Cc: Andreas Hindborg <a.hindborg@kernel.org> Cc: Björn Roy Baron <bjorn3_gh@protonmail.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Gary Guo <gary@garyguo.net> Cc: Miguel Ojeda <ojeda@kernel.org> Cc: Trevor Gross <tmgross@umich.edu> Cc: Uladzislau Rezki <urezki@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Deepanshu Kartikey
|
5747435e0f |
mm/vmalloc: prevent RCU stalls in kasan_release_vmalloc_node
When CONFIG_PAGE_OWNER is enabled, freeing KASAN shadow pages during vmalloc cleanup triggers expensive stack unwinding that acquires RCU read locks. Processing a large purge_list without rescheduling can cause the task to hold CPU for extended periods (10+ seconds), leading to RCU stalls and potential OOM conditions. The issue manifests in purge_vmap_node() -> kasan_release_vmalloc_node() where iterating through hundreds or thousands of vmap_area entries and freeing their associated shadow pages causes: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P6229/1:b..l ... task:kworker/0:17 state:R running task stack:28840 pid:6229 ... kasan_release_vmalloc_node+0x1ba/0xad0 mm/vmalloc.c:2299 purge_vmap_node+0x1ba/0xad0 mm/vmalloc.c:2299 Each call to kasan_release_vmalloc() can free many pages, and with page_owner tracking, each free triggers save_stack() which performs stack unwinding under RCU read lock. Without yielding, this creates an unbounded RCU critical section. Add periodic cond_resched() calls within the loop to allow: - RCU grace periods to complete - Other tasks to run - Scheduler to preempt when needed The fix uses need_resched() for immediate response under load, with a batch count of 32 as a guaranteed upper bound to prevent worst-case stalls even under light load. Link: https://lkml.kernel.org/r/20260112103612.627247-1-kartikey406@gmail.com Signed-off-by: Deepanshu Kartikey <kartikey406@gmail.com> Reported-by: syzbot+d8d4c31d40f868eaea30@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=d8d4c31d40f868eaea30 Link: https://lore.kernel.org/all/20260112084723.622910-1-kartikey406@gmail.com/T/ [v1] Suggested-by: Uladzislau Rezki <urezki@gmail.com> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Hillf Danton <hdanton@sina.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Andrey Ryabinin
|
9b47d4eea3 |
mm/kasan: fix KASAN poisoning in vrealloc()
A KASAN warning can be triggered when vrealloc() changes the requested
size to a value that is not aligned to KASAN_GRANULE_SIZE.
------------[ cut here ]------------
WARNING: CPU: 2 PID: 1 at mm/kasan/shadow.c:174 kasan_unpoison+0x40/0x48
...
pc : kasan_unpoison+0x40/0x48
lr : __kasan_unpoison_vmalloc+0x40/0x68
Call trace:
kasan_unpoison+0x40/0x48 (P)
vrealloc_node_align_noprof+0x200/0x320
bpf_patch_insn_data+0x90/0x2f0
convert_ctx_accesses+0x8c0/0x1158
bpf_check+0x1488/0x1900
bpf_prog_load+0xd20/0x1258
__sys_bpf+0x96c/0xdf0
__arm64_sys_bpf+0x50/0xa0
invoke_syscall+0x90/0x160
Introduce a dedicated kasan_vrealloc() helper that centralizes KASAN
handling for vmalloc reallocations. The helper accounts for KASAN granule
alignment when growing or shrinking an allocation and ensures that partial
granules are handled correctly.
Use this helper from vrealloc_node_align_noprof() to fix poisoning logic.
[ryabinin.a.a@gmail.com: move kasan_enabled() check, fix build]
Link: https://lkml.kernel.org/r/20260119144509.32767-1-ryabinin.a.a@gmail.com
Link: https://lkml.kernel.org/r/20260113191516.31015-1-ryabinin.a.a@gmail.com
Fixes:
|
||
|
Kevin Brodsky
|
0a096ab7a3 |
mm: introduce generic lazy_mmu helpers
The implementation of the lazy MMU mode is currently entirely
arch-specific; core code directly calls arch helpers:
arch_{enter,leave}_lazy_mmu_mode().
We are about to introduce support for nested lazy MMU sections. As things
stand we'd have to duplicate that logic in every arch implementing
lazy_mmu - adding to a fair amount of logic already duplicated across
lazy_mmu implementations.
This patch therefore introduces a new generic layer that calls the
existing arch_* helpers. Two pair of calls are introduced:
* lazy_mmu_mode_enable() ... lazy_mmu_mode_disable()
This is the standard case where the mode is enabled for a given
block of code by surrounding it with enable() and disable()
calls.
* lazy_mmu_mode_pause() ... lazy_mmu_mode_resume()
This is for situations where the mode is temporarily disabled
by first calling pause() and then resume() (e.g. to prevent any
batching from occurring in a critical section).
The documentation in <linux/pgtable.h> will be updated in a subsequent
patch.
No functional change should be introduced at this stage. The
implementation of enable()/resume() and disable()/pause() is currently
identical, but nesting support will change that.
Most of the call sites have been updated using the following Coccinelle
script:
@@
@@
{
...
- arch_enter_lazy_mmu_mode();
+ lazy_mmu_mode_enable();
...
- arch_leave_lazy_mmu_mode();
+ lazy_mmu_mode_disable();
...
}
@@
@@
{
...
- arch_leave_lazy_mmu_mode();
+ lazy_mmu_mode_pause();
...
- arch_enter_lazy_mmu_mode();
+ lazy_mmu_mode_resume();
...
}
A couple of notes regarding x86:
* Xen is currently the only case where explicit handling is required
for lazy MMU when context-switching. This is purely an
implementation detail and using the generic lazy_mmu_mode_*
functions would cause trouble when nesting support is introduced,
because the generic functions must be called from the current task.
For that reason we still use arch_leave() and arch_enter() there.
* x86 calls arch_flush_lazy_mmu_mode() unconditionally in a few
places, but only defines it if PARAVIRT_XXL is selected, and we
are removing the fallback in <linux/pgtable.h>. Add a new fallback
definition to <asm/pgtable.h> to keep things building.
Link: https://lkml.kernel.org/r/20251215150323.2218608-8-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky <kevin.brodsky@arm.com>
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Reviewed-by: Yeoreum Yun <yeoreum.yun@arm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: Borislav Betkov <bp@alien8.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: David Hildenbrand (Red Hat) <david@kernel.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Juegren Gross <jgross@suse.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Gleinxer <tglx@linutronix.de>
Cc: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Brendan Jackman
|
a03ed8f144 |
mm/vmalloc: clarify why vmap_range_noflush() might sleep
The only reason vmap_range_noflush() can sleep is because of pagetable allocations. The actual allocation mechanism is arch-specific so might_alloc() doesn't work here (what GFP flags would be used?). Hence, just add a comment. Also note that this might do a TLB shootdown. This is not actually sleeping but it requires IRQs on for x86, and might_sleep() incidentally serves to detect violations of that too. Link: https://lkml.kernel.org/r/20251215-b4-vmalloc-might_alloc-v3-1-92dd8e406868@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Bagas Sanjaya
|
eb3f781ab7 |
mm: vmalloc: fix up vrealloc_node_align() kernel-doc macro name
Sphinx reports kernel-doc warning:
WARNING: ./mm/vmalloc.c:4284 expecting prototype for vrealloc_node_align_noprof(). Prototype was for vrealloc_node_align() instead
Fix the macro name in vrealloc_node_align_noprof() kernel-doc comment.
Link: https://lkml.kernel.org/r/20251219014006.16328-5-bagasdotme@gmail.com
Fixes:
|
||
|
Maciej Wieczor-Retman
|
6f13db031e |
kasan: refactor pcpu kasan vmalloc unpoison
A KASAN tag mismatch, possibly causing a kernel panic, can be observed
on systems with a tag-based KASAN enabled and with multiple NUMA nodes.
It was reported on arm64 and reproduced on x86. It can be explained in
the following points:
1. There can be more than one virtual memory chunk.
2. Chunk's base address has a tag.
3. The base address points at the first chunk and thus inherits
the tag of the first chunk.
4. The subsequent chunks will be accessed with the tag from the
first chunk.
5. Thus, the subsequent chunks need to have their tag set to
match that of the first chunk.
Refactor code by reusing __kasan_unpoison_vmalloc in a new helper in
preparation for the actual fix.
Link: https://lkml.kernel.org/r/eb61d93b907e262eefcaa130261a08bcb6c5ce51.1764874575.git.m.wieczorretman@pm.me
Fixes:
|