mirror of
https://github.com/torvalds/linux.git
synced 2026-05-31 18:43:33 +02:00
Merge patch series "riscv: Memory Hot(Un)Plug support"
Björn Töpel <bjorn@kernel.org> says:
From: Björn Töpel <bjorn@rivosinc.com>
================================================================
Memory Hot(Un)Plug support (and ZONE_DEVICE) for the RISC-V port
================================================================
Introduction
============
To quote "Documentation/admin-guide/mm/memory-hotplug.rst": "Memory
hot(un)plug allows for increasing and decreasing the size of physical
memory available to a machine at runtime."
This series adds memory hot(un)plugging, and ZONE_DEVICE support for
the RISC-V Linux port.
MM configuration
================
RISC-V MM has the following configuration:
* Memory blocks are 128M, analogous to x86-64. It uses PMD
("hugepage") vmemmaps. From that follows that 2M (PMD) worth of
vmemmap spans 32768 pages á 4K which gets us 128M.
* The pageblock size is the minimum minimum virtio_mem size, and on
RISC-V it's 2M (2^9 * 4K).
Implementation
==============
The PGD table on RISC-V is shared/copied between for all processes. To
avoid doing page table synchronization, the first patch (patch 1)
pre-allocated the PGD entries for vmemmap/direct map. By doing that
the init_mm PGD will be fixed at kernel init, and synchronization can
be avoided all together.
The following two patches (patch 2-3) does some preparations, followed
by the actual MHP implementation (patch 4-5). Then, MHP and virtio-mem
are enabled (patch 6-7), and finally ZONE_DEVICE support is added
(patch 8).
MHP and locking
===============
TL;DR: The MHP does not step on any toes, except for ptdump.
Additional locking is required for ptdump.
Long version: For v2 I spent some time digging into init_mm
synchronization/update. Here are my findings, and I'd love them to be
corrected if incorrect.
It's been a gnarly path...
The `init_mm` structure is a special mm (perhaps not a "real" one).
It's a "lazy context" that tracks kernel page table resources, e.g.,
the kernel page table (swapper_pg_dir), a kernel page_table_lock (more
about the usage below), mmap_lock, and such.
`init_mm` does not track/contain any VMAs. Having the `init_mm` is
convenient, so that the regular kernel page table walk/modify
functions can be used.
Now, `init_mm` being special means that the locking for kernel page
tables are special as well.
On RISC-V the PGD (top-level page table structure), similar to x86, is
shared (copied) with user processes. If the kernel PGD is modified, it
has to be synched to user-mode processes PGDs. This is avoided by
pre-populating the PGD, so it'll be fixed from boot.
The in-kernel pgd regions are documented in
`Documentation/arch/riscv/vm-layout.rst`.
The distinct regions are:
* vmemmap
* vmalloc/ioremap space
* direct mapping of all physical memory
* kasan
* modules, BPF
* kernel
Memory hotplug is the process of adding/removing memory to/from the
kernel.
Adding is done in two phases:
1. Add the memory to the kernel
2. Online memory, making it available to the page allocator.
Step 1 is partially architecture dependent, and updates the init_mm
page table:
* Update the direct map page tables. The direct map is a linear map,
representing all physical memory: `virt = phys + PAGE_OFFSET`
* Add a `struct page` for each added page of memory. Update the
vmemmap (virtual mapping to the `struct page`, so we can easily
transform a kernel virtual address to a `struct page *` address.
From an MHP perspective, there are two regions of the PGD that are
updated:
* vmemmap
* direct mapping of all physical memory
The `struct mm_struct` has a couple of locks in play:
* `spinlock_t page_table_lock` protects the page table, and some
counters
* `struct rw_semaphore mmap_lock` protect an mm's VMAs
Note again that `init_mm` does not contain any VMAs, but still uses
the mmap_lock in some places.
The `page_table_lock` was originally used to to protect all pages
tables, but more recently a split page table lock has been introduced.
The split lock has a per-table lock for the PTE and PMD tables. If
split lock is disabled, all tables are guarded by
`mm->page_table_lock` (for user processes). Split page table locks are
not used for init_mm.
MHP operations is typically synchronized using
`DEFINE_STATIC_PERCPU_RWSEM(mem_hotplug_lock)`.
Actors
------
The following non-MHP actors in the kernel traverses (read), and/or
modifies the kernel PGD.
* `ptdump`
Walks the entire `init_mm`, via `ptdump_walk_pgd()` with the
`mmap_write_lock(init_mm)` taken.
Observation: ptdump can race with MHP, and needs additional locking
to avoid crashes/races.
* `set_direct_*` / `arch/riscv/mm/pageattr.c`
The `set_direct_*` functionality is used to "synchronize" the
direct map to other kernel mappings, e.g. modules/kernel text. The
direct map is using "as large huge table mappings as possible",
which means that the `set_direct_*` might need to split the direct
map.
The `set_direct_*` functions operates with the
`mmap_write_lock(init_mm)` taken.
Observation: `set_direct_*` uses the direct map, but will never
modify the same entry as MHP. If there is a mapping, that entry will
never race with MHP. Further, MHP acts when memory is offline.
* HVO / `mm/hugetlb_vmemmap`
HVO optimizes the backing `struct page` for hugetlb pages, which
means changing the "vmemmap" region. HVO can split (merge?) a
vmemmap pmd. However, it will never race with MHP, since HVO only
operates at online memory. HVO cannot touch memory being MHP added
or removed.
* `apply_to_page_range`
Walks a range, creates pages and applies a callback (setting
permissions) for the page.
When creating a table, it might use `int __pte_alloc_kernel(pmd_t
*pmd)` which takes the `init_mm.page_table_lock` to synchronize pmd
populate.
Used by: `mm/vmalloc.c` and `mm/kasan/shadow.c`. The KASAN callback
takes the `init_mm.page_table_lock` to synchronize pte creation.
Observations: `apply_to_page_range` applies to the "vmalloc/ioremap
space" region, and "kasan" region. *Not* affected by MHP.
* `apply_to_existing_page_range`
Walks a range, applies a callback (setting permissions) for the
page (no page creation).
Used by: `kernel/bpf/arena.c` and `mm/kasan/shadow.c`. The KASAN
callback takes the `init_mm.page_table_lock` to synchronize pte
creation. *Not* affected by MHP regions.
* `apply_to_existing_page_range` applies to the "vmalloc/ioremap
space" region, and "kasan" region. *Not* affected by MHP regions.
* `ioremap_page_range` and `vmap_page_range`
Uses the same internal function, and might create table entries at
the "vmalloc/ioremap space" region. Can call
`__pte_alloc_kernel()` which takes the `init_mm.page_table_lock`
synchronizing pmd populate in the region. *Not* affected by MHP
regions.
Summary:
* MHP add will never modify the same page table entries, as any of
the other actors.
* MHP remove is done when memory is offlined, and will not clash
with any of the actors.
* Functions that walk the entire kernel page table need
synchronization
* It's sufficient to add the MHP lock ptdump.
Testing
=======
This series adds basic DT supported hotplugging. There is a QEMU
series enabling MHP for the RISC-V "virt" machine here: [1]
ACPI/MSI support is still in the making for RISC-V, and prior proper
(ACPI) PCI MSI support lands [2] and NUMA SRAT support [3], it hard to
try it out.
I've prepared a QEMU branch with proper ACPI GED/PC-DIMM support [4],
and a this series with the required prerequisites [5] (AIA, ACPI AIA
MADT, ACPI NUMA SRAT).
To test with virtio-mem, e.g.:
| qemu-system-riscv64 \
| -machine virt,aia=aplic-imsic \
| -cpu rv64,v=true,vlen=256,elen=64,h=true,zbkb=on,zbkc=on,zbkx=on,zkr=on,zkt=on,svinval=on,svnapot=on,svpbmt=on \
| -nodefaults \
| -nographic -smp 8 -kernel rv64-u-boot.bin \
| -drive file=rootfs.img,format=raw,if=virtio \
| -device virtio-rng-pci \
| -m 16G,slots=3,maxmem=32G \
| -object memory-backend-ram,id=mem0,size=16G \
| -numa node,nodeid=0,memdev=mem0 \
| -serial chardev:char0 \
| -mon chardev=char0,mode=readline \
| -chardev stdio,mux=on,id=char0 \
| -device pci-serial,id=serial0,chardev=char0 \
| -object memory-backend-ram,id=vmem0,size=2G \
| -device virtio-mem-pci,id=vm0,memdev=vmem0,node=0
where "rv64-u-boot.bin" is U-boot with EFI/ACPI-support (use [6] if
you're lazy).
In the QEMU monitor:
| (qemu) info memory-devices
| (qemu) qom-set vm0 requested-size 1G
...to test DAX/KMEM, use the follow QEMU parameters:
| -object memory-backend-file,id=mem1,share=on,mem-path=virtio_pmem.img,size=4G \
| -device virtio-pmem-pci,memdev=mem1,id=nv1
and the regular ndctl/daxctl dance.
If you're brave to try the ACPI branch, add "acpi=on" to "-machine
virt", and test PC-DIMM MHP (in addition to virtio-{p},mem):
In the QEMU monitor:
| (qemu) object_add memory-backend-ram,id=mem1,size=1G
| (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
You can also try hot-remove with some QEMU options, say:
| -object memory-backend-file,id=mem-1,size=256M,mem-path=/pagesize-2MB
| -device pc-dimm,id=mem1,memdev=mem-1
| -object memory-backend-file,id=mem-2,size=1G,mem-path=/pagesize-1GB
| -device pc-dimm,id=mem2,memdev=mem-2
| -object memory-backend-file,id=mem-3,size=256M,mem-path=/pagesize-2MB
| -device pc-dimm,id=mem3,memdev=mem-3
Remove "acpi=on" to run with DT.
Thanks to Alex, Andrew, David, and Oscar for all
comments/tests/fixups.
References
==========
[1] https://lore.kernel.org/qemu-devel/20240521105635.795211-1-bjorn@kernel.org/
[2] https://lore.kernel.org/linux-riscv/20240501121742.1215792-1-sunilvl@ventanamicro.com/
[3] https://lore.kernel.org/linux-riscv/cover.1713778236.git.haibo1.xu@intel.com/
[4] https://github.com/bjoto/qemu/commits/virtio-mem-pc-dimm-mhp-acpi-v2/
[5] https://github.com/bjoto/linux/commits/mhp-v4-acpi
[6] https://github.com/bjoto/riscv-rootfs-utils/tree/acpi
* b4-shazam-merge:
riscv: Enable DAX VMEMMAP optimization
riscv: mm: Add support for ZONE_DEVICE
virtio-mem: Enable virtio-mem for RISC-V
riscv: Enable memory hotplugging for RISC-V
riscv: mm: Take memory hotplug read-lock during kernel page table dump
riscv: mm: Add memory hotplugging support
riscv: mm: Add pfn_to_kaddr() implementation
riscv: mm: Refactor create_linear_mapping_range() for memory hot add
riscv: mm: Change attribute from __init to __meminit for page functions
riscv: mm: Pre-allocate vmemmap/direct map/kasan PGD entries
riscv: mm: Properly forward vmemmap_populate() altmap parameter
Link: https://lore.kernel.org/r/20240605114100.315918-1-bjorn@kernel.org
Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
This commit is contained in:
Palmer Dabbelt
commit
60a6707f58
|
|
@ -16,6 +16,8 @@ config RISCV
|
|||
select ACPI_REDUCED_HARDWARE_ONLY if ACPI
|
||||
select ARCH_DMA_DEFAULT_COHERENT
|
||||
select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION
|
||||
select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP
|
||||
select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG
|
||||
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
|
||||
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
|
||||
select ARCH_HAS_BINFMT_FLAT
|
||||
|
|
@ -35,6 +37,7 @@ config RISCV
|
|||
select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
|
||||
select ARCH_HAS_PMEM_API
|
||||
select ARCH_HAS_PREPARE_SYNC_CORE_CMD
|
||||
select ARCH_HAS_PTE_DEVMAP if 64BIT && MMU
|
||||
select ARCH_HAS_PTE_SPECIAL
|
||||
select ARCH_HAS_SET_DIRECT_MAP if MMU
|
||||
select ARCH_HAS_SET_MEMORY if MMU
|
||||
|
|
@ -46,6 +49,7 @@ config RISCV
|
|||
select ARCH_HAS_UBSAN
|
||||
select ARCH_HAS_VDSO_DATA
|
||||
select ARCH_KEEP_MEMBLOCK if ACPI
|
||||
select ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE if 64BIT && MMU
|
||||
select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX
|
||||
select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT
|
||||
select ARCH_STACKWALK
|
||||
|
|
@ -69,6 +73,7 @@ config RISCV
|
|||
select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT
|
||||
select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
|
||||
select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL
|
||||
select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP
|
||||
select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP
|
||||
select ARCH_WANTS_NO_INSTR
|
||||
select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
|
||||
|
|
|
|||
|
|
@ -6,8 +6,6 @@
|
|||
|
||||
#ifndef __ASSEMBLY__
|
||||
|
||||
#ifdef CONFIG_KASAN
|
||||
|
||||
/*
|
||||
* The following comment was copied from arm64:
|
||||
* KASAN_SHADOW_START: beginning of the kernel virtual addresses.
|
||||
|
|
@ -34,6 +32,8 @@
|
|||
*/
|
||||
#define KASAN_SHADOW_START ((KASAN_SHADOW_END - KASAN_SHADOW_SIZE) & PGDIR_MASK)
|
||||
#define KASAN_SHADOW_END MODULES_LOWEST_VADDR
|
||||
|
||||
#ifdef CONFIG_KASAN
|
||||
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
|
||||
|
||||
void kasan_init(void);
|
||||
|
|
|
|||
|
|
@ -31,8 +31,8 @@ typedef struct {
|
|||
#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK)
|
||||
#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK)
|
||||
|
||||
void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa,
|
||||
phys_addr_t sz, pgprot_t prot);
|
||||
void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
|
||||
pgprot_t prot);
|
||||
#endif /* __ASSEMBLY__ */
|
||||
|
||||
#endif /* _ASM_RISCV_MMU_H */
|
||||
|
|
|
|||
|
|
@ -188,6 +188,11 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x);
|
|||
|
||||
unsigned long kaslr_offset(void);
|
||||
|
||||
static __always_inline void *pfn_to_kaddr(unsigned long pfn)
|
||||
{
|
||||
return __va(pfn << PAGE_SHIFT);
|
||||
}
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
|
||||
#define virt_addr_valid(vaddr) ({ \
|
||||
|
|
|
|||
|
|
@ -398,4 +398,24 @@ static inline struct page *pgd_page(pgd_t pgd)
|
|||
#define p4d_offset p4d_offset
|
||||
p4d_t *p4d_offset(pgd_t *pgd, unsigned long address);
|
||||
|
||||
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
||||
static inline int pte_devmap(pte_t pte);
|
||||
static inline pte_t pmd_pte(pmd_t pmd);
|
||||
|
||||
static inline int pmd_devmap(pmd_t pmd)
|
||||
{
|
||||
return pte_devmap(pmd_pte(pmd));
|
||||
}
|
||||
|
||||
static inline int pud_devmap(pud_t pud)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
static inline int pgd_devmap(pgd_t pgd)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* _ASM_RISCV_PGTABLE_64_H */
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@
|
|||
#define _PAGE_SOFT (3 << 8) /* Reserved for software */
|
||||
|
||||
#define _PAGE_SPECIAL (1 << 8) /* RSW: 0x1 */
|
||||
#define _PAGE_DEVMAP (1 << 9) /* RSW, devmap */
|
||||
#define _PAGE_TABLE _PAGE_PRESENT
|
||||
|
||||
/*
|
||||
|
|
|
|||
|
|
@ -165,7 +165,7 @@ struct pt_alloc_ops {
|
|||
#endif
|
||||
};
|
||||
|
||||
extern struct pt_alloc_ops pt_ops __initdata;
|
||||
extern struct pt_alloc_ops pt_ops __meminitdata;
|
||||
|
||||
#ifdef CONFIG_MMU
|
||||
/* Number of PGD entries that a user-mode program can use */
|
||||
|
|
@ -403,6 +403,13 @@ static inline int pte_special(pte_t pte)
|
|||
return pte_val(pte) & _PAGE_SPECIAL;
|
||||
}
|
||||
|
||||
#ifdef CONFIG_ARCH_HAS_PTE_DEVMAP
|
||||
static inline int pte_devmap(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_DEVMAP;
|
||||
}
|
||||
#endif
|
||||
|
||||
/* static inline pte_t pte_rdprotect(pte_t pte) */
|
||||
|
||||
static inline pte_t pte_wrprotect(pte_t pte)
|
||||
|
|
@ -444,6 +451,11 @@ static inline pte_t pte_mkspecial(pte_t pte)
|
|||
return __pte(pte_val(pte) | _PAGE_SPECIAL);
|
||||
}
|
||||
|
||||
static inline pte_t pte_mkdevmap(pte_t pte)
|
||||
{
|
||||
return __pte(pte_val(pte) | _PAGE_DEVMAP);
|
||||
}
|
||||
|
||||
static inline pte_t pte_mkhuge(pte_t pte)
|
||||
{
|
||||
return pte;
|
||||
|
|
@ -734,6 +746,11 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd)
|
|||
return pte_pmd(pte_mkdirty(pmd_pte(pmd)));
|
||||
}
|
||||
|
||||
static inline pmd_t pmd_mkdevmap(pmd_t pmd)
|
||||
{
|
||||
return pte_pmd(pte_mkdevmap(pmd_pte(pmd)));
|
||||
}
|
||||
|
||||
static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
|
||||
pmd_t *pmdp, pmd_t pmd)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@
|
|||
|
||||
#include <asm/fixmap.h>
|
||||
#include <asm/io.h>
|
||||
#include <asm/kasan.h>
|
||||
#include <asm/numa.h>
|
||||
#include <asm/pgtable.h>
|
||||
#include <asm/sections.h>
|
||||
|
|
@ -295,7 +296,7 @@ static void __init setup_bootmem(void)
|
|||
}
|
||||
|
||||
#ifdef CONFIG_MMU
|
||||
struct pt_alloc_ops pt_ops __initdata;
|
||||
struct pt_alloc_ops pt_ops __meminitdata;
|
||||
|
||||
pgd_t swapper_pg_dir[PTRS_PER_PGD] __page_aligned_bss;
|
||||
pgd_t trampoline_pg_dir[PTRS_PER_PGD] __page_aligned_bss;
|
||||
|
|
@ -357,7 +358,7 @@ static inline pte_t *__init get_pte_virt_fixmap(phys_addr_t pa)
|
|||
return (pte_t *)set_fixmap_offset(FIX_PTE, pa);
|
||||
}
|
||||
|
||||
static inline pte_t *__init get_pte_virt_late(phys_addr_t pa)
|
||||
static inline pte_t *__meminit get_pte_virt_late(phys_addr_t pa)
|
||||
{
|
||||
return (pte_t *) __va(pa);
|
||||
}
|
||||
|
|
@ -376,7 +377,7 @@ static inline phys_addr_t __init alloc_pte_fixmap(uintptr_t va)
|
|||
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
|
||||
}
|
||||
|
||||
static phys_addr_t __init alloc_pte_late(uintptr_t va)
|
||||
static phys_addr_t __meminit alloc_pte_late(uintptr_t va)
|
||||
{
|
||||
struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0);
|
||||
|
||||
|
|
@ -384,9 +385,8 @@ static phys_addr_t __init alloc_pte_late(uintptr_t va)
|
|||
return __pa((pte_t *)ptdesc_address(ptdesc));
|
||||
}
|
||||
|
||||
static void __init create_pte_mapping(pte_t *ptep,
|
||||
uintptr_t va, phys_addr_t pa,
|
||||
phys_addr_t sz, pgprot_t prot)
|
||||
static void __meminit create_pte_mapping(pte_t *ptep, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
|
||||
pgprot_t prot)
|
||||
{
|
||||
uintptr_t pte_idx = pte_index(va);
|
||||
|
||||
|
|
@ -440,7 +440,7 @@ static pmd_t *__init get_pmd_virt_fixmap(phys_addr_t pa)
|
|||
return (pmd_t *)set_fixmap_offset(FIX_PMD, pa);
|
||||
}
|
||||
|
||||
static pmd_t *__init get_pmd_virt_late(phys_addr_t pa)
|
||||
static pmd_t *__meminit get_pmd_virt_late(phys_addr_t pa)
|
||||
{
|
||||
return (pmd_t *) __va(pa);
|
||||
}
|
||||
|
|
@ -457,7 +457,7 @@ static phys_addr_t __init alloc_pmd_fixmap(uintptr_t va)
|
|||
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
|
||||
}
|
||||
|
||||
static phys_addr_t __init alloc_pmd_late(uintptr_t va)
|
||||
static phys_addr_t __meminit alloc_pmd_late(uintptr_t va)
|
||||
{
|
||||
struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0);
|
||||
|
||||
|
|
@ -465,9 +465,9 @@ static phys_addr_t __init alloc_pmd_late(uintptr_t va)
|
|||
return __pa((pmd_t *)ptdesc_address(ptdesc));
|
||||
}
|
||||
|
||||
static void __init create_pmd_mapping(pmd_t *pmdp,
|
||||
uintptr_t va, phys_addr_t pa,
|
||||
phys_addr_t sz, pgprot_t prot)
|
||||
static void __meminit create_pmd_mapping(pmd_t *pmdp,
|
||||
uintptr_t va, phys_addr_t pa,
|
||||
phys_addr_t sz, pgprot_t prot)
|
||||
{
|
||||
pte_t *ptep;
|
||||
phys_addr_t pte_phys;
|
||||
|
|
@ -503,7 +503,7 @@ static pud_t *__init get_pud_virt_fixmap(phys_addr_t pa)
|
|||
return (pud_t *)set_fixmap_offset(FIX_PUD, pa);
|
||||
}
|
||||
|
||||
static pud_t *__init get_pud_virt_late(phys_addr_t pa)
|
||||
static pud_t *__meminit get_pud_virt_late(phys_addr_t pa)
|
||||
{
|
||||
return (pud_t *)__va(pa);
|
||||
}
|
||||
|
|
@ -521,7 +521,7 @@ static phys_addr_t __init alloc_pud_fixmap(uintptr_t va)
|
|||
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
|
||||
}
|
||||
|
||||
static phys_addr_t alloc_pud_late(uintptr_t va)
|
||||
static phys_addr_t __meminit alloc_pud_late(uintptr_t va)
|
||||
{
|
||||
unsigned long vaddr;
|
||||
|
||||
|
|
@ -541,7 +541,7 @@ static p4d_t *__init get_p4d_virt_fixmap(phys_addr_t pa)
|
|||
return (p4d_t *)set_fixmap_offset(FIX_P4D, pa);
|
||||
}
|
||||
|
||||
static p4d_t *__init get_p4d_virt_late(phys_addr_t pa)
|
||||
static p4d_t *__meminit get_p4d_virt_late(phys_addr_t pa)
|
||||
{
|
||||
return (p4d_t *)__va(pa);
|
||||
}
|
||||
|
|
@ -559,7 +559,7 @@ static phys_addr_t __init alloc_p4d_fixmap(uintptr_t va)
|
|||
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
|
||||
}
|
||||
|
||||
static phys_addr_t alloc_p4d_late(uintptr_t va)
|
||||
static phys_addr_t __meminit alloc_p4d_late(uintptr_t va)
|
||||
{
|
||||
unsigned long vaddr;
|
||||
|
||||
|
|
@ -568,9 +568,8 @@ static phys_addr_t alloc_p4d_late(uintptr_t va)
|
|||
return __pa(vaddr);
|
||||
}
|
||||
|
||||
static void __init create_pud_mapping(pud_t *pudp,
|
||||
uintptr_t va, phys_addr_t pa,
|
||||
phys_addr_t sz, pgprot_t prot)
|
||||
static void __meminit create_pud_mapping(pud_t *pudp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
|
||||
pgprot_t prot)
|
||||
{
|
||||
pmd_t *nextp;
|
||||
phys_addr_t next_phys;
|
||||
|
|
@ -595,9 +594,8 @@ static void __init create_pud_mapping(pud_t *pudp,
|
|||
create_pmd_mapping(nextp, va, pa, sz, prot);
|
||||
}
|
||||
|
||||
static void __init create_p4d_mapping(p4d_t *p4dp,
|
||||
uintptr_t va, phys_addr_t pa,
|
||||
phys_addr_t sz, pgprot_t prot)
|
||||
static void __meminit create_p4d_mapping(p4d_t *p4dp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
|
||||
pgprot_t prot)
|
||||
{
|
||||
pud_t *nextp;
|
||||
phys_addr_t next_phys;
|
||||
|
|
@ -653,9 +651,8 @@ static void __init create_p4d_mapping(p4d_t *p4dp,
|
|||
#define create_pmd_mapping(__pmdp, __va, __pa, __sz, __prot) do {} while(0)
|
||||
#endif /* __PAGETABLE_PMD_FOLDED */
|
||||
|
||||
void __init create_pgd_mapping(pgd_t *pgdp,
|
||||
uintptr_t va, phys_addr_t pa,
|
||||
phys_addr_t sz, pgprot_t prot)
|
||||
void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
|
||||
pgprot_t prot)
|
||||
{
|
||||
pgd_next_t *nextp;
|
||||
phys_addr_t next_phys;
|
||||
|
|
@ -680,8 +677,7 @@ void __init create_pgd_mapping(pgd_t *pgdp,
|
|||
create_pgd_next_mapping(nextp, va, pa, sz, prot);
|
||||
}
|
||||
|
||||
static uintptr_t __init best_map_size(phys_addr_t pa, uintptr_t va,
|
||||
phys_addr_t size)
|
||||
static uintptr_t __meminit best_map_size(phys_addr_t pa, uintptr_t va, phys_addr_t size)
|
||||
{
|
||||
if (debug_pagealloc_enabled())
|
||||
return PAGE_SIZE;
|
||||
|
|
@ -717,7 +713,7 @@ asmlinkage void __init __copy_data(void)
|
|||
#endif
|
||||
|
||||
#ifdef CONFIG_STRICT_KERNEL_RWX
|
||||
static __init pgprot_t pgprot_from_va(uintptr_t va)
|
||||
static __meminit pgprot_t pgprot_from_va(uintptr_t va)
|
||||
{
|
||||
if (is_va_kernel_text(va))
|
||||
return PAGE_KERNEL_READ_EXEC;
|
||||
|
|
@ -742,7 +738,7 @@ void mark_rodata_ro(void)
|
|||
set_memory_ro);
|
||||
}
|
||||
#else
|
||||
static __init pgprot_t pgprot_from_va(uintptr_t va)
|
||||
static __meminit pgprot_t pgprot_from_va(uintptr_t va)
|
||||
{
|
||||
if (IS_ENABLED(CONFIG_64BIT) && !is_kernel_mapping(va))
|
||||
return PAGE_KERNEL;
|
||||
|
|
@ -1234,9 +1230,8 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa)
|
|||
pt_ops_set_fixmap();
|
||||
}
|
||||
|
||||
static void __init create_linear_mapping_range(phys_addr_t start,
|
||||
phys_addr_t end,
|
||||
uintptr_t fixed_map_size)
|
||||
static void __meminit create_linear_mapping_range(phys_addr_t start, phys_addr_t end,
|
||||
uintptr_t fixed_map_size, const pgprot_t *pgprot)
|
||||
{
|
||||
phys_addr_t pa;
|
||||
uintptr_t va, map_size;
|
||||
|
|
@ -1247,7 +1242,7 @@ static void __init create_linear_mapping_range(phys_addr_t start,
|
|||
best_map_size(pa, va, end - pa);
|
||||
|
||||
create_pgd_mapping(swapper_pg_dir, va, pa, map_size,
|
||||
pgprot_from_va(va));
|
||||
pgprot ? *pgprot : pgprot_from_va(va));
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1291,22 +1286,19 @@ static void __init create_linear_mapping_page_table(void)
|
|||
if (end >= __pa(PAGE_OFFSET) + memory_limit)
|
||||
end = __pa(PAGE_OFFSET) + memory_limit;
|
||||
|
||||
create_linear_mapping_range(start, end, 0);
|
||||
create_linear_mapping_range(start, end, 0, NULL);
|
||||
}
|
||||
|
||||
#ifdef CONFIG_STRICT_KERNEL_RWX
|
||||
create_linear_mapping_range(ktext_start, ktext_start + ktext_size, 0);
|
||||
create_linear_mapping_range(krodata_start,
|
||||
krodata_start + krodata_size, 0);
|
||||
create_linear_mapping_range(ktext_start, ktext_start + ktext_size, 0, NULL);
|
||||
create_linear_mapping_range(krodata_start, krodata_start + krodata_size, 0, NULL);
|
||||
|
||||
memblock_clear_nomap(ktext_start, ktext_size);
|
||||
memblock_clear_nomap(krodata_start, krodata_size);
|
||||
#endif
|
||||
|
||||
#ifdef CONFIG_KFENCE
|
||||
create_linear_mapping_range(kfence_pool,
|
||||
kfence_pool + KFENCE_POOL_SIZE,
|
||||
PAGE_SIZE);
|
||||
create_linear_mapping_range(kfence_pool, kfence_pool + KFENCE_POOL_SIZE, PAGE_SIZE, NULL);
|
||||
|
||||
memblock_clear_nomap(kfence_pool, KFENCE_POOL_SIZE);
|
||||
#endif
|
||||
|
|
@ -1438,7 +1430,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
|
|||
* memory hotplug, we are not able to update all the page tables with
|
||||
* the new PMDs.
|
||||
*/
|
||||
return vmemmap_populate_hugepages(start, end, node, NULL);
|
||||
return vmemmap_populate_hugepages(start, end, node, altmap);
|
||||
}
|
||||
#endif
|
||||
|
||||
|
|
@ -1492,11 +1484,19 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon
|
|||
panic("Failed to pre-allocate %s pages for %s area\n", lvl, area);
|
||||
}
|
||||
|
||||
#define PAGE_END KASAN_SHADOW_START
|
||||
|
||||
void __init pgtable_cache_init(void)
|
||||
{
|
||||
preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc");
|
||||
if (IS_ENABLED(CONFIG_MODULES))
|
||||
preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules");
|
||||
if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG)) {
|
||||
preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap");
|
||||
preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map");
|
||||
if (IS_ENABLED(CONFIG_KASAN))
|
||||
preallocate_pgd_pages_range(KASAN_SHADOW_START, KASAN_SHADOW_END, "kasan");
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
|
|
@ -1533,3 +1533,270 @@ struct execmem_info __init *execmem_arch_setup(void)
|
|||
}
|
||||
#endif /* CONFIG_MMU */
|
||||
#endif /* CONFIG_EXECMEM */
|
||||
|
||||
#ifdef CONFIG_MEMORY_HOTPLUG
|
||||
static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)
|
||||
{
|
||||
struct page *page = pmd_page(*pmd);
|
||||
struct ptdesc *ptdesc = page_ptdesc(page);
|
||||
pte_t *pte;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < PTRS_PER_PTE; i++) {
|
||||
pte = pte_start + i;
|
||||
if (!pte_none(*pte))
|
||||
return;
|
||||
}
|
||||
|
||||
pagetable_pte_dtor(ptdesc);
|
||||
if (PageReserved(page))
|
||||
free_reserved_page(page);
|
||||
else
|
||||
pagetable_free(ptdesc);
|
||||
pmd_clear(pmd);
|
||||
}
|
||||
|
||||
static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud)
|
||||
{
|
||||
struct page *page = pud_page(*pud);
|
||||
struct ptdesc *ptdesc = page_ptdesc(page);
|
||||
pmd_t *pmd;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < PTRS_PER_PMD; i++) {
|
||||
pmd = pmd_start + i;
|
||||
if (!pmd_none(*pmd))
|
||||
return;
|
||||
}
|
||||
|
||||
pagetable_pmd_dtor(ptdesc);
|
||||
if (PageReserved(page))
|
||||
free_reserved_page(page);
|
||||
else
|
||||
pagetable_free(ptdesc);
|
||||
pud_clear(pud);
|
||||
}
|
||||
|
||||
static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d)
|
||||
{
|
||||
struct page *page = p4d_page(*p4d);
|
||||
pud_t *pud;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < PTRS_PER_PUD; i++) {
|
||||
pud = pud_start + i;
|
||||
if (!pud_none(*pud))
|
||||
return;
|
||||
}
|
||||
|
||||
if (PageReserved(page))
|
||||
free_reserved_page(page);
|
||||
else
|
||||
free_pages((unsigned long)page_address(page), 0);
|
||||
p4d_clear(p4d);
|
||||
}
|
||||
|
||||
static void __meminit free_vmemmap_storage(struct page *page, size_t size,
|
||||
struct vmem_altmap *altmap)
|
||||
{
|
||||
int order = get_order(size);
|
||||
|
||||
if (altmap) {
|
||||
vmem_altmap_free(altmap, size >> PAGE_SHIFT);
|
||||
return;
|
||||
}
|
||||
|
||||
if (PageReserved(page)) {
|
||||
unsigned int nr_pages = 1 << order;
|
||||
|
||||
while (nr_pages--)
|
||||
free_reserved_page(page++);
|
||||
return;
|
||||
}
|
||||
|
||||
free_pages((unsigned long)page_address(page), order);
|
||||
}
|
||||
|
||||
static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, unsigned long end,
|
||||
bool is_vmemmap, struct vmem_altmap *altmap)
|
||||
{
|
||||
unsigned long next;
|
||||
pte_t *ptep, pte;
|
||||
|
||||
for (; addr < end; addr = next) {
|
||||
next = (addr + PAGE_SIZE) & PAGE_MASK;
|
||||
if (next > end)
|
||||
next = end;
|
||||
|
||||
ptep = pte_base + pte_index(addr);
|
||||
pte = ptep_get(ptep);
|
||||
if (!pte_present(*ptep))
|
||||
continue;
|
||||
|
||||
pte_clear(&init_mm, addr, ptep);
|
||||
if (is_vmemmap)
|
||||
free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap);
|
||||
}
|
||||
}
|
||||
|
||||
static void __meminit remove_pmd_mapping(pmd_t *pmd_base, unsigned long addr, unsigned long end,
|
||||
bool is_vmemmap, struct vmem_altmap *altmap)
|
||||
{
|
||||
unsigned long next;
|
||||
pte_t *pte_base;
|
||||
pmd_t *pmdp, pmd;
|
||||
|
||||
for (; addr < end; addr = next) {
|
||||
next = pmd_addr_end(addr, end);
|
||||
pmdp = pmd_base + pmd_index(addr);
|
||||
pmd = pmdp_get(pmdp);
|
||||
if (!pmd_present(pmd))
|
||||
continue;
|
||||
|
||||
if (pmd_leaf(pmd)) {
|
||||
pmd_clear(pmdp);
|
||||
if (is_vmemmap)
|
||||
free_vmemmap_storage(pmd_page(pmd), PMD_SIZE, altmap);
|
||||
continue;
|
||||
}
|
||||
|
||||
pte_base = (pte_t *)pmd_page_vaddr(*pmdp);
|
||||
remove_pte_mapping(pte_base, addr, next, is_vmemmap, altmap);
|
||||
free_pte_table(pte_base, pmdp);
|
||||
}
|
||||
}
|
||||
|
||||
static void __meminit remove_pud_mapping(pud_t *pud_base, unsigned long addr, unsigned long end,
|
||||
bool is_vmemmap, struct vmem_altmap *altmap)
|
||||
{
|
||||
unsigned long next;
|
||||
pud_t *pudp, pud;
|
||||
pmd_t *pmd_base;
|
||||
|
||||
for (; addr < end; addr = next) {
|
||||
next = pud_addr_end(addr, end);
|
||||
pudp = pud_base + pud_index(addr);
|
||||
pud = pudp_get(pudp);
|
||||
if (!pud_present(pud))
|
||||
continue;
|
||||
|
||||
if (pud_leaf(pud)) {
|
||||
if (pgtable_l4_enabled) {
|
||||
pud_clear(pudp);
|
||||
if (is_vmemmap)
|
||||
free_vmemmap_storage(pud_page(pud), PUD_SIZE, altmap);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
pmd_base = pmd_offset(pudp, 0);
|
||||
remove_pmd_mapping(pmd_base, addr, next, is_vmemmap, altmap);
|
||||
|
||||
if (pgtable_l4_enabled)
|
||||
free_pmd_table(pmd_base, pudp);
|
||||
}
|
||||
}
|
||||
|
||||
static void __meminit remove_p4d_mapping(p4d_t *p4d_base, unsigned long addr, unsigned long end,
|
||||
bool is_vmemmap, struct vmem_altmap *altmap)
|
||||
{
|
||||
unsigned long next;
|
||||
p4d_t *p4dp, p4d;
|
||||
pud_t *pud_base;
|
||||
|
||||
for (; addr < end; addr = next) {
|
||||
next = p4d_addr_end(addr, end);
|
||||
p4dp = p4d_base + p4d_index(addr);
|
||||
p4d = p4dp_get(p4dp);
|
||||
if (!p4d_present(p4d))
|
||||
continue;
|
||||
|
||||
if (p4d_leaf(p4d)) {
|
||||
if (pgtable_l5_enabled) {
|
||||
p4d_clear(p4dp);
|
||||
if (is_vmemmap)
|
||||
free_vmemmap_storage(p4d_page(p4d), P4D_SIZE, altmap);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
pud_base = pud_offset(p4dp, 0);
|
||||
remove_pud_mapping(pud_base, addr, next, is_vmemmap, altmap);
|
||||
|
||||
if (pgtable_l5_enabled)
|
||||
free_pud_table(pud_base, p4dp);
|
||||
}
|
||||
}
|
||||
|
||||
static void __meminit remove_pgd_mapping(unsigned long va, unsigned long end, bool is_vmemmap,
|
||||
struct vmem_altmap *altmap)
|
||||
{
|
||||
unsigned long addr, next;
|
||||
p4d_t *p4d_base;
|
||||
pgd_t *pgd;
|
||||
|
||||
for (addr = va; addr < end; addr = next) {
|
||||
next = pgd_addr_end(addr, end);
|
||||
pgd = pgd_offset_k(addr);
|
||||
|
||||
if (!pgd_present(*pgd))
|
||||
continue;
|
||||
|
||||
if (pgd_leaf(*pgd))
|
||||
continue;
|
||||
|
||||
p4d_base = p4d_offset(pgd, 0);
|
||||
remove_p4d_mapping(p4d_base, addr, next, is_vmemmap, altmap);
|
||||
}
|
||||
|
||||
flush_tlb_all();
|
||||
}
|
||||
|
||||
static void __meminit remove_linear_mapping(phys_addr_t start, u64 size)
|
||||
{
|
||||
unsigned long va = (unsigned long)__va(start);
|
||||
unsigned long end = (unsigned long)__va(start + size);
|
||||
|
||||
remove_pgd_mapping(va, end, false, NULL);
|
||||
}
|
||||
|
||||
struct range arch_get_mappable_range(void)
|
||||
{
|
||||
struct range mhp_range;
|
||||
|
||||
mhp_range.start = __pa(PAGE_OFFSET);
|
||||
mhp_range.end = __pa(PAGE_END - 1);
|
||||
return mhp_range;
|
||||
}
|
||||
|
||||
int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params)
|
||||
{
|
||||
int ret = 0;
|
||||
|
||||
create_linear_mapping_range(start, start + size, 0, ¶ms->pgprot);
|
||||
ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, params);
|
||||
if (ret) {
|
||||
remove_linear_mapping(start, size);
|
||||
goto out;
|
||||
}
|
||||
|
||||
max_pfn = PFN_UP(start + size);
|
||||
max_low_pfn = max_pfn;
|
||||
|
||||
out:
|
||||
flush_tlb_all();
|
||||
return ret;
|
||||
}
|
||||
|
||||
void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
|
||||
{
|
||||
__remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap);
|
||||
remove_linear_mapping(start, size);
|
||||
flush_tlb_all();
|
||||
}
|
||||
|
||||
void __ref vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap)
|
||||
{
|
||||
remove_pgd_mapping(start, end, true, altmap);
|
||||
}
|
||||
#endif /* CONFIG_MEMORY_HOTPLUG */
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@
|
|||
#include <linux/efi.h>
|
||||
#include <linux/init.h>
|
||||
#include <linux/debugfs.h>
|
||||
#include <linux/memory_hotplug.h>
|
||||
#include <linux/seq_file.h>
|
||||
#include <linux/ptdump.h>
|
||||
|
||||
|
|
@ -370,7 +371,9 @@ bool ptdump_check_wx(void)
|
|||
|
||||
static int ptdump_show(struct seq_file *m, void *v)
|
||||
{
|
||||
get_online_mems();
|
||||
ptdump_walk(m, m->private);
|
||||
put_online_mems();
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -122,7 +122,7 @@ config VIRTIO_BALLOON
|
|||
|
||||
config VIRTIO_MEM
|
||||
tristate "Virtio mem driver"
|
||||
depends on X86_64 || ARM64
|
||||
depends on X86_64 || ARM64 || RISCV
|
||||
depends on VIRTIO
|
||||
depends on MEMORY_HOTPLUG
|
||||
depends on MEMORY_HOTREMOVE
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user