From 0ee2c883b62deb74d8c43007e8049dc006d3a2fb Mon Sep 17 00:00:00 2001 From: Shivank Garg Date: Tue, 1 Sep 2026 07:20:38 +0000 Subject: [PATCH 01/60] KVM: guest_memfd: take the invalidate lock when unbinding a dying file kvm_gmem_unbind() takes mapping->invalidate_lock only if the get_file_active() succeeds. When the guest_memfd file is dying, its reference count is already zero so get_file_active fails and the binding is removed without holding the invalidate lock. kvm_gmem_invalidate_{start,end}() checks f->bindings independently to decide whether to begin or end KVM MMU invalidation. So, the bindings must therefore remain stable between the two calls. Otherwise, unbind can remove a binding after start increments mmu_invalidate_in_progress, but before end finds the binding and decrements it. Example, unbind race with memory failure: CPU 0: memory failure CPU 1: memslot delete ---------------------------------- --------------------------- (guest_memfd file is dying) kvm_gmem_error_folio() kvm_gmem_invalidate_start() finds binding mmu_invalidate_in_progress++ kvm_gmem_unbind() get_file_active() fails removes binding kvm_gmem_invalidate_end() no binding found counter stays elevated mmu_invalidate_retry() then returns 1 forever, so guest page faults retry without ever installing a mapping and the guest hangs. Remove the distinction between live and dying files in the unbind path. Always use slot->gmem.file and take the invalidate lock when it's non-NULL. Normal unbind callers hold slots_lock, which prevents release from clearing the pointer or freeing the file state until unbind completes. Final VM teardown can only see a NULL pointer because guest_memfd pins the KVM until release has cleared all bindings. Fixes: ae431059e75d ("KVM: guest_memfd: Remove bindings on memslot deletion when gmem is dying") Cc: stable@vger.kernel.org Reported-by: Sashiko Closes: https://lore.kernel.org/all/20260728092027.225CF1F000E9@smtp.kernel.org Suggested-by: Sean Christopherson Signed-off-by: Shivank Garg Reviewed-by: David Hildenbrand (Arm) Link: https://patch.msgid.link/20260901-gmem-unbind-fix-v2-1-12febe1d2216@amd.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 54 ++++++++++++++++++------------------------ 1 file changed, 23 insertions(+), 31 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 625e62e1a031..21bb3710edfe 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -668,11 +668,33 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, return r; } -static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct gmem_file *f) +void kvm_gmem_unbind(struct kvm_memory_slot *slot) { + struct file *file = slot->gmem.file; unsigned long start = slot->gmem.pgoff; unsigned long end = start + slot->npages; + struct gmem_file *f; + /* + * Nothing to do if the underlying file was _already_ closed, as + * kvm_gmem_release() invalidates and nullifies all bindings. + */ + if (!file) + return; + + /* + * However, if the file is _being_ closed, then the bindings need to be + * removed as kvm_gmem_release() might not run until after the memslot + * is freed. Modifying the bindings is safe even if the file is dying + * as kvm_gmem_release() nullifies slot->gmem.file under slots_lock, + * and only puts its reference to KVM after destroying all bindings. + * I.e. reaching this point means kvm_gmem_release() hasn't destroyed + * the bindings or freed the gmem_file and can't do so until the caller + * drops slots_lock, so there's no need to verify the file is live. + */ + f = file->private_data; + + filemap_invalidate_lock(file->f_mapping); xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); /* @@ -680,36 +702,6 @@ static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct gmem_file *f) * cannot see this memslot. */ WRITE_ONCE(slot->gmem.file, NULL); -} - -void kvm_gmem_unbind(struct kvm_memory_slot *slot) -{ - /* - * Nothing to do if the underlying file was _already_ closed, as - * kvm_gmem_release() invalidates and nullifies all bindings. - */ - if (!slot->gmem.file) - return; - - CLASS(gmem_get_file, file)(slot); - - /* - * However, if the file is _being_ closed, then the bindings need to be - * removed as kvm_gmem_release() might not run until after the memslot - * is freed. Note, modifying the bindings is safe even though the file - * is dying as kvm_gmem_release() nullifies slot->gmem.file under - * slots_lock, and only puts its reference to KVM after destroying all - * bindings. I.e. reaching this point means kvm_gmem_release() hasn't - * yet destroyed the bindings or freed the gmem_file, and can't do so - * until the caller drops slots_lock. - */ - if (!file) { - __kvm_gmem_unbind(slot, slot->gmem.file->private_data); - return; - } - - filemap_invalidate_lock(file->f_mapping); - __kvm_gmem_unbind(slot, file->private_data); filemap_invalidate_unlock(file->f_mapping); } From 53c34396c19e45cd45e6b20c47ef9f47f289698b Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Wed, 26 Aug 2026 09:56:47 -0700 Subject: [PATCH 02/60] KVM: guest_memfd: Elaborate on how release() vs. get_pfn() is safe against UAF Add more context and information to the comment in kvm_gmem_release() that explains why there's no synchronization on RCU _or_ kvm->srcu. Point (b) from commit 67b43038ce14 ("KVM: guest_memfd: Remove RCU-protected attribute from slot->gmem.file") b) kvm->srcu ensures that kvm_gmem_unbind() and freeing of a memslot occur after the memslot is no longer visible to kvm_gmem_get_pfn(). is especially difficult to fully grok, particularly in light of commit ae431059e75d ("KVM: guest_memfd: Remove bindings on memslot deletion when gmem is dying"), which addressed a race between unbind() and release(). See the extended on-list discussion[*] for more details about exactly what KVM guards against, and how. Opportunistically update the comment in kvm_gmem_unbind() to clarify the rules for unbinding, i.e. to be more explicit about why it's ok to nullify the file pointer even though kvm_gmem_get_pfn() doesn't take the invalidate lock. No functional change intended. Link: https://lore.kernel.org/all/CAEvNRgGmyd1yqQXsnz5hWRZpBZUs%3DpiEWbEaqP9%2Bcz9ZqEMQ6g@mail.gmail.com [*] Cc: Yan Zhao Cc: Vishal Annapurve Reviewed-by: Yan Zhao Link: https://patch.msgid.link/20260826165647.769231-1-seanjc@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 56 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 48 insertions(+), 8 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 21bb3710edfe..b8797382cc5b 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -300,17 +300,56 @@ static int kvm_gmem_release(struct inode *inode, struct file *file) * dereferencing the slot for existing bindings needs to be protected * against memslot updates, specifically so that unbind doesn't race * and free the memslot (kvm_gmem_get_file() will return NULL). - * - * Since .release is called only when the reference count is zero, - * after which file_ref_get() and get_file_active() fail, - * kvm_gmem_get_pfn() cannot be using the file concurrently. - * file_ref_put() provides a full barrier, and get_file_active() the - * matching acquire barrier. */ mutex_lock(&kvm->slots_lock); filemap_invalidate_lock(inode->i_mapping); + /* + * Note! synchronize_srcu() is _not_ needed after nullifying memslot + * bindings as slot->gmem.file cannot be set back to a non-null value + * without the memslot first being deleted. I.e. this relies on the + * synchronize_srcu_expedited() in kvm_swap_active_memslots() to ensure + * kvm_gmem_get_pfn() (which runs with kvm->srcu held for read) can't + * grab a reference to slot->gmem.file even if the struct file object + * is reallocated (for use as a different file). + * + * file_ref_put() provides a full barrier, and __get_file_rcu() the + * matching acquire barrier, to ensure that kvm_gmem_get_file() (via + * __get_file_rcu()) sees refcount==0 or fails the "file reloaded" + * check (file != NULL due to nullifying the file pointer here). + * + * Unlike most other users of get_file_rcu(), where callers don't care + * if they race with a write, only that they have a reference to _a_ + * live file, kvm_gmem_get_pfn() needs to get the exact file that is + * associated with the memslot. Without the aforementioned SRCU + * synchronization, the following could happen: + * + * CPU0 CPU1 + * kvm_gmem_get_pfn() + * slot->gmem.file == A + * kvm_gmem_release()) + * slot->gmem.file = NULL + * + * kvm_set_memory_region() + * slot deleted + * + * kvm_set_memory_region() + * slot created + * slot->gmem.file = B (alloc the same object) + * + * get_file_active() + * file = B + * file_reloaded = B + * + * + * + * Obviously KVM would be broken in many places if the synchronization + * were omitted, but it's important to note that get_file_active() does + * NOT guarantee a reference to the "original" file was obtained, only + * that it grabbed a reference for the returned file, i.e. didn't grab + * a reference for file A, but then returned a pointer to file B. + */ xa_for_each(&f->bindings, index, slot) WRITE_ONCE(slot->gmem.file, NULL); @@ -698,8 +737,9 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot) xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); /* - * synchronize_srcu(&kvm->srcu) ensured that kvm_gmem_get_pfn() - * cannot see this memslot. + * Note, the caller is responsible for ensuring the slot is unreachable + * before unbinding, e.g. by synchronizing SRCU after deleting the slot, + * to guarantee kvm_gmem_get_pfn() can't see the slot+file. */ WRITE_ONCE(slot->gmem.file, NULL); filemap_invalidate_unlock(file->f_mapping); From e6ecb88711d93b9e014532f17d3d29b57e80932c Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Mon, 21 Sep 2026 17:13:27 -0700 Subject: [PATCH 03/60] KVM: guest_memfd: Gracefully handle xarray errors when binding a memslot If inserting a memslot into a guest_memfd's bindings xarray fails, propagate the error back to the caller, i.e. fail memslot creation as well. Signalling success and continuing on with memslot creation results in use-after-free, as the guest_memfd instance will remain reachable via the memslot after the file is freed (kvm_gmem_release() won't nullify the file pointer due to lack of a valid binding). Opportunistically WARN and reject binding if KVM_MEMSLOT_GMEM_ONLY is already set, partly to guard against goofs elsewhere, but mostly so that KVM doesn't need to worry about clobbering flags when unwinding on failure. Regarding the unwind, the slot must be fully prepared before inserting it into the bindings, at which point the slot becomes reachable. I.e. waiting to update the slot in order to avoid the ugly unwind isn't an option. And as part of the unwind, explicitly nullify the relevant bindings, as xarray can store a subset of entries when populating a range. Fixes: a7800aa80ea4 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory") Cc: stable@vger.kernel.org Reported-by: Stefan Teodorescu Reported-by: Dennis Tighe Reported-by: Sashiko Bot Closes: https://lore.kernel.org/all/20260823135031.4F6DC1F000E9%40smtp.kernel.org Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Ackerley Tng Link: https://patch.msgid.link/20260922001332.1121266-2-seanjc@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index b8797382cc5b..751d0ad94f74 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -655,6 +655,9 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset)); BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff)); + if (WARN_ON_ONCE(slot->flags & KVM_MEMSLOT_GMEM_ONLY)) + return -EINVAL; + file = fget(fd); if (!file) return -EBADF; @@ -693,7 +696,13 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, if (kvm_gmem_supports_mmap(inode)) slot->flags |= KVM_MEMSLOT_GMEM_ONLY; - xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL); + r = xa_err(xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL)); + if (r) { + xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); + slot->gmem.file = NULL; + slot->gmem.pgoff = 0; + slot->flags &= ~KVM_MEMSLOT_GMEM_ONLY; + } filemap_invalidate_unlock(inode->i_mapping); /* @@ -701,7 +710,6 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, * not the other way 'round. Active bindings are invalidated if the * file is closed before memslots are destroyed. */ - r = 0; err: fput(file); return r; From 23a205797cec86f391d0be886b9bc741e1262063 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Mon, 21 Sep 2026 17:13:28 -0700 Subject: [PATCH 04/60] KVM: Use goto to handle errors during memslot preparation Use a goto to unwind early memslot changes if preparing for a memslot operation fails. This will allow moving the creation of guest_memfd bindings into kvm_set_memslot() without needing to copy+paste the unwind logic. No functional change intended. Cc: stable@vger.kernel.org Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Ackerley Tng Link: https://patch.msgid.link/20260922001332.1121266-3-seanjc@google.com Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 65eb26a0520d..3c0dbe60a5b4 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1931,21 +1931,8 @@ static int kvm_set_memslot(struct kvm *kvm, } r = kvm_prepare_memory_region(kvm, old, new, change); - if (r) { - /* - * For DELETE/MOVE, revert the above INVALID change. No - * modifications required since the original slot was preserved - * in the inactive slots. Changing the active memslots also - * release slots_arch_lock. - */ - if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) { - kvm_activate_memslot(kvm, invalid_slot, old); - kfree(invalid_slot); - } else { - mutex_unlock(&kvm->slots_arch_lock); - } - return r; - } + if (r) + goto err; /* * For DELETE and MOVE, the working slot is now active as the INVALID @@ -1977,6 +1964,20 @@ static int kvm_set_memslot(struct kvm *kvm, kvm_commit_memory_region(kvm, old, new, change); return 0; + +err: + /* + * For DELETE/MOVE, revert the above INVALID change. No modifications + * required since the original slot was preserved in the inactive slots. + * Changing the active memslots also release slots_arch_lock. + */ + if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) { + kvm_activate_memslot(kvm, invalid_slot, old); + kfree(invalid_slot); + } else { + mutex_unlock(&kvm->slots_arch_lock); + } + return r; } static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, From 513333a5546017da2a4a6173bfb70372f83733b3 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Mon, 21 Sep 2026 17:13:29 -0700 Subject: [PATCH 05/60] KVM: Only bind memslot to guest_memfd instance for CREATE operations For additional defense-in-depth, and to avoid having to handle impossible unwind scenarios when binding to a memslot fails, bind a memslot to a gmem instance only for CREATE operations, i.e. don't attempt to establish a binding for MOVE and FLAGS_ONLY operations. And when FLAGS_ONLY operations are eventually supported (this is currently all dead code), creating a new binding would be incorrect; KVM instead needs to do a 1:1 replacement of the existing binding, i.e. FLAGS_ONLY will need its own dedicated handling. Update the relevant TODO to make a better guess as to what needs to be done to support toggling dirty logging for guest_memfd memslots. Because it's dead code, no functional change intended. Cc: stable@vger.kernel.org Reviewed-by: David Hildenbrand (Arm) Link: https://patch.msgid.link/20260922001332.1121266-4-seanjc@google.com Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 3c0dbe60a5b4..c53a88cea1f2 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1742,10 +1742,10 @@ static void kvm_commit_memory_region(struct kvm *kvm, kvm_destroy_dirty_bitmap(old); /* - * Unbind the guest_memfd instance as needed; the @new slot has - * already created its own binding. TODO: Drop the WARN when - * dirty logging guest_memfd memslots is supported. Until then, - * flags-only changes on guest_memfd slots should be impossible. + * TODO: Drop the WARN and do the unbind() call only for MOVE + * when dirty logging guest_memfd memslots is supported. Until + * then, flags-only changes on guest_memfd slots should also be + * impossible; unbind the old memslot for defense-in-depth. */ if (WARN_ON_ONCE(old->flags & KVM_MEM_GUEST_MEMFD)) kvm_gmem_unbind(old); @@ -2106,7 +2106,7 @@ static int kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; - if (mem->flags & KVM_MEM_GUEST_MEMFD) { + if (change == KVM_MR_CREATE && (mem->flags & KVM_MEM_GUEST_MEMFD)) { r = kvm_gmem_bind(kvm, new, mem->guest_memfd, mem->guest_memfd_offset); if (r) goto out; @@ -2119,7 +2119,7 @@ static int kvm_set_memory_region(struct kvm *kvm, return 0; out_unbind: - if (mem->flags & KVM_MEM_GUEST_MEMFD) + if (change == KVM_MR_CREATE && (mem->flags & KVM_MEM_GUEST_MEMFD)) kvm_gmem_unbind(new); out: kfree(new); From 83411f32b8ad96c6b05b1511d1a588daab44bd25 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Mon, 21 Sep 2026 17:13:30 -0700 Subject: [PATCH 06/60] KVM: guest_memfd: Split bind() into prepare()+commit() phases Split binding a memslot to a guest_memfd instance into prepare() and commit() phases so that KVM can separate preparing the memslot from binding the memslot to the gmem instance, i.e. from committing the memslot. This will allow waiting to commit the memslot+gmem binding until the memslot is fully prepared, which is necessary as the memslot becomes reachable when the binding is created. As a bonus, drop the unwind-on-failure from the commit phase (other than nullifying the bindings), as the only reason bind() did the full unwind is because it technically didn't own the memslot, i.e. "needed" to leave memslot in the same state it started in. No functional change intended (the unwinding down on bind() failure was effectively dead code since KVM simply deletes the memslot on failure, i.e. there was nothing that could actually observe the unwind). Cc: stable@vger.kernel.org Reviewed-by: David Hildenbrand (Arm) Link: https://patch.msgid.link/20260922001332.1121266-5-seanjc@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 68 ++++++++++++++++++++++++------------------ virt/kvm/guest_memfd.h | 19 ++++++++---- virt/kvm/kvm_main.c | 18 ++++++++++- 3 files changed, 70 insertions(+), 35 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 751d0ad94f74..c813c20de7f3 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -642,15 +642,14 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) return __kvm_gmem_create(kvm, size, flags); } -int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset) +int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset) { uoff_t size = slot->npages << PAGE_SHIFT; - unsigned long start, end; struct gmem_file *f; struct inode *inode; struct file *file; - int r = -EINVAL; + BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset)); BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff)); @@ -674,44 +673,55 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode)) goto err; - filemap_invalidate_lock(inode->i_mapping); - - start = offset >> PAGE_SHIFT; - end = start + slot->npages; - - if (!xa_empty(&f->bindings) && - xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) { - r = -EEXIST; - filemap_invalidate_unlock(inode->i_mapping); - goto err; - } - /* * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so * kvm_gmem_bind() must occur on a new memslot. Because the memslot * is not visible yet, kvm_gmem_get_pfn() is guaranteed to see the file. */ WRITE_ONCE(slot->gmem.file, file); - slot->gmem.pgoff = start; + slot->gmem.pgoff = offset >> PAGE_SHIFT; if (kvm_gmem_supports_mmap(inode)) slot->flags |= KVM_MEMSLOT_GMEM_ONLY; - r = xa_err(xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL)); - if (r) { - xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); - slot->gmem.file = NULL; - slot->gmem.pgoff = 0; - slot->flags &= ~KVM_MEMSLOT_GMEM_ONLY; - } - filemap_invalidate_unlock(inode->i_mapping); - /* - * Drop the reference to the file, even on success. The file pins KVM, - * not the other way 'round. Active bindings are invalidated if the - * file is closed before memslots are destroyed. + * Gift the caller a reference to the file. The reference will be + * dropped after bindings are established, or if installing the new + * memslot ultimately fails. */ + return 0; + err: fput(file); + return -EINVAL; +} + +int kvm_gmem_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot) +{ + struct gmem_file *f = slot->gmem.file->private_data; + struct inode *inode = file_inode(slot->gmem.file); + unsigned long start, end; + int r; + + if (WARN_ON_ONCE(slot->gmem.file->f_op != &kvm_gmem_fops)) + return -EIO; + + filemap_invalidate_lock(inode->i_mapping); + + start = slot->gmem.pgoff; + end = start + slot->npages; + + if (!xa_empty(&f->bindings) && + xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) { + filemap_invalidate_unlock(inode->i_mapping); + return -EEXIST; + } + + r = xa_err(xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL)); + if (r) + xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); + + filemap_invalidate_unlock(inode->i_mapping); + return r; } diff --git a/virt/kvm/guest_memfd.h b/virt/kvm/guest_memfd.h index 0f9c6f840838..01bd359d27e3 100644 --- a/virt/kvm/guest_memfd.h +++ b/virt/kvm/guest_memfd.h @@ -8,8 +8,9 @@ int kvm_gmem_init(struct module *module); void kvm_gmem_exit(void); int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args); -int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset); +int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset); +int kvm_gmem_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot); void kvm_gmem_unbind(struct kvm_memory_slot *slot); #else static inline int kvm_gmem_init(struct module *module) @@ -17,9 +18,17 @@ static inline int kvm_gmem_init(struct module *module) return 0; } static inline void kvm_gmem_exit(void) {}; -static inline int kvm_gmem_bind(struct kvm *kvm, - struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset) + +static inline int kvm_gmem_prepare_memory_region(struct kvm *kvm, + struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset) +{ + WARN_ON_ONCE(1); + return -EIO; +} + +static inline int kvm_gmem_commit_memory_region(struct kvm *kvm, + struct kvm_memory_slot *slot) { WARN_ON_ONCE(1); return -EIO; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index c53a88cea1f2..c7526c1617f7 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2107,7 +2107,23 @@ static int kvm_set_memory_region(struct kvm *kvm, new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; if (change == KVM_MR_CREATE && (mem->flags & KVM_MEM_GUEST_MEMFD)) { - r = kvm_gmem_bind(kvm, new, mem->guest_memfd, mem->guest_memfd_offset); + r = kvm_gmem_prepare_memory_region(kvm, new, mem->guest_memfd, + mem->guest_memfd_offset); + if (r) + goto out; + + r = kvm_gmem_commit_memory_region(kvm, new); + + /* + * Drop the reference to the file, even on success. The file + * pins KVM, not the other way 'round. Active bindings are + * invalidated if the file is closed before memslots are + * destroyed. + */ +#ifdef CONFIG_KVM_GUEST_MEMFD + fput(new->gmem.file); +#endif + if (r) goto out; } From 3fe5d2944e5ac3fc5073344b0982c73703332e18 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Mon, 21 Sep 2026 17:13:31 -0700 Subject: [PATCH 07/60] KVM: guest_memfd: Establish memslot<=>guest_memfd bindings *after* memslot is ready Wait to bind a memslot to a guest_memfd instance until *after* the memslot is fully prepared, as creating the binding in guest_memfd will effectively expose the memslot to readers. As pointed out by Sashiko, binding the memslot before it's ready to be exposed to the rest of the world can break various memslot assumption and rules. E.g. x86 could observe a NULL rmap pointer if a PUNCH_HOLE hit the guest_memfd after the binding was created, but before KVM made it through kvm_prepare_memory_region(). Fixes: a7800aa80ea4 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory") Cc: stable@vger.kernel.org Reported-by: Sashiko Bot Closes: https://lore.kernel.org/all/20260826170551.BEF801F000E9@smtp.kernel.org Link: https://patch.msgid.link/20260922001332.1121266-6-seanjc@google.com Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index c7526c1617f7..f26986136bc6 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1934,6 +1934,15 @@ static int kvm_set_memslot(struct kvm *kvm, if (r) goto err; + if (change == KVM_MR_CREATE && (new->flags & KVM_MEM_GUEST_MEMFD)) { + r = kvm_gmem_commit_memory_region(kvm, new); + if (r) { + kvm_arch_free_memslot(kvm, new); + kvm_destroy_dirty_bitmap(new); + goto err; + } + } + /* * For DELETE and MOVE, the working slot is now active as the INVALID * version of the old slot. MOVE is particularly special as it reuses @@ -2111,32 +2120,25 @@ static int kvm_set_memory_region(struct kvm *kvm, mem->guest_memfd_offset); if (r) goto out; - - r = kvm_gmem_commit_memory_region(kvm, new); - - /* - * Drop the reference to the file, even on success. The file - * pins KVM, not the other way 'round. Active bindings are - * invalidated if the file is closed before memslots are - * destroyed. - */ -#ifdef CONFIG_KVM_GUEST_MEMFD - fput(new->gmem.file); -#endif - - if (r) - goto out; } r = kvm_set_memslot(kvm, old, new, change); + + /* + * Drop the reference to the gmem file, even on success. The file pins + * KVM, not the other way 'round. Active bindings are invalidated if + * the file is closed before memslots are destroyed. + */ +#ifdef CONFIG_KVM_GUEST_MEMFD + if (change == KVM_MR_CREATE && (mem->flags & KVM_MEM_GUEST_MEMFD)) + fput(new->gmem.file); +#endif + if (r) - goto out_unbind; + goto out; return 0; -out_unbind: - if (change == KVM_MR_CREATE && (mem->flags & KVM_MEM_GUEST_MEMFD)) - kvm_gmem_unbind(new); out: kfree(new); return r; From 48252615eb78e151f93183370a650f5eb7e572fb Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Mon, 21 Sep 2026 17:13:32 -0700 Subject: [PATCH 08/60] KVM: guest_memfd: Drop superfluous WRITE_ONCE() when binding a memslot Drop the superfluous WRITE_ONCE() when setting a memslot's guest_memfd file during initial binding, as the memslot *must* be inactive and unreachable. The superfluous WRITE_ONCE() was added by commit 67b43038ce14 ("KVM: guest_memfd: Remove RCU-protected attribute from slot->gmem.file") to maintain rough "parity" with the existing rcu_assign_pointer(), not realizing that the only reason rcu_assign_pointer() was used was to make sparse and other checkers happy. Cc: Yan Zhao Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Ackerley Tng Reviewed-by: Yan Zhao Link: https://patch.msgid.link/20260922001332.1121266-7-seanjc@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index c813c20de7f3..90ee0b9a9a6a 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -678,7 +678,7 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot * kvm_gmem_bind() must occur on a new memslot. Because the memslot * is not visible yet, kvm_gmem_get_pfn() is guaranteed to see the file. */ - WRITE_ONCE(slot->gmem.file, file); + slot->gmem.file = file; slot->gmem.pgoff = offset >> PAGE_SHIFT; if (kvm_gmem_supports_mmap(inode)) slot->flags |= KVM_MEMSLOT_GMEM_ONLY; From 211606532680fdb8113734b0aa8cbd490647523f Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Wed, 26 Aug 2026 09:01:46 +0000 Subject: [PATCH 09/60] KVM: SEV: Treat unassigned RMP entry as benign race on PSMASH failure When handling an RMP fault, KVM attempts to split a 2MB page via PSMASH. If PSMASH fails, the only expected return value is FAIL_BADADDR, which does not distinguish the reason for the bad address. Hence, another RMP entry lookup is required to determine whether the failure was benign. Specifically, KVM re-checks the RMP entry to determine if another CPU raced and already smashed the entry into 4KB pages. A concurrent operation (such as guest_memfd truncation or hole punching) can also race and transition the page to shared, removing the page from the RMP table and causing PSMASH to fail. This can happen even if the page is still referenced by KVM, because guest_memfd reclaim transitions the RMP entry to shared when the folio is removed from the page cache. Treat an unassigned RMP entry as an expected race when re-checking after a failed PSMASH, and skip logging an error warning. Fixes: c63cf135cc99 ("KVM: SEV: Add support to handle RMP nested page faults") Reviewed-by: Michael Roth Signed-off-by: Ackerley Tng Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-1-3bb9c1ddb4e3@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/sev.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 5705723f1f41..e198469eb074 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -5090,10 +5090,11 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) /* * Look it up again. If it's 4K now then the PSMASH may have * raced with another process and the issue has already resolved - * itself. + * itself. If it's not assigned, then this must have raced with + * another process that made this page shared. */ if (!snp_lookup_rmpentry(pfn, &assigned, &rmp_level) && - assigned && rmp_level == PG_LEVEL_4K) + ((assigned && rmp_level == PG_LEVEL_4K) || !assigned)) goto out; pr_warn_ratelimited("SEV: Unable to split RMP entry for GPA 0x%llx PFN 0x%llx ret %d\n", From 375b32931b5da6a86ccb017e7e12a67a2def03f0 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Wed, 26 Aug 2026 09:01:47 +0000 Subject: [PATCH 10/60] KVM: SEV: Drop page refcount early during RMP fault handling Rework KVM's handling of RMP faults to rely on MMU invalidation logic for safety, instead of the current approach of holding onto a folio reference until the RMP operations are complete. I.e. drop the reference gifted by guest_memfd immediately after getting the PFN, and instead do RMP updates under mmu_lock, after checking for relevant MMU invalidations. This will allow dropping guest_memfd's reference gifting entirely, which is ideally how KVM would operate for all "follow PFN" operations (GUP has many more complications, which is why KVM holds a reference across page faults *on top* of the standard MMU invalidation logic). Reviewed-by: Michael Roth Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-2-3bb9c1ddb4e3@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/sev.c | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index e198469eb074..0d027cff734c 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -5019,6 +5019,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) struct kvm_memory_slot *slot; struct kvm *kvm = vcpu->kvm; int order, rmp_level, ret; + unsigned long mmu_seq; struct page *page; bool assigned; kvm_pfn_t pfn; @@ -5046,18 +5047,22 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) return; } + mmu_seq = kvm->mmu_invalidate_seq; + smp_rmb(); + ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &page, &order); if (ret) { pr_warn_ratelimited("SEV: Unexpected RMP fault, no backing page for private GPA 0x%llx\n", gpa); return; } + kvm_release_page_unused(page); ret = snp_lookup_rmpentry(pfn, &assigned, &rmp_level); if (ret || !assigned) { pr_warn_ratelimited("SEV: Unexpected RMP fault, no assigned RMP entry found for GPA 0x%llx PFN 0x%llx error %d\n", gpa, pfn, ret); - goto out_no_trace; + return; } /* @@ -5085,27 +5090,31 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) if (rmp_level == PG_LEVEL_4K) goto out; - ret = snp_rmptable_psmash(pfn); - if (ret) { - /* - * Look it up again. If it's 4K now then the PSMASH may have - * raced with another process and the issue has already resolved - * itself. If it's not assigned, then this must have raced with - * another process that made this page shared. - */ - if (!snp_lookup_rmpentry(pfn, &assigned, &rmp_level) && - ((assigned && rmp_level == PG_LEVEL_4K) || !assigned)) + scoped_guard(read_lock, &kvm->mmu_lock) { + if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) goto out; - pr_warn_ratelimited("SEV: Unable to split RMP entry for GPA 0x%llx PFN 0x%llx ret %d\n", - gpa, pfn, ret); + ret = snp_rmptable_psmash(pfn); + if (ret) { + /* + * Look it up again. If it's 4K now then the PSMASH may + * have raced with another process and the issue has + * already resolved itself. If it's not assigned, then + * this must have raced with another process that made + * this page shared. + */ + if (!snp_lookup_rmpentry(pfn, &assigned, &rmp_level) && + ((assigned && rmp_level == PG_LEVEL_4K) || !assigned)) + goto out; + + pr_warn_ratelimited("SEV: Unable to split RMP entry for GPA 0x%llx PFN 0x%llx ret %d\n", + gpa, pfn, ret); + } } kvm_zap_gfn_range(kvm, gfn, gfn + PTRS_PER_PMD); out: trace_kvm_rmp_fault(vcpu, gpa, pfn, error_code, rmp_level, ret); -out_no_trace: - kvm_release_page_unused(page); } static bool is_pfn_range_shared(kvm_pfn_t start, kvm_pfn_t end) From acafcf32c83ec16631463eeed9e87e44177c2e49 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Wed, 26 Aug 2026 09:01:48 +0000 Subject: [PATCH 11/60] KVM: SEV: Check for invalidation before warning on unassigned RMP entry When handling an RMP fault, KVM looks up the RMP entry for the backing PFN to determine if a 2MB page needs to be split via PSMASH. If the RMP entry is not assigned (or lookup fails), KVM logs a rate-limited warning under the assumption that private memory should always have an assigned RMP entry. However, a concurrent invalidation (such as guest_memfd hole punching) can race with RMP fault handling and transition the page to shared, unassigning the RMP entry after KVM fetched the PFN. In such cases, not finding an assigned RMP entry is benign. Check mmu_invalidate_retry_gfn() under mmu_lock before warning about a missing or unassigned RMP entry, and suppress the spurious warning if an invalidation occurred for the faulting GFN. Fixes: c63cf135cc99 ("KVM: SEV: Add support to handle RMP nested page faults") Signed-off-by: Ackerley Tng Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-3-3bb9c1ddb4e3@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/sev.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 0d027cff734c..f9dce8acac8e 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -5060,8 +5060,12 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) ret = snp_lookup_rmpentry(pfn, &assigned, &rmp_level); if (ret || !assigned) { - pr_warn_ratelimited("SEV: Unexpected RMP fault, no assigned RMP entry found for GPA 0x%llx PFN 0x%llx error %d\n", - gpa, pfn, ret); + guard(read_lock)(&kvm->mmu_lock); + + if (!mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) + pr_warn_ratelimited("SEV: Unexpected RMP fault, no assigned RMP entry found for GPA 0x%llx PFN 0x%llx error %d\n", + gpa, pfn, ret); + return; } From 3c427e5c03476f6267a75116e11c8646683378e6 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Wed, 26 Aug 2026 09:01:49 +0000 Subject: [PATCH 12/60] KVM: SEV: Drop page refcount early in VMSA reload When reloading the guest VMSA for an SEV-SNP vCPU, KVM retrieves the PFN from guest_memfd. Drop the page reference immediately after retrieving the PFN instead of holding it across MMU lock acquisition in preparation for a follow-up patch to stop returning page pointers from guest_memfd PFN lookups. This is safe because the page's validity and presence are governed by KVM's MMU invalidation protocol rather than the page reference. No functional change intended. Reviewed-by: Michael Roth Signed-off-by: Ackerley Tng Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-4-3bb9c1ddb4e3@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/sev.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index f9dce8acac8e..f094b19226b9 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -4078,6 +4078,7 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) */ if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL)) return; + kvm_release_page_clean(page); read_lock(&kvm->mmu_lock); /* @@ -4092,8 +4093,6 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) else svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn); read_unlock(&kvm->mmu_lock); - - kvm_release_page_clean(page); } /* From 89fbe3de01ccfc1b4862af6dfc79ab65bdc69e19 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Wed, 26 Aug 2026 09:01:50 +0000 Subject: [PATCH 13/60] KVM: guest_memfd: Stop returning struct page from PFN lookup KVM currently expects guest_memfd PFN lookups to return a refcounted struct page, which callers hold across fault handling. Drop the page's refcount before returning from kvm_gmem_get_pfn() to prepare for the in-place conversions series. CoCo shared-to-private conversion handling must inspect folio refcounts to ensure exclusive ownership by guest_memfd. A concurrent guest page fault taking a temporary reference on the folio causes conversions to fail due to an elevated refcount. While this refcount is also taken on host userspace page faults, that refcount is taken on behalf of the host userspace page tables. This refcount will be dropped when conversions unmaps the page. Either way, once there's an mmap() or userspace mapping, the pages are open to way more refcounts, transient or not. This patch focuses on just dropping refcounts before handing KVM a page. guest_memfd already notifies KVM of page invalidations, so callers within KVM only need to respect the MMU invalidation protocol to safely rely on guest_memfd for page presence. Since the page refcounts are dropped, don't return the struct page pointer. Not returning the struct page from the guest_memfd PFN lookup moves KVM closer toward supporting memory backends that are not backed by struct page. Here are some notes on the cleanup in the callers of kvm_gmem_get_pfn(): kvm_release_faultin_page() in ARM's gmem_abort() originally also serves to set the page dirty and accessed under some conditions. The dirty and accessed flags don't matter for guest_memfd anyway, so it is safe to just drop the call to kvm_release_faultin_page(). For ARM's kvm_translate_vncr(), the local page pointer must be initialized to NULL so that the shared cleanup path that releases faulted-in pages safely no-ops for guest_memfd. For x86, no additional changes are required in the MMU fault path because the page fault tracking structure is zero-initialized at the start of page fault handling, ensuring the refcounted page pointer is already NULL. Reported-by: Yan Zhao Closes: https://lore.kernel.org/all/anZ4W9o5pTWIEgMY@yzhao56-desk.sh.intel.com/ Co-developed-by: Yan Zhao Signed-off-by: Yan Zhao Reviewed-by: Suzuki K Poulose Reviewed-by: Michael Roth Tested-by: Michael Roth Tested-by: Yan Zhao Reviewed-by: Fuad Tabba Tested-by: Fuad Tabba Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Reviewed-by: Gavin Shan Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-5-3bb9c1ddb4e3@google.com Signed-off-by: Sean Christopherson --- arch/arm64/kvm/mmu.c | 4 +--- arch/arm64/kvm/nested.c | 4 ++-- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/svm/sev.c | 8 ++------ include/linux/kvm_host.h | 6 ++---- virt/kvm/guest_memfd.c | 9 ++------- 6 files changed, 10 insertions(+), 23 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 9ba86450fe4a..d57f8181f1b7 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1613,7 +1613,6 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd) enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R; struct kvm_pgtable *pgt = s2fd->vcpu->arch.hw_mmu->pgt; unsigned long mmu_seq; - struct page *page; struct kvm *kvm = s2fd->vcpu->kvm; void *memcache = NULL; kvm_pfn_t pfn; @@ -1641,7 +1640,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd) /* Pairs with the smp_wmb() in kvm_mmu_invalidate_end(). */ smp_rmb(); - ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, gfn, &pfn, &page, NULL); + ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, gfn, &pfn, NULL); if (ret) { kvm_prepare_memory_fault_exit(s2fd->vcpu, s2fd->fault_ipa, PAGE_SIZE, write_fault, exec_fault, false); @@ -1681,7 +1680,6 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd) } out_unlock: - kvm_release_faultin_page(kvm, page, !!ret, prot & KVM_PGTABLE_PROT_W); kvm_fault_unlock(kvm); if ((prot & KVM_PGTABLE_PROT_W) && !ret) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 3c4fc566eafc..68acedc526c4 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -1415,7 +1415,7 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) bool write_fault, writable; unsigned long mmu_seq; struct vncr_tlb *vt; - struct page *page; + struct page *page = NULL; u64 va, pfn, gfn; int ret; @@ -1471,7 +1471,7 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) return -EFAULT; } } else { - ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, &page, NULL); + ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, NULL); if (ret) { kvm_prepare_memory_fault_exit(vcpu, vt->wr.pa, PAGE_SIZE, write_fault, false, false); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 064ecc33b926..947c9cd84345 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4628,7 +4628,7 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu, } r = kvm_gmem_get_pfn(vcpu->kvm, fault->slot, fault->gfn, &fault->pfn, - &fault->refcounted_page, &max_order); + &max_order); if (r) { kvm_mmu_prepare_memory_fault_exit(vcpu, fault); return r; diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index f094b19226b9..0a887f8e05d3 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -4032,7 +4032,6 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) struct kvm *kvm = vcpu->kvm; gfn_t gfn = gpa_to_gfn(gpa); unsigned long mmu_seq; - struct page *page; kvm_pfn_t pfn; lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex); @@ -4076,9 +4075,8 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) * The new VMSA will be private memory guest memory, so retrieve the * PFN from the gmem backend. */ - if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL)) + if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, NULL)) return; - kvm_release_page_clean(page); read_lock(&kvm->mmu_lock); /* @@ -5019,7 +5017,6 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) struct kvm *kvm = vcpu->kvm; int order, rmp_level, ret; unsigned long mmu_seq; - struct page *page; bool assigned; kvm_pfn_t pfn; gfn_t gfn; @@ -5049,13 +5046,12 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) mmu_seq = kvm->mmu_invalidate_seq; smp_rmb(); - ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &page, &order); + ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &order); if (ret) { pr_warn_ratelimited("SEV: Unexpected RMP fault, no backing page for private GPA 0x%llx\n", gpa); return; } - kvm_release_page_unused(page); ret = snp_lookup_rmpentry(pfn, &assigned, &rmp_level); if (ret || !assigned) { diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 03bfc92864b6..502465119ca0 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2586,13 +2586,11 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) #ifdef CONFIG_KVM_GUEST_MEMFD int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, - gfn_t gfn, kvm_pfn_t *pfn, struct page **page, - int *max_order); + gfn_t gfn, kvm_pfn_t *pfn, int *max_order); #else static inline int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, - kvm_pfn_t *pfn, struct page **page, - int *max_order) + kvm_pfn_t *pfn, int *max_order) { KVM_BUG_ON(1, kvm); return -EIO; diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 90ee0b9a9a6a..77876d20f5a8 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -801,8 +801,7 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file, } int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, - gfn_t gfn, kvm_pfn_t *pfn, struct page **page, - int *max_order) + gfn_t gfn, kvm_pfn_t *pfn, int *max_order) { pgoff_t index = kvm_gmem_get_index(slot, gfn); struct folio *folio; @@ -830,11 +829,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, #endif folio_unlock(folio); - - if (!r) - *page = folio_file_page(folio, index); - else - folio_put(folio); + folio_put(folio); return r; } From 54623a16bbdca9f273ce5658854e0b0d1637bc4d Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:27 -0700 Subject: [PATCH 14/60] KVM: guest_memfd: Optimize away conversion overheads via dead-code elimination Add and use kvm_arch_has_gmem_convert() to guard guest_memfd's invocation of arch hooks related to converting memory between private and shared, as only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is pure overhead for TDX). At this exact moment, the overhead is negligible, but that will change when in-place conversion comes along, at which point to-shared conversions will "need" to find all affected folios prior to calling into arch code. In quotes because very technically that work could be pushed to arch code, but that would bleed guest_memfd details into arch code and would be far worse than adding yet another kvm_arch_has... hook. Opportunistically provide the kvm_arch_gmem_make_private() declaration, and rely on dead-code elimination to eliminate the call to non-existent code when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n. Reported-by: Binbin Wu Closes: https://lore.kernel.org/all/1ec08cd8-3072-4753-ad5e-cd34956647f8@linux.intel.com Suggested-by: Ackerley Tng Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Xiaoyao Li Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-1-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 3 +++ include/linux/kvm_host.h | 3 ++- virt/kvm/guest_memfd.c | 5 ++--- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 683bb8bf43a9..6beddd8a0c74 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1854,6 +1854,9 @@ enum kvm_intr_type { #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem) #endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#define kvm_arch_has_gmem_convert() (!!kvm_x86_ops.gmem_make_private) +#endif #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 502465119ca0..b7516e19ef41 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2597,9 +2597,10 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm, } #endif /* CONFIG_KVM_GUEST_MEMFD */ -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages); +#ifndef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +#define kvm_arch_has_gmem_convert() false #endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 77876d20f5a8..551b19ad2296 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -822,11 +822,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, folio_mark_uptodate(folio); } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT - if (kvm_gmem_is_private_mem(file_inode(file), index)) + if (kvm_arch_has_gmem_convert() && + kvm_gmem_is_private_mem(file_inode(file), index)) r = kvm_arch_gmem_make_private(kvm, gfn, *pfn, (kvm_pfn_t)1 << *max_order); -#endif folio_unlock(folio); folio_put(folio); From fa7b8b1bb5d068ed737e64bcf6836922a70983d5 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:28 -0700 Subject: [PATCH 15/60] KVM: guest_memfd: Use kvm_mem_is_private() when populating guest_memfd memory Use kvm_mem_is_private() when populating guest_memfd instead of using an open coded equivalent. In addition to simplifying the populate code *now*, this avoids the need to provide a range-based gmem lookup API in the future as well. No functional change intended. Suggested-by: Xiaoyao Li Signed-off-by: Ackerley Tng Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Reviewed-by: Fuad Tabba Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-2-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 551b19ad2296..73974fece401 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -855,9 +855,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot, folio_unlock(folio); - if (!kvm_range_has_memory_attributes(kvm, gfn, gfn + 1, - KVM_MEMORY_ATTRIBUTE_PRIVATE, - KVM_MEMORY_ATTRIBUTE_PRIVATE)) { + if (!kvm_mem_is_private(kvm, gfn)) { ret = -EINVAL; goto out_put_folio; } From 1ae5be958ff702749b5ac8488dc18e2c6c175200 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:29 -0700 Subject: [PATCH 16/60] KVM: guest_memfd: Introduce per-gmem attributes, use to guard user mappings Start plumbing in guest_memfd support for in-place private<=>shared conversions by tracking attributes via a maple tree. KVM currently tracks private vs. shared attributes on a per-VM basis, which made sense when a guest_memfd _only_ supported private memory, but tracking per-VM simply can't work for in-place conversions as the shared/private status of a given page needs to be per-gmem_inode, not per-VM. Use the filemap invalidation lock to protect the maple tree, as taking the lock for read when faulting in memory (for userspace or the guest) isn't expected to result in meaningful contention, and using a separate lock would add significant complexity (avoiding deadlock is quite difficult). In kvm_gmem_get_pfn(), drop the folio refcount before releasing filemap_invalidate_lock(). This ensures that a competing conversion request from userspace (to be added in a later patch), which also takes the filemap_invalidate_lock(), will never see an elevated refcount due to kvm_gmem_get_pfn(). Co-developed-by: Vishal Annapurve Signed-off-by: Vishal Annapurve Co-developed-by: Fuad Tabba Signed-off-by: Fuad Tabba Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-3-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 136 +++++++++++++++++++++++++++++++++++------ 1 file changed, 119 insertions(+), 17 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 73974fece401..fc92c45bb249 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,13 @@ struct gmem_inode { struct list_head gmem_file_list; u64 flags; + /* + * Every index in this inode, whether memory is populated or + * not, is tracked in attributes. The entire range of indices, + * corresponding to the size of this inode, is represented in + * this maple tree. + */ + struct maple_tree attributes; }; static __always_inline struct gmem_inode *GMEM_I(struct inode *inode) @@ -61,9 +69,28 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) return gfn - slot->base_gfn + slot->gmem.pgoff; } +static u64 kvm_gmem_get_default_attributes(struct inode *inode) +{ + bool init_shared = GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED; + + return init_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE; +} + +static u64 kvm_gmem_get_attributes(struct inode *inode, void *entry) +{ + if (WARN_ON_ONCE(!entry)) + return kvm_gmem_get_default_attributes(inode); + + return xa_to_value(entry); +} + static bool kvm_gmem_is_private_mem(struct inode *inode, pgoff_t index) { - return !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED); + struct maple_tree *mt = &GMEM_I(inode)->attributes; + void *entry = mtree_load(mt, index); + + return kvm_gmem_get_attributes(inode, entry) & + KVM_MEMORY_ATTRIBUTE_PRIVATE; } static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index) @@ -403,10 +430,13 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS; - if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff)) - return VM_FAULT_SIGBUS; + filemap_invalidate_lock_shared(inode->i_mapping); + if (kvm_gmem_is_shared_mem(inode, vmf->pgoff)) + folio = kvm_gmem_get_folio(inode, vmf->pgoff); + else + folio = ERR_PTR(-EACCES); + filemap_invalidate_unlock_shared(inode->i_mapping); - folio = kvm_gmem_get_folio(inode, vmf->pgoff); if (IS_ERR(folio)) { if (PTR_ERR(folio) == -EAGAIN) return VM_FAULT_RETRY; @@ -559,6 +589,51 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm) return true; } +static int kvm_gmem_init_inode(struct inode *inode, loff_t size, u64 flags) +{ + struct gmem_inode *gi = GMEM_I(inode); + MA_STATE(mas, &gi->attributes, 0, (size >> PAGE_SHIFT) - 1); + u64 attrs; + int r; + + inode->i_op = &kvm_gmem_iops; + inode->i_mapping->a_ops = &kvm_gmem_aops; + inode->i_mode |= S_IFREG; + inode->i_size = size; + mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER); + + /* + * guest_memfd memory is neither migratable nor swappable: set + * inaccessible to gate off both. + */ + mapping_set_inaccessible(inode->i_mapping); + WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping)); + + gi->flags = flags; + + mt_set_external_lock(&gi->attributes, + &inode->i_mapping->invalidate_lock); + + /* + * Store default attributes for the entire gmem instance. Ensuring every + * index is represented in the maple tree at all times simplifies the + * conversion and merging logic. + */ + attrs = kvm_gmem_get_default_attributes(inode); + + /* + * Acquire the invalidation lock purely to make lockdep happy. The + * maple tree library expects all stores to be protected via the lock, + * and the library can't know when the tree is reachable only by the + * caller, as is the case here. + */ + filemap_invalidate_lock(inode->i_mapping); + r = mas_store_gfp(&mas, xa_mk_value(attrs), GFP_KERNEL); + filemap_invalidate_unlock(inode->i_mapping); + + return r; +} + static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) { static const char *name = "[kvm-gmem]"; @@ -589,16 +664,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) goto err_fops; } - inode->i_op = &kvm_gmem_iops; - inode->i_mapping->a_ops = &kvm_gmem_aops; - inode->i_mode |= S_IFREG; - inode->i_size = size; - mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER); - mapping_set_inaccessible(inode->i_mapping); - /* Unmovable mappings are supposed to be marked unevictable as well. */ - WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping)); - - GMEM_I(inode)->flags = flags; + err = kvm_gmem_init_inode(inode, size, flags); + if (err) + goto err_inode; file = alloc_file_pseudo(inode, kvm_gmem_mnt, name, O_RDWR, &kvm_gmem_fops); if (IS_ERR(file)) { @@ -813,9 +881,13 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, if (!file) return -EFAULT; + filemap_invalidate_lock_shared(file_inode(file)->i_mapping); + folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order); - if (IS_ERR(folio)) - return PTR_ERR(folio); + if (IS_ERR(folio)) { + r = PTR_ERR(folio); + goto out; + } if (!folio_test_uptodate(folio)) { clear_highpage(folio_page(folio, 0)); @@ -830,6 +902,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, folio_unlock(folio); folio_put(folio); +out: + filemap_invalidate_unlock_shared(file_inode(file)->i_mapping); return r; } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_get_pfn); @@ -959,6 +1033,15 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb) mpol_shared_policy_init(&gi->policy, NULL); + /* + * Memory attributes are protected by the filemap invalidation lock, but + * the lock structure isn't available at this time. Immediately mark + * maple tree as using external locking so that accessing the tree + * before it's fully initialized results in NULL pointer dereferences + * and not more subtle bugs. + */ + mt_init_flags(&gi->attributes, MT_FLAGS_LOCK_EXTERN | MT_FLAGS_USE_RCU); + gi->flags = 0; INIT_LIST_HEAD(&gi->gmem_file_list); return &gi->vfs_inode; @@ -966,7 +1049,26 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb) static void kvm_gmem_destroy_inode(struct inode *inode) { - mpol_free_shared_policy(&GMEM_I(inode)->policy); + struct gmem_inode *gi = GMEM_I(inode); + + mpol_free_shared_policy(&gi->policy); + + /* + * Note! Checking for an empty tree is functionally necessary + * to avoid explosions if the tree hasn't been fully + * initialized, i.e. if the inode is being destroyed before + * guest_memfd can set the external lock, lockdep would find + * that the tree's internal ma_lock was not held. + */ + if (!mtree_empty(&gi->attributes)) { + /* + * Acquire the invalidation lock purely to make lockdep happy, + * the inode is unreachable at this point. + */ + filemap_invalidate_lock(inode->i_mapping); + __mt_destroy(&gi->attributes); + filemap_invalidate_unlock(inode->i_mapping); + } } static void kvm_gmem_free_inode(struct inode *inode) From 44e17d0a3663eeba1629d4ec5f14a16dc445d659 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:30 -0700 Subject: [PATCH 17/60] KVM: Rename KVM_GENERIC_MEMORY_ATTRIBUTES to KVM_VM_MEMORY_ATTRIBUTES Rename the per-VM memory attributes Kconfig to make it explicitly about per-VM attributes in anticipation of adding memory attributes support to guest_memfd, at which point it will be possible (and desirable) to have memory attributes without the per-VM support, even in x86. No functional change intended. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-4-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 6 +++--- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/x86.c | 2 +- include/linux/kvm_host.h | 8 ++++---- include/trace/events/kvm.h | 4 ++-- virt/kvm/Kconfig | 2 +- virt/kvm/kvm_main.c | 14 +++++++------- 8 files changed, 20 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 6beddd8a0c74..744c1f6ff03e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1851,7 +1851,7 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI))) -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem) #endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index 538ed1e80332..2f8e1b5a8a72 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -84,7 +84,7 @@ config KVM_SW_PROTECTED_VM bool "Enable support for KVM software-protected VMs" depends on EXPERT depends on KVM_X86 && X86_64 - select KVM_GENERIC_MEMORY_ATTRIBUTES + select KVM_VM_MEMORY_ATTRIBUTES help Enable support for KVM software-protected VMs. Currently, software- protected VMs are purely a development and testing vehicle for @@ -135,7 +135,7 @@ config KVM_INTEL_TDX bool "Intel Trust Domain Extensions (TDX) support" default y depends on INTEL_TDX_HOST - select KVM_GENERIC_MEMORY_ATTRIBUTES + select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_POPULATE help Provides support for launching Intel Trust Domain Extensions (TDX) @@ -159,7 +159,7 @@ config KVM_AMD_SEV depends on KVM_AMD && X86_64 depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM - select KVM_GENERIC_MEMORY_ATTRIBUTES + select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_CONVERT select HAVE_KVM_ARCH_GMEM_RECLAIM select HAVE_KVM_ARCH_GMEM_INVALIDATE diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 947c9cd84345..b88c00e11854 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -8097,7 +8097,7 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm) vhost_task_stop(kvm->arch.nx_huge_page_recovery_thread); } -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static bool hugepage_test_mixed(struct kvm_memory_slot *slot, gfn_t gfn, int level) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 79468ddfe473..229224957031 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10140,7 +10140,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm, } } -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES kvm_mmu_init_memslot_memory_attributes(kvm, slot); #endif diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index b7516e19ef41..cffa16647f2a 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu) } #endif -#ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifndef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static inline bool kvm_arch_has_private_mem(struct kvm *kvm) { return false; @@ -872,7 +872,7 @@ struct kvm { #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER struct notifier_block pm_notifier; #endif -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES /* Protected by slots_lock (for writes) and RCU (for reads) */ struct xarray mem_attr_array; #endif @@ -2560,7 +2560,7 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot) return slot->flags & KVM_MEMSLOT_GMEM_ONLY; } -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn) { return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)); @@ -2582,7 +2582,7 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) { return false; } -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ #ifdef CONFIG_KVM_GUEST_MEMFD int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index b282e3a86769..1ba72bd73ea2 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -358,7 +358,7 @@ TRACE_EVENT(kvm_dirty_ring_exit, TP_printk("vcpu %d", __entry->vcpu_id) ); -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES /* * @start: Starting address of guest memory range * @end: End address of guest memory range @@ -383,7 +383,7 @@ TRACE_EVENT(kvm_vm_set_mem_attributes, TP_printk("%#016llx -- %#016llx [0x%lx]", __entry->start, __entry->end, __entry->attr) ); -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ TRACE_EVENT(kvm_unmap_hva_range, TP_PROTO(unsigned long start, unsigned long end), diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index c3c0ee253fc7..df575449d97b 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,7 +100,7 @@ config KVM_ELIDE_TLB_FLUSH_IF_YOUNG config KVM_MMU_LOCKLESS_AGING bool -config KVM_GENERIC_MEMORY_ATTRIBUTES +config KVM_VM_MEMORY_ATTRIBUTES bool config KVM_GUEST_MEMFD diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index f26986136bc6..2b425b4607b9 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1116,7 +1116,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname) spin_lock_init(&kvm->mn_invalidate_lock); rcuwait_init(&kvm->mn_memslots_update_rcuwait); xa_init(&kvm->vcpu_array); -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES xa_init(&kvm->mem_attr_array); #endif @@ -1301,7 +1301,7 @@ static void kvm_destroy_vm(struct kvm *kvm) cleanup_srcu_struct(&kvm->irq_srcu); srcu_barrier(&kvm->srcu); cleanup_srcu_struct(&kvm->srcu); -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES xa_destroy(&kvm->mem_attr_array); #endif kvm_arch_free_vm(kvm); @@ -2438,7 +2438,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, } #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static u64 kvm_supported_mem_attributes(struct kvm *kvm) { if (!kvm || kvm_arch_has_private_mem(kvm)) @@ -2643,7 +2643,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes); } -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn) { @@ -4960,7 +4960,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) case KVM_CAP_SYSTEM_EVENT_DATA: case KVM_CAP_DEVICE_CTRL: return 1; -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES case KVM_CAP_MEMORY_ATTRIBUTES: return kvm_supported_mem_attributes(kvm); #endif @@ -5364,7 +5364,7 @@ static long kvm_vm_ioctl(struct file *filp, break; } #endif /* CONFIG_HAVE_KVM_IRQ_ROUTING */ -#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES case KVM_SET_MEMORY_ATTRIBUTES: { struct kvm_memory_attributes attrs; @@ -5375,7 +5375,7 @@ static long kvm_vm_ioctl(struct file *filp, r = kvm_vm_ioctl_set_mem_attributes(kvm, &attrs); break; } -#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ case KVM_CREATE_DEVICE: { struct kvm_create_device cd; From c2ae2bc1a311975b1c9c808cf9ad2758faf94495 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:31 -0700 Subject: [PATCH 18/60] KVM: Enumerate support for PRIVATE memory iff kvm_arch_has_private_mem is defined Explicitly guard reporting support for KVM_MEMORY_ATTRIBUTE_PRIVATE based on kvm_arch_has_private_mem being #defined in anticipation of tracking PRIVATE vs. SHARED state per-guest_memfd, not per-VM (to allow in-place conversion). guest_memfd support for memory attributes is expected to be unconditional to avoid yet more macros (all architectures that support guest_memfd are expected to use per-gmem attributes at some point), at which point enumerating support KVM_MEMORY_ATTRIBUTE_PRIVATE based solely on memory attributes being supported by KVM at-large would result in a system-scope check (NULL @kvm) over-reporting support on arm64. Give architectures full control over overriding the default definition of kvm_arch_has_private_mem() by removing the coupling with CONFIG_KVM_VM_MEMORY_ATTRIBUTES. In a later patch, kvm_arch_has_private_mem() will be defined based on whether architectural features are compiled in, and made orthogonal to CONFIG_KVM_VM_MEMORY_ATTRIBUTES. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: Xiaoyao Li Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-5-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- include/linux/kvm_host.h | 2 +- virt/kvm/kvm_main.c | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index cffa16647f2a..b45fb80b0aca 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu) } #endif -#ifndef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +#ifndef kvm_arch_has_private_mem static inline bool kvm_arch_has_private_mem(struct kvm *kvm) { return false; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 2b425b4607b9..798afcf14194 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2441,8 +2441,10 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static u64 kvm_supported_mem_attributes(struct kvm *kvm) { +#ifdef kvm_arch_has_private_mem if (!kvm || kvm_arch_has_private_mem(kvm)) return KVM_MEMORY_ATTRIBUTE_PRIVATE; +#endif return 0; } From a84ef89cbf91e88fff493b01b142cc949bb9bd7b Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:32 -0700 Subject: [PATCH 19/60] KVM: Rename memory attribute APIs to prepare for in-place gmem conversion Rename memory attribute APIs to add a "vm_" in the name in anticipation of moving PRIVATE tracking into guest_memfd, to allow in-place conversion between SHARED and PRIVATE. At that point, there will effectively be two (potential) sources of memory attributes: the VM and guest_memfd. kvm_vm_set_mem_attributes() already has "vm" in the name to indicate that it is a VM ioctl. Rename it to kvm_set_vm_mem_attributes() to show that it is setting the VM's memory attributes. (Drop the VM-ioctl scoping since it is a helper local to the file.) Update the accompanying trace function to match. No functional change intended. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-6-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 14 +++++++------- include/linux/kvm_host.h | 16 ++++++++-------- include/trace/events/kvm.h | 2 +- virt/kvm/kvm_main.c | 32 ++++++++++++++++---------------- 4 files changed, 32 insertions(+), 32 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b88c00e11854..10e59507d252 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -8116,8 +8116,8 @@ static void hugepage_set_mixed(struct kvm_memory_slot *slot, gfn_t gfn, lpage_info_slot(gfn, slot, level)->disallow_lpage |= KVM_LPAGE_MIXED_FLAG; } -bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm, - struct kvm_gfn_range *range) +bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range) { struct kvm_memory_slot *slot = range->slot; int level; @@ -8186,18 +8186,18 @@ static bool hugepage_has_attrs(struct kvm *kvm, struct kvm_memory_slot *slot, const unsigned long end = start + KVM_PAGES_PER_HPAGE(level); if (level == PG_LEVEL_2M) - return kvm_range_has_memory_attributes(kvm, start, end, ~0, attrs); + return kvm_range_has_vm_memory_attributes(kvm, start, end, ~0, attrs); for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1)) { if (hugepage_test_mixed(slot, gfn, level - 1) || - attrs != kvm_get_memory_attributes(kvm, gfn)) + attrs != kvm_get_vm_memory_attributes(kvm, gfn)) return false; } return true; } -bool kvm_arch_post_set_memory_attributes(struct kvm *kvm, - struct kvm_gfn_range *range) +bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range) { unsigned long attrs = range->arg.attributes; struct kvm_memory_slot *slot = range->slot; @@ -8290,7 +8290,7 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, * be manually checked as the attributes may already be mixed. */ for (gfn = start; gfn < end; gfn += nr_pages) { - unsigned long attrs = kvm_get_memory_attributes(kvm, gfn); + unsigned long attrs = kvm_get_vm_memory_attributes(kvm, gfn); if (hugepage_has_attrs(kvm, slot, gfn, level, attrs)) hugepage_clear_mixed(slot, gfn, level); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index b45fb80b0aca..6ca81f5349db 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2561,21 +2561,21 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot) } #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES -static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn) +static inline unsigned long kvm_get_vm_memory_attributes(struct kvm *kvm, gfn_t gfn) { return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)); } -bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, - unsigned long mask, unsigned long attrs); -bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm, - struct kvm_gfn_range *range); -bool kvm_arch_post_set_memory_attributes(struct kvm *kvm, - struct kvm_gfn_range *range); +bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, + unsigned long mask, unsigned long attrs); +bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range); +bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range); static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) { - return kvm_get_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE; + return kvm_get_vm_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE; } #else static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 1ba72bd73ea2..c4d514d2350c 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -364,7 +364,7 @@ TRACE_EVENT(kvm_dirty_ring_exit, * @end: End address of guest memory range * @attr: The value of the attribute being set. */ -TRACE_EVENT(kvm_vm_set_mem_attributes, +TRACE_EVENT(kvm_set_vm_mem_attributes, TP_PROTO(gfn_t start, gfn_t end, unsigned long attr), TP_ARGS(start, end, attr), diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 798afcf14194..960060a442bc 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2439,7 +2439,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES -static u64 kvm_supported_mem_attributes(struct kvm *kvm) +static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm) { #ifdef kvm_arch_has_private_mem if (!kvm || kvm_arch_has_private_mem(kvm)) @@ -2453,19 +2453,19 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm) * Returns true if _all_ gfns in the range [@start, @end) have attributes * such that the bits in @mask match @attrs. */ -bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, - unsigned long mask, unsigned long attrs) +bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, + unsigned long mask, unsigned long attrs) { XA_STATE(xas, &kvm->mem_attr_array, start); unsigned long index; void *entry; - mask &= kvm_supported_mem_attributes(kvm); + mask &= kvm_supported_vm_mem_attributes(kvm); if (attrs & ~mask) return false; if (end == start + 1) - return (kvm_get_memory_attributes(kvm, start) & mask) == attrs; + return (kvm_get_vm_memory_attributes(kvm, start) & mask) == attrs; guard(rcu)(); if (!attrs) @@ -2536,8 +2536,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm, KVM_MMU_UNLOCK(kvm); } -static bool kvm_pre_set_memory_attributes(struct kvm *kvm, - struct kvm_gfn_range *range) +static bool kvm_pre_set_vm_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range) { /* * Unconditionally add the range to the invalidation set, regardless of @@ -2552,18 +2552,18 @@ static bool kvm_pre_set_memory_attributes(struct kvm *kvm, */ kvm_mmu_invalidate_range_add(kvm, range->start, range->end); - return kvm_arch_pre_set_memory_attributes(kvm, range); + return kvm_arch_pre_set_vm_memory_attributes(kvm, range); } /* Set @attributes for the gfn range [@start, @end). */ -static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, +static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, unsigned long attributes) { struct kvm_mmu_notifier_range pre_set_range = { .start = start, .end = end, .arg.attributes = attributes, - .handler = kvm_pre_set_memory_attributes, + .handler = kvm_pre_set_vm_memory_attributes, .on_lock = kvm_mmu_invalidate_start, .flush_on_ret = true, .may_block = true, @@ -2572,7 +2572,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, .start = start, .end = end, .arg.attributes = attributes, - .handler = kvm_arch_post_set_memory_attributes, + .handler = kvm_arch_post_set_vm_memory_attributes, .on_lock = kvm_mmu_invalidate_end, .may_block = true, }; @@ -2582,12 +2582,12 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, entry = attributes ? xa_mk_value(attributes) : NULL; - trace_kvm_vm_set_mem_attributes(start, end, attributes); + trace_kvm_set_vm_mem_attributes(start, end, attributes); mutex_lock(&kvm->slots_lock); /* Nothing to do if the entire range has the desired attributes. */ - if (kvm_range_has_memory_attributes(kvm, start, end, ~0, attributes)) + if (kvm_range_has_vm_memory_attributes(kvm, start, end, ~0, attributes)) goto out_unlock; /* @@ -2626,7 +2626,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, /* flags is currently not used. */ if (attrs->flags) return -EINVAL; - if (attrs->attributes & ~kvm_supported_mem_attributes(kvm)) + if (attrs->attributes & ~kvm_supported_vm_mem_attributes(kvm)) return -EINVAL; if (attrs->size == 0 || attrs->address + attrs->size < attrs->address) return -EINVAL; @@ -2643,7 +2643,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, */ BUILD_BUG_ON(sizeof(attrs->attributes) != sizeof(unsigned long)); - return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes); + return kvm_set_vm_mem_attributes(kvm, start, end, attrs->attributes); } #endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ @@ -4964,7 +4964,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) return 1; #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES case KVM_CAP_MEMORY_ATTRIBUTES: - return kvm_supported_mem_attributes(kvm); + return kvm_supported_vm_mem_attributes(kvm); #endif #ifdef CONFIG_KVM_GUEST_MEMFD case KVM_CAP_GUEST_MEMFD: From 98fd6e30bd148d6c76aa39407a98281ea4256e3c Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:33 -0700 Subject: [PATCH 20/60] KVM: Rename kvm_mem_is_private() to kvm_is_private_gfn() Rename kvm_mem_is_private() to kvm_is_private_gfn() to prepare for in-place conversion, where there will be two lookup functions, kvm_vm_is_private_gfn() and kvm_gmem_is_private_gfn(). This renaming allows consistent prefixing of "vm" vs "gmem" for kvm_*_is_private_gfn(), as opposed to kvm_gmem_mem_is_private(), which looks like a typo. Suggested-by: Sean Christopherson Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: Xiaoyao Li Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-7-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 12 ++++++------ arch/x86/kvm/svm/sev.c | 2 +- include/linux/kvm_host.h | 4 ++-- virt/kvm/guest_memfd.c | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 10e59507d252..62f751952ad8 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3421,7 +3421,7 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_page_fault *fault, is_private = fault->is_private; } else { max_level = PG_LEVEL_NUM; - is_private = kvm_mem_is_private(kvm, gfn); + is_private = kvm_is_private_gfn(kvm, gfn); } max_level = min(max_level, max_huge_page_level); @@ -3634,13 +3634,13 @@ static bool page_fault_can_be_fast(struct kvm *kvm, struct kvm_page_fault *fault * guest spinning on a #PF indefinitely, so don't attempt the fast path * in this case. * - * Note that the kvm_mem_is_private() check might race with an + * Note that the kvm_is_private_gfn() check might race with an * attribute update, but this will either result in the guest spinning * on RET_PF_SPURIOUS until the update completes, or an actual spurious * case might go down the slow path. Either case will resolve itself. */ if (kvm->arch.has_private_mem && - fault->is_private != kvm_mem_is_private(kvm, fault->gfn)) + fault->is_private != kvm_is_private_gfn(kvm, fault->gfn)) return false; /* @@ -4708,7 +4708,7 @@ static int kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu, * Now that we have a snapshot of mmu_invalidate_seq we can check for a * private vs. shared mismatch. */ - if (fault->is_private != kvm_mem_is_private(kvm, fault->gfn)) { + if (fault->is_private != kvm_is_private_gfn(kvm, fault->gfn)) { kvm_mmu_prepare_memory_fault_exit(vcpu, fault); return -EFAULT; } @@ -5111,7 +5111,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, direct_bits = 0; if (kvm_arch_has_private_mem(vcpu->kvm) && - kvm_mem_is_private(vcpu->kvm, gpa_to_gfn(range->gpa))) + kvm_is_private_gfn(vcpu->kvm, gpa_to_gfn(range->gpa))) error_code |= PFERR_PRIVATE_ACCESS; else direct_bits = gfn_to_gpa(kvm_gfn_direct_bits(vcpu->kvm)); @@ -6584,7 +6584,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err if (IS_ENABLED(CONFIG_KVM_SW_PROTECTED_VM) && !(error_code & PFERR_RSVD_MASK) && vcpu->kvm->arch.vm_type == KVM_X86_SW_PROTECTED_VM && - kvm_mem_is_private(vcpu->kvm, gpa_to_gfn(cr2_or_gpa))) + kvm_is_private_gfn(vcpu->kvm, gpa_to_gfn(cr2_or_gpa))) error_code |= PFERR_PRIVATE_ACCESS; r = RET_PF_INVALID; diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 0a887f8e05d3..dff684b88f15 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -5030,7 +5030,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) * userspace via KVM_EXIT_MEMORY_FAULT events, however, so RMP faults * for shared pages should not end up here. */ - if (!kvm_mem_is_private(kvm, gfn)) { + if (!kvm_is_private_gfn(kvm, gfn)) { pr_warn_ratelimited("SEV: Unexpected RMP fault for non-private GPA 0x%llx\n", gpa); return; diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 6ca81f5349db..9b97266f61ec 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2573,12 +2573,12 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm, bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm, struct kvm_gfn_range *range); -static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) +static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) { return kvm_get_vm_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE; } #else -static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) +static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) { return false; } diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index fc92c45bb249..3b05c6443f0f 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -929,7 +929,7 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot, folio_unlock(folio); - if (!kvm_mem_is_private(kvm, gfn)) { + if (!kvm_is_private_gfn(kvm, gfn)) { ret = -EINVAL; goto out_put_folio; } From 8ab5719fbcdc96f5e636b67f14204ef7670abde1 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:34 -0700 Subject: [PATCH 21/60] KVM: Provide generic interface for checking memory private/shared status Introduce a generic kvm_is_private_gfn() interface using a static call to determine if a GFN is private. This allows the implementation for checking a GFN's private/shared status to be set at runtime. In preparation for choosing implementations between a guest_memfd lookup and the existing VM attribute lookup, rename the existing VM-attribute-based check to kvm_vm_is_private_gfn() to emphasize that it looks up VM attributes. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Xiaoyao Li Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-8-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- include/linux/kvm_host.h | 14 ++++++++++++-- virt/kvm/kvm_main.c | 15 +++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 9b97266f61ec..3579feb63177 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2573,16 +2573,26 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm, bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm, struct kvm_gfn_range *range); -static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) +static inline bool kvm_vm_is_private_gfn(struct kvm *kvm, gfn_t gfn) { return kvm_get_vm_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE; } +#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ + +#ifdef kvm_arch_has_private_mem +typedef bool (kvm_is_private_gfn_t)(struct kvm *kvm, gfn_t gfn); +DECLARE_STATIC_CALL(__kvm_is_private_gfn, kvm_is_private_gfn_t); + +static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) +{ + return static_call(__kvm_is_private_gfn)(kvm, gfn); +} #else static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) { return false; } -#endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ +#endif /* kvm_arch_has_private_mem */ #ifdef CONFIG_KVM_GUEST_MEMFD int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 960060a442bc..19098a51c35d 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2647,6 +2647,20 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, } #endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ +#ifdef kvm_arch_has_private_mem +DEFINE_STATIC_CALL_RET0(__kvm_is_private_gfn, kvm_is_private_gfn_t); +EXPORT_STATIC_CALL_GPL(__kvm_is_private_gfn); + +static void kvm_init_memory_attributes(void) +{ +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES + static_call_update(__kvm_is_private_gfn, kvm_vm_is_private_gfn); +#endif +} +#else +static void kvm_init_memory_attributes(void) { } +#endif + struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn) { return __gfn_to_memslot(kvm_memslots(kvm), gfn); @@ -6560,6 +6574,7 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) kvm_preempt_ops.sched_in = kvm_sched_in; kvm_preempt_ops.sched_out = kvm_sched_out; + kvm_init_memory_attributes(); kvm_init_debug(); r = kvm_vfio_ops_init(); From ee1168fb73136dc1bd6b6f26142fdaa2399a2d60 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:35 -0700 Subject: [PATCH 22/60] KVM: guest_memfd: Stub in ability to enable in-place shared<=>private conversion Stub in global variable to enable in-place guest_memfd private<=>shared memory conversion, which will eventually be exposed to userspace via a module param, and wire up the __kvm_is_private_gfn() static call to the guest_memfd version when in-place conversion is enabled, i.e. when gmem is the sole authority on private vs. shared memory. Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Cc: Fuad Tabba Cc: Xiaoyao Li Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-9-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- Documentation/virt/kvm/api.rst | 17 ++++++++++++----- include/linux/kvm_host.h | 6 ++++++ virt/kvm/guest_memfd.c | 26 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 12 +++++++++++- 4 files changed, 55 insertions(+), 6 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index e0430cc750c9..09967e030803 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6381,11 +6381,16 @@ mapping for userspace_addr is not required to be valid/populated at the time of KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. -When mapping a gfn into the guest, KVM selects shared vs. private, i.e consumes -userspace_addr vs. guest_memfd, based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE -state. At VM creation time, all memory is shared, i.e. the PRIVATE attribute -is '0' for all gfns. Userspace can control whether memory is shared/private by -toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via KVM_SET_MEMORY_ATTRIBUTES as needed. +When mapping a gfn into the guest, KVM selects shared vs. private, i.e. consumes +userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the +sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` +to find out more about the creation-time shared/private status. + +If in-place conversion is disabled, KVM selects shared vs. private based on the +gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE state. At VM creation time, all memory is +shared, i.e. the PRIVATE attribute is '0' for all gfns. Userspace can control +whether memory is shared/private by toggling KVM_MEMORY_ATTRIBUTE_PRIVATE via +KVM_SET_MEMORY_ATTRIBUTES as needed. S390: ^^^^^ @@ -6429,6 +6434,8 @@ the state of a gfn/page as needed. The "flags" field is reserved for future extensions and must be '0'. +.. _KVM_CREATE_GUEST_MEMFD: + 4.142 KVM_CREATE_GUEST_MEMFD ---------------------------- diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 3579feb63177..b0110143d2ff 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2580,6 +2580,8 @@ static inline bool kvm_vm_is_private_gfn(struct kvm *kvm, gfn_t gfn) #endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */ #ifdef kvm_arch_has_private_mem +extern bool gmem_in_place_conversion; + typedef bool (kvm_is_private_gfn_t)(struct kvm *kvm, gfn_t gfn); DECLARE_STATIC_CALL(__kvm_is_private_gfn, kvm_is_private_gfn_t); @@ -2588,6 +2590,8 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) return static_call(__kvm_is_private_gfn)(kvm, gfn); } #else +#define gmem_in_place_conversion false + static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) { return false; @@ -2595,6 +2599,8 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) #endif /* kvm_arch_has_private_mem */ #ifdef CONFIG_KVM_GUEST_MEMFD +bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn); + int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, kvm_pfn_t *pfn, int *max_order); #else diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 3b05c6443f0f..134fed0e24fa 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -517,6 +517,32 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma) return 0; } +bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn) +{ + struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn); + struct inode *inode; + + if (!slot) + return 0; + + CLASS(gmem_get_file, file)(slot); + if (!file) + return 0; + + inode = file_inode(file); + + /* + * Rely on the maple tree's internal RCU lock to ensure a stable result. + * This result can become stale as soon as the lock is dropped, so the + * caller _must_ protect consumption of private vs. shared either by + * holding guest_memfd's invalidate lock for the entire duration, or by + * checking mmu_invalidate_retry_gfn() under mmu_lock to serialize + * against concurrent attribute updates. + */ + return kvm_gmem_is_private_mem(inode, kvm_gmem_get_index(slot, gfn)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_is_private_gfn); + static struct file_operations kvm_gmem_fops = { .mmap = kvm_gmem_mmap, .open = generic_file_open, diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 19098a51c35d..2532d1252a10 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -102,6 +102,10 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink); static bool __ro_after_init allow_unsafe_mappings; module_param(allow_unsafe_mappings, bool, 0444); +#ifdef kvm_arch_has_private_mem +bool __ro_after_init gmem_in_place_conversion = false; +#endif + /* * Ordering of locks: * @@ -2442,6 +2446,9 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm) { #ifdef kvm_arch_has_private_mem + if (gmem_in_place_conversion) + return 0; + if (!kvm || kvm_arch_has_private_mem(kvm)) return KVM_MEMORY_ATTRIBUTE_PRIVATE; #endif @@ -2653,8 +2660,11 @@ EXPORT_STATIC_CALL_GPL(__kvm_is_private_gfn); static void kvm_init_memory_attributes(void) { + if (gmem_in_place_conversion) + static_call_update(__kvm_is_private_gfn, kvm_gmem_is_private_gfn); #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES - static_call_update(__kvm_is_private_gfn, kvm_vm_is_private_gfn); + else + static_call_update(__kvm_is_private_gfn, kvm_vm_is_private_gfn); #endif } #else From bcc640eabbb289b96acd06c35adc623739d20548 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:36 -0700 Subject: [PATCH 23/60] KVM: Consolidate private memory and guest_memfd ifdeffery in kvm_host.h Move the kvm_arch_has_private_mem() stub and a few guest_memfd function definitions/declarations "down" in kvm_host.h to utilize existing #ifdefs, and so that related code is clustered together. No functional change intended. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Reviewed-by: Fuad Tabba Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-10-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- include/linux/kvm_host.h | 37 ++++++++++++++++--------------------- 1 file changed, 16 insertions(+), 21 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index b0110143d2ff..ab87effdd221 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -722,27 +722,6 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu) } #endif -#ifndef kvm_arch_has_private_mem -static inline bool kvm_arch_has_private_mem(struct kvm *kvm) -{ - return false; -} -#endif - -#ifdef CONFIG_KVM_GUEST_MEMFD -bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm); - -static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm) -{ - u64 flags = GUEST_MEMFD_FLAG_MMAP; - - if (!kvm || kvm_arch_supports_gmem_init_shared(kvm)) - flags |= GUEST_MEMFD_FLAG_INIT_SHARED; - - return flags; -} -#endif - #ifndef kvm_arch_has_readonly_mem static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm) { @@ -2592,6 +2571,11 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) #else #define gmem_in_place_conversion false +static inline bool kvm_arch_has_private_mem(struct kvm *kvm) +{ + return false; +} + static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) { return false; @@ -2600,6 +2584,17 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn) #ifdef CONFIG_KVM_GUEST_MEMFD bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn); +bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm); + +static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm) +{ + u64 flags = GUEST_MEMFD_FLAG_MMAP; + + if (!kvm || kvm_arch_supports_gmem_init_shared(kvm)) + flags |= GUEST_MEMFD_FLAG_INIT_SHARED; + + return flags; +} int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, kvm_pfn_t *pfn, int *max_order); From e088c7f86befedcfb391ab54c924d65d36f0c328 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:37 -0700 Subject: [PATCH 24/60] KVM: guest_memfd: Invalidate both SHARED and PRIVATE mappings for in-place conversions When removing one or more folios from a guest_memfd instance, invalidate both SHARED and PRIVATE mappings if in-place conversion is enabled, because stating the obvious, KVM needs to ensure that all mappings to the folio(s) are dropped. Opportunistically rename the helper to capture that it returns a filter for all gfns in anticipation of zapping only the previous mapping types on conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED mappings need to be zapped (ignoring that KVM would ideally not invalidate ranges whose attributes aren't changing in the first place). Note, precisely zapping only the possible mapping types when in-place conversion is disabled is important for functional correctness, not just for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will incorrectly zap SHARED mappings that have nothing to do with that gmem instance (because they're mapped via a VMA, not a gmem fd). The incorrect over-zapping of SHARED memory that doesn't belong to the gmem fd requesting the zapping will be resolved in a later patch, where, if in-place conversion is enabled, KVM will use both shared and private memory from the guest_memfd. If both shared and private memory are from the guest_memfd, invalidation will only zap memory belonging to the given gmem instance. Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-11-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 134fed0e24fa..2af2b6f0ab61 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -139,8 +139,11 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index) return folio; } -static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode) +static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inode) { + if (gmem_in_place_conversion) + return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE; + if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED) return KVM_FILTER_SHARED; @@ -191,11 +194,9 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start, pgoff_t end) { - enum kvm_gfn_range_filter attr_filter; + enum kvm_gfn_range_filter attr_filter = kvm_gmem_get_all_gfns_filter(inode); struct gmem_file *f; - attr_filter = kvm_gmem_get_invalidate_filter(inode); - kvm_gmem_for_each_file(f, inode) __kvm_gmem_invalidate_start(f, start, end, attr_filter); } @@ -386,7 +387,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file) * memory, as its lifetime is associated with the inode, not the file. */ __kvm_gmem_invalidate_start(f, 0, -1ul, - kvm_gmem_get_invalidate_filter(inode)); + kvm_gmem_get_all_gfns_filter(inode)); __kvm_gmem_invalidate_end(f, 0, -1ul); list_del(&f->entry); From 2e646bcfcf900cf35d5982e5d8d9bb8dab828380 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:38 -0700 Subject: [PATCH 25/60] KVM: guest_memfd: Always fault from guest_memfd if in-place conversion is enabled If a guest_memfd memslot is created but the guest_memfd does not have the GUEST_MEMFD_FLAG_MMAP set, KVM still fulfils guest faults by looking up the memslot's userspace_addr. Set KVM_MEMSLOT_GMEM_ONLY if in-place conversion is enabled so that the guest_memfd's memory will be used for both shared and private memory. With in-place conversion, guest_memfd will be the only backing memory for the memslot. No validation is performed to require userspace_addr to be a mapping from the associated guest_memfd because even after validation, userspace is free to remap something else at the provided userspace_addr. userspace_addr will still be used by functions like kvm_read_guest(), and if userspace_addr does not match up with the corresponding memory in the memslot's guest_memfd (whether userspace_addr points to the wrong offset or some non-guest_memfd memory, etc), that is a user error. Requiring both shared and private memory to come from the only associated guest_memfd simplifies invalidation in stage 2 page tables. On a PUNCH_HOLE operation on a guest_memfd, the invalidation is now guaranteed to be invalidating only memory mapped from the given guest_memfd. Suggested-by: Sean Christopherson Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-12-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- Documentation/virt/kvm/api.rst | 22 ++++++++++++++-------- virt/kvm/guest_memfd.c | 2 +- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 09967e030803..35d94c51cc56 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6381,10 +6381,16 @@ mapping for userspace_addr is not required to be valid/populated at the time of KVM_SET_USER_MEMORY_REGION2, e.g. shared memory can be lazily mapped/allocated on-demand. -When mapping a gfn into the guest, KVM selects shared vs. private, i.e. consumes -userspace_addr vs. guest_memfd, based on the state in guest_memfd, which is the -sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` -to find out more about the creation-time shared/private status. +When mapping a gfn into the guest, guest faults are always serviced from +guest_memfd regardless of whether memory is shared or private. KVM determines +shared vs. private based on the state in guest_memfd, which is the sole +authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` to +find out more about the creation-time shared/private status. + +userspace_addr is expected to be the mmap()-ed address corresponding to the +right offset within the guest_memfd. Any mismatch between userspace_addr and +guest_memfd is not validated and is a user error. userspace_addr is only used +for host-side guest accesses such as kvm_read_guest(). If in-place conversion is disabled, KVM selects shared vs. private based on the gfn's KVM_MEMORY_ATTRIBUTE_PRIVATE state. At VM creation time, all memory is @@ -6490,10 +6496,10 @@ specified via KVM_CREATE_GUEST_MEMFD. Currently defined flags: page tables. Private memory cannot. ============================ ================================================ -When the KVM MMU performs a PFN lookup to service a guest fault and the backing -guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be -consumed from guest_memfd, regardless of whether it is a shared or a private -fault. +When the KVM MMU performs a PFN lookup to service a guest fault, the fault will +always be consumed from guest_memfd, regardless of whether it is a shared or a +private fault (unless in-place conversion is disabled and the backing +guest_memfd does not have the GUEST_MEMFD_FLAG_MMAP flag set). See KVM_SET_USER_MEMORY_REGION2 for additional details. diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 2af2b6f0ab61..10831f9d7ab0 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -775,7 +775,7 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot */ slot->gmem.file = file; slot->gmem.pgoff = offset >> PAGE_SHIFT; - if (kvm_gmem_supports_mmap(inode)) + if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode)) slot->flags |= KVM_MEMSLOT_GMEM_ONLY; /* From 2f7f4e102880706a2cadb46ba77859b5e968768c Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:39 -0700 Subject: [PATCH 26/60] KVM: guest_memfd: Pass mapping type filter to invalidation helper Accept the mapping type filter as a parameter in the invalidation start helper instead of querying it internally. This allows callers to specify which mappings (shared, private, or both) should be invalidated. In the next patch, the conversion process will use this new parameter to invalidate mappings only when they're different from the target state of the conversion, i.e. invalidate only shared mappings on a shared to private conversion and not both. No functional change intended. Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-13-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 10831f9d7ab0..a7615a8bf50c 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -192,13 +192,13 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, } static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start, - pgoff_t end) + pgoff_t end, + enum kvm_gfn_range_filter filter) { - enum kvm_gfn_range_filter attr_filter = kvm_gmem_get_all_gfns_filter(inode); struct gmem_file *f; kvm_gmem_for_each_file(f, inode) - __kvm_gmem_invalidate_start(f, start, end, attr_filter); + __kvm_gmem_invalidate_start(f, start, end, filter); } static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start, @@ -224,6 +224,7 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start, static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len) { + enum kvm_gfn_range_filter filter = kvm_gmem_get_all_gfns_filter(inode); pgoff_t start = offset >> PAGE_SHIFT; pgoff_t end = (offset + len) >> PAGE_SHIFT; @@ -233,7 +234,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len) */ filemap_invalidate_lock(inode->i_mapping); - kvm_gmem_invalidate_start(inode, start, end); + kvm_gmem_invalidate_start(inode, start, end, filter); truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1); @@ -561,6 +562,8 @@ static int kvm_gmem_migrate_folio(struct address_space *mapping, static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folio) { + struct inode *inode = mapping->host; + enum kvm_gfn_range_filter filter; pgoff_t start, end; filemap_invalidate_lock_shared(mapping); @@ -568,7 +571,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol start = folio->index; end = start + folio_nr_pages(folio); - kvm_gmem_invalidate_start(mapping->host, start, end); + filter = kvm_gmem_get_all_gfns_filter(inode); + kvm_gmem_invalidate_start(inode, start, end, filter); /* * Do not truncate the range, what action is taken in response to the From 799c9fadc639949beeab9328930f99fbf873d01a Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:40 -0700 Subject: [PATCH 27/60] KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2 Add a new ioctl (and matching struct), KVM_SET_MEMORY_ATTRIBUTES2, using the same base ioctl number (0xd2), but with R/W semantics for the kernel instead of just read semantics. "Officially" documenting that KVM writes to the payload will allow KVM to support partial/incremental conversions, instead of all-or-nothing updates (which requires complex unwinding), by recording the failing offset if an error occurs. Opportunistically add a new struct as well, even though KVM could squeeze the error offset into "struct kvm_memory_attributes", as there's no cost to doing so in practice. Pad the struct with a pile of extra space to try and avoid ending up with "struct kvm_memory_attributes3" in the future. Use the same layout for the fields common to version 1 of the struct, e.g. to ease upgrading userspace, and to provide flexibility if KVM ever adds support for KVM_SET_MEMORY_ATTRIBUTES2 at VM scope. Introduce KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES to advertise the availability of the KVM_SET_MEMORY_ATTRIBUTES2 ioctl. Update the KVM API documentation to define the new ioctl and its behavior, and add the necessary UAPI definitions and capability checks. The process of setting memory attributes has a clear point of no return because, for CoCo VMs, zapping stage 2 page tables is a destructive operation. Unlike regular VMs, where re-faulting pages into the stage 2 page tables merely incurs a performance penalty, CoCo guests must (re-):accept pages after every fault. To preserve CoCo security guarantees, guests will not accept pages they did not explicitly request faults for. Consequently, during memory conversions, any operation that could cause the process to abort must be completed before the stage 2 page tables are zapped. Zap only the ranges that are not already in the requested state to avoid inadvertently destroying (CoCo) data. ARM CCA guests will try to mark the entire DRAM as private at boot. If there are no shared pages at all, the to-private conversion can be skipped, but the existence of a single shared page would require the conversion process to proceed, and if it proceeds, zapping both shared and private pages would destroy data and break the guest. Suggested-by: Michael Roth Suggested-by: Suzuki K Poulose Co-developed-by: Vishal Annapurve Signed-off-by: Vishal Annapurve Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: Suzuki K Poulose Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-14-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- Documentation/virt/kvm/api.rst | 71 +++++++++++++++++++- include/uapi/linux/kvm.h | 15 +++++ virt/kvm/guest_memfd.c | 119 +++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 23 +++++-- 4 files changed, 219 insertions(+), 9 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 35d94c51cc56..027156508d04 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -117,7 +117,7 @@ description: x86 includes both i386 and x86_64. Type: - system, vm, or vcpu. + system, vm, vcpu or guest_memfd. Parameters: what parameters are accepted by the ioctl. @@ -6385,7 +6385,9 @@ When mapping a gfn into the guest, guest faults are always serviced from guest_memfd regardless of whether memory is shared or private. KVM determines shared vs. private based on the state in guest_memfd, which is the sole authority on private vs. shared memory. See :ref:`KVM_CREATE_GUEST_MEMFD` to -find out more about the creation-time shared/private status. +find out more about the creation-time shared/private status. Userspace can +control whether memory is shared/private by toggling +KVM_MEMORY_ATTRIBUTE_PRIVATE via :ref:`KVM_SET_MEMORY_ATTRIBUTES2` as needed. userspace_addr is expected to be the mmap()-ed address corresponding to the right offset within the guest_memfd. Any mismatch between userspace_addr and @@ -6404,6 +6406,8 @@ S390: Returns -EINVAL if the VM has the KVM_VM_S390_UCONTROL flag set. Returns -EINVAL if called on a protected VM. +.. _KVM_SET_MEMORY_ATTRIBUTES: + 4.141 KVM_SET_MEMORY_ATTRIBUTES ------------------------------- @@ -6440,6 +6444,8 @@ the state of a gfn/page as needed. The "flags" field is reserved for future extensions and must be '0'. +See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES2`. + .. _KVM_CREATE_GUEST_MEMFD: 4.142 KVM_CREATE_GUEST_MEMFD @@ -6676,6 +6682,67 @@ significant bit): Userspace should use the defined constants from ```` rather than hardcoding bit positions. +.. _KVM_SET_MEMORY_ATTRIBUTES2: + +4.146 KVM_SET_MEMORY_ATTRIBUTES2 +--------------------------------- + +:Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES +:Architectures: all +:Type: guest_memfd ioctl +:Parameters: struct kvm_memory_attributes2 (in) +:Returns: 0 on success, <0 on error + +Errors: + + ========== =============================================================== + EINVAL The specified `offset` or `size` was invalid (e.g. not + page aligned, causes an overflow, or size is zero). + EFAULT The parameter address was invalid. + ENOMEM Ran out of memory trying to track private/shared state + ========== =============================================================== + +KVM_SET_MEMORY_ATTRIBUTES2 is an extension to +KVM_SET_MEMORY_ATTRIBUTES that supports returning (writing) values to +userspace. The original (pre-extension) fields are shared with +KVM_SET_MEMORY_ATTRIBUTES identically. + +Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES. + +:: + + struct kvm_memory_attributes2 { + union { + __u64 address; + __u64 offset; + }; + __u64 size; + __u64 attributes; + __u64 flags; + __u64 reserved[12]; + }; + + #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) + +Set attributes for a range of offsets within a guest_memfd to +KVM_MEMORY_ATTRIBUTE_PRIVATE to limit the specified guest_memfd backed +memory range for guest use. Even if KVM_CAP_GUEST_MEMFD_MMAP is +supported, after a successful call to set +KVM_MEMORY_ATTRIBUTE_PRIVATE, the requested range will not be mappable +into host userspace and will only be mappable by the guest. + +To allow the range to be mappable into host userspace again, call +KVM_SET_MEMORY_ATTRIBUTES2 on the guest_memfd again with +KVM_MEMORY_ATTRIBUTE_PRIVATE unset. + +KVM does not directly manipulate the memory contents of pages during +attribute updates. However, the process of setting these attributes, +which includes operations such as unmapping pages from the host or +stage-2 page tables, may result in side effects on memory contents +that vary across different trusted firmware implementations. + +See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES`. + .. _kvm_run: 5. The kvm_run structure diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index ac2d77d14963..ac371a50041c 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -999,6 +999,7 @@ struct kvm_enable_cap { #define KVM_CAP_S390_HPAGE_2G 249 #define KVM_CAP_PPC_COMPAT_CAPS 250 #define KVM_CAP_ARM_PMU_V3_STRICT 251 +#define KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES 252 struct kvm_irq_routing_irqchip { __u32 irqchip; @@ -1653,6 +1654,20 @@ struct kvm_memory_attributes { __u64 flags; }; +/* Available with KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES */ +#define KVM_SET_MEMORY_ATTRIBUTES2 _IOWR(KVMIO, 0xd2, struct kvm_memory_attributes2) + +struct kvm_memory_attributes2 { + union { + __u64 address; + __u64 offset; + }; + __u64 size; + __u64 attributes; + __u64 flags; + __u64 reserved[12]; +}; + #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) #define KVM_CREATE_GUEST_MEMFD _IOWR(KVMIO, 0xd4, struct kvm_create_guest_memfd) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index a7615a8bf50c..0baf5c342d83 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -545,11 +545,130 @@ bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_is_private_gfn); +/* + * Preallocate memory for attributes to be stored on a maple tree, pointed to + * by mas. Adjacent ranges with attributes identical to the new attributes + * will be merged. Also sets mas's bounds up for storing attributes. + * + * This maintains the invariant that ranges with the same attributes will + * always be merged. + */ +static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, + pgoff_t start, size_t nr_pages) +{ + pgoff_t end = start + nr_pages; + pgoff_t last = end - 1; + void *entry; + + /* Try extending range. entry is NULL on overflow/wrap-around. */ + mas_set(mas, end); + entry = mas_find(mas, end); + if (entry && xa_to_value(entry) == attributes) + last = mas->last; + + if (start > 0) { + mas_set(mas, start - 1); + entry = mas_find(mas, start - 1); + if (entry && xa_to_value(entry) == attributes) + start = mas->index; + } + + mas_set_range(mas, start, last); + return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL); +} + +static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, + size_t nr_pages, uint64_t attrs) +{ + bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE; + struct address_space *mapping = inode->i_mapping; + struct gmem_inode *gi = GMEM_I(inode); + enum kvm_gfn_range_filter filter; + pgoff_t end = start + nr_pages; + struct maple_tree *mt; + struct ma_state mas; + int r; + + mt = &gi->attributes; + + filemap_invalidate_lock(mapping); + + mas_init(&mas, mt, start); + r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages); + if (r) + goto out; + + /* + * From this point on guest_memfd has performed necessary + * checks and can proceed to do guest-breaking changes. + */ + + filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE; + kvm_gmem_invalidate_start(inode, start, end, filter); + mas_store_prealloc(&mas, xa_mk_value(attrs)); + kvm_gmem_invalidate_end(inode, start, end); +out: + filemap_invalidate_unlock(mapping); + return r; +} + +static long kvm_gmem_set_attributes(struct file *file, void __user *argp) +{ + struct gmem_file *f = file->private_data; + struct inode *inode = file_inode(file); + struct kvm_memory_attributes2 attrs; + size_t nr_pages; + pgoff_t index; + int i; + + if (copy_from_user(&attrs, argp, sizeof(attrs))) + return -EFAULT; + + if (attrs.flags) + return -EINVAL; + for (i = 0; i < ARRAY_SIZE(attrs.reserved); i++) { + if (attrs.reserved[i]) + return -EINVAL; + } + if (!kvm_arch_has_private_mem(f->kvm)) + return -EINVAL; + if (attrs.attributes & ~KVM_MEMORY_ATTRIBUTE_PRIVATE) + return -EINVAL; + if (attrs.size == 0 || attrs.offset + attrs.size < attrs.offset) + return -EINVAL; + if (!PAGE_ALIGNED(attrs.offset) || !PAGE_ALIGNED(attrs.size)) + return -EINVAL; + + if (attrs.offset >= i_size_read(inode) || + attrs.offset + attrs.size > i_size_read(inode)) + return -EINVAL; + + nr_pages = attrs.size >> PAGE_SHIFT; + index = attrs.offset >> PAGE_SHIFT; + return __kvm_gmem_set_attributes(inode, index, nr_pages, + attrs.attributes); +} + +static long kvm_gmem_ioctl(struct file *file, unsigned int ioctl, + unsigned long arg) +{ + switch (ioctl) { + case KVM_SET_MEMORY_ATTRIBUTES2: + if (!gmem_in_place_conversion) + return -ENOTTY; + + return kvm_gmem_set_attributes(file, (void __user *)arg); + default: + return -ENOTTY; + } +} + static struct file_operations kvm_gmem_fops = { .mmap = kvm_gmem_mmap, .open = generic_file_open, .release = kvm_gmem_release, .fallocate = kvm_gmem_fallocate, + .unlocked_ioctl = kvm_gmem_ioctl, }; static int kvm_gmem_migrate_folio(struct address_space *mapping, diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 2532d1252a10..82e85a72c323 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2442,18 +2442,22 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, } #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ +#ifdef kvm_arch_has_private_mem +static u64 kvm_supports_private_mem(struct kvm *kvm) +{ + return !kvm || kvm_arch_has_private_mem(kvm); +} +#else +#define kvm_supports_private_mem(kvm) false +#endif + #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm) { -#ifdef kvm_arch_has_private_mem - if (gmem_in_place_conversion) + if (gmem_in_place_conversion || !kvm_supports_private_mem(kvm)) return 0; - if (!kvm || kvm_arch_has_private_mem(kvm)) - return KVM_MEMORY_ATTRIBUTE_PRIVATE; -#endif - - return 0; + return KVM_MEMORY_ATTRIBUTE_PRIVATE; } /* @@ -4995,6 +4999,11 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) return 1; case KVM_CAP_GUEST_MEMFD_FLAGS: return kvm_gmem_get_supported_flags(kvm); + case KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES: + if (!gmem_in_place_conversion || !kvm_supports_private_mem(kvm)) + return 0; + + return KVM_MEMORY_ATTRIBUTE_PRIVATE; #endif default: break; From dcde2f853ff46836aa2b2d6530a47ba164cb49ab Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:41 -0700 Subject: [PATCH 28/60] KVM: guest_memfd: Ensure pages are not in use before conversion When converting memory to private in guest_memfd, it is necessary to ensure that the pages are not currently being accessed by any other part of the kernel or userspace to avoid any current user writing to guest private memory. guest_memfd checks for any outstanding references to determine whether a page is still in use. The only expected references after unmapping the range requested for conversion are those that are held by guest_memfd itself. A folio will have outstanding references if it is present in a per-CPU lru_add fbatch. guest_memfd does not actually participate in LRU, but freshly-allocated folios are still added to the lru_add fbatch for batch LRU statistics processing. A folio may also have extra refcounts if it is on the mlock fbatch. These two known usages of the folio are handled by draining both the lru_add and mlock fbatches for the folio. After draining, if the refcount is still elevated, then there are truly outstanding references. If the page may be DMA-pinned, DMA is using it and hence there are outstanding references. Checking if the page may be DMA-pinned can have false positives, but that is only with a significant number of refcounts, at which point draining LRU is not going to move the needle - it can still be concluded that the folio has outstanding references. If the page is still mapped after guest_memfd tried to unmap it earlier in the conversion process, it also has outstanding references. Exit early to avoid unnecessary draining in these two cases. If an outstanding reference is detected, stop scanning, record the failing offset, and return immediately. Track the drain status to avoid repeatedly draining LRU caches across multiple folios while scanning the requested range. Update the kvm_memory_attributes2 structure to include an error_offset field. This allows KVM to report the exact offset where a conversion failed. If the safety check fails, return -EAGAIN and copy the error_offset back to userspace so that it can potentially retry the operation or handle the failure gracefully. Report error_offset if preallocating memory to track attributes fails with -ENOMEM as well, in which case the start offset of the range is returned. Update documentation to document the error_offset field and the possible -EAGAIN error. Suggested-by: David Hildenbrand Co-developed-by: Vishal Annapurve Signed-off-by: Vishal Annapurve Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Acked-by: Vlastimil Babka (SUSE) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-15-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- Documentation/virt/kvm/api.rst | 10 ++++ include/uapi/linux/kvm.h | 3 +- virt/kvm/guest_memfd.c | 90 ++++++++++++++++++++++++++++++++-- 3 files changed, 97 insertions(+), 6 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 027156508d04..67f0f290797a 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6741,6 +6741,16 @@ which includes operations such as unmapping pages from the host or stage-2 page tables, may result in side effects on memory contents that vary across different trusted firmware implementations. +If this ioctl returns -EAGAIN, the offset of the page with unexpected +refcounts will be returned in ``error_offset``. This can occur if +there are transient refcounts on the pages, taken by other parts of +the kernel. + +Userspace is expected to figure out how to remove all known refcounts +on the shared pages, such as refcounts taken by get_user_pages(), and +try the ioctl again. A possible source of these long term refcounts is +if the guest_memfd memory was pinned in IOMMU page tables. + See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES`. .. _kvm_run: diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index ac371a50041c..8dff2fc1972e 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1665,7 +1665,8 @@ struct kvm_memory_attributes2 { __u64 size; __u64 attributes; __u64 flags; - __u64 reserved[12]; + __u64 error_offset; + __u64 reserved[11]; }; #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 0baf5c342d83..7a244cb2be54 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -8,6 +8,7 @@ #include #include #include +#include #include "kvm_mm.h" #include "guest_memfd.h" @@ -577,8 +578,58 @@ static int kvm_gmem_mas_preallocate(struct ma_state *mas, u64 attributes, return mas_preallocate(mas, xa_mk_value(attributes), GFP_KERNEL); } +static bool __folio_has_outstanding_references(struct folio *folio, + enum lru_cache_drained *drained) +{ + if (folio_maybe_dma_pinned(folio) || folio_mapped(folio)) + return true; + + /* 1 reference held by filemap_get_folios() in the folio batch. */ + lru_cache_drain_for_folio(folio, 1, drained); + + /* + * Outstanding references are anything other than those from the page + * cache, plus 1 temporary reference held by filemap_get_folios() in the + * folio batch. + */ + return folio_ref_count(folio) != folio_nr_pages(folio) + 1; +} + +static bool kvm_gmem_has_outstanding_references(struct inode *inode, + pgoff_t start, size_t nr_pages, + pgoff_t *err_index) +{ + enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED; + struct address_space *mapping = inode->i_mapping; + pgoff_t last = start + nr_pages - 1; + struct folio_batch fbatch; + pgoff_t next; + int i; + + folio_batch_init(&fbatch); + + next = start; + while (filemap_get_folios(mapping, &next, last, &fbatch)) { + for (i = 0; i < folio_batch_count(&fbatch); ++i) { + struct folio *folio = fbatch.folios[i]; + + if (__folio_has_outstanding_references(folio, &drained)) { + *err_index = max(start, folio->index); + folio_batch_release(&fbatch); + return true; + } + } + + folio_batch_release(&fbatch); + cond_resched(); + } + + return false; +} + static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, - size_t nr_pages, uint64_t attrs) + size_t nr_pages, uint64_t attrs, + pgoff_t *err_index) { bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE; struct address_space *mapping = inode->i_mapping; @@ -595,8 +646,28 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, mas_init(&mas, mt, start); r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages); - if (r) + if (r) { + *err_index = start; goto out; + } + + if (to_private) { + /* + * Forcefully unmap the pages from all userspace page tables, + * and then verify there are no outstanding references, e.g. + * acquired via GUP or similar. Tell userspace to try again if + * there are outstanding references and hope that whatever has + * pinned the page will put its reference "soon". + */ + unmap_mapping_pages(mapping, start, nr_pages, false); + + if (kvm_gmem_has_outstanding_references(inode, start, nr_pages, + err_index)) { + mas_destroy(&mas); + r = -EAGAIN; + goto out; + } + } /* * From this point on guest_memfd has performed necessary @@ -617,9 +688,10 @@ static long kvm_gmem_set_attributes(struct file *file, void __user *argp) struct gmem_file *f = file->private_data; struct inode *inode = file_inode(file); struct kvm_memory_attributes2 attrs; + pgoff_t err_index; size_t nr_pages; pgoff_t index; - int i; + int i, r; if (copy_from_user(&attrs, argp, sizeof(attrs))) return -EFAULT; @@ -645,8 +717,16 @@ static long kvm_gmem_set_attributes(struct file *file, void __user *argp) nr_pages = attrs.size >> PAGE_SHIFT; index = attrs.offset >> PAGE_SHIFT; - return __kvm_gmem_set_attributes(inode, index, nr_pages, - attrs.attributes); + r = __kvm_gmem_set_attributes(inode, index, nr_pages, attrs.attributes, + &err_index); + if (r) { + attrs.error_offset = ((uint64_t)err_index) << PAGE_SHIFT; + + if (copy_to_user(argp, &attrs, sizeof(attrs))) + return -EFAULT; + } + + return r; } static long kvm_gmem_ioctl(struct file *file, unsigned int ioctl, From 6fa1dab756fd49d4b3162a68775de3aeec78646e Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:42 -0700 Subject: [PATCH 29/60] KVM: guest_memfd: Call arch make_shared callback for to-shared conversion When doing in-place conversion from PRIVATE to SHARED, immediately inform arch code of the conversion for all allocated pages/folios, e.g. so that arch code can put hardware metadata tables in the correct state. Eagerly updating the table for to SHARED conversions avoids having to implement on-demand updates, e.g. when faulting in host userspace mappings. Skip the entire flow if the arch doesn't implement conversion callbacks, as getting folios from the filemap is noticeably expensive, especially when converting large chunks of memory. Deliberately don't eagerly update the metadata table on conversions from SHARED to PRIVATE, because assigning a page to a VM (versus "returning" it to the host) requires the exact GFN associated with the page, i.e would require walking the memslot bindings. And because KVM *must* do on-demand metadata updates when getting a PFN for KVM-internal usage, as that's the only time a relevant memslot binding is guaranteed to exist. Note! Inform arch code of the conversion within the protection of the invalidation sequence, to ensure that any existing mappings are dropped before hardware is updated, and to ensure that new mappings can't be established until after the conversion is complete. Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-16-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/x86.c | 5 ++++ include/linux/kvm_host.h | 1 + virt/kvm/guest_memfd.c | 42 ++++++++++++++++++++++++++++++ 5 files changed, 50 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index e213c9ae3e30..67b43c167045 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -150,7 +150,7 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT KVM_X86_OP_OPTIONAL_RET0(gmem_make_private) #endif -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM) KVM_X86_OP_OPTIONAL(gmem_make_shared) #endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 744c1f6ff03e..83e26ce45fb7 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1732,7 +1732,7 @@ struct kvm_x86_ops { int (*gmem_make_private)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages); #endif -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM) void (*gmem_make_shared)(kvm_pfn_t pfn, kvm_pfn_t nr_pages); #endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 229224957031..75e03a2f79db 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10653,6 +10653,11 @@ int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, { return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages); } + +void kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages) +{ + kvm_x86_call(gmem_make_shared)(pfn, nr_pages); +} #endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index ab87effdd221..485f18454eb4 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2610,6 +2610,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm, int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages); +void kvm_arch_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages); #ifndef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT #define kvm_arch_has_gmem_convert() false #endif diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 7a244cb2be54..5b6b58ea2acb 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -627,6 +627,43 @@ static bool kvm_gmem_has_outstanding_references(struct inode *inode, return false; } +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) +{ + struct folio_batch fbatch; + pgoff_t next = start; + int i; + + folio_batch_init(&fbatch); + while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) { + for (i = 0; i < folio_batch_count(&fbatch); ++i) { + struct folio *folio = fbatch.folios[i]; + pgoff_t start_index, end_index; + kvm_pfn_t start_pfn; + kvm_pfn_t nr_pages; + + start_index = max(start, folio->index); + end_index = min(end, folio_next_index(folio)); + /* + * end_index is either in folio or points to + * the first page of the next folio. Hence, + * all pages in range [start_index, end_index) + * are contiguous. + */ + start_pfn = folio_file_pfn(folio, start_index); + nr_pages = end_index - start_index; + + kvm_arch_gmem_make_shared(start_pfn, nr_pages); + } + + folio_batch_release(&fbatch); + cond_resched(); + } +} +#else +static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {} +#endif + static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, size_t nr_pages, uint64_t attrs, pgoff_t *err_index) @@ -676,7 +713,12 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE; kvm_gmem_invalidate_start(inode, start, end, filter); + + if (!to_private && kvm_arch_has_gmem_convert()) + kvm_gmem_make_shared(inode, start, end); + mas_store_prealloc(&mas, xa_mk_value(attrs)); + kvm_gmem_invalidate_end(inode, start, end); out: filemap_invalidate_unlock(mapping); From 856852ef2108fd4e70f29d450487fe15324c647c Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:43 -0700 Subject: [PATCH 30/60] KVM: guest_memfd: Return early if range already has requested attributes Provide a function to check that a range has given attributes. Optimize setting memory attributes by returning early if all pages in the requested range already have the requested attributes. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Binbin Wu Reviewed-by: Xiaoyao Li Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-17-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 5b6b58ea2acb..8eedff42a96f 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -99,6 +99,24 @@ static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index) return !kvm_gmem_is_private_mem(inode, index); } +static bool kvm_gmem_range_has_attributes(struct inode *inode, + pgoff_t index, size_t nr_pages, + u64 attributes) +{ + struct maple_tree *mt = &GMEM_I(inode)->attributes; + pgoff_t end = index + nr_pages - 1; + void *entry; + + lockdep_assert(mt_lock_is_held(mt)); + + mt_for_each(mt, entry, index, end) { + if (kvm_gmem_get_attributes(inode, entry) != attributes) + return false; + } + + return true; +} + /* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest. @@ -675,12 +693,15 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start, pgoff_t end = start + nr_pages; struct maple_tree *mt; struct ma_state mas; - int r; + int r = 0; mt = &gi->attributes; filemap_invalidate_lock(mapping); + if (kvm_gmem_range_has_attributes(inode, start, nr_pages, attrs)) + goto out; + mas_init(&mas, mt, start); r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages); if (r) { From cb0544296a33f8dae0d60b24cfb4de839b332b98 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:44 -0700 Subject: [PATCH 31/60] KVM: guest_memfd: Zero page while getting pfn Move the folio initialization logic from kvm_gmem_get_pfn() into __kvm_gmem_get_pfn() to also zero pages if the page is to be used in kvm_gmem_populate(). With in-place conversion, the existing data in a guest_memfd page can be populated into guest memory through platform-specific ioctls. Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it might contain uninitialized host memory, which would leak to the guest if the populate completes. guest_memfd pages are zeroed at most once in the page's entire lifetime with guest_memfd, and that is tracked using the uptodate flag. Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in kvm_gmem_get_folio() since other flows, such as a future write() syscall, can get a page, write to the page and then set page uptodate without zeroing. There may be some performance penalty due to redundant zeroing, but this would pale in comparison to the cost of actually assigning the page to the VM. This aligns with the concept of zeroing before first use - the other place where zeroing happens is in kvm_gmem_fault_user_mapping(). On populate failure, the page is not re-zeroed, since on SNP, if firmware rejects a CPUID page, the expected CPUID values provided by firmware are returned to userspace via page contents. More generally, page contents may be modified on populate failure. Don't mark the page uptodate again after populating, since the page would already be marked uptodate before the post_populate() call. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-18-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 8eedff42a96f..a13445c26d9d 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -1154,6 +1154,11 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file, return ERR_PTR(-EHWPOISON); } + if (!folio_test_uptodate(folio)) { + clear_highpage(folio_page(folio, 0)); + folio_mark_uptodate(folio); + } + *pfn = folio_file_pfn(folio, index); if (max_order) *max_order = 0; @@ -1182,11 +1187,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, goto out; } - if (!folio_test_uptodate(folio)) { - clear_highpage(folio_page(folio, 0)); - folio_mark_uptodate(folio); - } - if (kvm_arch_has_gmem_convert() && kvm_gmem_is_private_mem(file_inode(file), index)) r = kvm_arch_gmem_make_private(kvm, gfn, *pfn, @@ -1228,8 +1228,6 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot, } ret = post_populate(kvm, gfn, pfn, src_page, opaque); - if (!ret) - folio_mark_uptodate(folio); out_put_folio: folio_put(folio); From edcc3f42981e9018d73c65ad8cfaf9e2ec70c07f Mon Sep 17 00:00:00 2001 From: Michael Roth Date: Thu, 10 Sep 2026 16:55:45 -0700 Subject: [PATCH 32/60] KVM: SEV: Make 'uaddr' parameter optional for KVM_SEV_SNP_LAUNCH_UPDATE Make the source page for populating an SNP guest_memfd instance optional if in-place conversion/population is enabled. If KVM can convert the page in-place, then it's possible for guest memory to be initialized directly from userspace by mmap()'ing the guest_memfd and writing to it while the corresponding GPA ranges are in a 'shared' state, before converting them to the 'private' state expected by KVM_SEV_SNP_LAUNCH_UPDATE. Update the handling/documentation for KVM_SEV_SNP_LAUNCH_UPDATE to allow for 'uaddr' to be set to NULL when in-place conversion is enabled, which SNP_LAUNCH_UPDATE will then use to determine when it should/shouldn't copy in data from a separate memory location. Continue to enforce non-NULL when PRIVATE is tracked per-VM, not per-guest_memfd. [ Sean: Moved condition to snp_launch_update ] Signed-off-by: Michael Roth Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-19-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../virt/kvm/x86/amd-memory-encryption.rst | 14 ++++++++++---- arch/x86/kvm/svm/sev.c | 11 ++++++----- virt/kvm/kvm_main.c | 1 + 3 files changed, 17 insertions(+), 9 deletions(-) diff --git a/Documentation/virt/kvm/x86/amd-memory-encryption.rst b/Documentation/virt/kvm/x86/amd-memory-encryption.rst index bd04a908a8db..5977fbe33b98 100644 --- a/Documentation/virt/kvm/x86/amd-memory-encryption.rst +++ b/Documentation/virt/kvm/x86/amd-memory-encryption.rst @@ -503,7 +503,8 @@ secrets. It is required that the GPA ranges initialized by this command have had the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute set in advance. See the documentation -for KVM_SET_MEMORY_ATTRIBUTES for more details on this aspect. +for KVM_SET_MEMORY_ATTRIBUTES/KVM_SET_MEMORY_ATTRIBUTES2 for more details on +this aspect. Upon success, this command is not guaranteed to have processed the entire range requested. Instead, the ``gfn_start``, ``uaddr``, and ``len`` fields of @@ -511,9 +512,14 @@ range requested. Instead, the ``gfn_start``, ``uaddr``, and ``len`` fields of remaining range that has yet to be processed. The caller should continue calling this command until those fields indicate the entire range has been processed, e.g. ``len`` is 0, ``gfn_start`` is equal to the last GFN in the -range plus 1, and ``uaddr`` is the last byte of the userspace-provided source -buffer address plus 1. In the case where ``type`` is KVM_SEV_SNP_PAGE_TYPE_ZERO, -``uaddr`` will be ignored completely. +range plus 1, and ``uaddr`` (if specified) is the last byte of the +userspace-provided source buffer address plus 1. + +In the case where ``type`` is KVM_SEV_SNP_PAGE_TYPE_ZERO, ``uaddr`` will be +ignored completely. For all other page types, ``uaddr`` is optional if in-place +conversion is enabled (i.e. when the data had been written directly to +guest_memfd while the page was in the shared state) and is required if in-place +conversion is disabled. Parameters (in): struct kvm_sev_snp_launch_update diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index dff684b88f15..63e8cfa9bf55 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -2400,7 +2400,7 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, */ if (ret && !snp_page_reclaim(kvm, pfn) && sev_populate_args->type == KVM_SEV_SNP_PAGE_TYPE_CPUID && - sev_populate_args->fw_error == SEV_RET_INVALID_PARAM) { + sev_populate_args->fw_error == SEV_RET_INVALID_PARAM && src_page) { void *src_vaddr = kmap_local_page(src_page); void *dst_vaddr = kmap_local_pfn(pfn); @@ -2433,8 +2433,8 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp) if (copy_from_user(¶ms, u64_to_user_ptr(argp->data), sizeof(params))) return -EFAULT; - pr_debug("%s: GFN start 0x%llx length 0x%llx type %d flags %d\n", __func__, - params.gfn_start, params.len, params.type, params.flags); + pr_debug("%s: GFN start 0x%llx length 0x%llx type %d flags %d src %llx\n", __func__, + params.gfn_start, params.len, params.type, params.flags, params.uaddr); if (!params.len || !PAGE_ALIGNED(params.len) || params.flags || (params.type != KVM_SEV_SNP_PAGE_TYPE_NORMAL && @@ -2446,7 +2446,8 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp) if (params.type == KVM_SEV_SNP_PAGE_TYPE_ZERO) src = NULL; - else if (!params.uaddr || !PAGE_ALIGNED(params.uaddr)) + else if ((!gmem_in_place_conversion && !params.uaddr) || + !PAGE_ALIGNED(params.uaddr)) return -EINVAL; else src = u64_to_user_ptr(params.uaddr); @@ -2493,7 +2494,7 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp) params.gfn_start += count; params.len -= count * PAGE_SIZE; - if (params.type != KVM_SEV_SNP_PAGE_TYPE_ZERO) + if (src && params.type != KVM_SEV_SNP_PAGE_TYPE_ZERO) params.uaddr += count * PAGE_SIZE; if (copy_to_user(u64_to_user_ptr(argp->data), ¶ms, sizeof(params))) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 82e85a72c323..94b50fbe6614 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -104,6 +104,7 @@ module_param(allow_unsafe_mappings, bool, 0444); #ifdef kvm_arch_has_private_mem bool __ro_after_init gmem_in_place_conversion = false; +EXPORT_SYMBOL_FOR_KVM_INTERNAL(gmem_in_place_conversion); #endif /* From af602a96f401124a788836239c5fd775a687d8e5 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:46 -0700 Subject: [PATCH 33/60] KVM: TDX: Make source page optional for KVM_TDX_INIT_MEM_REGION Update tdx_gmem_post_populate() to handle cases where userspace requests "no source page". To handle "no source page", populate (perform TDH.MEM.PAGE.ADD) using memory in-place at the target PFN. Allow "no source page" only when gmem_in_place_conversion is enabled, because retroactively adding support for out-of-place conversion would mean requiring a userspace update for a feature that's being deprecated. Also, KVM supporting "no source page" without in-place conversion would effectively be an obscure zero-page optimization that relies on the page being zeroed when it is allocated by guest_memfd. Rejecting "no source page" without in-place conversion scenario is valuable for KVM developers since it helps newcomers understand what exactly is and isn't possible. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Tested-by: Yan Zhao Reviewed-by: Binbin Wu Reviewed-by: Yan Zhao Reviewed-by: Xiaoyao Li Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-20-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- Documentation/virt/kvm/x86/intel-tdx.rst | 4 ++++ arch/x86/kvm/vmx/tdx.c | 8 +++++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Documentation/virt/kvm/x86/intel-tdx.rst b/Documentation/virt/kvm/x86/intel-tdx.rst index 6a222e9d0954..d8d9409120e6 100644 --- a/Documentation/virt/kvm/x86/intel-tdx.rst +++ b/Documentation/virt/kvm/x86/intel-tdx.rst @@ -158,6 +158,10 @@ KVM_TDX_INIT_MEM_REGION Initialize @nr_pages TDX guest private memory starting from @gpa with userspace provided data from @source_addr. @source_addr must be PAGE_SIZE-aligned. +If guest_memfd in-place conversion is enabled, pass 0 for @source_addr +to represent "no source page". A source page is required if in-place +conversion is not enabled or not supported. + Note, before calling this sub command, memory attribute of the range [gpa, gpa + nr_pages] needs to be private. Userspace can use KVM_SET_MEMORY_ATTRIBUTES to set the attribute. diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index b272c20586a7..7654a9f5fa12 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -3189,7 +3189,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm)) return -EIO; - kvm_tdx->page_add_src = src_page; + kvm_tdx->page_add_src = src_page ?: pfn_to_page(pfn); ret = kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn); kvm_tdx->page_add_src = NULL; @@ -3235,7 +3235,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c if (copy_from_user(®ion, u64_to_user_ptr(cmd->data), sizeof(region))) return -EFAULT; - if (!PAGE_ALIGNED(region.source_addr) || !region.source_addr || + if (!PAGE_ALIGNED(region.source_addr) || + (!gmem_in_place_conversion && !region.source_addr) || !PAGE_ALIGNED(region.gpa) || !region.nr_pages || region.gpa + (region.nr_pages << PAGE_SHIFT) <= region.gpa || !vt_is_tdx_private_gpa(kvm, region.gpa) || @@ -3266,7 +3267,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c break; } - region.source_addr += PAGE_SIZE; + if (region.source_addr) + region.source_addr += PAGE_SIZE; region.gpa += PAGE_SIZE; region.nr_pages--; From 31670c382efc5c7b3e40bb74ebc1b17bc8780fcc Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:47 -0700 Subject: [PATCH 34/60] KVM: Move KVM_VM_MEMORY_ATTRIBUTES config definition to x86 Bury KVM_VM_MEMORY_ATTRIBUTES in x86 to discourage other architectures from adding support for per-VM memory attributes, because tracking private vs. shared memory on a per-VM basis is now deprecated in favor of tracking on a per-guest_memfd basis, and while RWX memory attributes are on the horizon, they too are expected to be x86-only. This will also allow modifying KVM_VM_MEMORY_ATTRIBUTES to be user-selectable (in x86) without creating weirdness in KVM's Kconfigs. Now that guest_memfd supports in-place conversions, it's entirely possible to run x86 CoCo VMs without support for KVM_VM_MEMORY_ATTRIBUTES. Leave the code itself in common KVM so that it's trivial to undo this change if new per-VM attributes do come along. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Reviewed-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-21-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/Kconfig | 3 +++ virt/kvm/Kconfig | 3 --- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index 2f8e1b5a8a72..abb108886733 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -80,6 +80,9 @@ config KVM_WERROR If in doubt, say "N". +config KVM_VM_MEMORY_ATTRIBUTES + bool + config KVM_SW_PROTECTED_VM bool "Enable support for KVM software-protected VMs" depends on EXPERT diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index df575449d97b..a0678ef8ee3f 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,9 +100,6 @@ config KVM_ELIDE_TLB_FLUSH_IF_YOUNG config KVM_MMU_LOCKLESS_AGING bool -config KVM_VM_MEMORY_ATTRIBUTES - bool - config KVM_GUEST_MEMFD select XARRAY_MULTI bool From 537ec2b15fdd7396c6c50860c17ebbf21f6f47d1 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:48 -0700 Subject: [PATCH 35/60] KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes Allow the user to disable KVM_VM_MEMORY_ATTRIBUTES even when KVM supports PRIVATE and SHARED attributes, and expose gmem_in_place_conversion as a module parameter when per-VM attributes are supported. I.e. let userspace enable in-place PRIVATE<=>SHARED conversion of guest_memfd pages. Provide both a Kconfig option and a (conditional) module param so that deployments that use a custom kernel can fully disable per-VM tracking, while not forcing distros to ship two separate kernels in order to provide backwards compatibility for downstream users. Don't allow running VMs with mixed tracking for a given instance of KVM, i.e. disallow toggling the module param after KVM is loaded, as the extra complexity needed to handle per-VM behavior far outweighs any potential benefit. E.g. neither TDX nor SNP supports live migration, so in effect the requirement is that existing deployments that want to support both the old and the new models would need to tell their VMM which flavor of tracking to use. [ Xiaoyao: Define module_param only if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled ] Suggested-by: Xiaoyao Li Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Xiaoyao Li Acked-by: David Hildenbrand (Arm) Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-22-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../admin-guide/kernel-parameters.txt | 25 +++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 4 ++- arch/x86/kvm/Kconfig | 14 ++++++++--- virt/kvm/kvm_main.c | 5 +++- 4 files changed, 42 insertions(+), 6 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 33cd30996e47..49560b1b54c4 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3156,6 +3156,31 @@ Kernel parameters kvm.enable_vmware_backdoor=[KVM] Support VMware backdoor PV interface. Default is false (don't support). + kvm.gmem_in_place_conversion= + [KVM] Controls whether KVM enables in-place conversion + support for guest_memfd and tracks the private/shared + state of memory per guest_memfd instead of per VM. + + If enabled, KVM enables the KVM_SET_MEMORY_ATTRIBUTES2 + ioctl on guest_memfd file descriptors and disables the + legacy VM-scoped KVM_SET_MEMORY_ATTRIBUTES ioctl for + private memory state tracking. Only the + KVM_MEMORY_ATTRIBUTE_PRIVATE attribute moves to + per-guest_memfd tracking; other attributes remain + per-VM. + + This parameter toggles KVM's in-place conversion + capability support. Whether a VMM uses separate backends + or out-of-place memory management is determined by + userspace VMM design. + + Note, this parameter is only available when + CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When + CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place + conversion is unconditionally enabled. + + Default is N (off). + kvm.nx_huge_pages= [KVM] Controls the software workaround for the X86_BUG_ITLB_MULTIHIT bug. diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 83e26ce45fb7..e840418427a1 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1851,7 +1851,9 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI))) -#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +#if defined(CONFIG_KVM_SW_PROTECTED_VM) || \ + defined(CONFIG_KVM_INTEL_TDX) || \ + defined(CONFIG_KVM_AMD_SEV) #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem) #endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index abb108886733..2c3c22aeafa5 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -81,13 +81,21 @@ config KVM_WERROR If in doubt, say "N". config KVM_VM_MEMORY_ATTRIBUTES - bool + bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)" + depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV + help + Enable support for tracking PRIVATE vs. SHARED memory using per-VM + memory attributes. Using per-VM attributes is deprecated in favor of + tracking PRIVATE state in guest_memfd. Select this if you need to run + CoCo VMs using a VMM that doesn't support guest_memfd memory + attributes. + + If unsure, say N. config KVM_SW_PROTECTED_VM bool "Enable support for KVM software-protected VMs" depends on EXPERT depends on KVM_X86 && X86_64 - select KVM_VM_MEMORY_ATTRIBUTES help Enable support for KVM software-protected VMs. Currently, software- protected VMs are purely a development and testing vehicle for @@ -138,7 +146,6 @@ config KVM_INTEL_TDX bool "Intel Trust Domain Extensions (TDX) support" default y depends on INTEL_TDX_HOST - select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_POPULATE help Provides support for launching Intel Trust Domain Extensions (TDX) @@ -162,7 +169,6 @@ config KVM_AMD_SEV depends on KVM_AMD && X86_64 depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM - select KVM_VM_MEMORY_ATTRIBUTES select HAVE_KVM_ARCH_GMEM_CONVERT select HAVE_KVM_ARCH_GMEM_RECLAIM select HAVE_KVM_ARCH_GMEM_INVALIDATE diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 94b50fbe6614..3ed481c4a15b 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -103,7 +103,10 @@ static bool __ro_after_init allow_unsafe_mappings; module_param(allow_unsafe_mappings, bool, 0444); #ifdef kvm_arch_has_private_mem -bool __ro_after_init gmem_in_place_conversion = false; +bool __ro_after_init gmem_in_place_conversion = !IS_ENABLED(CONFIG_KVM_VM_MEMORY_ATTRIBUTES); +#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES +module_param(gmem_in_place_conversion, bool, 0444); +#endif EXPORT_SYMBOL_FOR_KVM_INTERNAL(gmem_in_place_conversion); #endif From b5ed1ee2a7a0a9b2e70308e6a7306b5ce04d5ab7 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:49 -0700 Subject: [PATCH 36/60] KVM: guest_memfd: Enable INIT_SHARED on guest_memfd for x86 Coco VMs Now that guest_memfd supports tracking private vs. shared within gmem itself, allow userspace to specify INIT_SHARED on a guest_memfd instance for x86 Confidential Computing (CoCo) VMs, so long as in-place conversion is enabled, i.e. when it's actually possible for a guest_memfd instance to contain shared memory. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Reviewed-by: Xiaoyao Li Reviewed-by: Binbin Wu Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-23-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 75e03a2f79db..c081e4b9b25d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10637,14 +10637,15 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu) } #ifdef CONFIG_KVM_GUEST_MEMFD -/* - * KVM doesn't yet support initializing guest_memfd memory as shared for VMs - * with private memory (the private vs. shared tracking needs to be moved into - * guest_memfd). - */ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm) { - return !kvm_arch_has_private_mem(kvm); + /* + * INIT_SHARED is supported if in-place conversion is enabled, or if + * the VM doesn't support private memory. If the VM has private memory + * and in-place conversion is disabled, then guest_memfd can _only_ be + * used for private memory. + */ + return gmem_in_place_conversion || !kvm_arch_has_private_mem(kvm); } #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT From b312cab4e48c654d58a09abc4100a488e344d4fb Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:50 -0700 Subject: [PATCH 37/60] KVM: selftests: Create gmem fd before "regular" fd when adding memslot When adding a memslot associated with a guest_memfd instance, create/dup the guest_memfd before creating the "normal" backing file. This will allow dup'ing the gmem fd as the normal fd when guest_memfd supports mmap(), i.e. to make guest_memfd the _only_ backing source for the memslot. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-24-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/lib/kvm_util.c | 45 +++++++++++----------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 9ddc047d5c27..3413d45e34fa 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1077,6 +1077,29 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, if (alignment > 1) region->mmap_size += alignment; + if (flags & KVM_MEM_GUEST_MEMFD) { + if (guest_memfd < 0) { + u32 guest_memfd_flags = 0; + + TEST_ASSERT(!guest_memfd_offset, + "Offset must be zero when creating new guest_memfd"); + guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags); + } else { + /* + * Install a unique fd for each memslot so that the fd + * can be closed when the region is deleted without + * needing to track if the fd is owned by the framework + * or by the caller. + */ + guest_memfd = kvm_dup(guest_memfd); + } + + region->region.guest_memfd = guest_memfd; + region->region.guest_memfd_offset = guest_memfd_offset; + } else { + region->region.guest_memfd = -1; + } + region->fd = -1; if (backing_src_is_shared(src_type)) region->fd = kvm_memfd_alloc(region->mmap_size, @@ -1106,28 +1129,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, region->backing_src_type = src_type; - if (flags & KVM_MEM_GUEST_MEMFD) { - if (guest_memfd < 0) { - u32 guest_memfd_flags = 0; - TEST_ASSERT(!guest_memfd_offset, - "Offset must be zero when creating new guest_memfd"); - guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags); - } else { - /* - * Install a unique fd for each memslot so that the fd - * can be closed when the region is deleted without - * needing to track if the fd is owned by the framework - * or by the caller. - */ - guest_memfd = kvm_dup(guest_memfd); - } - - region->region.guest_memfd = guest_memfd; - region->region.guest_memfd_offset = guest_memfd_offset; - } else { - region->region.guest_memfd = -1; - } - region->unused_phy_pages = sparsebit_alloc(); if (vm_arch_has_protected_memory(vm)) region->protected_phy_pages = sparsebit_alloc(); From 8d5f49000f8bb92a88f712c967ed8582d3f08904 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:51 -0700 Subject: [PATCH 38/60] KVM: selftests: Rename guest_memfd{,_offset} to gmem_{fd,offset} Rename local variables and function parameters for the guest memory file descriptor and its offset to use a "gmem_" prefix instead of "guest_memfd_". No functional change intended. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-25-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../testing/selftests/kvm/include/kvm_util.h | 6 ++--- tools/testing/selftests/kvm/lib/kvm_util.c | 26 +++++++++---------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 5a7a455b5387..cf49a84de02e 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -690,17 +690,17 @@ int __vm_set_user_memory_region(struct kvm_vm *vm, u32 slot, u32 flags, gpa_t gpa, u64 size, void *hva); void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, gpa_t gpa, u64 size, void *hva, - u32 guest_memfd, u64 guest_memfd_offset); + u32 gmem_fd, u64 gmem_offset); int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, gpa_t gpa, u64 size, void *hva, - u32 guest_memfd, u64 guest_memfd_offset); + u32 gmem_fd, u64 gmem_offset); void vm_userspace_mem_region_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, gpa_t gpa, u32 slot, u64 npages, u32 flags); void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, gpa_t gpa, u32 slot, u64 npages, u32 flags, - int guest_memfd_fd, u64 guest_memfd_offset); + int gmem_fd, u64 gmem_offset); #ifndef vm_arch_has_protected_memory static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 3413d45e34fa..af501a27211a 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -970,7 +970,7 @@ void vm_set_user_memory_region(struct kvm_vm *vm, u32 slot, u32 flags, int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, gpa_t gpa, u64 size, void *hva, - u32 guest_memfd, u64 guest_memfd_offset) + u32 gmem_fd, u64 gmem_offset) { struct kvm_userspace_memory_region2 region = { .slot = slot, @@ -978,8 +978,8 @@ int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, .guest_phys_addr = gpa, .memory_size = size, .userspace_addr = (uintptr_t)hva, - .guest_memfd = guest_memfd, - .guest_memfd_offset = guest_memfd_offset, + .guest_memfd = gmem_fd, + .guest_memfd_offset = gmem_offset, }; TEST_REQUIRE_SET_USER_MEMORY_REGION2(); @@ -989,10 +989,10 @@ int __vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, gpa_t gpa, u64 size, void *hva, - u32 guest_memfd, u64 guest_memfd_offset) + u32 gmem_fd, u64 gmem_offset) { int ret = __vm_set_user_memory_region2(vm, slot, flags, gpa, size, hva, - guest_memfd, guest_memfd_offset); + gmem_fd, gmem_offset); TEST_ASSERT(!ret, "KVM_SET_USER_MEMORY_REGION2 failed, errno = %d (%s)", errno, strerror(errno)); @@ -1002,7 +1002,7 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, /* FIXME: This thing needs to be ripped apart and rewritten. */ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, gpa_t gpa, u32 slot, u64 npages, u32 flags, - int guest_memfd, u64 guest_memfd_offset) + int gmem_fd, u64 gmem_offset) { int ret; struct userspace_mem_region *region; @@ -1078,12 +1078,12 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, region->mmap_size += alignment; if (flags & KVM_MEM_GUEST_MEMFD) { - if (guest_memfd < 0) { - u32 guest_memfd_flags = 0; + if (gmem_fd < 0) { + u32 gmem_flags = 0; - TEST_ASSERT(!guest_memfd_offset, + TEST_ASSERT(!gmem_offset, "Offset must be zero when creating new guest_memfd"); - guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags); + gmem_fd = vm_create_guest_memfd(vm, mem_size, gmem_flags); } else { /* * Install a unique fd for each memslot so that the fd @@ -1091,11 +1091,11 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, * needing to track if the fd is owned by the framework * or by the caller. */ - guest_memfd = kvm_dup(guest_memfd); + gmem_fd = kvm_dup(gmem_fd); } - region->region.guest_memfd = guest_memfd; - region->region.guest_memfd_offset = guest_memfd_offset; + region->region.guest_memfd = gmem_fd; + region->region.guest_memfd_offset = gmem_offset; } else { region->region.guest_memfd = -1; } From ee455183c92be7661ab08eedd68cd2bed0874975 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:52 -0700 Subject: [PATCH 39/60] KVM: selftests: Add support for mmap() on guest_memfd in core library Accept gmem_flags in vm_mem_add() to be able to create a guest_memfd within vm_mem_add(). When vm_mem_add() is used to set up a guest_memfd for a memslot, set up the provided (or created) gmem_fd as the fd for the user memory region. This makes it available to be mmap()-ed from just like fds from other memory sources. For guest_memfds, mmap() using gmem_offset instead of 0 all the time. Always use MAP_SHARED if mmap-ing from guest_memfd instead of reading flag from the configured src_type, which doesn't include guest_memfd. Add a kvm_slot_to_fd() helper to provide convenient access to the file descriptor of a memslot. Update existing callers of vm_mem_add() to pass 0 for gmem_flags to preserve existing behavior. Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-26-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../testing/selftests/kvm/include/kvm_util.h | 7 ++++- tools/testing/selftests/kvm/lib/kvm_util.c | 28 +++++++++++-------- .../kvm/x86/private_mem_conversions_test.c | 2 +- 3 files changed, 23 insertions(+), 14 deletions(-) diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index cf49a84de02e..a091c495442a 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -700,7 +700,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm, gpa_t gpa, u32 slot, u64 npages, u32 flags); void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, gpa_t gpa, u32 slot, u64 npages, u32 flags, - int gmem_fd, u64 gmem_offset); + int gmem_fd, u64 gmem_offset, u64 gmem_flags); #ifndef vm_arch_has_protected_memory static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm) @@ -732,6 +732,11 @@ void *addr_gva2hva(struct kvm_vm *vm, gva_t gva); gpa_t addr_hva2gpa(struct kvm_vm *vm, void *hva); void *addr_gpa2alias(struct kvm_vm *vm, gpa_t gpa); +static inline int kvm_slot_to_fd(struct kvm_vm *vm, u32 slot) +{ + return memslot2region(vm, slot)->fd; +} + #ifndef vcpu_arch_put_guest #define vcpu_arch_put_guest(mem, val) do { (mem) = (val); } while (0) #endif diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index af501a27211a..08d0bbcd421e 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1002,12 +1002,14 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, u32 slot, u32 flags, /* FIXME: This thing needs to be ripped apart and rewritten. */ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, gpa_t gpa, u32 slot, u64 npages, u32 flags, - int gmem_fd, u64 gmem_offset) + int gmem_fd, u64 gmem_offset, u64 gmem_flags) { int ret; struct userspace_mem_region *region; size_t backing_src_pagesz = get_backing_src_pagesz(src_type); + int mmap_flags = vm_mem_backing_src_alias(src_type)->flag; size_t mem_size = npages * vm->page_size; + off_t mmap_offset = 0; size_t alignment = 1; TEST_REQUIRE_SET_USER_MEMORY_REGION2(); @@ -1079,8 +1081,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, if (flags & KVM_MEM_GUEST_MEMFD) { if (gmem_fd < 0) { - u32 gmem_flags = 0; - TEST_ASSERT(!gmem_offset, "Offset must be zero when creating new guest_memfd"); gmem_fd = vm_create_guest_memfd(vm, mem_size, gmem_flags); @@ -1101,13 +1101,17 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, } region->fd = -1; - if (backing_src_is_shared(src_type)) + if (flags & KVM_MEM_GUEST_MEMFD && gmem_flags & GUEST_MEMFD_FLAG_MMAP) { + region->fd = kvm_dup(gmem_fd); + mmap_flags = MAP_SHARED; + mmap_offset = gmem_offset; + } else if (backing_src_is_shared(src_type)) { region->fd = kvm_memfd_alloc(region->mmap_size, src_type == VM_MEM_SRC_SHARED_HUGETLB); + } - region->mmap_start = kvm_mmap(region->mmap_size, PROT_READ | PROT_WRITE, - vm_mem_backing_src_alias(src_type)->flag, - region->fd); + region->mmap_start = __kvm_mmap(region->mmap_size, PROT_READ | PROT_WRITE, + mmap_flags, region->fd, mmap_offset); TEST_ASSERT(!is_backing_src_hugetlb(src_type) || region->mmap_start == align_ptr_up(region->mmap_start, backing_src_pagesz), @@ -1153,10 +1157,10 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, /* If shared memory, create an alias. */ if (region->fd >= 0) { - region->mmap_alias = kvm_mmap(region->mmap_size, - PROT_READ | PROT_WRITE, - vm_mem_backing_src_alias(src_type)->flag, - region->fd); + region->mmap_alias = __kvm_mmap(region->mmap_size, + PROT_READ | PROT_WRITE, + mmap_flags, region->fd, + mmap_offset); /* Align host alias address */ region->host_alias = align_ptr_up(region->mmap_alias, alignment); @@ -1167,7 +1171,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, gpa_t gpa, u32 slot, u64 npages, u32 flags) { - vm_mem_add(vm, src_type, gpa, slot, npages, flags, -1, 0); + vm_mem_add(vm, src_type, gpa, slot, npages, flags, -1, 0, 0); } /* diff --git a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c index 3aa0673f79f3..f3de2a4ab055 100644 --- a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c @@ -399,7 +399,7 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v for (i = 0; i < nr_memslots; i++) vm_mem_add(vm, src_type, BASE_DATA_GPA + slot_size * i, BASE_DATA_SLOT + i, slot_size / vm->page_size, - KVM_MEM_GUEST_MEMFD, memfd, slot_size * i); + KVM_MEM_GUEST_MEMFD, memfd, slot_size * i, 0); for (i = 0; i < nr_vcpus; i++) { gpa_t gpa = BASE_DATA_GPA + i * per_cpu_size; From c3e48c166e35a5031df811459c96c303a2774cc1 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:53 -0700 Subject: [PATCH 40/60] KVM: selftests: Add selftests global for guest memory attributes capability Add a global variable, kvm_has_gmem_attributes, to make the result of checking for KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES available to all tests. kvm_has_gmem_attributes is true if guest_memfd tracks memory attributes, as opposed to VM-level tracking. This global variable is synced to the guest for testing convenience, to avoid introducing subtle bugs when host/guest state is desynced. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-27-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/include/test_util.h | 2 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index a6a3e1657895..d5825211f272 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -116,6 +116,8 @@ struct kvm_random_state { extern u32 kvm_random_seed; extern struct kvm_random_state kvm_rng; +extern bool kvm_has_gmem_attributes; + struct kvm_random_state new_kvm_random_state(u32 seed); u32 kvm_random_u32(struct kvm_random_state *state); diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 08d0bbcd421e..cd468b2bb834 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -32,6 +32,8 @@ static void kvm_seed_rng(u32 seed) kvm_rng = new_kvm_random_state(kvm_random_seed); } +bool kvm_has_gmem_attributes; + static size_t vcpu_mmap_sz(void); int __open_path_or_exit(const char *path, int flags, const char *enoent_help) @@ -527,6 +529,7 @@ struct kvm_vm *__vm_create(struct vm_shape shape, u32 nr_runnable_vcpus, kvm_seed_rng(kvm_random_seed); sync_global_to_guest(vm, kvm_rng); + sync_global_to_guest(vm, kvm_has_gmem_attributes); kvm_arch_vm_post_create(vm, nr_runnable_vcpus); @@ -2311,6 +2314,8 @@ void __attribute((constructor)) kvm_selftest_init(void) srandom(time(0)); kvm_seed_rng(random()); + kvm_has_gmem_attributes = kvm_has_cap(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES); + kvm_selftest_arch_init(); } From 0bbbd2b02bf0f66a65e4c6cc427fefbad512a861 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:55:54 -0700 Subject: [PATCH 41/60] KVM: selftests: Add helpers for calling ioctls on guest_memfd Add helper functions to kvm_util.h to support calling ioctls, specifically KVM_SET_MEMORY_ATTRIBUTES2, on a guest_memfd file descriptor. Introduce gmem_ioctl() and __gmem_ioctl() macros, modeled after the existing vm_ioctl() helpers, to provide a standard way to call ioctls on a guest_memfd. Add gmem_set_memory_attributes() and its derivatives (gmem_set_private(), gmem_set_shared()) to set memory attributes on a guest_memfd region. Also provide "__" variants that return the ioctl error code instead of aborting the test. These helpers will be used by upcoming guest_memfd tests. To avoid code duplication, factor out the check for supported memory attributes into a new macro, TEST_ASSERT_SUPPORTED_ATTRIBUTES, and use it in both the existing vm_set_memory_attributes() and the new gmem_set_memory_attributes() helpers. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-28-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../testing/selftests/kvm/include/kvm_util.h | 94 +++++++++++++++++-- 1 file changed, 86 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index a091c495442a..575032e9791d 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -392,6 +392,16 @@ static __always_inline void static_assert_is_vcpu(struct kvm_vcpu *vcpu) { } __TEST_ASSERT_VM_VCPU_IOCTL(!ret, #cmd, ret, (vcpu)->vm); \ }) +#define __gmem_ioctl(gmem_fd, cmd, arg) \ + kvm_do_ioctl(gmem_fd, cmd, arg) + +#define gmem_ioctl(gmem_fd, cmd, arg) \ +({ \ + int ret = __gmem_ioctl(gmem_fd, cmd, arg); \ + \ + TEST_ASSERT(!ret, __KVM_IOCTL_ERROR(#cmd, ret)); \ +}) + /* * Looks up and returns the value corresponding to the capability * (KVM_CAP_*) given by cap. @@ -418,8 +428,16 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0) vm_ioctl(vm, KVM_ENABLE_CAP, &enable_cap); } +/* + * KVM_SET_MEMORY_ATTRIBUTES{,2} overwrites _all_ attributes. These + * flows need significant enhancements to support multiple attributes. + */ +#define TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes) \ + TEST_ASSERT(!(attributes) || (attributes) == KVM_MEMORY_ATTRIBUTE_PRIVATE, \ + "Update me to support multiple attributes!") + static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa, - u64 size, u64 attributes) + size_t size, u64 attributes) { struct kvm_memory_attributes attr = { .attributes = attributes, @@ -428,17 +446,11 @@ static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa, .flags = 0, }; - /* - * KVM_SET_MEMORY_ATTRIBUTES overwrites _all_ attributes. These flows - * need significant enhancements to support multiple attributes. - */ - TEST_ASSERT(!attributes || attributes == KVM_MEMORY_ATTRIBUTE_PRIVATE, - "Update me to support multiple attributes!"); + TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes); vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr); } - static inline void vm_mem_set_private(struct kvm_vm *vm, gpa_t gpa, u64 size) { @@ -451,6 +463,72 @@ static inline void vm_mem_set_shared(struct kvm_vm *vm, gpa_t gpa, vm_set_memory_attributes(vm, gpa, size, 0); } +static inline int __gmem_set_memory_attributes(int fd, u64 offset, + size_t size, u64 attributes, + u64 *error_offset) +{ + struct kvm_memory_attributes2 attr = { + .attributes = attributes, + .offset = offset, + .size = size, + .flags = 0, + .error_offset = 0, + }; + int r; + + r = __gmem_ioctl(fd, KVM_SET_MEMORY_ATTRIBUTES2, &attr); + + /* Copy error_offset regardless of r so caller can check. */ + if (error_offset) + *error_offset = attr.error_offset; + + return r; +} + +static inline int __gmem_set_private(int fd, u64 offset, size_t size, + u64 *error_offset) +{ + return __gmem_set_memory_attributes(fd, offset, size, + KVM_MEMORY_ATTRIBUTE_PRIVATE, + error_offset); +} + +static inline int __gmem_set_shared(int fd, u64 offset, size_t size, + u64 *error_offset) +{ + return __gmem_set_memory_attributes(fd, offset, size, 0, + error_offset); +} + +static inline void gmem_set_memory_attributes(int fd, u64 offset, + size_t size, u64 attributes) +{ + struct kvm_memory_attributes2 attr = { + .attributes = attributes, + .offset = offset, + .size = size, + .flags = 0, + }; + + TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes); + + __TEST_REQUIRE(kvm_check_cap(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES) > 0, + "No valid attributes for guest_memfd ioctl!"); + + gmem_ioctl(fd, KVM_SET_MEMORY_ATTRIBUTES2, &attr); +} + +static inline void gmem_set_private(int fd, u64 offset, size_t size) +{ + gmem_set_memory_attributes(fd, offset, size, + KVM_MEMORY_ATTRIBUTE_PRIVATE); +} + +static inline void gmem_set_shared(int fd, u64 offset, size_t size) +{ + gmem_set_memory_attributes(fd, offset, size, 0); +} + void vm_guest_mem_fallocate(struct kvm_vm *vm, gpa_t gpa, u64 size, bool punch_hole); From eaef4ed5fabcb62890bfc4596230c43703841384 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:55 -0700 Subject: [PATCH 42/60] KVM: selftests: Test basic single-page conversion flow Add a selftest for the guest_memfd memory attribute conversion ioctls. The test starts the guest_memfd as all-private (the default state), and verifies the basic flow of converting a single page to shared and then back to private. Add infrastructure that supports extensions to other conversion flow tests. This infrastructure will be used in upcoming patches for other conversion tests. Add test as an x86-specific test since guest_memfd's testing vehicle (KVM_X86_SW_PROTECTED_VM) is x86-specific. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-29-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../kvm/x86/guest_memfd_conversions_test.c | 200 ++++++++++++++++++ 2 files changed, 201 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm index 96bab7002d39..239bc61ea384 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -155,6 +155,7 @@ TEST_GEN_PROGS_x86 += x86/max_vcpuid_cap_test TEST_GEN_PROGS_x86 += x86/triple_fault_event_test TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test TEST_GEN_PROGS_x86 += x86/aperfmperf_test +TEST_GEN_PROGS_x86 += x86/guest_memfd_conversions_test TEST_GEN_PROGS_x86 += access_tracking_perf_test TEST_GEN_PROGS_x86 += coalesced_io_test TEST_GEN_PROGS_x86 += dirty_log_perf_test diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c new file mode 100644 index 000000000000..77f95d49349c --- /dev/null +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, Google LLC. + */ +#include +#include + +#include +#include +#include + +#include "kvm_util.h" +#include "kselftest_harness.h" +#include "test_util.h" +#include "ucall_common.h" + +FIXTURE(gmem_conversions) { + struct kvm_vcpu *vcpu; + int gmem_fd; + /* HVA of the first byte of the memory mmap()-ed from gmem_fd. */ + char *mem; +}; + +typedef FIXTURE_DATA(gmem_conversions) test_data_t; + +FIXTURE_SETUP(gmem_conversions) { } + +static size_t page_size; + +static void guest_do_rmw(void); +#define GUEST_MEMFD_SHARING_TEST_GVA 0x90000000ULL + +/* + * Defer setup until the individual test is invoked so that tests can specify + * the number of pages and flags for the guest_memfd instance. + */ +static void gmem_conversions_do_setup(test_data_t *t, int nr_pages, + int gmem_flags) +{ + const struct vm_shape shape = { + .mode = VM_MODE_DEFAULT, + .type = KVM_X86_SW_PROTECTED_VM, + }; + /* + * Use high GPA above APIC_DEFAULT_PHYS_BASE to avoid clashing with + * APIC_DEFAULT_PHYS_BASE. + */ + const gpa_t gpa = SZ_4G; + const u32 slot = 1; + struct kvm_vm *vm; + + vm = __vm_create_shape_with_one_vcpu(shape, &t->vcpu, nr_pages, guest_do_rmw); + + vm_mem_add(vm, VM_MEM_SRC_SHMEM, gpa, slot, nr_pages, + KVM_MEM_GUEST_MEMFD, -1, 0, gmem_flags); + + t->gmem_fd = kvm_slot_to_fd(vm, slot); + t->mem = addr_gpa2hva(vm, gpa); + virt_map(vm, GUEST_MEMFD_SHARING_TEST_GVA, gpa, nr_pages); +} + +static void gmem_conversions_do_teardown(test_data_t *t) +{ + /* No need to close gmem_fd, it's owned by the VM structure. */ + kvm_vm_free(t->vcpu->vm); +} + +FIXTURE_TEARDOWN(gmem_conversions) +{ + gmem_conversions_do_teardown(self); +} + +/* + * In these test definition macros, __nr_pages and nr_pages is used to set up + * the total number of pages in the guest_memfd under test. This will be + * available in the test definitions as nr_pages. + */ + +#define __GMEM_CONVERSION_TEST(test, __nr_pages, flags) \ +static void __gmem_conversions_##test(test_data_t *t, int nr_pages); \ + \ +TEST_F(gmem_conversions, test) \ +{ \ + const int nr = (__nr_pages); \ + \ + gmem_conversions_do_setup(self, nr, flags); \ + __gmem_conversions_##test(self, nr); \ +} \ +static void __gmem_conversions_##test(test_data_t *t, int nr_pages) \ + +#define GMEM_CONVERSION_TEST(test, __nr_pages, flags) \ + __GMEM_CONVERSION_TEST(test, __nr_pages, (flags) | GUEST_MEMFD_FLAG_MMAP) + +#define __GMEM_CONVERSION_TEST_INIT_PRIVATE(test, __nr_pages) \ + GMEM_CONVERSION_TEST(test, __nr_pages, 0) + +#define GMEM_CONVERSION_TEST_INIT_PRIVATE(test) \ + __GMEM_CONVERSION_TEST_INIT_PRIVATE(test, 1) + +struct guest_check_data { + void *mem; + char expected_val; + char write_val; +}; +static struct guest_check_data guest_data; + +static void guest_do_rmw(void) +{ + for (;;) { + char *mem = READ_ONCE(guest_data.mem); + + GUEST_ASSERT_EQ(READ_ONCE(*mem), READ_ONCE(guest_data.expected_val)); + WRITE_ONCE(*mem, READ_ONCE(guest_data.write_val)); + + GUEST_SYNC(0); + } +} + +static void run_guest_do_rmw(struct kvm_vcpu *vcpu, u64 pgoff, + char expected_val, char write_val) +{ + struct ucall uc; + int r; + + guest_data.mem = (void *)GUEST_MEMFD_SHARING_TEST_GVA + pgoff * page_size; + guest_data.expected_val = expected_val; + guest_data.write_val = write_val; + sync_global_to_guest(vcpu->vm, guest_data); + + do { + r = __vcpu_run(vcpu); + } while (r == -1 && errno == EINTR); + + TEST_ASSERT_EQ(r, 0); + + switch (get_ucall(vcpu, &uc)) { + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + case UCALL_SYNC: + break; + default: + TEST_FAIL("Unexpected ucall %lu", uc.cmd); + } +} + +static void host_do_rmw(char *mem, u64 pgoff, char expected_val, + char write_val) +{ + TEST_ASSERT_EQ(READ_ONCE(mem[pgoff * page_size]), expected_val); + WRITE_ONCE(mem[pgoff * page_size], write_val); +} + +static void test_private(test_data_t *t, u64 pgoff, char starting_val, + char write_val) +{ + TEST_EXPECT_SIGBUS(WRITE_ONCE(t->mem[pgoff * page_size], write_val)); + run_guest_do_rmw(t->vcpu, pgoff, starting_val, write_val); + TEST_EXPECT_SIGBUS(READ_ONCE(t->mem[pgoff * page_size])); +} + +static void test_convert_to_private(test_data_t *t, u64 pgoff, + char starting_val, char write_val) +{ + gmem_set_private(t->gmem_fd, pgoff * page_size, page_size); + test_private(t, pgoff, starting_val, write_val); +} + +static void test_shared(test_data_t *t, u64 pgoff, char starting_val, + char host_write_val, char write_val) +{ + host_do_rmw(t->mem, pgoff, starting_val, host_write_val); + run_guest_do_rmw(t->vcpu, pgoff, host_write_val, write_val); + TEST_ASSERT_EQ(READ_ONCE(t->mem[pgoff * page_size]), write_val); +} + +static void test_convert_to_shared(test_data_t *t, u64 pgoff, + char starting_val, char host_write_val, + char write_val) +{ + gmem_set_shared(t->gmem_fd, pgoff * page_size, page_size); + test_shared(t, pgoff, starting_val, host_write_val, write_val); +} + +GMEM_CONVERSION_TEST_INIT_PRIVATE(init_private) +{ + test_private(t, 0, 0, 'A'); + test_convert_to_shared(t, 0, 'A', 'B', 'C'); + test_convert_to_private(t, 0, 'C', 'E'); +} + +int main(int argc, char *argv[]) +{ + TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); + TEST_REQUIRE(kvm_check_cap(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES) & + KVM_MEMORY_ATTRIBUTE_PRIVATE); + + page_size = getpagesize(); + + return test_harness_run(argc, argv); +} From e5fa04dc920e4542404da89b31a8eb5a2d3360eb Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:56 -0700 Subject: [PATCH 43/60] KVM: selftests: Test conversion flow when INIT_SHARED Add a test case to verify that conversions between private and shared memory work correctly when the memory is initially created as shared. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-30-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/guest_memfd_conversions_test.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c index 77f95d49349c..4c20d3d8e934 100644 --- a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -97,6 +97,12 @@ static void __gmem_conversions_##test(test_data_t *t, int nr_pages) \ #define GMEM_CONVERSION_TEST_INIT_PRIVATE(test) \ __GMEM_CONVERSION_TEST_INIT_PRIVATE(test, 1) +#define __GMEM_CONVERSION_TEST_INIT_SHARED(test, __nr_pages) \ + GMEM_CONVERSION_TEST(test, __nr_pages, GUEST_MEMFD_FLAG_INIT_SHARED) + +#define GMEM_CONVERSION_TEST_INIT_SHARED(test) \ + __GMEM_CONVERSION_TEST_INIT_SHARED(test, 1) + struct guest_check_data { void *mem; char expected_val; @@ -188,6 +194,13 @@ GMEM_CONVERSION_TEST_INIT_PRIVATE(init_private) test_convert_to_private(t, 0, 'C', 'E'); } +GMEM_CONVERSION_TEST_INIT_SHARED(init_shared) +{ + test_shared(t, 0, 0, 'A', 'B'); + test_convert_to_private(t, 0, 'B', 'C'); + test_convert_to_shared(t, 0, 'C', 'D', 'E'); +} + int main(int argc, char *argv[]) { TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); From 492080a3d4e5812c6c232ee69758a856c303f723 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:57 -0700 Subject: [PATCH 44/60] KVM: selftests: Test conversion precision in guest_memfd The existing guest_memfd conversion tests only use single-page memory regions. This provides no coverage for multi-page guest_memfd objects, specifically whether KVM correctly handles the page index for conversion operations. An incorrect implementation could, for example, always operate on the first page regardless of the index provided. Add a new test case to verify that conversions between private and shared memory correctly target the specified page within a multi-page guest_memfd. This test also verifies the precision of memory conversions by converting a single page and then iterating through all other pages to ensure they remain in their original state. To support this test, add a new GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED macro that handles setting up and tearing down the VM for each page iteration. The teardown logic is adjusted to prevent a double-free in this new scenario. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-31-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/guest_memfd_conversions_test.c | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c index 4c20d3d8e934..a8d89df53e46 100644 --- a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -61,8 +61,13 @@ static void gmem_conversions_do_setup(test_data_t *t, int nr_pages, static void gmem_conversions_do_teardown(test_data_t *t) { + /* Use NULL to avoid second free in FIXTURE_TEARDOWN (multipage tests). */ + if (!t->vcpu) + return; + /* No need to close gmem_fd, it's owned by the VM structure. */ kvm_vm_free(t->vcpu->vm); + t->vcpu = NULL; } FIXTURE_TEARDOWN(gmem_conversions) @@ -103,6 +108,30 @@ static void __gmem_conversions_##test(test_data_t *t, int nr_pages) \ #define GMEM_CONVERSION_TEST_INIT_SHARED(test) \ __GMEM_CONVERSION_TEST_INIT_SHARED(test, 1) +/* + * Repeats test over nr_pages in a guest_memfd of size nr_pages, providing each + * test iteration with test_page, the index of the page under test in + * guest_memfd. test_page takes values 0..(nr_pages - 1) inclusive. + */ +#define GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(test, __nr_pages) \ +static void __gmem_conversions_multipage_##test(test_data_t *t, int nr_pages, \ + const int test_page); \ + \ +TEST_F(gmem_conversions, test) \ +{ \ + const u64 flags = GUEST_MEMFD_FLAG_MMAP | GUEST_MEMFD_FLAG_INIT_SHARED; \ + const int nr = (__nr_pages); \ + int i; \ + \ + for (i = 0; i < nr; ++i) { \ + gmem_conversions_do_setup(self, nr, flags); \ + __gmem_conversions_multipage_##test(self, nr, i); \ + gmem_conversions_do_teardown(self); \ + } \ +} \ +static void __gmem_conversions_multipage_##test(test_data_t *t, int nr_pages, \ + const int test_page) + struct guest_check_data { void *mem; char expected_val; @@ -201,6 +230,44 @@ GMEM_CONVERSION_TEST_INIT_SHARED(init_shared) test_convert_to_shared(t, 0, 'C', 'D', 'E'); } +GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(indexing, 4) +{ + int i; + + /* Get a char that varies with both i and n. */ +#define combine(x, n) (((x) << 4) + (n)) +#define i_(n) (combine(i, n)) +#define t_(n) (combine(test_page, n)) + + /* + * Start with the highest index, to catch any errors when, perhaps, the + * first page is returned even for the last index. + */ + for (i = nr_pages - 1; i >= 0; --i) + test_shared(t, i, 0, i_(0), i_(2)); + + test_convert_to_private(t, test_page, t_(2), t_(3)); + + for (i = 0; i < nr_pages; ++i) { + if (i == test_page) + test_private(t, test_page, t_(3), t_(4)); + else + test_shared(t, i, i_(2), i_(3), i_(4)); + } + + test_convert_to_shared(t, test_page, t_(4), t_(5), t_(6)); + + for (i = 0; i < nr_pages; ++i) { + char expected = i == test_page ? t_(6) : i_(4); + + test_shared(t, i, expected, i_(7), i_(8)); + } + +#undef t_ +#undef i_ +#undef combine +} + int main(int argc, char *argv[]) { TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); From 97137c1403607ed8316865867186382a71b4ea66 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:58 -0700 Subject: [PATCH 45/60] KVM: selftests: Test conversion before allocation Add two test cases to the guest_memfd conversions selftest to cover the scenario where a conversion is requested before any memory has been allocated in the guest_memfd region. The KVM_SET_MEMORY_ATTRIBUTES2 ioctl can be called on a memory region at any time. If the guest had not yet faulted in any pages for that region, the kernel must record the conversion request and apply the requested state when the pages are eventually allocated. The new tests cover both conversion directions. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-32-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/guest_memfd_conversions_test.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c index a8d89df53e46..f46f1530a94b 100644 --- a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -268,6 +268,20 @@ GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(indexing, 4) #undef combine } +/* + * Test that even if there are no folios yet, conversion requests are recorded + * in guest_memfd. + */ +GMEM_CONVERSION_TEST_INIT_SHARED(before_allocation_shared) +{ + test_convert_to_private(t, 0, 0, 'A'); +} + +GMEM_CONVERSION_TEST_INIT_PRIVATE(before_allocation_private) +{ + test_convert_to_shared(t, 0, 0, 'A', 'B'); +} + int main(int argc, char *argv[]) { TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); From 5362c5fb88f6df1a00eda33ab8487ce09ebdb71a Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:55:59 -0700 Subject: [PATCH 46/60] KVM: selftests: Convert with allocated folios in different layouts Add a guest_memfd selftest to verify that memory conversions work correctly with allocated folios in different layouts. By iterating through which pages are initially faulted, the test covers various layouts of contiguous allocated and unallocated regions, exercising conversion with different range layouts. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-33-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/guest_memfd_conversions_test.c | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c index f46f1530a94b..acdff2d15404 100644 --- a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -282,6 +282,36 @@ GMEM_CONVERSION_TEST_INIT_PRIVATE(before_allocation_private) test_convert_to_shared(t, 0, 0, 'A', 'B'); } +/* + * Test that when some of the folios in the conversion range are allocated, + * conversion requests are handled correctly in guest_memfd. Vary the ranges + * allocated before conversion, using test_page, to cover various layouts of + * contiguous allocated and unallocated regions. + */ +GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(unallocated_folios, 8) +{ + const int second_page_to_fault = 4; + int i; + + /* + * Fault 2 of the pages to test filemap range operations except when + * test_page == second_page_to_fault. + */ + host_do_rmw(t->mem, test_page, 0, 'A'); + if (test_page != second_page_to_fault) + host_do_rmw(t->mem, second_page_to_fault, 0, 'A'); + + gmem_set_private(t->gmem_fd, 0, nr_pages * page_size); + for (i = 0; i < nr_pages; ++i) { + char expected = (i == test_page || i == second_page_to_fault) ? 'A' : 0; + + test_private(t, i, expected, 'B'); + } + + for (i = 0; i < nr_pages; ++i) + test_convert_to_shared(t, i, 'B', 'C', 'D'); +} + int main(int argc, char *argv[]) { TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); From bfb9b5a286ea6a190bfed191de3a6b6f0bf801c4 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:00 -0700 Subject: [PATCH 47/60] KVM: selftests: Test that truncation does not change shared/private status Add a test to verify that deallocating a page in a guest memfd region via fallocate() with FALLOC_FL_PUNCH_HOLE does not alter the shared or private status of the corresponding memory range. When a page backing a guest memfd mapping is deallocated, e.g., by punching a hole or truncating the file, and then subsequently faulted back in, the new page must inherit the correct shared/private status tracked by guest_memfd. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-34-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/guest_memfd_conversions_test.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c index acdff2d15404..b7a54d2a9092 100644 --- a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -10,6 +10,7 @@ #include #include "kvm_util.h" +#include "kvm_syscalls.h" #include "kselftest_harness.h" #include "test_util.h" #include "ucall_common.h" @@ -312,6 +313,19 @@ GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(unallocated_folios, 8) test_convert_to_shared(t, i, 'B', 'C', 'D'); } +/* Truncation should not affect shared/private status. */ +GMEM_CONVERSION_TEST_INIT_SHARED(truncate) +{ + host_do_rmw(t->mem, 0, 0, 'A'); + kvm_fallocate(t->gmem_fd, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, 0, page_size); + host_do_rmw(t->mem, 0, 0, 'A'); + + test_convert_to_private(t, 0, 'A', 'B'); + + kvm_fallocate(t->gmem_fd, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, 0, page_size); + test_private(t, 0, 0, 'A'); +} + int main(int argc, char *argv[]) { TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); From 8b72eaa766b6b11a38faafa0b936d1952f1b7ba6 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:56:01 -0700 Subject: [PATCH 48/60] KVM: selftests: Test that shared/private status is consistent across processes Add a test to verify that a guest_memfd's shared/private status is consistent across processes, and that any shared pages previously mapped in any process are unmapped from all processes. The test forks a child process after creating the shared guest_memfd region so that the second process exists alongside the main process for the entire test. The processes then take turns to access memory to check that the shared/private status is consistent across processes. Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-35-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/guest_memfd_conversions_test.c | 118 ++++++++++++++++++ 1 file changed, 118 insertions(+) diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c index b7a54d2a9092..a972a09c5c6e 100644 --- a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -2,7 +2,9 @@ /* * Copyright (c) 2024, Google LLC. */ +#include #include +#include #include #include @@ -326,6 +328,122 @@ GMEM_CONVERSION_TEST_INIT_SHARED(truncate) test_private(t, 0, 0, 'A'); } +/* Test that shared/private memory protections work and are seen from any process. */ +GMEM_CONVERSION_TEST_INIT_SHARED(forked_accesses) +{ + enum test_state { + STATE_INIT, + STATE_CHECK_SHARED, + STATE_DONE_CHECKING_SHARED, + STATE_CHECK_PRIVATE, + STATE_DONE_CHECKING_PRIVATE, + }; + + struct sync_state { + pthread_mutex_t mutex; + pthread_cond_t cond; + enum test_state step; + } *sync; + + pthread_mutexattr_t mattr; + pthread_condattr_t cattr; + pid_t child_pid, parent_pid; + int status; + + sync = kvm_mmap(sizeof(*sync), PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, -1); + + pthread_mutexattr_init(&mattr); + pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED); + pthread_mutex_init(&sync->mutex, &mattr); + pthread_mutexattr_destroy(&mattr); + + pthread_condattr_init(&cattr); + pthread_condattr_setpshared(&cattr, PTHREAD_PROCESS_SHARED); + pthread_cond_init(&sync->cond, &cattr); + pthread_condattr_destroy(&cattr); + + sync->step = STATE_INIT; + +#define TEST_STATE_AWAIT(__state) \ + do { \ + pthread_mutex_lock(&sync->mutex); \ + while (sync->step != (__state)) { \ + struct timespec ts, stop; \ + int ret; \ + \ + clock_gettime(CLOCK_REALTIME, &ts); \ + stop = timespec_add_ns(ts, 100 * 1000000UL); \ + \ + ret = pthread_cond_timedwait(&sync->cond, &sync->mutex, &stop); \ + if (ret == ETIMEDOUT) { \ + bool alive = (child_pid == 0) ? \ + (getppid() == parent_pid) : \ + (waitpid(child_pid, NULL, WNOHANG) == 0); \ + TEST_ASSERT(alive, "Other process exited prematurely"); \ + } else { \ + TEST_ASSERT(!ret, "pthread_cond_timedwait failed"); \ + } \ + } \ + pthread_mutex_unlock(&sync->mutex); \ + } while (0) + +#define TEST_STATE_SET(__state) \ + do { \ + pthread_mutex_lock(&sync->mutex); \ + sync->step = (__state); \ + pthread_cond_broadcast(&sync->cond); \ + pthread_mutex_unlock(&sync->mutex); \ + } while (0) + + parent_pid = getpid(); + child_pid = fork(); + TEST_ASSERT(child_pid != -1, "fork failed"); + + if (child_pid == 0) { + const char inconsequential = 0xdd; + + TEST_STATE_AWAIT(STATE_CHECK_SHARED); + + /* + * This maps the pages into the child process as well, and tests + * that the conversion process will unmap the guest_memfd memory + * from all processes. + */ + host_do_rmw(t->mem, 0, 0xB, 0xC); + + TEST_STATE_SET(STATE_DONE_CHECKING_SHARED); + TEST_STATE_AWAIT(STATE_CHECK_PRIVATE); + + TEST_EXPECT_SIGBUS(READ_ONCE(t->mem[0])); + TEST_EXPECT_SIGBUS(WRITE_ONCE(t->mem[0], inconsequential)); + + TEST_STATE_SET(STATE_DONE_CHECKING_PRIVATE); + exit(0); + } + + test_shared(t, 0, 0, 0xA, 0xB); + + TEST_STATE_SET(STATE_CHECK_SHARED); + TEST_STATE_AWAIT(STATE_DONE_CHECKING_SHARED); + + test_convert_to_private(t, 0, 0xC, 0xD); + + TEST_STATE_SET(STATE_CHECK_PRIVATE); + TEST_STATE_AWAIT(STATE_DONE_CHECKING_PRIVATE); + + TEST_ASSERT_EQ(waitpid(child_pid, &status, 0), child_pid); + TEST_ASSERT(WIFEXITED(status) && WEXITSTATUS(status) == 0, + "Child exited with unexpected status"); + + pthread_mutex_destroy(&sync->mutex); + pthread_cond_destroy(&sync->cond); + kvm_munmap(sync, sizeof(*sync)); + +#undef TEST_STATE_SET +#undef TEST_STATE_AWAIT +} + int main(int argc, char *argv[]) { TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); From e8a40b9a604e560b99f4dc4fbb4971143c75807e Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:02 -0700 Subject: [PATCH 49/60] KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST Add helper functions to allow KVM selftests to pin memory using CONFIG_GUP_TEST. This is useful for creating test scenarios where some page has an increased refcount, such as when testing guest_memfd in-place conversion. The helpers open /sys/kernel/debug/gup_test and invoke the PIN_LONGTERM_TEST_START and PIN_LONGTERM_TEST_STOP ioctls. Since this functionality depends on the kernel being built with CONFIG_GUP_TEST, provide stub implementations that trigger a test failure if the configuration is missing. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-36-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../testing/selftests/kvm/include/kvm_util.h | 3 +++ tools/testing/selftests/kvm/lib/kvm_util.c | 25 +++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 575032e9791d..3cfe243d7c8b 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -1206,6 +1206,9 @@ static inline int pin_self_to_any_cpu(void) return pin_task_to_any_cpu(pthread_self()); } +void pin_pages(void *vaddr, uint64_t size); +void unpin_pages(void); + void kvm_print_vcpu_pinning_help(void); void kvm_parse_vcpu_pinning(const char *pcpus_string, u32 vcpu_to_pcpu[], int nr_vcpus); diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index cd468b2bb834..b1456ed76e27 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -19,6 +19,8 @@ #include #include +#include "../../../../../mm/gup_test.h" + #define KVM_UTIL_MIN_PFN 2 u32 kvm_random_seed; @@ -645,6 +647,29 @@ int __pin_task_to_cpu(pthread_t task, int cpu) return pthread_setaffinity_np(task, sizeof(cpuset), &cpuset); } +static int gup_test_fd = -1; + +void pin_pages(void *vaddr, uint64_t size) +{ + const struct pin_longterm_test args = { + .addr = (uint64_t)vaddr, + .size = size, + .flags = PIN_LONGTERM_TEST_FLAG_USE_WRITE, + }; + + gup_test_fd = __open_path_or_exit("/sys/kernel/debug/gup_test", O_RDWR, + "Is CONFIG_GUP_TEST enabled?"); + + TEST_ASSERT_EQ(ioctl(gup_test_fd, PIN_LONGTERM_TEST_START, &args), 0); +} + +void unpin_pages(void) +{ + TEST_ASSERT_EQ(ioctl(gup_test_fd, PIN_LONGTERM_TEST_STOP), 0); + + kvm_free_fd(gup_test_fd); +} + static u32 parse_pcpu(const char *cpu_str, const cpu_set_t *allowed_mask) { u32 pcpu = atoi_non_negative("CPU number", cpu_str); From 0db3f14f40c42dd0ac6a85b5e8fd605d37c8128a Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:03 -0700 Subject: [PATCH 50/60] KVM: selftests: Test conversion with elevated page refcount Add a selftest to verify that converting a shared guest_memfd page to a private page fails if the page has an elevated reference count. When KVM converts a shared page to a private one, it expects the page to have a reference count equal to the reference counts taken by the filemap. If another kernel subsystem holds a reference to the page, the conversion must be aborted. The test asserts that both bulk and single-page conversion attempts correctly fail with EAGAIN for the pinned page. After the page is unpinned, the test verifies that subsequent conversions succeed. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-37-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/guest_memfd_conversions_test.c | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c index a972a09c5c6e..e1c84df0f059 100644 --- a/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/guest_memfd_conversions_test.c @@ -444,6 +444,62 @@ GMEM_CONVERSION_TEST_INIT_SHARED(forked_accesses) #undef TEST_STATE_AWAIT } +static void test_convert_to_private_fails(test_data_t *t, u64 pgoff, + size_t nr_pages, + u64 expected_error_offset) +{ + /* +1 to make it anything but expected_error_offset. */ + u64 error_offset = expected_error_offset + 1; + u64 offset = pgoff * page_size; + int ret; + + do { + ret = __gmem_set_private(t->gmem_fd, offset, + nr_pages * page_size, &error_offset); + } while (ret == -1 && errno == EINTR); + TEST_ASSERT(ret == -1 && errno == EAGAIN, + "Wanted EAGAIN on page %lu, got %d (ret = %d)", pgoff, + errno, ret); + TEST_ASSERT_EQ(error_offset, expected_error_offset); +} + +GMEM_CONVERSION_MULTIPAGE_TEST_INIT_SHARED(elevated_refcount, 4) +{ + int i; + + pin_pages(t->mem + test_page * page_size, page_size); + + for (i = 0; i < nr_pages; i++) + test_shared(t, i, 0, 'A', 'B'); + + /* + * Converting in bulk should fail as long any page in the range has + * unexpected refcounts. + */ + test_convert_to_private_fails(t, 0, nr_pages, test_page * page_size); + + for (i = 0; i < nr_pages; i++) { + /* + * Converting page-wise should also fail as long any page in the + * range has unexpected refcounts. + */ + if (i == test_page) + test_convert_to_private_fails(t, i, 1, test_page * page_size); + else + test_convert_to_private(t, i, 'B', 'C'); + } + + unpin_pages(); + + gmem_set_private(t->gmem_fd, 0, nr_pages * page_size); + + for (i = 0; i < nr_pages; i++) { + char expected = i == test_page ? 'B' : 'C'; + + test_private(t, i, expected, 'D'); + } +} + int main(int argc, char *argv[]) { TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM)); From c235250e8bc1826ddab92565b659350e06e9d40c Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:04 -0700 Subject: [PATCH 51/60] KVM: selftests: Reset shared memory after hole-punching private_mem_conversions_test used to reset the shared memory that was used for the test to an initial pattern at the end of each test iteration. Then, it would punch out the pages, which would zero memory. Without in-place conversion, the resetting would write shared memory, and hole-punching will zero private memory, hence resetting the test to the state at the beginning of the for loop. With in-place conversion, resetting writes memory as shared, and hole-punching zeroes the same physical memory, hence undoing the reset done before the hole punch. Move the resetting after the hole-punching, and reset the entire PER_CPU_DATA_SIZE instead of just the tested range. With in-place conversion, this zeroes and then resets the same physical memory. Without in-place conversion, the private memory is zeroed, and the shared memory is reset to init_p. This is sufficient since at each test stage, the memory is assumed to start as shared, and private memory is always assumed to start zeroed. Conversion zeroes memory, so the future test stages will work as expected. Fixes: 43f623f350ce1 ("KVM: selftests: Add x86-only selftest for private memory conversions") Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-38-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../selftests/kvm/x86/private_mem_conversions_test.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c index f3de2a4ab055..cf50e9a332c5 100644 --- a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c @@ -202,15 +202,18 @@ static void guest_test_explicit_conversion(u64 base_gpa, bool do_fallocate) guest_sync_shared(gpa, size, p3, p4); memcmp_g(gpa, p4, size); - /* Reset the shared memory back to the initial pattern. */ - memset((void *)gpa, init_p, size); - /* * Free (via PUNCH_HOLE) *all* private memory so that the next * iteration starts from a clean slate, e.g. with respect to * whether or not there are pages/folios in guest_mem. */ guest_map_shared(base_gpa, PER_CPU_DATA_SIZE, true); + + /* + * Hole-punching above zeroed private memory. Reset shared + * memory in preparation for the next GUEST_STAGE. + */ + memset((void *)base_gpa, init_p, PER_CPU_DATA_SIZE); } } From ee6f4c381903e015b52a6797a7ef066966dc65ec Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:05 -0700 Subject: [PATCH 52/60] KVM: selftests: Provide function to look up guest_memfd details from gpa Introduce a new helper, kvm_gpa_to_guest_memfd(), to find the guest_memfd-related details of a memory region that contains a given guest physical address (GPA). The function returns the file descriptor for the memfd, the offset into the file that corresponds to the GPA, and the number of bytes remaining in the region from that GPA. kvm_gpa_to_guest_memfd() was factored out from vm_guest_mem_fallocate(); refactor vm_guest_mem_fallocate() to use the new helper. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-39-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../testing/selftests/kvm/include/kvm_util.h | 3 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 37 ++++++++++++------- 2 files changed, 26 insertions(+), 14 deletions(-) diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 3cfe243d7c8b..795c9eb31107 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -428,6 +428,9 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0) vm_ioctl(vm, KVM_ENABLE_CAP, &enable_cap); } +int kvm_gpa_to_guest_memfd(struct kvm_vm *vm, gpa_t gpa, off_t *fd_offset, + size_t *nr_bytes); + /* * KVM_SET_MEMORY_ATTRIBUTES{,2} overwrites _all_ attributes. These * flows need significant enhancements to support multiple attributes. diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index b1456ed76e27..1188d0a6c0a5 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1332,27 +1332,20 @@ void vm_guest_mem_fallocate(struct kvm_vm *vm, u64 base, u64 size, bool punch_hole) { const int mode = FALLOC_FL_KEEP_SIZE | (punch_hole ? FALLOC_FL_PUNCH_HOLE : 0); - struct userspace_mem_region *region; u64 end = base + size; - gpa_t gpa, len; off_t fd_offset; - int ret; + int fd, ret; + size_t len; + gpa_t gpa; for (gpa = base; gpa < end; gpa += len) { - u64 offset; + fd = kvm_gpa_to_guest_memfd(vm, gpa, &fd_offset, &len); + len = min(end - gpa, len); - region = userspace_mem_region_find(vm, gpa, gpa); - TEST_ASSERT(region && region->region.flags & KVM_MEM_GUEST_MEMFD, - "Private memory region not found for GPA 0x%lx", gpa); - - offset = gpa - region->region.guest_phys_addr; - fd_offset = region->region.guest_memfd_offset + offset; - len = min_t(u64, end - gpa, region->region.memory_size - offset); - - ret = fallocate(region->region.guest_memfd, mode, fd_offset, len); + ret = fallocate(fd, mode, fd_offset, len); TEST_ASSERT(!ret, "fallocate() failed to %s at %lx (len = %lu), fd = %d, mode = %x, offset = %lx", punch_hole ? "punch hole" : "allocate", gpa, len, - region->region.guest_memfd, mode, fd_offset); + fd, mode, fd_offset); } } @@ -1689,6 +1682,22 @@ void *addr_gpa2alias(struct kvm_vm *vm, gpa_t gpa) return (void *) ((uintptr_t) region->host_alias + offset); } +int kvm_gpa_to_guest_memfd(struct kvm_vm *vm, gpa_t gpa, off_t *fd_offset, + size_t *nr_bytes) +{ + struct userspace_mem_region *region; + gpa_t gpa_offset; + + region = userspace_mem_region_find(vm, gpa, gpa); + TEST_ASSERT(region && region->region.flags & KVM_MEM_GUEST_MEMFD, + "guest_memfd memory region not found for GPA 0x%lx", gpa); + + gpa_offset = gpa - region->region.guest_phys_addr; + *fd_offset = region->region.guest_memfd_offset + gpa_offset; + *nr_bytes = region->region.memory_size - gpa_offset; + return region->region.guest_memfd; +} + /* Create an interrupt controller chip for the specified VM. */ void vm_create_irqchip(struct kvm_vm *vm) { From f49dd8ba330e89457c23448d4137b9f984cb0904 Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:56:06 -0700 Subject: [PATCH 53/60] KVM: selftests: Provide common function to set memory attributes Introduce vm_mem_set_memory_attributes(), which handles setting of memory attributes for a range of guest physical addresses, regardless of whether the attributes should be set via guest_memfd or via the memory attributes at the VM level. Refactor existing vm_mem_set_{shared,private} functions to use the new function. Opportunistically update the size parameter to use size_t instead of u64. Co-developed-by: Ackerley Tng Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-40-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../testing/selftests/kvm/include/kvm_util.h | 46 ++++++++++++++----- 1 file changed, 34 insertions(+), 12 deletions(-) diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 795c9eb31107..777fa3dbf88d 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -454,18 +454,6 @@ static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa, vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr); } -static inline void vm_mem_set_private(struct kvm_vm *vm, gpa_t gpa, - u64 size) -{ - vm_set_memory_attributes(vm, gpa, size, KVM_MEMORY_ATTRIBUTE_PRIVATE); -} - -static inline void vm_mem_set_shared(struct kvm_vm *vm, gpa_t gpa, - u64 size) -{ - vm_set_memory_attributes(vm, gpa, size, 0); -} - static inline int __gmem_set_memory_attributes(int fd, u64 offset, size_t size, u64 attributes, u64 *error_offset) @@ -532,6 +520,40 @@ static inline void gmem_set_shared(int fd, u64 offset, size_t size) gmem_set_memory_attributes(fd, offset, size, 0); } +static inline void vm_mem_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa, + size_t size, u64 attrs) +{ + if (kvm_has_gmem_attributes) { + gpa_t end = gpa + size; + off_t fd_offset; + gpa_t addr; + size_t len; + int fd; + + for (addr = gpa; addr < end; addr += len) { + fd = kvm_gpa_to_guest_memfd(vm, addr, &fd_offset, &len); + len = min(end - addr, len); + + gmem_set_memory_attributes(fd, fd_offset, len, attrs); + } + } else { + vm_set_memory_attributes(vm, gpa, size, attrs); + } +} + +static inline void vm_mem_set_private(struct kvm_vm *vm, gpa_t gpa, + size_t size) +{ + vm_mem_set_memory_attributes(vm, gpa, size, + KVM_MEMORY_ATTRIBUTE_PRIVATE); +} + +static inline void vm_mem_set_shared(struct kvm_vm *vm, gpa_t gpa, + size_t size) +{ + vm_mem_set_memory_attributes(vm, gpa, size, 0); +} + void vm_guest_mem_fallocate(struct kvm_vm *vm, gpa_t gpa, u64 size, bool punch_hole); From ef72e71da099c0cfb4fe581ac94168531ad2c1ea Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:07 -0700 Subject: [PATCH 54/60] KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe The TEST_EXPECT_SIGBUS macro is not thread-safe as it uses a global sigjmp_buf and installs a global SIGBUS signal handler. If multiple threads execute the macro concurrently, they will race on installing the signal handler and stomp on other threads' jump buffers, leading to incorrect test behavior. Make TEST_EXPECT_SIGBUS thread-safe with the following changes: Share the KVM tests' global signal handler. sigaction() applies to all threads; without sharing a global signal handler, one thread may have removed the signal handler that another thread added, hence leading to unexpected signals. The alternative of layering signal handlers was considered, but calling sigaction() within TEST_EXPECT_SIGBUS() necessarily creates a race. To avoid adding new setup and teardown routines to do sigaction() and keep usage of TEST_EXPECT_SIGBUS() simple, share the KVM tests' global signal handler. Opportunistically rename report_unexpected_signal to catchall_signal_handler. To continue to only expect SIGBUS within specific regions of code, use a thread-specific variable, expecting_sigbus, to replace installing and removing signal handlers. Make the execution environment for the thread, sigjmp_buf, a thread-specific variable. As part of TEST_EXPECT_SIGBUS(), assert the prerequisite for this setup, that the current signal handler is the catchall_signal_handler. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-41-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../testing/selftests/kvm/include/test_util.h | 30 ++++++++++--------- tools/testing/selftests/kvm/lib/kvm_util.c | 18 +++++++---- tools/testing/selftests/kvm/lib/test_util.c | 7 ----- 3 files changed, 29 insertions(+), 26 deletions(-) diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index d5825211f272..75e98ec14107 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -83,21 +83,23 @@ do { \ __builtin_unreachable(); \ } while (0) -extern sigjmp_buf expect_sigbus_jmpbuf; -void expect_sigbus_handler(int signum); +extern __thread sigjmp_buf expect_sigbus_jmpbuf; +extern __thread volatile sig_atomic_t expecting_sigbus; +void catchall_signal_handler(int signum); -#define TEST_EXPECT_SIGBUS(action) \ -do { \ - struct sigaction sa_old, sa_new = { \ - .sa_handler = expect_sigbus_handler, \ - }; \ - \ - sigaction(SIGBUS, &sa_new, &sa_old); \ - if (sigsetjmp(expect_sigbus_jmpbuf, 1) == 0) { \ - action; \ - TEST_FAIL("'%s' should have triggered SIGBUS", #action); \ - } \ - sigaction(SIGBUS, &sa_old, NULL); \ +#define TEST_EXPECT_SIGBUS(action) \ +do { \ + struct sigaction __sa = {}; \ + \ + TEST_ASSERT_EQ(sigaction(SIGBUS, NULL, &__sa), 0); \ + TEST_ASSERT_EQ(__sa.sa_handler, &catchall_signal_handler); \ + \ + expecting_sigbus = true; \ + if (sigsetjmp(expect_sigbus_jmpbuf, 1) == 0) { \ + action; \ + TEST_FAIL("'%s' should have triggered SIGBUS", #action);\ + } \ + expecting_sigbus = false; \ } while (0) size_t parse_size(const char *size); diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 1188d0a6c0a5..cac5624e9977 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -2316,13 +2316,20 @@ __weak void kvm_selftest_arch_init(void) { } -static void report_unexpected_signal(int signum) +__thread sigjmp_buf expect_sigbus_jmpbuf; +__thread volatile sig_atomic_t expecting_sigbus; + +void catchall_signal_handler(int signum) { + switch (signum) { + case SIGBUS: { + if (expecting_sigbus) + siglongjmp(expect_sigbus_jmpbuf, 1); + + TEST_FAIL("Unexpected SIGBUS (%d)\n", signum); + } #define KVM_CASE_SIGNUM(sig) \ case sig: TEST_FAIL("Unexpected " #sig " (%d)\n", signum) - - switch (signum) { - KVM_CASE_SIGNUM(SIGBUS); KVM_CASE_SIGNUM(SIGSEGV); KVM_CASE_SIGNUM(SIGILL); KVM_CASE_SIGNUM(SIGFPE); @@ -2334,12 +2341,13 @@ static void report_unexpected_signal(int signum) void __attribute((constructor)) kvm_selftest_init(void) { struct sigaction sig_sa = { - .sa_handler = report_unexpected_signal, + .sa_handler = catchall_signal_handler, }; /* Tell stdout not to buffer its content. */ setbuf(stdout, NULL); + expecting_sigbus = false; sigaction(SIGBUS, &sig_sa, NULL); sigaction(SIGSEGV, &sig_sa, NULL); sigaction(SIGILL, &sig_sa, NULL); diff --git a/tools/testing/selftests/kvm/lib/test_util.c b/tools/testing/selftests/kvm/lib/test_util.c index 4dff72f6bd34..89d6d1b57295 100644 --- a/tools/testing/selftests/kvm/lib/test_util.c +++ b/tools/testing/selftests/kvm/lib/test_util.c @@ -19,13 +19,6 @@ #include "test_util.h" #include "kvm_syscalls.h" -sigjmp_buf expect_sigbus_jmpbuf; - -void __attribute__((used)) expect_sigbus_handler(int signum) -{ - siglongjmp(expect_sigbus_jmpbuf, 1); -} - /* * Random number generator that is usable from guest code. This is the * Park-Miller LCG using standard constants. From 710169986d9b7df56e435a0c5890b355ac8603d8 Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:08 -0700 Subject: [PATCH 55/60] KVM: selftests: Set up page size and alignment independently for guest_memfd Currently, vm_mem_add derives the backing source page size, alignment padding, and mmap size from the backing source type upfront before checking if guest_memfd is being mmapped. With shared memory also mmap()-ed from guest_memfd, the alignment of the mmap-ed address needs to respect guest_memfd's backing page size. Refactor the backing store setup to configure the backing source page size, alignment, mmap flags, and mmap offset directly for guest_memfd when it is mmapped, ignoring the backing source type. Skip hugepage validation and anonymous memory madvise calls when mmapping from guest_memfd, since those are not applicable when mmapping guest_memfd. Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-42-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/lib/kvm_util.c | 74 +++++++++++++--------- 1 file changed, 45 insertions(+), 29 deletions(-) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index cac5624e9977..3f559f207408 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1034,11 +1034,12 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, { int ret; struct userspace_mem_region *region; - size_t backing_src_pagesz = get_backing_src_pagesz(src_type); - int mmap_flags = vm_mem_backing_src_alias(src_type)->flag; size_t mem_size = npages * vm->page_size; - off_t mmap_offset = 0; - size_t alignment = 1; + size_t backing_src_pagesz; + off_t mmap_offset; + bool is_gmem_mmap; + size_t alignment; + int mmap_flags; TEST_REQUIRE_SET_USER_MEMORY_REGION2(); @@ -1090,19 +1091,31 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, /* Allocate and initialize new mem region structure. */ region = calloc(1, sizeof(*region)); TEST_ASSERT(region != NULL, "Insufficient Memory"); - region->mmap_size = mem_size; - /* - * When using THP mmap is not guaranteed to returned a hugepage aligned - * address so we have to pad the mmap. Padding is not needed for HugeTLB - * because mmap will always return an address aligned to the HugeTLB - * page size. - */ - if (src_type == VM_MEM_SRC_ANONYMOUS_THP) - alignment = max(backing_src_pagesz, alignment); + is_gmem_mmap = (flags & KVM_MEM_GUEST_MEMFD) && + (gmem_flags & GUEST_MEMFD_FLAG_MMAP); + + if (is_gmem_mmap) { + backing_src_pagesz = getpagesize(); + alignment = 1; + mmap_flags = MAP_SHARED; + mmap_offset = gmem_offset; + } else { + backing_src_pagesz = get_backing_src_pagesz(src_type); + /* + * When using THP mmap is not guaranteed to returned a hugepage aligned + * address so we have to pad the mmap. Padding is not needed for HugeTLB + * because mmap will always return an address aligned to the HugeTLB + * page size. + */ + alignment = src_type == VM_MEM_SRC_ANONYMOUS_THP ? backing_src_pagesz : 1; + mmap_flags = vm_mem_backing_src_alias(src_type)->flag; + mmap_offset = 0; + } TEST_ASSERT_EQ(gpa, align_up(gpa, backing_src_pagesz)); + region->mmap_size = mem_size; /* Add enough memory to align up if necessary */ if (alignment > 1) region->mmap_size += alignment; @@ -1129,10 +1142,8 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, } region->fd = -1; - if (flags & KVM_MEM_GUEST_MEMFD && gmem_flags & GUEST_MEMFD_FLAG_MMAP) { + if (is_gmem_mmap) { region->fd = kvm_dup(gmem_fd); - mmap_flags = MAP_SHARED; - mmap_offset = gmem_offset; } else if (backing_src_is_shared(src_type)) { region->fd = kvm_memfd_alloc(region->mmap_size, src_type == VM_MEM_SRC_SHARED_HUGETLB); @@ -1141,22 +1152,27 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, region->mmap_start = __kvm_mmap(region->mmap_size, PROT_READ | PROT_WRITE, mmap_flags, region->fd, mmap_offset); - TEST_ASSERT(!is_backing_src_hugetlb(src_type) || - region->mmap_start == align_ptr_up(region->mmap_start, backing_src_pagesz), - "mmap_start %p is not aligned to HugeTLB page size 0x%lx", - region->mmap_start, backing_src_pagesz); - /* Align host address */ region->host_mem = align_ptr_up(region->mmap_start, alignment); - /* As needed perform madvise */ - if ((src_type == VM_MEM_SRC_ANONYMOUS || - src_type == VM_MEM_SRC_ANONYMOUS_THP) && thp_configured()) { - ret = madvise(region->host_mem, mem_size, - src_type == VM_MEM_SRC_ANONYMOUS ? MADV_NOHUGEPAGE : MADV_HUGEPAGE); - TEST_ASSERT(ret == 0, "madvise failed, addr: %p length: 0x%lx src_type: %s", - region->host_mem, mem_size, - vm_mem_backing_src_alias(src_type)->name); + if (!is_gmem_mmap) { + TEST_ASSERT(!is_backing_src_hugetlb(src_type) || + region->mmap_start == + align_ptr_up(region->mmap_start, backing_src_pagesz), + "mmap_start %p is not aligned to HugeTLB page size 0x%lx", + region->mmap_start, backing_src_pagesz); + + /* As needed perform madvise */ + if ((src_type == VM_MEM_SRC_ANONYMOUS || + src_type == VM_MEM_SRC_ANONYMOUS_THP) && thp_configured()) { + int advice = src_type == VM_MEM_SRC_ANONYMOUS ? + MADV_NOHUGEPAGE : MADV_HUGEPAGE; + + ret = madvise(region->host_mem, mem_size, advice); + TEST_ASSERT(ret == 0, "madvise failed, addr: %p length: 0x%lx src_type: %s", + region->host_mem, mem_size, + vm_mem_backing_src_alias(src_type)->name); + } } region->backing_src_type = src_type; From e24e534ff850b51520a45699cc9c7c1e2396c67d Mon Sep 17 00:00:00 2001 From: Ackerley Tng Date: Thu, 10 Sep 2026 16:56:09 -0700 Subject: [PATCH 56/60] KVM: selftests: Update private_mem_conversions_test for in-place conversions Update private_mem_conversions_test for in-place conversions. In-place conversions support is detected in selftests with kvm_has_gmem_attributes. With in-place conversions, specifying userspace_addr from some other memory provider that is not the guest_memfd associated with the memslot is a user error, since KVM will only use both shared and private memory from the guest_memfd. Hence, when kvm_has_gmem_attributes, only test in-place conversions with single backing, where guest_memfd provides both shared and private memory. For single backing, guest_memfd must be created with GUEST_MEMFD_FLAG_MMAP. Initialize the guest_memfd as shared to align with how memory would default to shared when shared/private state was tracked at the VM level (the test expects this, it was written for non-in-place-conversions). When handling a hypercall to set attributes, use vm_mem_set_memory_attributes() to send the ioctl to the guest_memfd instead of the VM. When testing in-place conversions (single-backing), don't allow the user to configure src_type, since src_type will be ignored. Don't use src_type to determine alignment for rounding up per-cpu test memory size, since guest_memfd's backing page size is always the system PAGE_SIZE. Co-developed-by: Sean Christopherson Signed-off-by: Ackerley Tng Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-43-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/private_mem_conversions_test.c | 57 ++++++++++++++----- 1 file changed, 44 insertions(+), 13 deletions(-) diff --git a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c index cf50e9a332c5..33b610383693 100644 --- a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c @@ -306,9 +306,12 @@ static void handle_exit_hypercall(struct kvm_vcpu *vcpu) if (do_fallocate) vm_guest_mem_fallocate(vm, gpa, size, map_shared); - if (set_attributes) - vm_set_memory_attributes(vm, gpa, size, - map_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE); + if (set_attributes) { + u64 attrs = map_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE; + + vm_mem_set_memory_attributes(vm, gpa, size, attrs); + } + run->hypercall.ret = 0; } @@ -352,8 +355,21 @@ static void *__test_mem_conversions(void *__vcpu) size_t nr_bytes = min_t(size_t, vm->page_size, size - i); u8 *hva = addr_gpa2hva(vm, gpa + i); - /* In all cases, the host should observe the shared data. */ - memcmp_h(hva, gpa + i, uc.args[3], nr_bytes); + if (kvm_has_gmem_attributes && + uc.args[0] == SYNC_PRIVATE) { + TEST_EXPECT_SIGBUS(READ_ONCE(*hva)); + } else { + /* + * If not testing in-place conversion + * (dual backing), the host should + * always observe shared data, since the + * shared memory is a separate page. + * + * For SYNC_SHARED, test that the host + * can see shared memory. + */ + memcmp_h(hva, gpa + i, uc.args[3], nr_bytes); + } /* For shared, write the new pattern to guest memory. */ if (uc.args[0] == SYNC_SHARED) @@ -369,20 +385,29 @@ static void *__test_mem_conversions(void *__vcpu) } } +/* Align each vCPU's chunk of memory naturally to the size of the backing store. */ +static size_t compute_per_cpu_size(enum vm_mem_backing_src_type src_type) +{ + size_t alignment; + + if (kvm_has_gmem_attributes) + alignment = getpagesize(); + else + alignment = get_backing_src_pagesz(src_type); + + return align_up(PER_CPU_DATA_SIZE, max_t(size_t, SZ_2M, alignment)); +} + static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_vcpus, u32 nr_memslots) { - /* - * Allocate enough memory so that each vCPU's chunk of memory can be - * naturally aligned with respect to the size of the backing store. - */ - const size_t alignment = max_t(size_t, SZ_2M, get_backing_src_pagesz(src_type)); - const size_t per_cpu_size = align_up(PER_CPU_DATA_SIZE, alignment); + const size_t per_cpu_size = compute_per_cpu_size(src_type); const size_t memfd_size = per_cpu_size * nr_vcpus; const size_t slot_size = memfd_size / nr_memslots; struct kvm_vcpu *vcpus[KVM_MAX_VCPUS]; pthread_t threads[KVM_MAX_VCPUS]; struct kvm_vm *vm; + u64 gmem_flags; int memfd, i; const struct vm_shape shape = { @@ -397,12 +422,16 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v vm_enable_cap(vm, KVM_CAP_EXIT_HYPERCALL, (1 << KVM_HC_MAP_GPA_RANGE)); - memfd = vm_create_guest_memfd(vm, memfd_size, 0); + gmem_flags = 0; + if (kvm_has_gmem_attributes) + gmem_flags = GUEST_MEMFD_FLAG_MMAP | GUEST_MEMFD_FLAG_INIT_SHARED; + + memfd = vm_create_guest_memfd(vm, memfd_size, gmem_flags); for (i = 0; i < nr_memslots; i++) vm_mem_add(vm, src_type, BASE_DATA_GPA + slot_size * i, BASE_DATA_SLOT + i, slot_size / vm->page_size, - KVM_MEM_GUEST_MEMFD, memfd, slot_size * i, 0); + KVM_MEM_GUEST_MEMFD, memfd, slot_size * i, gmem_flags); for (i = 0; i < nr_vcpus; i++) { gpa_t gpa = BASE_DATA_GPA + i * per_cpu_size; @@ -462,6 +491,8 @@ int main(int argc, char *argv[]) while ((opt = getopt(argc, argv, "hm:s:n:")) != -1) { switch (opt) { case 's': + TEST_ASSERT(!kvm_has_gmem_attributes, + "src_type is only configurable when testing without in-place conversion"); src_type = parse_backing_src_type(optarg); break; case 'n': From 37e0791600e5f1e2837a270c885296a172f5825e Mon Sep 17 00:00:00 2001 From: Sean Christopherson Date: Thu, 10 Sep 2026 16:56:10 -0700 Subject: [PATCH 57/60] KVM: selftests: Update private memory exits test to work with per-gmem attributes Skip setting memory to private in the private memory exits test when using per-gmem memory attributes, as memory is initialized to private by default for guest_memfd, and using vm_mem_set_private() on a guest_memfd instance requires creating guest_memfd with GUEST_MEMFD_FLAG_MMAP (which is totally doable, but would need to be conditional and is ultimately unnecessary). Expect an emulated MMIO instead of a memory fault exit when attributes are per-gmem, as deleting the memslot effectively drops the private status, i.e. the GPA becomes shared and thus supports emulated MMIO. Skip the "memslot not private" test entirely, as private vs. shared state for x86 software-protected VMs comes from the memory attributes themselves, and so when doing in-place conversions there can never be a disconnect between the expected and actual states. Signed-off-by: Ackerley Tng Tested-by: Shivank Garg Reviewed-by: Fuad Tabba Tested-by: Yan Zhao Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-44-dd6fbf94f4e1@google.com Signed-off-by: Sean Christopherson --- .../kvm/x86/private_mem_kvm_exits_test.c | 36 +++++++++++++++---- 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c b/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c index e6d16e9d6120..f7228e0f1ac6 100644 --- a/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c +++ b/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c @@ -62,8 +62,9 @@ static void test_private_access_memslot_deleted(void) virt_map(vm, EXITS_TEST_GVA, EXITS_TEST_GPA, EXITS_TEST_NPAGES); - /* Request to access page privately */ - vm_mem_set_private(vm, EXITS_TEST_GPA, EXITS_TEST_SIZE); + /* Request to access page privately. */ + if (!kvm_has_gmem_attributes) + vm_mem_set_private(vm, EXITS_TEST_GPA, EXITS_TEST_SIZE); kvm_pthread_create(&vm_thread, NULL, (pthread_fn_t)run_vcpu_get_exit_reason, (void *)vcpu); @@ -73,10 +74,26 @@ static void test_private_access_memslot_deleted(void) kvm_pthread_join(vm_thread, &thread_return); exit_reason = (u32)(u64)thread_return; - TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MEMORY_FAULT); - TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, KVM_MEMORY_EXIT_FLAG_PRIVATE); - TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, EXITS_TEST_GPA); - TEST_ASSERT_EQ(vcpu->run->memory_fault.size, EXITS_TEST_SIZE); + /* + * If attributes are tracked per-gmem, deleting the memslot that points + * at the gmem instance effectively makes the memory shared, and so the + * read should trigger emulated MMIO. + * + * If attributes are tracked per-VM, deleting the memslot shouldn't + * affect the private attribute, and so KVM should generate a memory + * fault exit (emulated MMIO on private GPAs is disallowed). + */ + if (kvm_has_gmem_attributes) { + TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MMIO); + TEST_ASSERT_EQ(vcpu->run->mmio.phys_addr, EXITS_TEST_GPA); + TEST_ASSERT_EQ(vcpu->run->mmio.len, sizeof(u64)); + TEST_ASSERT_EQ(vcpu->run->mmio.is_write, false); + } else { + TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MEMORY_FAULT); + TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, KVM_MEMORY_EXIT_FLAG_PRIVATE); + TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, EXITS_TEST_GPA); + TEST_ASSERT_EQ(vcpu->run->memory_fault.size, EXITS_TEST_SIZE); + } kvm_vm_free(vm); } @@ -87,6 +104,13 @@ static void test_private_access_memslot_not_private(void) struct kvm_vcpu *vcpu; u32 exit_reason; + /* + * Accessing non-private memory as private with a software-protected VM + * isn't possible when doing in-place conversions. + */ + if (kvm_has_gmem_attributes) + return; + vm = vm_create_shape_with_one_vcpu(protected_vm_shape, &vcpu, guest_repeatedly_read); From 681f3b68f4356f223ad32e1fa2253b2929750685 Mon Sep 17 00:00:00 2001 From: Fuad Tabba Date: Mon, 28 Sep 2026 19:02:50 +0100 Subject: [PATCH 58/60] KVM: guest_memfd: Fix the in-place conversion documentation Commit dcde2f853ff46 ("KVM: guest_memfd: Ensure pages are not in use before conversion") added error_offset to struct kvm_memory_attributes2, written back on failure, and a -EAGAIN return. The KVM_SET_MEMORY_ATTRIBUTES2 section of api.rst still lists the old struct, marks it input-only and leaves EAGAIN out of the Errors table. It also refers to KVM_CAP_GUEST_MEMFD_MMAP, which had become KVM_CAP_GUEST_MEMFD_FLAGS before this ioctl was added. kvm.gmem_in_place_conversion's entry says in-place conversion is unconditionally enabled without CONFIG_KVM_VM_MEMORY_ATTRIBUTES, but a build without kvm_arch_has_private_mem has it disabled: arm64 always, and x86 without CONFIG_KVM_SW_PROTECTED_VM, CONFIG_KVM_INTEL_TDX or CONFIG_KVM_AMD_SEV. Bring both in line with the code: replace the stale capability with GUEST_MEMFD_FLAG_MMAP, and describe KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES the way the KVM_CREATE_GUEST_MEMFD section describes KVM_CAP_GUEST_MEMFD_FLAGS. Fixes: 799c9fadc6399 ("KVM: guest_memfd: Add base support for KVM_SET_MEMORY_ATTRIBUTES2") Fixes: dcde2f853ff46 ("KVM: guest_memfd: Ensure pages are not in use before conversion") Fixes: 537ec2b15fdd7 ("KVM: Let userspace disable per-VM mem attributes, enable per-gmem attributes") Signed-off-by: Fuad Tabba Link: https://patch.msgid.link/20260928180250.3075584-1-fuad.tabba@linux.dev Signed-off-by: Sean Christopherson --- Documentation/admin-guide/kernel-parameters.txt | 3 ++- Documentation/virt/kvm/api.rst | 14 ++++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 49560b1b54c4..27e1f05c44c0 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3177,7 +3177,8 @@ Kernel parameters Note, this parameter is only available when CONFIG_KVM_VM_MEMORY_ATTRIBUTES=y. When CONFIG_KVM_VM_MEMORY_ATTRIBUTES is not set, in-place - conversion is unconditionally enabled. + conversion is enabled (and cannot be disabled) if KVM + is built with private memory support. Default is N (off). diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 67f0f290797a..3d6a4bf01c70 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6690,7 +6690,7 @@ significant bit): :Capability: KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES :Architectures: all :Type: guest_memfd ioctl -:Parameters: struct kvm_memory_attributes2 (in) +:Parameters: struct kvm_memory_attributes2 (in/out) :Returns: 0 on success, <0 on error Errors: @@ -6700,6 +6700,8 @@ Errors: page aligned, causes an overflow, or size is zero). EFAULT The parameter address was invalid. ENOMEM Ran out of memory trying to track private/shared state + EAGAIN Pages in the range have outstanding references, see + ``error_offset`` below. ========== =============================================================== KVM_SET_MEMORY_ATTRIBUTES2 is an extension to @@ -6719,15 +6721,19 @@ Attribute values are shared with KVM_SET_MEMORY_ATTRIBUTES. __u64 size; __u64 attributes; __u64 flags; - __u64 reserved[12]; + __u64 error_offset; + __u64 reserved[11]; }; #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) +The capability KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES enumerates the attributes +that can be set via KVM_SET_MEMORY_ATTRIBUTES2. + Set attributes for a range of offsets within a guest_memfd to KVM_MEMORY_ATTRIBUTE_PRIVATE to limit the specified guest_memfd backed -memory range for guest use. Even if KVM_CAP_GUEST_MEMFD_MMAP is -supported, after a successful call to set +memory range for guest use. Even if the guest_memfd was created with +GUEST_MEMFD_FLAG_MMAP, after a successful call to set KVM_MEMORY_ATTRIBUTE_PRIVATE, the requested range will not be mappable into host userspace and will only be mappable by the guest. From ad06cf5338bc5501bd989584db6dc71972bb436a Mon Sep 17 00:00:00 2001 From: Hao Zhang Date: Mon, 27 Jul 2026 10:37:41 +0800 Subject: [PATCH 59/60] KVM: guest_memfd: Avoid deadlock in error_remove_folio() memory_failure() invokes ->error_remove_folio() while holding the poisoned folio lock. KVM's guest_memfd callback then takes mapping->invalidate_lock for read before invalidating guest mappings. This can deadlock with paths that hold mapping->invalidate_lock for write and then try to lock the same folio, e.g. truncation from MADV_REMOVE or FALLOC_FL_PUNCH_HOLE. Because rwsems are fair, the problem is not limited to guest_memfd punch-hole: a pending writer can also block future readers, so ->error_remove_folio() must not block on the invalidate lock while holding the poisoned folio lock. Use filemap_invalidate_trylock_shared() in kvm_gmem_error_folio(). If the invalidate lock is contended, fail the recovery instead of potentially deadlocking the memory-failure path. Fixes: a7800aa80ea4 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory") Signed-off-by: Hao Zhang Link: https://lore.kernel.org/all/tencent_7DA064CAC9B463EB9B651801011E2CE4230A@qq.com/ Link: https://patch.msgid.link/ambEdSPjerZIVN0b@192.168.1.215 Signed-off-by: Sean Christopherson --- virt/kvm/guest_memfd.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index a13445c26d9d..93f3d29d5751 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -828,7 +828,17 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol enum kvm_gfn_range_filter filter; pgoff_t start, end; - filemap_invalidate_lock_shared(mapping); + /* + * memory_failure() invokes ->error_remove_folio() while holding the + * poisoned folio lock. Do not block on mapping->invalidate_lock, as + * a writer that already holds the invalidate lock can be waiting for + * the same folio, e.g. truncation from MADV_REMOVE/FALLOC_FL_PUNCH_HOLE. + * + * If the invalidate lock is contended, fail the recovery instead of + * risking a deadlock in the memory-failure path. + */ + if (!filemap_invalidate_trylock_shared(mapping)) + return -EBUSY; start = folio->index; end = start + folio_nr_pages(folio); From a6270ed4bec0caa25913082e71ce22254e9cd3ed Mon Sep 17 00:00:00 2001 From: Lisa Wang Date: Thu, 1 Oct 2026 19:37:40 +0000 Subject: [PATCH 60/60] KVM: selftests: Support guest_memfd in-place conversion With guest_memfd in-place conversion, the memory region can be set as shared, eliminating the need for dual-backed memory. By setting the region to shared initially, the KVM selftest framework can populate the mmap()-ed guest_memfd memory just like any standard memory provider. For CoCo VMs, pages that need to be private are explicitly set to private before executing the VM. Signed-off-by: Lisa Wang Link: https://patch.msgid.link/20261001-tdx-selftests-v15-13-7c62a5d8a992@google.com [sean: anchor comment to its relevant place, drop unnecessary code changes] Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/lib/kvm_util.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 3f559f207408..0c67d964a9aa 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -494,6 +494,7 @@ struct kvm_vm *__vm_create(struct vm_shape shape, u32 nr_runnable_vcpus, u64 nr_pages = vm_nr_pages_required(shape.mode, nr_runnable_vcpus, nr_extra_pages); struct userspace_mem_region *slot0; + u64 gmem_flags = 0; struct kvm_vm *vm; int i, flags; @@ -509,10 +510,20 @@ struct kvm_vm *__vm_create(struct vm_shape shape, u32 nr_runnable_vcpus, * for CoCo VMs that require GUEST_MEMFD backed private memory. */ flags = 0; - if (is_guest_memfd_required(shape)) + if (is_guest_memfd_required(shape)) { flags |= KVM_MEM_GUEST_MEMFD; - vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, 0, 0, nr_pages, flags); + /* + * Allow mmap() and initialize the gmem instance as SHARED if + * in-place conversions are supported, so that the memory can + * be populated/initialized like any other memory region. + */ + if (kvm_has_gmem_attributes) + gmem_flags |= GUEST_MEMFD_FLAG_INIT_SHARED | + GUEST_MEMFD_FLAG_MMAP; + } + + vm_mem_add(vm, VM_MEM_SRC_ANONYMOUS, 0, 0, nr_pages, flags, -1, 0, gmem_flags); for (i = 0; i < NR_MEM_REGIONS; i++) vm->memslots[i] = 0;