kfd_dev_mapping caches the address_space of the first /dev/kfd opener
so that the GPU reset path can call unmap_mapping_range() to zap all
userspace mappings of doorbell and MMIO ranges. This design has two
bugs that both manifest under SRIOV with multiple containers:
1. Use-after-free / rwsem deadlock. The cached pointer refers to an
inode owned by the first opener's container. When that container
exits and its inode is released, kfd_dev_mapping becomes a dangling
pointer. A subsequent GPU reset dereferences it inside
unmap_mapping_range(), which takes i_mmap_rwsem on the freed inode,
causing a hard hang observable as an uninterruptible rwsem wait.
2. Multi-container gap. Only the first opener's address_space is cached;
VMAs created by later openers live in a different address_space and
are never reached by unmap_mapping_range(). After a GPU reset those
stale mappings keep doorbell and MMIO pages accessible to guest
userspace with no GPU behind them, risking PCIe transaction timeouts
and NMI panics.
Fix both bugs with the same approach used by DRM core (drm_drv.c):
create a private pseudo-filesystem at module init time and allocate one
anonymous inode from it. In kfd_open() redirect every opener's
file->f_mapping to that inode's address_space. The inode is
module-owned, lives exactly as long as the amdgpu module, and collects
VMAs from all openers in one address_space. A single
unmap_mapping_range() call in the reset path then correctly reaches
every container's mappings with no dangling pointer risk.
The hang manifests as an NMI backtrace on the GPU reset workqueue stuck
spinning in rwsem_down_read_slowpath() with a corrupted i_mmap_rwsem:
Workqueue: amdgpu-reset-dev xgpu_ai_mailbox_flr_work [amdgpu]
Call Trace:
<TASK>
kvm_wait+0x1f/0x40
__pv_queued_spin_lock_slowpath+0x31d/0x3a0
_raw_spin_lock_irq+0x51/0x80
rwsem_down_read_slowpath+0xb3/0x550
down_read+0x48/0xd0
unmap_mapping_range+0x71/0x140
kfd_dev_unmap_mapping_range+0x5b/0x140 [amdgpu]
amdgpu_amdkfd_clear_kfd_mapping+0xd8/0x190 [amdgpu]
amdgpu_device_gpu_recover+0x232/0x450 [amdgpu]
xgpu_ai_mailbox_flr_work+0xb5/0xc0 [amdgpu]
process_one_work+0x18e/0x3e0
worker_thread+0x2e3/0x420
kthread+0x10a/0x230
Fixes: 70cadefcc6 ("drm/amdgpu: unmap all user mappings of framebuffer and doorbell before mode1 reset")
Signed-off-by: Asad Kamal <asad.kamal@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 1128b4a52de1572e87431de837fd9850cb99542c)
Cc: stable@vger.kernel.org
The low 6 bits of cp_hqd_eop_control store the base-2 logarithm
of the EOP ring size. This was calculated as
order_base_2(q->eop_ring_buffer_size / 4) - 1
But order_base_2 can in theory return 0, so this could underflow
(although in practice the ring buffer size cannot be less than 4096).
Change this to
order_base_2(q->eop_ring_buffer_size / 8)
using properties of logarithms.
Also add to the above comment to make the mathematics more clear.
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit f0f43fcf8b2b3a924cad9444340921c96ed5f634)
Cc: stable@vger.kernel.org
The low 6 bits of cp_hqd_eop_control store the base-2 logarithm
of the EOP ring size. This was calculated as
ffs(q->eop_ring_buffer_size / sizeof(unsigned int)) - 1 - 1
But ffs can in theory return 1 or 0, so this could underflow
(although in practice the ring buffer size cannot be less than 4096).
Change this to
ffs(q->eop_ring_buffer_size / sizeof(unsigned int) / 4)
using properties of logarithms.
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 4f18c56630383c14bfc6b2d65f88f2f895d2121a)
Cc: stable@vger.kernel.org
kfd_mqd_manager_v12.c (GFX 12.0) and kfd_mqd_manager_v12_1.c (GFX 12.1)
do not implement restore_mqd callbacks, leaving the function pointers
NULL and causing CRIU restore to return -EOPNOTSUPP on GFX12.
Implement restore_mqd for both compute and SDMA queues in
kfd_mqd_manager_v12.c and kfd_mqd_manager_v12_1.c, modeled after the
GFX 11 implementation with the following improvements:
- update cp_mqd_base_addr_lo/hi to the newly allocated MQD address,
fixing a pre-existing gap shared with v11 where the in-MQD copy
still pointed at the old checkpoint-time address after restore
- memset the full allocation before memcpy for compute queues to avoid
stale data in the GTT sub-allocator tail; SDMA MQDs use sizeof(*m)
since they are packed at mqd_size stride in a shared BO
checkpoint_mqd registration is deferred to a follow-up patch that also
implements get_checkpoint_info, so that checkpoint and restore are
enabled together as a complete and testable unit.
Note: GFX12.1 restore handles XCC0 only. Multi-XCC CRIU restore is
currently unreachable due to a separate validation issue in
kfd_criu_restore_queue(). A pr_warn_once() is emitted if a multi-XCC
device is encountered.
Signed-off-by: Vladimir Marioukhine <Vladimir.Marioukhine@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit b1f9601237d050f5df478464cf51bf1fff29a256)
Cc: stable@vger.kernel.org
- Run scripts/coccinelle/api/kmalloc_objs.cocci for v7.3
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRSPkdeREjth1dHnSE2KwveOeQkuwUCapuwVwAKCRA2KwveOeQk
u5EeAP9TS7K4iVlw3KlZHuLIK2q+CQfALPepcu+ME2lO5dta4gEAxCTi0ZXmU7OT
XbmWUd+DTkKNYCBW8E6Lvn72Er13uQs=
=ZtN4
-----END PGP SIGNATURE-----
Merge tag 'kmalloc_obj-v7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux
Pull kmalloc_obj conversions from Kees Cook:
"Another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules
in scripts/coccinelle/api/kmalloc_objs.cocci"
* tag 'kmalloc_obj-v7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux:
treewide: refresh kmalloc_obj() conversions
drm/amd/display: Fix harmless type mismatch in allocation
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
Reading /sys/kernel/debug/kfd/mqds while a process holds an active KFD
queue triggers a NULL pointer dereference because the for loop that
calls mqd_mgr->debugfs_show_mqd() is incorrectly placed outside the
if (pqn->q) block that initializes mqd_mgr.
The queue list can contain entries where pqn->q is NULL (kernel queues
where only pqn->kq is valid). In the original code:
if (pqn->q) {
...
mqd_mgr = q->device->dqm->mqd_mgrs[mqd_type];
size = mqd_mgr->mqd_stride(...);
}
for (xcc = 0; xcc < num_xccs; xcc++) { // WRONG: outside if block
mqd = q->mqd + size * xcc;
r = mqd_mgr->debugfs_show_mqd(m, mqd);
}
When iterating over a queue node where pqn->q is NULL:
1. The if (pqn->q) block is skipped
2. mqd_mgr remains uninitialized (NULL from declaration)
3. The for loop executes anyway
4. mqd_mgr->debugfs_show_mqd(m, mqd) dereferences NULL
The crash manifests as:
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor instruction fetch in kernel mode
RIP: 0010:0x0
Call Trace:
pqm_debugfs_mqds+0x10c/0x1d0 [amdgpu]
kfd_debugfs_mqds_by_process+0x9b/0x110 [amdgpu]
seq_read_iter+0x132/0x4b0
...
Fix by moving the for loop inside the if (pqn->q) block, so mqd_mgr
and related variables are only used when properly initialized.
Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5689
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Link: https://patch.msgid.link/20260831130051.2031435-1-mario.limonciello@amd.com
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 8bfe29d5c798940f797aa24135d2734c3ffce9de)
Cc: stable@vger.kernel.org
MES (Micro Engine Scheduler) does not perform heavy-weight TLB
invalidation after unmapping queues, unlike HWS which does this
automatically. This causes a race condition where in-flight DMA
descriptors can access memory that has been unmapped, leading to page
faults and GPU queue hangs during SVM page migration.
The issue manifests as KFDSVMRangeTest.MultiThreadMigrationTest
failures on gfx1151 (Strix Point) with XNACK mode 1 enabled - the GPU
compute queue hangs with packets submitted but never consumed.
Add kfd_flush_tlb() calls after MES queue removal in two locations:
- evict_process_queues_cpsch(): after all queues removed during eviction
- suspend_queues(): after debug/criu queue suspension (with mem_fence barrier)
This ensures all in-flight memory accesses from unmapped queues are
flushed before memory is freed or migrated.
Signed-off-by: Priya Hosur <Priya.Hosur@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit f5c4f88e0f9c45a8fb9dfac0c1df726c95e41b77)
Cc: stable@vger.kernel.org
Both create_queue_cpsch() and create_queue_nocpsch() unconditionally
call mqd_mgr->restore_mqd() when a CRIU restore is in progress
(qd != NULL), with no NULL guard. On any system where restore_mqd is
not implemented for the given queue type, a user holding
CAP_CHECKPOINT_RESTORE can trigger a kernel NULL pointer dereference
and panic the machine by issuing KFD_IOC_CRIU_OP_RESTORE with a
crafted queue restore object. Note that checkpoint_mqd is likewise
unimplemented on GFX12, so no legitimate CRIU image can reach this
path — only a hand-crafted restore payload.
Add a NULL guard for restore_mqd immediately after mqd_mgr is
resolved, unwinding via the existing error labels and returning
-EOPNOTSUPP if the callback is not implemented. This mirrors the
existing checkpoint_mqd guard in checkpoint_mqd().
Fixes: 48f0bdf4e3 ("drm/amdkfd: Added MQD manager files for GFX12.")
Cc: stable@vger.kernel.org
Signed-off-by: Vladimir Marioukhine <Vladimir.Marioukhine@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
When migration vm range is hole at cpu side(MIGRATE_PFN_MIGRATE set +
MIGRATE_PFN_VALID unset) driver still allocates device pages. There is no
dma map of src pages and migration. j is 0 and svm_migrate_copy_memory_gart()
will return an uninitialized r. That can trigger out_free_vram_pages to drop
all VRAM just set up.
Initialize r and only call the last svm_migrate_copy_memory_gart if j > 0.
Current code postponed the last page to the final copy. This patch flushes on
the last page when reach to the end of current drm_buddy_block; avoids another
svm_migrate_copy_memory_gart.
Cc: stable@vger.kernel.org
Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
If page migration from device to sys ram fails for some reasons driver needs
release and unlock allocated system pages. To do that driver should use page
physical address, or pfn, then get struct page*. Current driver uses dma
address(for adev) that is not correct with IOMMU enabled, or even in general.
The patch releases and unlocks allocated system pages based on where migration
failed by struct page* of sys ram pages. Also dma_unmap correspodent system
ram pages at error path.
Cc: stable@vger.kernel.org
Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
KFD_IOC_ALLOC_MEMORY_OF_GPU with flag
KFD_IOC_ALLOC_MEM_FLAGS_AQL_QUEUE_MEM and size=1 triggers the AQL
wraparound workaround (size >>= 1), reducing size to 0. The resulting
zero passes through PAGE_ALIGN(0) = 0 without validation, bypassing the
per-process VRAM quota check in reserve_mem_limit()
(vram_used + 0 > vram_available is always false).
The fix adds post-halving zero-size validation in the primary
allocation path (amdgpu_amdkfd_gpuvm.c). The check happens after size
halving but before reserve_mem_limit(), and uses err_alignment_size
error path to properly clean up the allocated kgd_mem structure and
mutex.
Cc: stable@vger.kernel.org
Signed-off-by: Sunday Clement <Sunday.Clement@amd.com>
Reviewed-by: Alex Deucher <Alexander.Deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
profile_lock_device() may return negive error code,
so the type of the return value should be int,
not uint32
Signed-off-by: Zhu Lingshan <lingshan.zhu@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
On gfx9 ASICs with mqd_on_vram(), a compute queue MQD lives in a pinned
VRAM buffer object. Pinned BOs are skipped by the VRAM eviction done at S4
suspend, so the MQD contents are lost across hibernation and the first
submission after resume page-faults on a stale MQD.
Unpin the MQD BO at suspend so the eviction migrates it into the
hibernation image, and pin it back to VRAM on resume. The BO may return at
a different VRAM address, so refresh the kernel mapping and cached GPU
addresses and patch the MQD self-address via a new update_mqd_gpu_addr()
mqd_manager op; skip eviction with a warning if that op is not implemented.
v3: use unpin/repin instead of shadowing the MQD into a separate buffer.
v4: drop the explicit VRAM->GTT placement at evict (a bare unpin is enough
for the eviction pass to move the BO out of VRAM), and also repin at queue
destroy. KFD queue restore runs late - user processes thaw before it, and
under SR-IOV it is deferred until the VF exits full access - so once the
VM has resumed an application can destroy a queue before its MQD BO is
repinned, which would otherwise unpin an already-unpinned BO and touch a
stale q->mqd.
v5: drop support for no-HWS mode, and set q->mqd to NULL at eviction.
Signed-off-by: Shikang Fan <shikang.fan@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Enabled RS64mem for KFD queues by integrating
process and gang context index allocation in
the per KFD device process and queue creation.
Signed-off-by: Prike Liang <Prike.Liang@amd.com>
Reviewed-by: Michael Chen <michael.chen@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
When KFD_IOCTL_SVM_ATTR_NO_ACCESS is applied to a GPU that has an SVM
range mapped, unmap the range from that GPU. Once no GPU maps the range,
the MMU notifier can skip queue eviction on CPU page faults.
Replace the mapped_to_gpu boolean with bitmap_mapped to track which
GPUs currently have the range mapped. Set bits in svm_range_map_to_gpus()
and clear them in svm_range_unmap_from_gpus(). This is separate from
bitmap_access/bitmap_aip which track user-requested attributes and must
not be used to determine mapping state.
Add bitmap_needs_unmap to svm_range, set for each GPU given no-access.
Add svm_range_needs_unmap() to unmap the range from those GPUs when the
app sets the no-access attribute.
Bump the checkpoint timestamp on unmap so retry faults queued before the
no-access unmap are dropped instead of restoring the mapping.
v4:
- Rename and set prange->mapping_done to false if validate and map not
complete successfully (Felix)
v3:
- Correct error handling, support app retry update mapping (Felix)
v2:
- Add bitmap_mapped to not break get_attr (Felix)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Extract svm_range_update_checkpoint_timestamp() from
svm_range_unmap_from_cpu(). The next patch calls it when the app sets
the no-access attribute.
Change checkpoint_ts in svm_range_list from uint64_t to atomic64_t so
svm_range_restore_pages() can read it from the page fault handler
without holding the svms lock.
No functional change, preparation for the next patch.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
SVM BOs are now migrated back to system memory synchronously from the TTM
eviction path (svm_range_evict_svm_bo), so the per-svm_bo eviction fence
is no longer used.
Remove the eviction fence from svm_range_bo, drop the
amdgpu_amdkfd_fence->svm_bo back pointer and the
amdgpu_amdkfd_evict_svm_bo() helper, and stop special-casing svm_bo
fences in the KFD fence enable_signaling and check_mm paths. Embed struct
amdgpu_bo directly in svm_range_bo with a dedicated svm_range_bo_destroy()
callback, and keep the owning mm via mmgrab()/mmdrop() instead of through
the fence.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
svm_range_evict_svm_bo_worker() migrated an SVM BO's pages back to system
memory from a work item that took mmap_read_lock. When an mmap writer was
pending, that read lock blocked behind the writer while the thread
allocating a new migration VRAM BO waited on this BO's eviction fence - a
circular wait that hung the SVM workers.
Evict the SVM BO synchronously from the TTM eviction path
(amdgpu_ttm_bo_eviction_valuable) instead of deferring to a work item.
The BO is already reserved and the lock order is mmap_lock -> BO
reservation, so only trylock the owning process's mmap lock; on
contention return -EBUSY so TTM skips this BO. This removes the eviction
work item and the enable_signaling path, so no worker can block on
mmap_read_lock.
The SVM BO uses AMDGPU_GEM_CREATE_DISCARDABLE, so ttm_bo_evict takes the
pipeline_gutting path and skips allocating a system memory placement.
That would be wasted work, since svm_migrate_vram_to_ram allocates the
system pages and copies the data back itself.
Eviction now migrates ranges directly, so it must serialize with the
owning process: it trylocks migrate_mutex under svm_bo->list_lock before
unlinking the range, and svm_range_free() unlinks the range then waits on
migrate_mutex, so a concurrent eviction cannot free a range under it.
Drop the mm reference with mmput_async so exit_mmap() does not run under
the BO reservation.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
When a mode reset happens, driver needs to notify the process on that
GPU a reset event is happening. The existing code assumes the process is
using the GPU that is getting mode reset, which is not always true. For
example, on a 8G system, the process may be only using GPU 0~4 but a
mode 2 reset is resetting the all 8 GPUs connected by XGMI. Trying to
find a process on GPU 5~7 will fail, which is fine and should skip the
event signal.
Signed-off-by: Amber Lin <amber.lin@amd.com>
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The debugger sets the CWSR grace period via AMDKFD_IOC_DBG_TRAP and may
request a value of 0. An earlier change clamped 0 to 1 unconditionally
because most firmware revisions locked up when given an infinite (0)
grace period.
Firmware has since been fixed on most ASICs, so gate the clamp on ASIC
type and MEC firmware version and allow 0 where the running firmware is
known to handle it. Navi3x and MI350 support 0 in every firmware
revision and need no version check. MI100 never received the firmware
fix and is kept clamped. Any unlisted or future ASIC defaults to the
safe (clamped) behaviour.
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
kfd_criu_checkpoint_events() counts the entries in p->event_idr via
kfd_get_num_events(), allocates an array sized to that count, and then
walks the same IDR to fill it. Neither the count nor the walk holds
p->event_mutex.
The CRIU checkpoint caller holds only p->mutex. Event create and destroy
(kfd_event_create()/kfd_event_destroy()) take p->event_mutex and do not
take p->mutex, so a second thread in the same process can insert or remove
events between the count and the walk. If an event is inserted, the walk
iterates more entries than were counted and writes past the end of the
ev_privs allocation; if an event is removed, the walk dereferences an
entry that is being freed.
Hold p->event_mutex across the count and the walk so both observe a
consistent view of p->event_idr. The lock is released before
copy_to_user(), which only touches the local buffer. The caller already
holds p->mutex and the create/destroy paths never take p->mutex, so the
p->mutex -> p->event_mutex order is not inverted and no deadlock is
introduced.
Fixes: 40e8a766a7 ("drm/amdkfd: CRIU checkpoint and restore events")
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Alysa Liu <Alysa.Liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Convert ptl_hw_supported to enum with three states:
- AMDGPU_PTL_HW_UNINIT: not yet initialized
- AMDGPU_PTL_HW_SUPPORTED: initialized and supported
- AMDGPU_PTL_HW_NOT_SUPPORTED: initialized and not supported
This allows skipping PTL initialization attempts when hardware is
known to not support it, avoiding repeated initialization failures
after GPU resets.
v2:move ptl_hw_supported_state to AMDGPU_PTL_HW_NOT_SUPPORTED regardless
of error code during first time initialization. Print init fail log when
error code is not EOPNOTSUPP.
Signed-off-by: Victor Zhao <Victor.Zhao@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
eop_ring_buffer_size in struct queue_properties is a u32. In
kfd_queue_acquire_buffers() the expected EOP buffer size is computed as
ALIGN(eop_ring_buffer_size, PAGE_SIZE); ALIGN uses typeof(x), so the
addition is done in 32-bit. A user-supplied size of 0xFFFFF001 wraps to
0, causing kfd_queue_buffer_get() to skip its exact-size check (gated on
size != 0) and accept any BO mapped at the address. On GFX8/GFX9 the MQD
cp_hqd_eop_control is then programmed for an 8KB EOP ring backed by a 4KB
BO, so CP EOP writes can land past the buffer and fault the GPU.
Cast the operand to u64 so the alignment is computed in 64-bit; the size
check in kfd_queue_buffer_get() then rejects the oversized request.
Fixes: 42ea9cf2f1 ("drm/amdkfd: Relax size checking during queue buffer get")
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Alysa Liu <Alysa.Liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Prevent unauthorized termination of active GPU debug sessions.
Previously, users with /dev/kfd access could terminate another process's
debug session without proper ownership or ptrace authorization.
Signed-off-by: Gang Ba <Gang.Ba@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
In kfd_criu_restore_event, there was no handling for
the event priv data having an invalid event type. The priv
data here is untrusted and can be invalid.
In that case, fail with EINVAL.
Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
When MES is enabled and amdgpu_amdkfd_alloc_kernel_mem() fails during
the first queue creation for a process, pqm_create_queue() returns
early via 'return retval' without going through the err_create_queue
cleanup label.
This means clear_bit(*qid, pqm->queue_slot_bitmap) is never called,
leaving the reserved QID bit permanently set in queue_slot_bitmap.
Over time this leaks QID slots, potentially exhausting all available
queue slots.
Fix this by replacing 'return retval' with 'goto err_allocate_pqn'
so that clear_bit() is always called on the error path without
touching the uninitialized pqn pointer.
AILIKFD-813
Reported-by: Deucher, Alexander <alexander.deucher@amd.com>
Signed-off-by: Vladimir Marioukhine <Vladimir.Marioukhine@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The CRAT parser validates that the subtype header fits within the image,
but does not verify that the advertised subtype length fits. A malformed
CRAT table with an oversized length field causes out-of-bounds reads when
kfd_parse_subtype() casts the header to specific subtype structures.
Add validation that sub_type_hdr + length does not exceed the image
boundary before parsing the subtype contents.
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Alysa Liu <Alysa.Liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Move the job output parameter to be the last parameter in the
amdgpu_job_alloc() and amdgpu_job_alloc_with_ib() function signature. This
aligns with the common kernel coding convention where output parameters
typically come last.
Assisted-by: Claude:Sonnet 4
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
We need to increment the reset counter, force fence completion,
and set the wedged event when a user queue is reset, but only
for the guilty queue. We don't want additional events for
collateral damage.
Only increment the reset counter and set the wedged event
in the top level userq reset callers (both KGD and KFD).
This aligns with how this is managed for kernel queue resets.
v2: fix task info refcount handling, align with
kfd error reporting.
v3: check for has_reset_queue in KFD path (Prike)
Cc: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Cc: Prike Liang <Prike.Liang@amd.com>
Cc: Sunil Khatri <sunil.khatri@amd.com>
Cc: Lazar, Lijo <lijo.lazar@amd.com>
Reviewed-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Prike Liang <Prike.Liang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The change referenced by the Fixes tag releases the HIQ SDMA MQD trunk
buffer when device_queue_manager_init() fails after it has been
allocated.
However, the same failure path can also be reached after
init_mqd_managers() has succeeded. At that point dqm->mqd_mgrs[] contains
per-type MQD manager objects owned by the device queue manager. The
normal teardown path frees those objects from uninitialize(), but the
initialization error path only frees dqm itself.
Free the MQD managers from the initialization error path as well. This is
safe for earlier failures because dqm is zeroed when allocated and
init_mqd_managers() clears the entries it rolls back internally.
Fixes: b7cccc8286 ("drm/amdkfd: fix a memory leak in device_queue_manager_init()")
Signed-off-by: Guangshuo Li <lgs201920130244@gmail.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
debugfs is intended for debugging only, and failures to create debugfs
entries should not affect normal operation.
Remove the check for debugfs_create_dir() in kfd_debugfs_add_process().
If debugfs entries cannot be created, continue without them instead of
reporting an unnecessary error.
Fixes: 325de6bf33 ("drm/amdkfd: expose pasid of secondary contexts by debugfs")
Reported-by: Dan Carpenter <error27@gmail.com>
Cc: Zhu Lingshan <lingshan.zhu@amd.com>
Cc: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
total_cwsr_size was computed in 32-bit before being used as a BO/SVM
allocation size.
With large ctx_save_restore_area_size and debug_memory_size
multiplied by the XCC count, the product can wrap,
yielding an undersized CWSR save area that firmware later overruns.
Promote total_cwsr_size to u64 and use check_add_overflow()/
check_mul_overflow() in both kfd_queue_acquire_buffers() and
kfd_queue_release_buffers().
Signed-off-by: Yongqiang Sun <Yongqiang.Sun@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The valid amdkfd event ids go from 0 to KFD_SIGNAL_EVENT_LIMIT - 1.
During CRIU restore, ensure that the provided event ids are
in that range.
v2: No need for lower bound check since idr_alloc rejects negative
inputs
v3: Also change error message to reflect new error condition
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Current kfd debugfs interfaces only expose pasid
of the primary process, this commit exposes
pasid of secondary contexts by debugfs
Just like entries under sysfs,
the secondary contexts are named as
context_<id> under its primary kfd process.
The layout:
/sys/kernel/debug/kfd/proc# tree
.
└── 5802
├── context_0
│ ├── pasid_1025
│ └── pasid_63266
├── context_1
│ ├── pasid_1025
│ └── pasid_63266
├── pasid_1025
└── pasid_63266
Another fix is, kfd_debugfs_add_process may fail,
this commit change it to return a meaningful
value other than void
Signed-off-by: Zhu Lingshan <lingshan.zhu@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The valid event ids go from 0 to KFD_SIGNAL_EVENT_LIMIT
allocate_event_notification_slot has an option to specify
an event id to allocate at, used by CRIU. We weren't checking
the bounds on that value.
Check them.
v2: Lower bounds check is unecessary because of idr_alloc
already rejecting negative numbers. Upper bounds check should
be KFD_SIGNAL_EVENT_LIMIT since the signal mode mappings might
not yet exist
Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Function kfd_process_free_id() should skip over
the primary kfd process because its context id
is fixed assigned, not allocated through the ida table.
This function should only work on secondary contexts.
Fixes: fac682a1d1 ("amdkfd: identify a secondary kfd process by its id")
Signed-off-by: Zhu Lingshan <lingshan.zhu@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
kfd_criu_restore_queue's call of kfd_queue_acquire_buffers was
failing for multiple reasons
- The ctl_stack_size set by the CRIU plugin doesn't match
what is expected by acquire_buffers
- The svm buffer cannot be acquired at this point because
CRIU may not have restored it, or may have restored it
to a different address.
The only reason acquire_buffers was necessary here was to
avoid a null ptr dereference in init_user_queue.
Just put in a check for that dereference; it doesn't appear to
come up in real use cases right now. That is, there is no
usage of CRIU with shared MES.
This is a partial revert of
commit 20a5e7ffdf ("drm/amdkfd: Properly acquire queue buffers in CRIU restore")
Fixes: 20a5e7ffdf ("drm/amdkfd: Properly acquire queue buffers in CRIU restore")
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
We weren't checking whether the values provided in the private
data in kfd CRIU restore were within bounds.
For queue type, add a KFD_QUEUE_TYPE_MAX and ensure the provided
type is less than it.
For mqd_size, add new function mqd_size_from_queue_type and confirm
that the provided mqd_size matches expectations.
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
After moving TBA/TMA from GTT to VRAM for GFX9.4.2+ in commit
5088a1ba6d6d ("drm/amdkfd: move TBA/TMA from system to device memory"),
direct pointer dereferences to CWSR buffers became unsafe because VRAM
is accessed via MMIO (PCI BAR mappings).
Direct writes like 'tma[2] = enabled' and memcpy() can fail or
produce incorrect results on non-x86 architectures because:
- MMIO requires specific accessor functions (writeq/readq)
- Compiler optimizations may generate invalid instruction sequences
- No guarantee of proper memory barriers or atomic access
This patch converts CWSR buffer access to use struct iosys_map,
which automatically handles both system memory (GTT) and MMIO
(VRAM) correctly by:
- Using writeq/writel/memcpy_toio for MMIO regions
- Using WRITE_ONCE/memcpy for system memory
- Providing proper memory barriers and access guarantees
Changes:
- Replace void *cwsr_kaddr with struct iosys_map cwsr_map
- Detect MMIO vs system memory using TTM_BO_MAP_IOMEM_MASK
- Use iosys_map_wr() for writing trap handler addresses and flags
- Use iosys_map_memcpy_to() for copying CWSR ISA code
This ensures correct operation on all architectures while maintaining
backward compatibility with older GPUs and APUs that use GTT.
Assisted-by: Claude:Claude-Opus-4.6
Signed-off-by: James Zhu <James.Zhu@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Duplicated code from the merge. Drop it.
Fixes: 0461ba9a79 ("Merge tag 'amd-drm-next-7.3-2026-07-02' of https://gitlab.freedesktop.org/agd5f/linux into drm-next")
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Running RCCL unit tests on a system with a 64K PAGE_SIZE triggers
the following warning and causes the test to terminate on latest
upstream kernel:
WARNING: drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:1335 at
amdgpu_bo_release_notify+0x1bc/0x280 [amdgpu],
CPU#18: rccl-UnitTests/33151
Call trace:
amdgpu_bo_release_notify
ttm_bo_release
amdgpu_gem_object_free
drm_gem_object_free
amdgpu_bo_unref
amdgpu_bo_create
amdgpu_bo_create_user
amdgpu_gem_object_create
amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu
kfd_ioctl_alloc_memory_of_gpu
kfd_ioctl
sys_ioctl
The warning is triggered because
amdgpu_ttm_next_clear_entity() returns NULL when a clear buffer
operation is requested. This happens because the GART window
allocation for the default_entity, clear_entity and move_entity
fails during initialization.
Commit [1] introduced separate GART windows for the
default_entity, clear_entity and move_entity of each SDMA
instance. Their sizes are derived from
AMDGPU_GTT_MAX_TRANSFER_SIZE, which is currently defined as 1024
pages. This implicitly assumes a 4K PAGE_SIZE, where 1024 pages
correspond to a 4MB transfer. On a 64K PAGE_SIZE system, however,
the same value expands to 64MB.
The default_entity and clear_entity each allocate one
AMDGPU_GTT_MAX_TRANSFER_SIZE GART window, while the move_entity
allocates two such windows. This results in 16MB of GART space
per SDMA instance on a 4K PAGE_SIZE system, but 256MB per SDMA
instance on a 64K PAGE_SIZE system.
On an MI210 system with five SDMA instances and a 512MB GART
aperture, the total GART space required becomes 1.25GB,
exceeding the available GART aperture. Consequently, GART window
allocation fails, amdgpu_ttm_next_clear_entity() returns NULL,
and the above warning is triggered.
Redefine AMDGPU_GTT_MAX_TRANSFER_SIZE in bytes instead of page
units. Where a page count is required, convert it using
PAGE_SHIFT. This preserves the existing 4MB transfer size across
all PAGE_SIZE configurations while keeping GART window
allocations within the available GART aperture.
[1] https://lore.kernel.org/all/20260408100327.1372-3-pierre-eric.pelloux-prayer@amd.com/#t
Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5435
Fixes: 897ee11ec0 ("drm/amdgpu: create multiple clear/move ttm entities")
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 27213b776a)
Cc: stable@vger.kernel.org
There were a few instances in kfd_chardev.c of kvzalloc being
used to allocate memory for an array.
Switch those to kvcalloc, which
- is the standard way of allocating a zero-initialized array
- does a check for the mul overflowing
Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 60b048c93f)
Cc: stable@vger.kernel.org
CRIU checkpoint copies the MQD control stack using cp_hqd_cntl_stack_size
from hardware without bounding it to the allocated BO region. If the HW
field is larger than the queue's control stack allocation, memcpy reads
past the BO into adjacent GTT memory and can leak kernel data to userspace.
Store the page-aligned control stack BO size in mqd_manager and clamp
checkpoint copies and reported checkpoint sizes to
min(cp_hqd_cntl_stack_size, mm->ctl_stack_size). Apply the same bound
for multi-XCC v9.4.3 checkpoint layout.
Signed-off-by: Yongqiang Sun <Yongqiang.Sun@amd.com>
Reviewed-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 6c2abd0ec0)
Cc: stable@vger.kernel.org
