github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-06-02 19:43:40 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

UAPI Changes:

Browse Source 
- Drop L3 bank mask reporting from the media GT on Xe3 and later. Only
    do that for the primary GT. No userspace needs or uses it for media
    and some platforms may report bogus values.
  - Add SLPC power_profile sysfs interface with support for base and
    power_saving modes (Vinay Belgaumkar, Rodrigo Vivi)
  - Add configfs attributes to add post/mid context-switch commands
    (Lucas De Marchi)
 
 Cross-subsystem Changes:
  - Fix hmm_pfn_to_map_order() usage in gpusvm and refactor APIs to
    align with pieces previous handled by xe_hmm (Matthew Auld)
 
 Core Changes:
  - Add MEI driver for Late Binding Firmware Update/Upload
    (Alexander Usyskin)
 
 Driver Changes:
  - Fix GuC CT teardown wrt TLB invalidation (Satyanarayana)
  - Fix CCS save/restore on VF (Satyanarayana)
  - Increase default GuC crash buffer size (Zhanjun)
  - Allow to clear GT stats in debugfs to aid debugging (Matthew Brost)
  - Add more SVM GT stats to debugfs (Matthew Brost)
  - Fix error handling in VMA attr query (Himal)
  - Move sa_info in debugfs to be per tile (Michal Wajdeczko)
  - Limit number of retries upon receiving NO_RESPONSE_RETRY from GuC to
    avoid endless loop (Michal Wajdeczko)
  - Fix configfs handling for survivability_mode undoing user choice when
    unbinding the module (Michal Wajdeczko)
  - Refactor configfs attribute visibility to future-proof it and stop
    exposing survivability_mode if not applicable (Michal Wajdeczko)
  - Constify some functions (Harish Chegondi, Michal Wajdeczko)
  - Add/extend more HW workarounds for Xe2 and Xe3
    (Harish Chegondi, Tangudu Tilak Tirumalesh)
  - Replace xe_hmm with gpusvm (Matthew Auld)
  - Improve fake pci and WA kunit handling for testing new platforms
    (Michal Wajdeczko)
  - Reduce unnecessary PTE writes when migrating (Sanjay Yadav)
  - Cleanup GuC interface definitions and log message (John Harrison)
  - Small improvements around VF CCS (Michal Wajdeczko)
  - Enable bus mastering for the I2C controller (Raag Jadav)
  - Prefer devm_mutex of hand rolling it (Christophe JAILLET)
  - Drop sysfs and debugfs attributes not available for VF (Michal Wajdeczko)
  - GuC CT devm actions improvements (Michal Wajdeczko)
  - Recommend new GuC versions for PTL and BMG (Julia Filipchuk)
  - Improveme driver handling for exhaustive eviction using new
    xe_validation wrapper around drm_exec (Thomas Hellström)
  - Add and use printk wrappers for tile and device (Michal Wajdeczko)
  - Better document workaround handling in Xe (Lucas De Marchi)
  - Improvements on ARRAY_SIZE  and ERR_CAST usage (Lucas De Marchi,
    Fushuai Wang)
  - Align CSS firmware headers with the GuC APIs (John Harrison)
  - Test GuC to GuC (G2G) communication to aid debug in pre-production
    firmware (John Harrison)
  - Bail out driver probing if GuC fails to load (John Harrison)
  - Allow error injection in xe_pxp_exec_queue_add()
    (Daniele Ceraolo Spurio)
  - Minor refactors in xe_svm (Shuicheng Lin)
  - Fix madvise ioctl error handling (Shuicheng Lin)
  - Use attribute groups to simplify sysfs registration
    (Michal Wajdeczko)
  - Add Late Binding Firmware implementation in Xe to work together with
    the MEI component (Badal Nilawar, Daniele Ceraolo Spurio, Rodrigo
    Vivi)
  - Fix build with CONFIG_MODULES=n (Lucas De Marchi)
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEE6rM8lpABPHM5FqyDm6KlpjDL6lMFAmjNZ8AACgkQm6KlpjDL
 6lOiug/+L7kz6+diug2n6HjM0USxw9V9x3YuguSDErPief6c5ew2xMej72gEM9F6
 VfPKvt6e3VsQ0nhhBhA0aZMzLgAD+FOxx9NzBu8QTFNCLENuSEBqfCdQ8pHhFLx4
 CX4iU98pr11UPSnNQ8DKpegn0vEyBbIcT0Yxq1rm/rbF5l2vphh6AQw0ffE8RoaW
 3721KFBReOVowARJUQkf9dUOSf9HowybhAJWXy5JSqNN1MlqWUfYGxkuS4NFGRz3
 BBA0z+5JBqNKShVXOJbW+z04X/u08/VC6vCw9W7sOxEvfgTV2X1/xHzIFOwuU/hD
 oUypLAjhXDdjvOBuXWc7C4+IlNCmoSZiubU/hCBsObElxX7VbkYna345r83Qv7K8
 mCwjL3zjFgpWwvNblNL23TRQQJdMcXSlwuecnYelX2G0dPxZw17py1wMRVs7oVXQ
 xdMnM8MOLlFzaSgNtWKoJcPgBrCLpkvtV3yNOa4kL8IekkiWHun7k2ruQIOb7ux+
 kxWP62QyfJUAJA5pj5LPkym+6LYMcCDmz+n0wzvt5Nm8HSGX+YQXvmlHjwAmIhXP
 USBKYSJrijtDoji9ng9eAD+Mkm0cH/NQeqbmHMsAhb9tatROzZGsH9lBDi5Kqxxt
 2UyIOFM7K/oZ5PGGxTvvp62i0Psu/c9biTyAcrL/3TUGXHScDJ0=
 =2NXV
 -----END PGP SIGNATURE-----

Merge tag 'drm-xe-next-2025-09-19' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-next

UAPI Changes:
 - Drop L3 bank mask reporting from the media GT on Xe3 and later. Only
   do that for the primary GT. No userspace needs or uses it for media
   and some platforms may report bogus values.
 - Add SLPC power_profile sysfs interface with support for base and
   power_saving modes (Vinay Belgaumkar, Rodrigo Vivi)
 - Add configfs attributes to add post/mid context-switch commands
   (Lucas De Marchi)

Cross-subsystem Changes:
 - Fix hmm_pfn_to_map_order() usage in gpusvm and refactor APIs to
   align with pieces previous handled by xe_hmm (Matthew Auld)

Core Changes:
 - Add MEI driver for Late Binding Firmware Update/Upload
   (Alexander Usyskin)

Driver Changes:
 - Fix GuC CT teardown wrt TLB invalidation (Satyanarayana)
 - Fix CCS save/restore on VF (Satyanarayana)
 - Increase default GuC crash buffer size (Zhanjun)
 - Allow to clear GT stats in debugfs to aid debugging (Matthew Brost)
 - Add more SVM GT stats to debugfs (Matthew Brost)
 - Fix error handling in VMA attr query (Himal)
 - Move sa_info in debugfs to be per tile (Michal Wajdeczko)
 - Limit number of retries upon receiving NO_RESPONSE_RETRY from GuC to
   avoid endless loop (Michal Wajdeczko)
 - Fix configfs handling for survivability_mode undoing user choice when
   unbinding the module (Michal Wajdeczko)
 - Refactor configfs attribute visibility to future-proof it and stop
   exposing survivability_mode if not applicable (Michal Wajdeczko)
 - Constify some functions (Harish Chegondi, Michal Wajdeczko)
 - Add/extend more HW workarounds for Xe2 and Xe3
   (Harish Chegondi, Tangudu Tilak Tirumalesh)
 - Replace xe_hmm with gpusvm (Matthew Auld)
 - Improve fake pci and WA kunit handling for testing new platforms
   (Michal Wajdeczko)
 - Reduce unnecessary PTE writes when migrating (Sanjay Yadav)
 - Cleanup GuC interface definitions and log message (John Harrison)
 - Small improvements around VF CCS (Michal Wajdeczko)
 - Enable bus mastering for the I2C controller (Raag Jadav)
 - Prefer devm_mutex of hand rolling it (Christophe JAILLET)
 - Drop sysfs and debugfs attributes not available for VF (Michal Wajdeczko)
 - GuC CT devm actions improvements (Michal Wajdeczko)
 - Recommend new GuC versions for PTL and BMG (Julia Filipchuk)
 - Improveme driver handling for exhaustive eviction using new
   xe_validation wrapper around drm_exec (Thomas Hellström)
 - Add and use printk wrappers for tile and device (Michal Wajdeczko)
 - Better document workaround handling in Xe (Lucas De Marchi)
 - Improvements on ARRAY_SIZE  and ERR_CAST usage (Lucas De Marchi,
   Fushuai Wang)
 - Align CSS firmware headers with the GuC APIs (John Harrison)
 - Test GuC to GuC (G2G) communication to aid debug in pre-production
   firmware (John Harrison)
 - Bail out driver probing if GuC fails to load (John Harrison)
 - Allow error injection in xe_pxp_exec_queue_add()
   (Daniele Ceraolo Spurio)
 - Minor refactors in xe_svm (Shuicheng Lin)
 - Fix madvise ioctl error handling (Shuicheng Lin)
 - Use attribute groups to simplify sysfs registration
   (Michal Wajdeczko)
 - Add Late Binding Firmware implementation in Xe to work together with
   the MEI component (Badal Nilawar, Daniele Ceraolo Spurio, Rodrigo
   Vivi)
 - Fix build with CONFIG_MODULES=n (Lucas De Marchi)

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Lucas De Marchi <lucas.demarchi@intel.com>
Link: https://lore.kernel.org/r/c2et6dnkst2apsgt46dklej4nprqdukjosb55grpaknf3pvcxy@t7gtn3hqtp6n
This commit is contained in:
Dave Airlie 2025-09-22 07:42:05 +10:00
commit 0faeb8cf99
133 changed files with 6884 additions and 2355 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

297
drivers/gpu/drm/drm_gpusvm.c
View File

@ -373,6 +373,12 @@ static const struct mmu_interval_notifier_ops drm_gpusvm_notifier_ops = {
*
* This function initializes the GPU SVM.
*
* Note: If only using the simple drm_gpusvm_pages API (get/unmap/free),
* then only @gpusvm, @name, and @drm are expected. However, the same base
* @gpusvm can also be used with both modes together in which case the full
* setup is needed, where the core drm_gpusvm_pages API will simply never use
* the other fields.
*
* Return: 0 on success, a negative error code on failure.
*/
int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
@ -383,8 +389,16 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
const struct drm_gpusvm_ops *ops,
const unsigned long *chunk_sizes, int num_chunks)
{
if (!ops->invalidate || !num_chunks)
return -EINVAL;
if (mm) {
if (!ops->invalidate || !num_chunks)
return -EINVAL;
mmgrab(mm);
} else {
/* No full SVM mode, only core drm_gpusvm_pages API. */
if (ops || num_chunks || mm_range || notifier_size ||
device_private_page_owner)
return -EINVAL;
}
gpusvm->name = name;
gpusvm->drm = drm;
@ -397,7 +411,6 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
gpusvm->chunk_sizes = chunk_sizes;
gpusvm->num_chunks = num_chunks;
mmgrab(mm);
gpusvm->root = RB_ROOT_CACHED;
INIT_LIST_HEAD(&gpusvm->notifier_list);
@ -489,7 +502,8 @@ void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
drm_gpusvm_range_remove(gpusvm, range);
}
mmdrop(gpusvm->mm);
if (gpusvm->mm)
mmdrop(gpusvm->mm);
WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
}
EXPORT_SYMBOL_GPL(drm_gpusvm_fini);
@ -629,12 +643,41 @@ drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
range->itree.start = ALIGN_DOWN(fault_addr, chunk_size);
range->itree.last = ALIGN(fault_addr + 1, chunk_size) - 1;
INIT_LIST_HEAD(&range->entry);
range->notifier_seq = LONG_MAX;
range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
range->pages.notifier_seq = LONG_MAX;
range->pages.flags.migrate_devmem = migrate_devmem ? 1 : 0;
return range;
}
/**
* drm_gpusvm_hmm_pfn_to_order() - Get the largest CPU mapping order.
* @hmm_pfn: The current hmm_pfn.
* @hmm_pfn_index: Index of the @hmm_pfn within the pfn array.
* @npages: Number of pages within the pfn array i.e the hmm range size.
*
* To allow skipping PFNs with the same flags (like when they belong to
* the same huge PTE) when looping over the pfn array, take a given a hmm_pfn,
* and return the largest order that will fit inside the CPU PTE, but also
* crucially accounting for the original hmm range boundaries.
*
* Return: The largest order that will safely fit within the size of the hmm_pfn
* CPU PTE.
*/
static unsigned int drm_gpusvm_hmm_pfn_to_order(unsigned long hmm_pfn,
unsigned long hmm_pfn_index,
unsigned long npages)
{
unsigned long size;
size = 1UL << hmm_pfn_to_map_order(hmm_pfn);
size -= (hmm_pfn & ~HMM_PFN_FLAGS) & (size - 1);
hmm_pfn_index += size;
if (hmm_pfn_index > npages)
size -= (hmm_pfn_index - npages);
return ilog2(size);
}
/**
* drm_gpusvm_check_pages() - Check pages
* @gpusvm: Pointer to the GPU SVM structure
@ -693,7 +736,7 @@ static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
err = -EFAULT;
goto err_free;
}
i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
i += 0x1 << drm_gpusvm_hmm_pfn_to_order(pfns[i], i, npages);
}
err_free:
@ -951,31 +994,31 @@ drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
EXPORT_SYMBOL_GPL(drm_gpusvm_range_find_or_insert);
/**
* __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range (internal)
* __drm_gpusvm_unmap_pages() - Unmap pages associated with GPU SVM pages (internal)
* @gpusvm: Pointer to the GPU SVM structure
* @range: Pointer to the GPU SVM range structure
* @svm_pages: Pointer to the GPU SVM pages structure
* @npages: Number of pages to unmap
*
* This function unmap pages associated with a GPU SVM range. Assumes and
* This function unmap pages associated with a GPU SVM pages struct. Assumes and
* asserts correct locking is in place when called.
*/
static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_range *range,
unsigned long npages)
static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_pages *svm_pages,
unsigned long npages)
{
unsigned long i, j;
struct drm_pagemap *dpagemap = range->dpagemap;
struct drm_pagemap *dpagemap = svm_pages->dpagemap;
struct device *dev = gpusvm->drm->dev;
unsigned long i, j;
lockdep_assert_held(&gpusvm->notifier_lock);
if (range->flags.has_dma_mapping) {
struct drm_gpusvm_range_flags flags = {
.__flags = range->flags.__flags,
if (svm_pages->flags.has_dma_mapping) {
struct drm_gpusvm_pages_flags flags = {
.__flags = svm_pages->flags.__flags,
};
for (i = 0, j = 0; i < npages; j++) {
struct drm_pagemap_addr *addr = &range->dma_addr[j];
struct drm_pagemap_addr *addr = &svm_pages->dma_addr[j];
if (addr->proto == DRM_INTERCONNECT_SYSTEM)
dma_unmap_page(dev,
@ -991,30 +1034,51 @@ static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
/* WRITE_ONCE pairs with READ_ONCE for opportunistic checks */
flags.has_devmem_pages = false;
flags.has_dma_mapping = false;
WRITE_ONCE(range->flags.__flags, flags.__flags);
WRITE_ONCE(svm_pages->flags.__flags, flags.__flags);
range->dpagemap = NULL;
svm_pages->dpagemap = NULL;
}
}
/**
* drm_gpusvm_range_free_pages() - Free pages associated with a GPU SVM range
* __drm_gpusvm_free_pages() - Free dma array associated with GPU SVM pages
* @gpusvm: Pointer to the GPU SVM structure
* @range: Pointer to the GPU SVM range structure
* @svm_pages: Pointer to the GPU SVM pages structure
*
* This function frees the dma address array associated with a GPU SVM range.
*/
static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_range *range)
static void __drm_gpusvm_free_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_pages *svm_pages)
{
lockdep_assert_held(&gpusvm->notifier_lock);
if (range->dma_addr) {
kvfree(range->dma_addr);
range->dma_addr = NULL;
if (svm_pages->dma_addr) {
kvfree(svm_pages->dma_addr);
svm_pages->dma_addr = NULL;
}
}
/**
* drm_gpusvm_free_pages() - Free dma-mapping associated with GPU SVM pages
* struct
* @gpusvm: Pointer to the GPU SVM structure
* @svm_pages: Pointer to the GPU SVM pages structure
* @npages: Number of mapped pages
*
* This function unmaps and frees the dma address array associated with a GPU
* SVM pages struct.
*/
void drm_gpusvm_free_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_pages *svm_pages,
unsigned long npages)
{
drm_gpusvm_notifier_lock(gpusvm);
__drm_gpusvm_unmap_pages(gpusvm, svm_pages, npages);
__drm_gpusvm_free_pages(gpusvm, svm_pages);
drm_gpusvm_notifier_unlock(gpusvm);
}
EXPORT_SYMBOL_GPL(drm_gpusvm_free_pages);
/**
* drm_gpusvm_range_remove() - Remove GPU SVM range
* @gpusvm: Pointer to the GPU SVM structure
@ -1040,8 +1104,8 @@ void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
return;
drm_gpusvm_notifier_lock(gpusvm);
__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
drm_gpusvm_range_free_pages(gpusvm, range);
__drm_gpusvm_unmap_pages(gpusvm, &range->pages, npages);
__drm_gpusvm_free_pages(gpusvm, &range->pages);
__drm_gpusvm_range_remove(notifier, range);
drm_gpusvm_notifier_unlock(gpusvm);
@ -1106,6 +1170,28 @@ void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
}
EXPORT_SYMBOL_GPL(drm_gpusvm_range_put);
/**
* drm_gpusvm_pages_valid() - GPU SVM range pages valid
* @gpusvm: Pointer to the GPU SVM structure
* @svm_pages: Pointer to the GPU SVM pages structure
*
* This function determines if a GPU SVM range pages are valid. Expected be
* called holding gpusvm->notifier_lock and as the last step before committing a
* GPU binding. This is akin to a notifier seqno check in the HMM documentation
* but due to wider notifiers (i.e., notifiers which span multiple ranges) this
* function is required for finer grained checking (i.e., per range) if pages
* are valid.
*
* Return: True if GPU SVM range has valid pages, False otherwise
*/
static bool drm_gpusvm_pages_valid(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_pages *svm_pages)
{
lockdep_assert_held(&gpusvm->notifier_lock);
return svm_pages->flags.has_devmem_pages || svm_pages->flags.has_dma_mapping;
}
/**
* drm_gpusvm_range_pages_valid() - GPU SVM range pages valid
* @gpusvm: Pointer to the GPU SVM structure
@ -1123,9 +1209,7 @@ EXPORT_SYMBOL_GPL(drm_gpusvm_range_put);
bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_range *range)
{
lockdep_assert_held(&gpusvm->notifier_lock);
return range->flags.has_devmem_pages || range->flags.has_dma_mapping;
return drm_gpusvm_pages_valid(gpusvm, &range->pages);
}
EXPORT_SYMBOL_GPL(drm_gpusvm_range_pages_valid);
@ -1139,66 +1223,71 @@ EXPORT_SYMBOL_GPL(drm_gpusvm_range_pages_valid);
*
* Return: True if GPU SVM range has valid pages, False otherwise
*/
static bool
drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_range *range)
static bool drm_gpusvm_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_pages *svm_pages)
{
bool pages_valid;
if (!range->dma_addr)
if (!svm_pages->dma_addr)
return false;
drm_gpusvm_notifier_lock(gpusvm);
pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range);
pages_valid = drm_gpusvm_pages_valid(gpusvm, svm_pages);
if (!pages_valid)
drm_gpusvm_range_free_pages(gpusvm, range);
__drm_gpusvm_free_pages(gpusvm, svm_pages);
drm_gpusvm_notifier_unlock(gpusvm);
return pages_valid;
}
/**
* drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range
* drm_gpusvm_get_pages() - Get pages and populate GPU SVM pages struct
* @gpusvm: Pointer to the GPU SVM structure
* @range: Pointer to the GPU SVM range structure
* @svm_pages: The SVM pages to populate. This will contain the dma-addresses
* @mm: The mm corresponding to the CPU range
* @notifier: The corresponding notifier for the given CPU range
* @pages_start: Start CPU address for the pages
* @pages_end: End CPU address for the pages (exclusive)
* @ctx: GPU SVM context
*
* This function gets pages for a GPU SVM range and ensures they are mapped for
* DMA access.
* This function gets and maps pages for CPU range and ensures they are
* mapped for DMA access.
*
* Return: 0 on success, negative error code on failure.
*/
int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_range *range,
const struct drm_gpusvm_ctx *ctx)
int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_pages *svm_pages,
struct mm_struct *mm,
struct mmu_interval_notifier *notifier,
unsigned long pages_start, unsigned long pages_end,
const struct drm_gpusvm_ctx *ctx)
{
struct mmu_interval_notifier *notifier = &range->notifier->notifier;
struct hmm_range hmm_range = {
.default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only ? 0 :
HMM_PFN_REQ_WRITE),
.notifier = notifier,
.start = drm_gpusvm_range_start(range),
.end = drm_gpusvm_range_end(range),
.start = pages_start,
.end = pages_end,
.dev_private_owner = gpusvm->device_private_page_owner,
};
struct mm_struct *mm = gpusvm->mm;
void *zdd;
unsigned long timeout =
jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
unsigned long i, j;
unsigned long npages = npages_in_range(drm_gpusvm_range_start(range),
drm_gpusvm_range_end(range));
unsigned long npages = npages_in_range(pages_start, pages_end);
unsigned long num_dma_mapped;
unsigned int order = 0;
unsigned long *pfns;
int err = 0;
struct dev_pagemap *pagemap;
struct drm_pagemap *dpagemap;
struct drm_gpusvm_range_flags flags;
struct drm_gpusvm_pages_flags flags;
enum dma_data_direction dma_dir = ctx->read_only ? DMA_TO_DEVICE :
DMA_BIDIRECTIONAL;
retry:
hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range))
if (drm_gpusvm_pages_valid_unlocked(gpusvm, svm_pages))
goto set_seqno;
pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
@ -1238,7 +1327,7 @@ int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
*/
drm_gpusvm_notifier_lock(gpusvm);
flags.__flags = range->flags.__flags;
flags.__flags = svm_pages->flags.__flags;
if (flags.unmapped) {
drm_gpusvm_notifier_unlock(gpusvm);
err = -EFAULT;
@ -1251,13 +1340,12 @@ int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
goto retry;
}
if (!range->dma_addr) {
if (!svm_pages->dma_addr) {
/* Unlock and restart mapping to allocate memory. */
drm_gpusvm_notifier_unlock(gpusvm);
range->dma_addr = kvmalloc_array(npages,
sizeof(*range->dma_addr),
GFP_KERNEL);
if (!range->dma_addr) {
svm_pages->dma_addr =
kvmalloc_array(npages, sizeof(*svm_pages->dma_addr), GFP_KERNEL);
if (!svm_pages->dma_addr) {
err = -ENOMEM;
goto err_free;
}
@ -1270,7 +1358,7 @@ int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
for (i = 0, j = 0; i < npages; ++j) {
struct page *page = hmm_pfn_to_page(pfns[i]);
order = hmm_pfn_to_map_order(pfns[i]);
order = drm_gpusvm_hmm_pfn_to_order(pfns[i], i, npages);
if (is_device_private_page(page) ||
is_device_coherent_page(page)) {
if (zdd != page->zone_device_data && i > 0) {
@ -1296,13 +1384,13 @@ int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
goto err_unmap;
}
}
range->dma_addr[j] =
svm_pages->dma_addr[j] =
dpagemap->ops->device_map(dpagemap,
gpusvm->drm->dev,
page, order,
DMA_BIDIRECTIONAL);
dma_dir);
if (dma_mapping_error(gpusvm->drm->dev,
range->dma_addr[j].addr)) {
svm_pages->dma_addr[j].addr)) {
err = -EFAULT;
goto err_unmap;
}
@ -1322,15 +1410,15 @@ int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
addr = dma_map_page(gpusvm->drm->dev,
page, 0,
PAGE_SIZE << order,
DMA_BIDIRECTIONAL);
dma_dir);
if (dma_mapping_error(gpusvm->drm->dev, addr)) {
err = -EFAULT;
goto err_unmap;
}
range->dma_addr[j] = drm_pagemap_addr_encode
svm_pages->dma_addr[j] = drm_pagemap_addr_encode
(addr, DRM_INTERCONNECT_SYSTEM, order,
DMA_BIDIRECTIONAL);
dma_dir);
}
i += 1 << order;
num_dma_mapped = i;
@ -1339,21 +1427,21 @@ int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
if (pagemap) {
flags.has_devmem_pages = true;
range->dpagemap = dpagemap;
svm_pages->dpagemap = dpagemap;
}
/* WRITE_ONCE pairs with READ_ONCE for opportunistic checks */
WRITE_ONCE(range->flags.__flags, flags.__flags);
WRITE_ONCE(svm_pages->flags.__flags, flags.__flags);
drm_gpusvm_notifier_unlock(gpusvm);
kvfree(pfns);
set_seqno:
range->notifier_seq = hmm_range.notifier_seq;
svm_pages->notifier_seq = hmm_range.notifier_seq;
return 0;
err_unmap:
__drm_gpusvm_range_unmap_pages(gpusvm, range, num_dma_mapped);
__drm_gpusvm_unmap_pages(gpusvm, svm_pages, num_dma_mapped);
drm_gpusvm_notifier_unlock(gpusvm);
err_free:
kvfree(pfns);
@ -1361,11 +1449,62 @@ int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
goto retry;
return err;
}
EXPORT_SYMBOL_GPL(drm_gpusvm_get_pages);
/**
* drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range
* @gpusvm: Pointer to the GPU SVM structure
* @range: Pointer to the GPU SVM range structure
* @ctx: GPU SVM context
*
* This function gets pages for a GPU SVM range and ensures they are mapped for
* DMA access.
*
* Return: 0 on success, negative error code on failure.
*/
int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_range *range,
const struct drm_gpusvm_ctx *ctx)
{
return drm_gpusvm_get_pages(gpusvm, &range->pages, gpusvm->mm,
&range->notifier->notifier,
drm_gpusvm_range_start(range),
drm_gpusvm_range_end(range), ctx);
}
EXPORT_SYMBOL_GPL(drm_gpusvm_range_get_pages);
/**
* drm_gpusvm_unmap_pages() - Unmap GPU svm pages
* @gpusvm: Pointer to the GPU SVM structure
* @svm_pages: Pointer to the GPU SVM pages structure
* @npages: Number of pages in @svm_pages.
* @ctx: GPU SVM context
*
* This function unmaps pages associated with a GPU SVM pages struct. If
* @in_notifier is set, it is assumed that gpusvm->notifier_lock is held in
* write mode; if it is clear, it acquires gpusvm->notifier_lock in read mode.
* Must be called in the invalidate() callback of the corresponding notifier for
* IOMMU security model.
*/
void drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
struct drm_gpusvm_pages *svm_pages,
unsigned long npages,
const struct drm_gpusvm_ctx *ctx)
{
if (ctx->in_notifier)
lockdep_assert_held_write(&gpusvm->notifier_lock);
else
drm_gpusvm_notifier_lock(gpusvm);
__drm_gpusvm_unmap_pages(gpusvm, svm_pages, npages);
if (!ctx->in_notifier)
drm_gpusvm_notifier_unlock(gpusvm);
}
EXPORT_SYMBOL_GPL(drm_gpusvm_unmap_pages);
/**
* drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range
* drm_gpusvm_range_evict() - Evict GPU SVM range
* @gpusvm: Pointer to the GPU SVM structure
* @range: Pointer to the GPU SVM range structure
* @ctx: GPU SVM context
@ -1383,15 +1522,7 @@ void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
unsigned long npages = npages_in_range(drm_gpusvm_range_start(range),
drm_gpusvm_range_end(range));
if (ctx->in_notifier)
lockdep_assert_held_write(&gpusvm->notifier_lock);
else
drm_gpusvm_notifier_lock(gpusvm);
__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
if (!ctx->in_notifier)
drm_gpusvm_notifier_unlock(gpusvm);
return drm_gpusvm_unmap_pages(gpusvm, &range->pages, npages, ctx);
}
EXPORT_SYMBOL_GPL(drm_gpusvm_range_unmap_pages);
@ -1489,10 +1620,10 @@ void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
{
lockdep_assert_held_write(&range->gpusvm->notifier_lock);
range->flags.unmapped = true;
range->pages.flags.unmapped = true;
if (drm_gpusvm_range_start(range) < mmu_range->start ||
drm_gpusvm_range_end(range) > mmu_range->end)
range->flags.partial_unmap = true;
range->pages.flags.partial_unmap = true;
}
EXPORT_SYMBOL_GPL(drm_gpusvm_range_set_unmapped);

2
drivers/gpu/drm/xe/Kconfig
View File

@ -40,12 +40,12 @@ config DRM_XE
select DRM_TTM
select DRM_TTM_HELPER
select DRM_EXEC
select DRM_GPUSVM if !UML && DEVICE_PRIVATE
select DRM_GPUVM
select DRM_SCHED
select MMU_NOTIFIER
select WANT_DEV_COREDUMP
select AUXILIARY_BUS
select HMM_MIRROR
select REGMAP if I2C
help
Driver for Intel Xe2 series GPUs and later. Experimental support

1
drivers/gpu/drm/xe/Kconfig.debug
View File

@ -104,6 +104,7 @@ config DRM_XE_DEBUG_GUC
config DRM_XE_USERPTR_INVAL_INJECT
bool "Inject userptr invalidation -EINVAL errors"
depends on DRM_GPUSVM
default n
help
Choose this option when debugging error paths that

5
drivers/gpu/drm/xe/Makefile
View File

@ -84,6 +84,7 @@ xe-y += xe_bb.o \
xe_hw_error.o \
xe_hw_fence.o \
xe_irq.o \
xe_late_bind_fw.o \
xe_lrc.o \
xe_migrate.o \
xe_mmio.o \
@ -130,6 +131,7 @@ xe-y += xe_bb.o \
xe_tuning.o \
xe_uc.o \
xe_uc_fw.o \
xe_validation.o \
xe_vm.o \
xe_vm_madvise.o \
xe_vram.o \
@ -140,8 +142,8 @@ xe-y += xe_bb.o \
xe_wopcm.o
xe-$(CONFIG_I2C) += xe_i2c.o
xe-$(CONFIG_HMM_MIRROR) += xe_hmm.o
xe-$(CONFIG_DRM_XE_GPUSVM) += xe_svm.o
xe-$(CONFIG_DRM_GPUSVM) += xe_userptr.o
# graphics hardware monitoring (HWMON) support
xe-$(CONFIG_HWMON) += xe_hwmon.o
@ -326,6 +328,7 @@ ifeq ($(CONFIG_DEBUG_FS),y)
xe_gt_stats.o \
xe_guc_debugfs.o \
xe_huc_debugfs.o \
xe_tile_debugfs.o \
xe_uc_debugfs.o
xe-$(CONFIG_PCI_IOV) += xe_gt_sriov_pf_debugfs.o

3
drivers/gpu/drm/xe/abi/guc_actions_abi.h
View File

@ -117,6 +117,7 @@ enum xe_guc_action {
XE_GUC_ACTION_ENTER_S_STATE = 0x501,
XE_GUC_ACTION_EXIT_S_STATE = 0x502,
XE_GUC_ACTION_GLOBAL_SCHED_POLICY_CHANGE = 0x506,
XE_GUC_ACTION_UPDATE_SCHEDULING_POLICIES_KLV = 0x509,
XE_GUC_ACTION_SCHED_CONTEXT = 0x1000,
XE_GUC_ACTION_SCHED_CONTEXT_MODE_SET = 0x1001,
XE_GUC_ACTION_SCHED_CONTEXT_MODE_DONE = 0x1002,
@ -154,6 +155,8 @@ enum xe_guc_action {
XE_GUC_ACTION_NOTIFY_FLUSH_LOG_BUFFER_TO_FILE = 0x8003,
XE_GUC_ACTION_NOTIFY_CRASH_DUMP_POSTED = 0x8004,
XE_GUC_ACTION_NOTIFY_EXCEPTION = 0x8005,
XE_GUC_ACTION_TEST_G2G_SEND = 0xF001,
XE_GUC_ACTION_TEST_G2G_RECV = 0xF002,
XE_GUC_ACTION_LIMIT
};

5
drivers/gpu/drm/xe/abi/guc_actions_slpc_abi.h
View File

@ -210,6 +210,11 @@ struct slpc_shared_data {
u8 reserved_mode_definition[4096];
} __packed;
enum slpc_power_profile {
SLPC_POWER_PROFILE_BASE = 0x0,
SLPC_POWER_PROFILE_POWER_SAVING = 0x1
};
/**
* DOC: SLPC H2G MESSAGE FORMAT
*

25
drivers/gpu/drm/xe/abi/guc_klvs_abi.h
View File

@ -17,6 +17,7 @@
* | 0 | 31:16 | **KEY** - KLV key identifier |
* | | | - `GuC Self Config KLVs`_ |
* | | | - `GuC Opt In Feature KLVs`_ |
* | | | - `GuC Scheduling Policies KLVs`_ |
* | | | - `GuC VGT Policy KLVs`_ |
* | | | - `GuC VF Configuration KLVs`_ |
* | | | |
@ -152,6 +153,30 @@ enum {
#define GUC_KLV_OPT_IN_FEATURE_DYNAMIC_INHIBIT_CONTEXT_SWITCH_KEY 0x4003
#define GUC_KLV_OPT_IN_FEATURE_DYNAMIC_INHIBIT_CONTEXT_SWITCH_LEN 0u
/**
* DOC: GuC Scheduling Policies KLVs
*
* `GuC KLV`_ keys available for use with UPDATE_SCHEDULING_POLICIES_KLV.
*
* _`GUC_KLV_SCHEDULING_POLICIES_RENDER_COMPUTE_YIELD` : 0x1001
* Some platforms do not allow concurrent execution of RCS and CCS
* workloads from different address spaces. By default, the GuC prioritizes
* RCS submissions over CCS ones, which can lead to CCS workloads being
* significantly (or completely) starved of execution time. This KLV allows
* the driver to specify a quantum (in ms) and a ratio (percentage value
* between 0 and 100), and the GuC will prioritize the CCS for that
* percentage of each quantum. For example, specifying 100ms and 30% will
* make the GuC prioritize the CCS for 30ms of every 100ms.
* Note that this does not necessarly mean that RCS and CCS engines will
* only be active for their percentage of the quantum, as the restriction
* only kicks in if both classes are fully busy with non-compatible address
* spaces; i.e., if one engine is idle or running the same address space,
* a pending job on the other engine will still be submitted to the HW no
* matter what the ratio is
*/
#define GUC_KLV_SCHEDULING_POLICIES_RENDER_COMPUTE_YIELD_KEY 0x1001
#define GUC_KLV_SCHEDULING_POLICIES_RENDER_COMPUTE_YIELD_LEN 2u
/**
* DOC: GuC VGT Policy KLVs
*

15
drivers/gpu/drm/xe/compat-i915-headers/gem/i915_gem_stolen.h
View File

@ -8,6 +8,7 @@
#include "xe_ttm_stolen_mgr.h"
#include "xe_res_cursor.h"
#include "xe_validation.h"
struct xe_bo;
@ -21,7 +22,7 @@ static inline int i915_gem_stolen_insert_node_in_range(struct xe_device *xe,
u32 start, u32 end)
{
struct xe_bo *bo;
int err;
int err = 0;
u32 flags = XE_BO_FLAG_PINNED | XE_BO_FLAG_STOLEN;
if (start < SZ_4K)
@ -32,21 +33,13 @@ static inline int i915_gem_stolen_insert_node_in_range(struct xe_device *xe,
start = ALIGN(start, align);
}
bo = xe_bo_create_locked_range(xe, xe_device_get_root_tile(xe),
NULL, size, start, end,
ttm_bo_type_kernel, flags, 0);
bo = xe_bo_create_pin_range_novm(xe, xe_device_get_root_tile(xe),
size, start, end, ttm_bo_type_kernel, flags);
if (IS_ERR(bo)) {
err = PTR_ERR(bo);
bo = NULL;
return err;
}
err = xe_bo_pin(bo);
xe_bo_unlock_vm_held(bo);
if (err) {
xe_bo_put(fb->bo);
bo = NULL;
}
fb->bo = bo;

18
drivers/gpu/drm/xe/display/intel_fbdev_fb.c
View File

@ -42,11 +42,11 @@ struct intel_framebuffer *intel_fbdev_fb_alloc(struct drm_fb_helper *helper,
obj = ERR_PTR(-ENODEV);
if (!IS_DGFX(xe) && !XE_GT_WA(xe_root_mmio_gt(xe), 22019338487_display)) {
obj = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe),
NULL, size,
ttm_bo_type_kernel, XE_BO_FLAG_SCANOUT |
XE_BO_FLAG_STOLEN |
XE_BO_FLAG_GGTT);
obj = xe_bo_create_pin_map_novm(xe, xe_device_get_root_tile(xe),
size,
ttm_bo_type_kernel, XE_BO_FLAG_SCANOUT |
XE_BO_FLAG_STOLEN |
XE_BO_FLAG_GGTT, false);
if (!IS_ERR(obj))
drm_info(&xe->drm, "Allocated fbdev into stolen\n");
else
@ -54,10 +54,10 @@ struct intel_framebuffer *intel_fbdev_fb_alloc(struct drm_fb_helper *helper,
}
if (IS_ERR(obj)) {
obj = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe), NULL, size,
ttm_bo_type_kernel, XE_BO_FLAG_SCANOUT |
XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
XE_BO_FLAG_GGTT);
obj = xe_bo_create_pin_map_novm(xe, xe_device_get_root_tile(xe), size,
ttm_bo_type_kernel, XE_BO_FLAG_SCANOUT |
XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
XE_BO_FLAG_GGTT, false);
}
if (IS_ERR(obj)) {

10
drivers/gpu/drm/xe/display/xe_dsb_buffer.c
View File

@ -43,11 +43,11 @@ bool intel_dsb_buffer_create(struct intel_crtc *crtc, struct intel_dsb_buffer *d
return false;
/* Set scanout flag for WC mapping */
obj = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe),
NULL, PAGE_ALIGN(size),
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
XE_BO_FLAG_SCANOUT | XE_BO_FLAG_GGTT);
obj = xe_bo_create_pin_map_novm(xe, xe_device_get_root_tile(xe),
PAGE_ALIGN(size),
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
XE_BO_FLAG_SCANOUT | XE_BO_FLAG_GGTT, false);
if (IS_ERR(obj)) {
kfree(vma);
return false;

71
drivers/gpu/drm/xe/display/xe_fb_pin.c
View File

@ -102,29 +102,29 @@ static int __xe_pin_fb_vma_dpt(const struct intel_framebuffer *fb,
XE_PAGE_SIZE);
if (IS_DGFX(xe))
dpt = xe_bo_create_pin_map_at_aligned(xe, tile0, NULL,
dpt_size, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM0 |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_PAGETABLE,
alignment);
dpt = xe_bo_create_pin_map_at_novm(xe, tile0,
dpt_size, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM0 |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_PAGETABLE,
alignment, false);
else
dpt = xe_bo_create_pin_map_at_aligned(xe, tile0, NULL,
dpt_size, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_STOLEN |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_PAGETABLE,
alignment);
dpt = xe_bo_create_pin_map_at_novm(xe, tile0,
dpt_size, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_STOLEN |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_PAGETABLE,
alignment, false);
if (IS_ERR(dpt))
dpt = xe_bo_create_pin_map_at_aligned(xe, tile0, NULL,
dpt_size, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_PAGETABLE,
alignment);
dpt = xe_bo_create_pin_map_at_novm(xe, tile0,
dpt_size, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_PAGETABLE,
alignment, false);
if (IS_ERR(dpt))
return PTR_ERR(dpt);
@ -281,7 +281,9 @@ static struct i915_vma *__xe_pin_fb_vma(const struct intel_framebuffer *fb,
struct i915_vma *vma = kzalloc(sizeof(*vma), GFP_KERNEL);
struct drm_gem_object *obj = intel_fb_bo(&fb->base);
struct xe_bo *bo = gem_to_xe_bo(obj);
int ret;
struct xe_validation_ctx ctx;
struct drm_exec exec;
int ret = 0;
if (!vma)
return ERR_PTR(-ENODEV);
@ -308,17 +310,22 @@ static struct i915_vma *__xe_pin_fb_vma(const struct intel_framebuffer *fb,
* Pin the framebuffer, we can't use xe_bo_(un)pin functions as the
* assumptions are incorrect for framebuffers
*/
ret = ttm_bo_reserve(&bo->ttm, false, false, NULL);
if (ret)
goto err;
xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {.interruptible = true},
ret) {
ret = drm_exec_lock_obj(&exec, &bo->ttm.base);
drm_exec_retry_on_contention(&exec);
if (ret)
break;
if (IS_DGFX(xe))
ret = xe_bo_migrate(bo, XE_PL_VRAM0);
else
ret = xe_bo_validate(bo, NULL, true);
if (!ret)
ttm_bo_pin(&bo->ttm);
ttm_bo_unreserve(&bo->ttm);
if (IS_DGFX(xe))
ret = xe_bo_migrate(bo, XE_PL_VRAM0, NULL, &exec);
else
ret = xe_bo_validate(bo, NULL, true, &exec);
drm_exec_retry_on_contention(&exec);
xe_validation_retry_on_oom(&ctx, &ret);
if (!ret)
ttm_bo_pin(&bo->ttm);
}
if (ret)
goto err;

8
drivers/gpu/drm/xe/display/xe_hdcp_gsc.c
View File

@ -72,10 +72,10 @@ static int intel_hdcp_gsc_initialize_message(struct xe_device *xe,
int ret = 0;
/* allocate object of two page for HDCP command memory and store it */
bo = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe), NULL, PAGE_SIZE * 2,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT);
bo = xe_bo_create_pin_map_novm(xe, xe_device_get_root_tile(xe), PAGE_SIZE * 2,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT, false);
if (IS_ERR(bo)) {
drm_err(&xe->drm, "Failed to allocate bo for HDCP streaming command!\n");

4
drivers/gpu/drm/xe/display/xe_plane_initial.c
View File

@ -140,8 +140,8 @@ initial_plane_bo(struct xe_device *xe,
page_size);
size -= base;
bo = xe_bo_create_pin_map_at(xe, tile0, NULL, size, phys_base,
ttm_bo_type_kernel, flags);
bo = xe_bo_create_pin_map_at_novm(xe, tile0, size, phys_base,
ttm_bo_type_kernel, flags, 0, false);
if (IS_ERR(bo)) {
drm_dbg(&xe->drm,
"Failed to create bo phys_base=%pa size %u with flags %x: %li\n",

1
drivers/gpu/drm/xe/regs/xe_gt_regs.h
View File

@ -522,6 +522,7 @@
#define TDL_CHICKEN XE_REG_MCR(0xe5f4, XE_REG_OPTION_MASKED)
#define QID_WAIT_FOR_THREAD_NOT_RUN_DISABLE REG_BIT(12)
#define EUSTALL_PERF_SAMPLING_DISABLE REG_BIT(5)
#define LSC_CHICKEN_BIT_0 XE_REG_MCR(0xe7c8)
#define DISABLE_D8_D16_COASLESCE REG_BIT(30)

3
drivers/gpu/drm/xe/regs/xe_lrc_layout.h
View File

@ -40,7 +40,4 @@
#define INDIRECT_CTX_RING_START_UDW (0x08 + 1)
#define INDIRECT_CTX_RING_CTL (0x0a + 1)
#define CTX_INDIRECT_CTX_OFFSET_MASK REG_GENMASK(15, 6)
#define CTX_INDIRECT_CTX_OFFSET_DEFAULT REG_FIELD_PREP(CTX_INDIRECT_CTX_OFFSET_MASK, 0xd)
#endif

36
drivers/gpu/drm/xe/tests/xe_bo.c
View File

@ -23,7 +23,7 @@
static int ccs_test_migrate(struct xe_tile *tile, struct xe_bo *bo,
bool clear, u64 get_val, u64 assign_val,
struct kunit *test)
struct kunit *test, struct drm_exec *exec)
{
struct dma_fence *fence;
struct ttm_tt *ttm;
@ -35,7 +35,7 @@ static int ccs_test_migrate(struct xe_tile *tile, struct xe_bo *bo,
u32 offset;
/* Move bo to VRAM if not already there. */
ret = xe_bo_validate(bo, NULL, false);
ret = xe_bo_validate(bo, NULL, false, exec);
if (ret) {
KUNIT_FAIL(test, "Failed to validate bo.\n");
return ret;
@ -60,7 +60,7 @@ static int ccs_test_migrate(struct xe_tile *tile, struct xe_bo *bo,
}
/* Evict to system. CCS data should be copied. */
ret = xe_bo_evict(bo);
ret = xe_bo_evict(bo, exec);
if (ret) {
KUNIT_FAIL(test, "Failed to evict bo.\n");
return ret;
@ -132,14 +132,15 @@ static void ccs_test_run_tile(struct xe_device *xe, struct xe_tile *tile,
/* TODO: Sanity check */
unsigned int bo_flags = XE_BO_FLAG_VRAM_IF_DGFX(tile);
struct drm_exec *exec = XE_VALIDATION_OPT_OUT;
if (IS_DGFX(xe))
kunit_info(test, "Testing vram id %u\n", tile->id);
else
kunit_info(test, "Testing system memory\n");
bo = xe_bo_create_user(xe, NULL, NULL, SZ_1M, DRM_XE_GEM_CPU_CACHING_WC,
bo_flags);
bo = xe_bo_create_user(xe, NULL, SZ_1M, DRM_XE_GEM_CPU_CACHING_WC,
bo_flags, exec);
if (IS_ERR(bo)) {
KUNIT_FAIL(test, "Failed to create bo.\n");
return;
@ -149,18 +150,18 @@ static void ccs_test_run_tile(struct xe_device *xe, struct xe_tile *tile,
kunit_info(test, "Verifying that CCS data is cleared on creation.\n");
ret = ccs_test_migrate(tile, bo, false, 0ULL, 0xdeadbeefdeadbeefULL,
test);
test, exec);
if (ret)
goto out_unlock;
kunit_info(test, "Verifying that CCS data survives migration.\n");
ret = ccs_test_migrate(tile, bo, false, 0xdeadbeefdeadbeefULL,
0xdeadbeefdeadbeefULL, test);
0xdeadbeefdeadbeefULL, test, exec);
if (ret)
goto out_unlock;
kunit_info(test, "Verifying that CCS data can be properly cleared.\n");
ret = ccs_test_migrate(tile, bo, true, 0ULL, 0ULL, test);
ret = ccs_test_migrate(tile, bo, true, 0ULL, 0ULL, test, exec);
out_unlock:
xe_bo_unlock(bo);
@ -210,6 +211,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
struct xe_bo *bo, *external;
unsigned int bo_flags = XE_BO_FLAG_VRAM_IF_DGFX(tile);
struct xe_vm *vm = xe_migrate_get_vm(xe_device_get_root_tile(xe)->migrate);
struct drm_exec *exec = XE_VALIDATION_OPT_OUT;
struct xe_gt *__gt;
int err, i, id;
@ -218,25 +220,25 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
for (i = 0; i < 2; ++i) {
xe_vm_lock(vm, false);
bo = xe_bo_create_user(xe, NULL, vm, 0x10000,
bo = xe_bo_create_user(xe, vm, 0x10000,
DRM_XE_GEM_CPU_CACHING_WC,
bo_flags);
bo_flags, exec);
xe_vm_unlock(vm);
if (IS_ERR(bo)) {
KUNIT_FAIL(test, "bo create err=%pe\n", bo);
break;
}
external = xe_bo_create_user(xe, NULL, NULL, 0x10000,
external = xe_bo_create_user(xe, NULL, 0x10000,
DRM_XE_GEM_CPU_CACHING_WC,
bo_flags);
bo_flags, NULL);
if (IS_ERR(external)) {
KUNIT_FAIL(test, "external bo create err=%pe\n", external);
goto cleanup_bo;
}
xe_bo_lock(external, false);
err = xe_bo_pin_external(external, false);
err = xe_bo_pin_external(external, false, exec);
xe_bo_unlock(external);
if (err) {
KUNIT_FAIL(test, "external bo pin err=%pe\n",
@ -294,7 +296,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
if (i) {
down_read(&vm->lock);
xe_vm_lock(vm, false);
err = xe_bo_validate(bo, bo->vm, false);
err = xe_bo_validate(bo, bo->vm, false, exec);
xe_vm_unlock(vm);
up_read(&vm->lock);
if (err) {
@ -303,7 +305,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
goto cleanup_all;
}
xe_bo_lock(external, false);
err = xe_bo_validate(external, NULL, false);
err = xe_bo_validate(external, NULL, false, exec);
xe_bo_unlock(external);
if (err) {
KUNIT_FAIL(test, "external bo valid err=%pe\n",
@ -495,9 +497,9 @@ static int shrink_test_run_device(struct xe_device *xe)
INIT_LIST_HEAD(&link->link);
/* We can create bos using WC caching here. But it is slower. */
bo = xe_bo_create_user(xe, NULL, NULL, XE_BO_SHRINK_SIZE,
bo = xe_bo_create_user(xe, NULL, XE_BO_SHRINK_SIZE,
DRM_XE_GEM_CPU_CACHING_WB,
XE_BO_FLAG_SYSTEM);
XE_BO_FLAG_SYSTEM, NULL);
if (IS_ERR(bo)) {
if (bo != ERR_PTR(-ENOMEM) && bo != ERR_PTR(-ENOSPC) &&
bo != ERR_PTR(-EINTR) && bo != ERR_PTR(-ERESTARTSYS))

16
drivers/gpu/drm/xe/tests/xe_dma_buf.c
View File

@ -27,7 +27,8 @@ static bool is_dynamic(struct dma_buf_test_params *params)
}
static void check_residency(struct kunit *test, struct xe_bo *exported,
struct xe_bo *imported, struct dma_buf *dmabuf)
struct xe_bo *imported, struct dma_buf *dmabuf,
struct drm_exec *exec)
{
struct dma_buf_test_params *params = to_dma_buf_test_params(test->priv);
u32 mem_type;
@ -62,7 +63,7 @@ static void check_residency(struct kunit *test, struct xe_bo *exported,
* importer is on a different device. If they're on the same device,
* the exporter and the importer should be the same bo.
*/
ret = xe_bo_evict(exported);
ret = xe_bo_evict(exported, exec);
if (ret) {
if (ret != -EINTR && ret != -ERESTARTSYS)
KUNIT_FAIL(test, "Evicting exporter failed with err=%d.\n",
@ -77,7 +78,7 @@ static void check_residency(struct kunit *test, struct xe_bo *exported,
}
/* Re-validate the importer. This should move also exporter in. */
ret = xe_bo_validate(imported, NULL, false);
ret = xe_bo_validate(imported, NULL, false, exec);
if (ret) {
if (ret != -EINTR && ret != -ERESTARTSYS)
KUNIT_FAIL(test, "Validating importer failed with err=%d.\n",
@ -113,8 +114,8 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
size = SZ_64K;
kunit_info(test, "running %s\n", __func__);
bo = xe_bo_create_user(xe, NULL, NULL, size, DRM_XE_GEM_CPU_CACHING_WC,
params->mem_mask);
bo = xe_bo_create_user(xe, NULL, size, DRM_XE_GEM_CPU_CACHING_WC,
params->mem_mask, NULL);
if (IS_ERR(bo)) {
KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
PTR_ERR(bo));
@ -142,11 +143,12 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
KUNIT_FAIL(test,
"xe_gem_prime_import() succeeded when it shouldn't have\n");
} else {
struct drm_exec *exec = XE_VALIDATION_OPT_OUT;
int err;
/* Is everything where we expect it to be? */
xe_bo_lock(import_bo, false);
err = xe_bo_validate(import_bo, NULL, false);
err = xe_bo_validate(import_bo, NULL, false, exec);
/* Pinning in VRAM is not allowed. */
if (!is_dynamic(params) &&
@ -159,7 +161,7 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
err == -ERESTARTSYS);
if (!err)
check_residency(test, bo, import_bo, dmabuf);
check_residency(test, bo, import_bo, dmabuf, exec);
xe_bo_unlock(import_bo);
}
drm_gem_object_put(import);

776
drivers/gpu/drm/xe/tests/xe_guc_g2g_test.c Normal file
View File

@ -0,0 +1,776 @@
// SPDX-License-Identifier: GPL-2.0 AND MIT
/*
* Copyright © 2025 Intel Corporation
*/
#include <linux/delay.h>
#include <kunit/test.h>
#include <kunit/visibility.h>
#include "tests/xe_kunit_helpers.h"
#include "tests/xe_pci_test.h"
#include "tests/xe_test.h"
#include "xe_bo.h"
#include "xe_device.h"
#include "xe_pm.h"
/*
* There are different ways to allocate the G2G buffers. The plan for this test
* is to make sure that all the possible options work. The particular option
* chosen by the driver may vary from one platform to another, it may also change
* with time. So to ensure consistency of testing, the relevant driver code is
* replicated here to guarantee it won't change without the test being updated
* to keep testing the other options.
*
* In order to test the actual code being used by the driver, there is also the
* 'default' scheme. That will use the official driver routines to test whatever
* method the driver is using on the current platform at the current time.
*/
enum {
/* Driver defined allocation scheme */
G2G_CTB_TYPE_DEFAULT,
/* Single buffer in host memory */
G2G_CTB_TYPE_HOST,
/* Single buffer in a specific tile, loops across all tiles */
G2G_CTB_TYPE_TILE,
};
/*
* Payload is opaque to GuC. So KMD can define any structure or size it wants.
*/
struct g2g_test_payload {
u32 tx_dev;
u32 tx_tile;
u32 rx_dev;
u32 rx_tile;
u32 seqno;
};
static void g2g_test_send(struct kunit *test, struct xe_guc *guc,
u32 far_tile, u32 far_dev,
struct g2g_test_payload *payload)
{
struct xe_device *xe = guc_to_xe(guc);
struct xe_gt *gt = guc_to_gt(guc);
u32 *action, total;
size_t payload_len;
int ret;
static_assert(IS_ALIGNED(sizeof(*payload), sizeof(u32)));
payload_len = sizeof(*payload) / sizeof(u32);
total = 4 + payload_len;
action = kunit_kmalloc_array(test, total, sizeof(*action), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, action);
action[0] = XE_GUC_ACTION_TEST_G2G_SEND;
action[1] = far_tile;
action[2] = far_dev;
action[3] = payload_len;
memcpy(action + 4, payload, payload_len * sizeof(u32));
atomic_inc(&xe->g2g_test_count);
/*
* Should specify the expected response notification here. Problem is that
* the response will be coming from a different GuC. By the end, it should
* all add up as long as an equal number of messages are sent from each GuC
* and to each GuC. However, in the middle negative reservation space errors
* and such like can occur. Rather than add intrusive changes to the CT layer
* it is simpler to just not bother counting it at all. The system should be
* idle when running the selftest, and the selftest's notification total size
* is well within the G2H allocation size. So there should be no issues with
* needing to block for space, which is all the tracking code is really for.
*/
ret = xe_guc_ct_send(&guc->ct, action, total, 0, 0);
kunit_kfree(test, action);
KUNIT_ASSERT_EQ_MSG(test, 0, ret, "G2G send failed: %d [%d:%d -> %d:%d]\n", ret,
gt_to_tile(gt)->id, G2G_DEV(gt), far_tile, far_dev);
}
/*
* NB: Can't use KUNIT_ASSERT and friends in here as this is called asynchronously
* from the G2H notification handler. Need that to actually complete rather than
* thread-abort in order to keep the rest of the driver alive!
*/
int xe_guc_g2g_test_notification(struct xe_guc *guc, u32 *msg, u32 len)
{
struct xe_device *xe = guc_to_xe(guc);
struct xe_gt *rx_gt = guc_to_gt(guc), *test_gt, *tx_gt = NULL;
u32 tx_tile, tx_dev, rx_tile, rx_dev, idx, got_len;
struct g2g_test_payload *payload;
size_t payload_len;
int ret = 0, i;
payload_len = sizeof(*payload) / sizeof(u32);
if (unlikely(len != (G2H_LEN_DW_G2G_NOTIFY_MIN + payload_len))) {
xe_gt_err(rx_gt, "G2G test notification invalid length %u", len);
ret = -EPROTO;
goto done;
}
tx_tile = msg[0];
tx_dev = msg[1];
got_len = msg[2];
payload = (struct g2g_test_payload *)(msg + 3);
rx_tile = gt_to_tile(rx_gt)->id;
rx_dev = G2G_DEV(rx_gt);
if (got_len != payload_len) {
xe_gt_err(rx_gt, "G2G: Invalid payload length: %u vs %zu\n", got_len, payload_len);
ret = -EPROTO;
goto done;
}
if (payload->tx_dev != tx_dev || payload->tx_tile != tx_tile ||
payload->rx_dev != rx_dev || payload->rx_tile != rx_tile) {
xe_gt_err(rx_gt, "G2G: Invalid payload: %d:%d -> %d:%d vs %d:%d -> %d:%d! [%d]\n",
payload->tx_tile, payload->tx_dev, payload->rx_tile, payload->rx_dev,
tx_tile, tx_dev, rx_tile, rx_dev, payload->seqno);
ret = -EPROTO;
goto done;
}
if (!xe->g2g_test_array) {
xe_gt_err(rx_gt, "G2G: Missing test array!\n");
ret = -ENOMEM;
goto done;
}
for_each_gt(test_gt, xe, i) {
if (gt_to_tile(test_gt)->id != tx_tile)
continue;
if (G2G_DEV(test_gt) != tx_dev)
continue;
if (tx_gt) {
xe_gt_err(rx_gt, "G2G: Got duplicate TX GTs: %d vs %d for %d:%d!\n",
tx_gt->info.id, test_gt->info.id, tx_tile, tx_dev);
ret = -EINVAL;
goto done;
}
tx_gt = test_gt;
}
if (!tx_gt) {
xe_gt_err(rx_gt, "G2G: Failed to find a TX GT for %d:%d!\n", tx_tile, tx_dev);
ret = -EINVAL;
goto done;
}
idx = (tx_gt->info.id * xe->info.gt_count) + rx_gt->info.id;
if (xe->g2g_test_array[idx] != payload->seqno - 1) {
xe_gt_err(rx_gt, "G2G: Seqno mismatch %d vs %d for %d:%d -> %d:%d!\n",
xe->g2g_test_array[idx], payload->seqno - 1,
tx_tile, tx_dev, rx_tile, rx_dev);
ret = -EINVAL;
goto done;
}
xe->g2g_test_array[idx] = payload->seqno;
done:
atomic_dec(&xe->g2g_test_count);
return ret;
}
/*
* Send the given seqno from all GuCs to all other GuCs in tile/GT order
*/
static void g2g_test_in_order(struct kunit *test, struct xe_device *xe, u32 seqno)
{
struct xe_gt *near_gt, *far_gt;
int i, j;
for_each_gt(near_gt, xe, i) {
u32 near_tile = gt_to_tile(near_gt)->id;
u32 near_dev = G2G_DEV(near_gt);
for_each_gt(far_gt, xe, j) {
u32 far_tile = gt_to_tile(far_gt)->id;
u32 far_dev = G2G_DEV(far_gt);
struct g2g_test_payload payload;
if (far_gt->info.id == near_gt->info.id)
continue;
payload.tx_dev = near_dev;
payload.tx_tile = near_tile;
payload.rx_dev = far_dev;
payload.rx_tile = far_tile;
payload.seqno = seqno;
g2g_test_send(test, &near_gt->uc.guc, far_tile, far_dev, &payload);
}
}
}
#define WAIT_TIME_MS 100
#define WAIT_COUNT (1000 / WAIT_TIME_MS)
static void g2g_wait_for_complete(void *_xe)
{
struct xe_device *xe = (struct xe_device *)_xe;
struct kunit *test = kunit_get_current_test();
int wait = 0;
/* Wait for all G2H messages to be received */
while (atomic_read(&xe->g2g_test_count)) {
if (++wait > WAIT_COUNT)
break;
msleep(WAIT_TIME_MS);
}
KUNIT_ASSERT_EQ_MSG(test, 0, atomic_read(&xe->g2g_test_count),
"Timed out waiting for notifications\n");
kunit_info(test, "Got all notifications back\n");
}
#undef WAIT_TIME_MS
#undef WAIT_COUNT
static void g2g_clean_array(void *_xe)
{
struct xe_device *xe = (struct xe_device *)_xe;
xe->g2g_test_array = NULL;
}
#define NUM_LOOPS 16
static void g2g_run_test(struct kunit *test, struct xe_device *xe)
{
u32 seqno, max_array;
int ret, i, j;
max_array = xe->info.gt_count * xe->info.gt_count;
xe->g2g_test_array = kunit_kcalloc(test, max_array, sizeof(u32), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, xe->g2g_test_array);
ret = kunit_add_action_or_reset(test, g2g_clean_array, xe);
KUNIT_ASSERT_EQ_MSG(test, 0, ret, "Failed to register clean up action\n");
/*
* Send incrementing seqnos from all GuCs to all other GuCs in tile/GT order.
* Tile/GT order doesn't really mean anything to the hardware but it is going
* to be a fixed sequence every time.
*
* Verify that each one comes back having taken the correct route.
*/
ret = kunit_add_action(test, g2g_wait_for_complete, xe);
KUNIT_ASSERT_EQ_MSG(test, 0, ret, "Failed to register clean up action\n");
for (seqno = 1; seqno < NUM_LOOPS; seqno++)
g2g_test_in_order(test, xe, seqno);
seqno--;
kunit_release_action(test, &g2g_wait_for_complete, xe);
/* Check for the final seqno in each slot */
for (i = 0; i < xe->info.gt_count; i++) {
for (j = 0; j < xe->info.gt_count; j++) {
u32 idx = (j * xe->info.gt_count) + i;
if (i == j)
KUNIT_ASSERT_EQ_MSG(test, 0, xe->g2g_test_array[idx],
"identity seqno modified: %d for %dx%d!\n",
xe->g2g_test_array[idx], i, j);
else
KUNIT_ASSERT_EQ_MSG(test, seqno, xe->g2g_test_array[idx],
"invalid seqno: %d vs %d for %dx%d!\n",
xe->g2g_test_array[idx], seqno, i, j);
}
}
kunit_kfree(test, xe->g2g_test_array);
kunit_release_action(test, &g2g_clean_array, xe);
kunit_info(test, "Test passed\n");
}
#undef NUM_LOOPS
static void g2g_ct_stop(struct xe_guc *guc)
{
struct xe_gt *remote_gt, *gt = guc_to_gt(guc);
struct xe_device *xe = gt_to_xe(gt);
int i, t;
for_each_gt(remote_gt, xe, i) {
u32 tile, dev;
if (remote_gt->info.id == gt->info.id)
continue;
tile = gt_to_tile(remote_gt)->id;
dev = G2G_DEV(remote_gt);
for (t = 0; t < XE_G2G_TYPE_LIMIT; t++)
guc_g2g_deregister(guc, tile, dev, t);
}
}
/* Size of a single allocation that contains all G2G CTBs across all GTs */
static u32 g2g_ctb_size(struct kunit *test, struct xe_device *xe)
{
unsigned int count = xe->info.gt_count;
u32 num_channels = (count * (count - 1)) / 2;
kunit_info(test, "Size: (%d * %d / 2) * %d * 0x%08X + 0x%08X => 0x%08X [%d]\n",
count, count - 1, XE_G2G_TYPE_LIMIT, G2G_BUFFER_SIZE, G2G_DESC_AREA_SIZE,
num_channels * XE_G2G_TYPE_LIMIT * G2G_BUFFER_SIZE + G2G_DESC_AREA_SIZE,
num_channels * XE_G2G_TYPE_LIMIT);
return num_channels * XE_G2G_TYPE_LIMIT * G2G_BUFFER_SIZE + G2G_DESC_AREA_SIZE;
}
/*
* Use the driver's regular CTB allocation scheme.
*/
static void g2g_alloc_default(struct kunit *test, struct xe_device *xe)
{
struct xe_gt *gt;
int i;
kunit_info(test, "Default [tiles = %d, GTs = %d]\n",
xe->info.tile_count, xe->info.gt_count);
for_each_gt(gt, xe, i) {
struct xe_guc *guc = &gt->uc.guc;
int ret;
ret = guc_g2g_alloc(guc);
KUNIT_ASSERT_EQ_MSG(test, 0, ret, "G2G alloc failed: %pe", ERR_PTR(ret));
continue;
}
}
static void g2g_distribute(struct kunit *test, struct xe_device *xe, struct xe_bo *bo)
{
struct xe_gt *root_gt, *gt;
int i;
root_gt = xe_device_get_gt(xe, 0);
root_gt->uc.guc.g2g.bo = bo;
root_gt->uc.guc.g2g.owned = true;
kunit_info(test, "[%d.%d] Assigned 0x%p\n", gt_to_tile(root_gt)->id, root_gt->info.id, bo);
for_each_gt(gt, xe, i) {
if (gt->info.id != 0) {
gt->uc.guc.g2g.owned = false;
gt->uc.guc.g2g.bo = xe_bo_get(bo);
kunit_info(test, "[%d.%d] Pinned 0x%p\n",
gt_to_tile(gt)->id, gt->info.id, gt->uc.guc.g2g.bo);
}
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, gt->uc.guc.g2g.bo);
}
}
/*
* Allocate a single blob on the host and split between all G2G CTBs.
*/
static void g2g_alloc_host(struct kunit *test, struct xe_device *xe)
{
struct xe_bo *bo;
u32 g2g_size;
kunit_info(test, "Host [tiles = %d, GTs = %d]\n", xe->info.tile_count, xe->info.gt_count);
g2g_size = g2g_ctb_size(test, xe);
bo = xe_managed_bo_create_pin_map(xe, xe_device_get_root_tile(xe), g2g_size,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_ALL |
XE_BO_FLAG_GGTT_INVALIDATE);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bo);
kunit_info(test, "[HST] G2G buffer create: 0x%p\n", bo);
xe_map_memset(xe, &bo->vmap, 0, 0, g2g_size);
g2g_distribute(test, xe, bo);
}
/*
* Allocate a single blob on the given tile and split between all G2G CTBs.
*/
static void g2g_alloc_tile(struct kunit *test, struct xe_device *xe, struct xe_tile *tile)
{
struct xe_bo *bo;
u32 g2g_size;
KUNIT_ASSERT_TRUE(test, IS_DGFX(xe));
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, tile);
kunit_info(test, "Tile %d [tiles = %d, GTs = %d]\n",
tile->id, xe->info.tile_count, xe->info.gt_count);
g2g_size = g2g_ctb_size(test, xe);
bo = xe_managed_bo_create_pin_map(xe, tile, g2g_size,
XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_ALL |
XE_BO_FLAG_GGTT_INVALIDATE);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bo);
kunit_info(test, "[%d.*] G2G buffer create: 0x%p\n", tile->id, bo);
xe_map_memset(xe, &bo->vmap, 0, 0, g2g_size);
g2g_distribute(test, xe, bo);
}
static void g2g_free(struct kunit *test, struct xe_device *xe)
{
struct xe_gt *gt;
struct xe_bo *bo;
int i;
for_each_gt(gt, xe, i) {
bo = gt->uc.guc.g2g.bo;
if (!bo)
continue;
if (gt->uc.guc.g2g.owned) {
xe_managed_bo_unpin_map_no_vm(bo);
kunit_info(test, "[%d.%d] Unmapped 0x%p\n",
gt_to_tile(gt)->id, gt->info.id, bo);
} else {
xe_bo_put(bo);
kunit_info(test, "[%d.%d] Unpinned 0x%p\n",
gt_to_tile(gt)->id, gt->info.id, bo);
}
gt->uc.guc.g2g.bo = NULL;
}
}
static void g2g_stop(struct kunit *test, struct xe_device *xe)
{
struct xe_gt *gt;
int i;
for_each_gt(gt, xe, i) {
struct xe_guc *guc = &gt->uc.guc;
if (!guc->g2g.bo)
continue;
g2g_ct_stop(guc);
}
g2g_free(test, xe);
}
/*
* Generate a unique id for each bi-directional CTB for each pair of
* near and far tiles/devices. The id can then be used as an index into
* a single allocation that is sub-divided into multiple CTBs.
*
* For example, with two devices per tile and two tiles, the table should
* look like:
* Far <tile>.<dev>
* 0.0 0.1 1.0 1.1
* N 0.0 --/-- 00/01 02/03 04/05
* e 0.1 01/00 --/-- 06/07 08/09
* a 1.0 03/02 07/06 --/-- 10/11
* r 1.1 05/04 09/08 11/10 --/--
*
* Where each entry is Rx/Tx channel id.
*
* So GuC #3 (tile 1, dev 1) talking to GuC #2 (tile 1, dev 0) would
* be reading from channel #11 and writing to channel #10. Whereas,
* GuC #2 talking to GuC #3 would be read on #10 and write to #11.
*/
static int g2g_slot_flat(u32 near_tile, u32 near_dev, u32 far_tile, u32 far_dev,
u32 type, u32 max_inst, bool have_dev)
{
u32 near = near_tile, far = far_tile;
u32 idx = 0, x, y, direction;
int i;
if (have_dev) {
near = (near << 1) | near_dev;
far = (far << 1) | far_dev;
}
/* No need to send to one's self */
if (far == near)
return -1;
if (far > near) {
/* Top right table half */
x = far;
y = near;
/* T/R is 'forwards' direction */
direction = type;
} else {
/* Bottom left table half */
x = near;
y = far;
/* B/L is 'backwards' direction */
direction = (1 - type);
}
/* Count the rows prior to the target */
for (i = y; i > 0; i--)
idx += max_inst - i;
/* Count this row up to the target */
idx += (x - 1 - y);
/* Slots are in Rx/Tx pairs */
idx *= 2;
/* Pick Rx/Tx direction */
idx += direction;
return idx;
}
static int g2g_register_flat(struct xe_guc *guc, u32 far_tile, u32 far_dev, u32 type, bool have_dev)
{
struct xe_gt *gt = guc_to_gt(guc);
struct xe_device *xe = gt_to_xe(gt);
u32 near_tile = gt_to_tile(gt)->id;
u32 near_dev = G2G_DEV(gt);
u32 max = xe->info.gt_count;
int idx;
u32 base, desc, buf;
if (!guc->g2g.bo)
return -ENODEV;
idx = g2g_slot_flat(near_tile, near_dev, far_tile, far_dev, type, max, have_dev);
xe_assert(xe, idx >= 0);
base = guc_bo_ggtt_addr(guc, guc->g2g.bo);
desc = base + idx * G2G_DESC_SIZE;
buf = base + idx * G2G_BUFFER_SIZE + G2G_DESC_AREA_SIZE;
xe_assert(xe, (desc - base + G2G_DESC_SIZE) <= G2G_DESC_AREA_SIZE);
xe_assert(xe, (buf - base + G2G_BUFFER_SIZE) <= xe_bo_size(guc->g2g.bo));
return guc_action_register_g2g_buffer(guc, type, far_tile, far_dev,
desc, buf, G2G_BUFFER_SIZE);
}
static void g2g_start(struct kunit *test, struct xe_guc *guc)
{
struct xe_gt *remote_gt, *gt = guc_to_gt(guc);
struct xe_device *xe = gt_to_xe(gt);
unsigned int i;
int t, ret;
bool have_dev;
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, guc->g2g.bo);
/* GuC interface will need extending if more GT device types are ever created. */
KUNIT_ASSERT_TRUE(test,
(gt->info.type == XE_GT_TYPE_MAIN) ||
(gt->info.type == XE_GT_TYPE_MEDIA));
/* Channel numbering depends on whether there are multiple GTs per tile */
have_dev = xe->info.gt_count > xe->info.tile_count;
for_each_gt(remote_gt, xe, i) {
u32 tile, dev;
if (remote_gt->info.id == gt->info.id)
continue;
tile = gt_to_tile(remote_gt)->id;
dev = G2G_DEV(remote_gt);
for (t = 0; t < XE_G2G_TYPE_LIMIT; t++) {
ret = g2g_register_flat(guc, tile, dev, t, have_dev);
KUNIT_ASSERT_EQ_MSG(test, 0, ret, "G2G register failed: %pe", ERR_PTR(ret));
}
}
}
static void g2g_reinit(struct kunit *test, struct xe_device *xe, int ctb_type, struct xe_tile *tile)
{
struct xe_gt *gt;
int i, found = 0;
g2g_stop(test, xe);
for_each_gt(gt, xe, i) {
struct xe_guc *guc = &gt->uc.guc;
KUNIT_ASSERT_NULL(test, guc->g2g.bo);
}
switch (ctb_type) {
case G2G_CTB_TYPE_DEFAULT:
g2g_alloc_default(test, xe);
break;
case G2G_CTB_TYPE_HOST:
g2g_alloc_host(test, xe);
break;
case G2G_CTB_TYPE_TILE:
g2g_alloc_tile(test, xe, tile);
break;
default:
KUNIT_ASSERT_TRUE(test, false);
}
for_each_gt(gt, xe, i) {
struct xe_guc *guc = &gt->uc.guc;
if (!guc->g2g.bo)
continue;
if (ctb_type == G2G_CTB_TYPE_DEFAULT)
guc_g2g_start(guc);
else
g2g_start(test, guc);
found++;
}
KUNIT_ASSERT_GT_MSG(test, found, 1, "insufficient G2G channels running: %d", found);
kunit_info(test, "Testing across %d GTs\n", found);
}
static void g2g_recreate_ctb(void *_xe)
{
struct xe_device *xe = (struct xe_device *)_xe;
struct kunit *test = kunit_get_current_test();
g2g_stop(test, xe);
if (xe_guc_g2g_wanted(xe))
g2g_reinit(test, xe, G2G_CTB_TYPE_DEFAULT, NULL);
}
static void g2g_pm_runtime_put(void *_xe)
{
struct xe_device *xe = (struct xe_device *)_xe;
xe_pm_runtime_put(xe);
}
static void g2g_pm_runtime_get(struct kunit *test)
{
struct xe_device *xe = test->priv;
int ret;
xe_pm_runtime_get(xe);
ret = kunit_add_action_or_reset(test, g2g_pm_runtime_put, xe);
KUNIT_ASSERT_EQ_MSG(test, 0, ret, "Failed to register runtime PM action\n");
}
static void g2g_check_skip(struct kunit *test)
{
struct xe_device *xe = test->priv;
struct xe_gt *gt;
int i;
if (IS_SRIOV_VF(xe))
kunit_skip(test, "not supported from a VF");
if (xe->info.gt_count <= 1)
kunit_skip(test, "not enough GTs");
for_each_gt(gt, xe, i) {
struct xe_guc *guc = &gt->uc.guc;
if (guc->fw.build_type == CSS_UKERNEL_INFO_BUILDTYPE_PROD)
kunit_skip(test,
"G2G test interface not available in production firmware builds\n");
}
}
/*
* Simple test that does not try to recreate the CTBs.
* Requires that the platform already enables G2G comms
* but has no risk of leaving the system in a broken state
* afterwards.
*/
static void xe_live_guc_g2g_kunit_default(struct kunit *test)
{
struct xe_device *xe = test->priv;
if (!xe_guc_g2g_wanted(xe))
kunit_skip(test, "G2G not enabled");
g2g_check_skip(test);
g2g_pm_runtime_get(test);
kunit_info(test, "Testing default CTBs\n");
g2g_run_test(test, xe);
kunit_release_action(test, &g2g_pm_runtime_put, xe);
}
/*
* More complex test that re-creates the CTBs in various location to
* test access to each location from each GuC. Can be run even on
* systems that do not enable G2G by default. On the other hand,
* because it recreates the CTBs, if something goes wrong it could
* leave the system with broken G2G comms.
*/
static void xe_live_guc_g2g_kunit_allmem(struct kunit *test)
{
struct xe_device *xe = test->priv;
int ret;
g2g_check_skip(test);
g2g_pm_runtime_get(test);
/* Make sure to leave the system as we found it */
ret = kunit_add_action_or_reset(test, g2g_recreate_ctb, xe);
KUNIT_ASSERT_EQ_MSG(test, 0, ret, "Failed to register CTB re-creation action\n");
kunit_info(test, "Testing CTB type 'default'...\n");
g2g_reinit(test, xe, G2G_CTB_TYPE_DEFAULT, NULL);
g2g_run_test(test, xe);
kunit_info(test, "Testing CTB type 'host'...\n");
g2g_reinit(test, xe, G2G_CTB_TYPE_HOST, NULL);
g2g_run_test(test, xe);
if (IS_DGFX(xe)) {
struct xe_tile *tile;
int id;
for_each_tile(tile, xe, id) {
kunit_info(test, "Testing CTB type 'tile: #%d'...\n", id);
g2g_reinit(test, xe, G2G_CTB_TYPE_TILE, tile);
g2g_run_test(test, xe);
}
} else {
kunit_info(test, "Skipping local memory on integrated platform\n");
}
kunit_release_action(test, g2g_recreate_ctb, xe);
kunit_release_action(test, g2g_pm_runtime_put, xe);
}
static struct kunit_case xe_guc_g2g_tests[] = {
KUNIT_CASE_PARAM(xe_live_guc_g2g_kunit_default, xe_pci_live_device_gen_param),
KUNIT_CASE_PARAM(xe_live_guc_g2g_kunit_allmem, xe_pci_live_device_gen_param),
{}
};
VISIBLE_IF_KUNIT
struct kunit_suite xe_guc_g2g_test_suite = {
.name = "xe_guc_g2g",
.test_cases = xe_guc_g2g_tests,
.init = xe_kunit_helper_xe_device_live_test_init,
};
EXPORT_SYMBOL_IF_KUNIT(xe_guc_g2g_test_suite);

2
drivers/gpu/drm/xe/tests/xe_live_test_mod.c
View File

@ -10,12 +10,14 @@ extern struct kunit_suite xe_bo_shrink_test_suite;
extern struct kunit_suite xe_dma_buf_test_suite;
extern struct kunit_suite xe_migrate_test_suite;
extern struct kunit_suite xe_mocs_test_suite;
extern struct kunit_suite xe_guc_g2g_test_suite;
kunit_test_suite(xe_bo_test_suite);
kunit_test_suite(xe_bo_shrink_test_suite);
kunit_test_suite(xe_dma_buf_test_suite);
kunit_test_suite(xe_migrate_test_suite);
kunit_test_suite(xe_mocs_test_suite);
kunit_test_suite(xe_guc_g2g_test_suite);
MODULE_AUTHOR("Intel Corporation");
MODULE_LICENSE("GPL");

66
drivers/gpu/drm/xe/tests/xe_migrate.c
View File

@ -70,7 +70,7 @@ static int run_sanity_job(struct xe_migrate *m, struct xe_device *xe,
} } while (0)
static void test_copy(struct xe_migrate *m, struct xe_bo *bo,
struct kunit *test, u32 region)
struct kunit *test, u32 region, struct drm_exec *exec)
{
struct xe_device *xe = tile_to_xe(m->tile);
u64 retval, expected = 0;
@ -84,14 +84,15 @@ static void test_copy(struct xe_migrate *m, struct xe_bo *bo,
ttm_bo_type_kernel,
region |
XE_BO_FLAG_NEEDS_CPU_ACCESS |
XE_BO_FLAG_PINNED);
XE_BO_FLAG_PINNED,
exec);
if (IS_ERR(remote)) {
KUNIT_FAIL(test, "Failed to allocate remote bo for %s: %pe\n",
str, remote);
return;
}
err = xe_bo_validate(remote, NULL, false);
err = xe_bo_validate(remote, NULL, false, exec);
if (err) {
KUNIT_FAIL(test, "Failed to validate system bo for %s: %i\n",
str, err);
@ -161,13 +162,13 @@ static void test_copy(struct xe_migrate *m, struct xe_bo *bo,
}
static void test_copy_sysmem(struct xe_migrate *m, struct xe_bo *bo,
struct kunit *test)
struct drm_exec *exec, struct kunit *test)
{
test_copy(m, bo, test, XE_BO_FLAG_SYSTEM);
test_copy(m, bo, test, XE_BO_FLAG_SYSTEM, exec);
}
static void test_copy_vram(struct xe_migrate *m, struct xe_bo *bo,
struct kunit *test)
struct drm_exec *exec, struct kunit *test)
{
u32 region;
@ -178,10 +179,11 @@ static void test_copy_vram(struct xe_migrate *m, struct xe_bo *bo,
region = XE_BO_FLAG_VRAM1;
else
region = XE_BO_FLAG_VRAM0;
test_copy(m, bo, test, region);
test_copy(m, bo, test, region, exec);
}
static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)
static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test,
struct drm_exec *exec)
{
struct xe_tile *tile = m->tile;
struct xe_device *xe = tile_to_xe(tile);
@ -202,7 +204,8 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)
big = xe_bo_create_pin_map(xe, tile, m->q->vm, SZ_4M,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile));
XE_BO_FLAG_VRAM_IF_DGFX(tile),
exec);
if (IS_ERR(big)) {
KUNIT_FAIL(test, "Failed to allocate bo: %li\n", PTR_ERR(big));
goto vunmap;
@ -210,7 +213,8 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)
pt = xe_bo_create_pin_map(xe, tile, m->q->vm, XE_PAGE_SIZE,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile));
XE_BO_FLAG_VRAM_IF_DGFX(tile),
exec);
if (IS_ERR(pt)) {
KUNIT_FAIL(test, "Failed to allocate fake pt: %li\n",
PTR_ERR(pt));
@ -220,7 +224,8 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)
tiny = xe_bo_create_pin_map(xe, tile, m->q->vm,
2 * SZ_4K,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile));
XE_BO_FLAG_VRAM_IF_DGFX(tile),
exec);
if (IS_ERR(tiny)) {
KUNIT_FAIL(test, "Failed to allocate tiny fake pt: %li\n",
PTR_ERR(tiny));
@ -290,10 +295,10 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)
check(retval, expected, "Command clear small last value", test);
kunit_info(test, "Copying small buffer object to system\n");
test_copy_sysmem(m, tiny, test);
test_copy_sysmem(m, tiny, exec, test);
if (xe->info.tile_count > 1) {
kunit_info(test, "Copying small buffer object to other vram\n");
test_copy_vram(m, tiny, test);
test_copy_vram(m, tiny, exec, test);
}
/* Clear a big bo */
@ -312,10 +317,10 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)
check(retval, expected, "Command clear big last value", test);
kunit_info(test, "Copying big buffer object to system\n");
test_copy_sysmem(m, big, test);
test_copy_sysmem(m, big, exec, test);
if (xe->info.tile_count > 1) {
kunit_info(test, "Copying big buffer object to other vram\n");
test_copy_vram(m, big, test);
test_copy_vram(m, big, exec, test);
}
out:
@ -343,10 +348,11 @@ static int migrate_test_run_device(struct xe_device *xe)
for_each_tile(tile, xe, id) {
struct xe_migrate *m = tile->migrate;
struct drm_exec *exec = XE_VALIDATION_OPT_OUT;
kunit_info(test, "Testing tile id %d.\n", id);
xe_vm_lock(m->q->vm, false);
xe_migrate_sanity_test(m, test);
xe_migrate_sanity_test(m, test, exec);
xe_vm_unlock(m->q->vm);
}
@ -490,7 +496,7 @@ static struct dma_fence *blt_copy(struct xe_tile *tile,
static void test_migrate(struct xe_device *xe, struct xe_tile *tile,
struct xe_bo *sys_bo, struct xe_bo *vram_bo, struct xe_bo *ccs_bo,
struct kunit *test)
struct drm_exec *exec, struct kunit *test)
{
struct dma_fence *fence;
u64 expected, retval;
@ -509,7 +515,7 @@ static void test_migrate(struct xe_device *xe, struct xe_tile *tile,
dma_fence_put(fence);
kunit_info(test, "Evict vram buffer object\n");
ret = xe_bo_evict(vram_bo);
ret = xe_bo_evict(vram_bo, exec);
if (ret) {
KUNIT_FAIL(test, "Failed to evict bo.\n");
return;
@ -538,7 +544,7 @@ static void test_migrate(struct xe_device *xe, struct xe_tile *tile,
dma_fence_put(fence);
kunit_info(test, "Restore vram buffer object\n");
ret = xe_bo_validate(vram_bo, NULL, false);
ret = xe_bo_validate(vram_bo, NULL, false, exec);
if (ret) {
KUNIT_FAIL(test, "Failed to validate vram bo for: %li\n", ret);
return;
@ -636,13 +642,14 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
{
struct xe_bo *sys_bo, *vram_bo = NULL, *ccs_bo = NULL;
unsigned int bo_flags = XE_BO_FLAG_VRAM_IF_DGFX(tile);
struct drm_exec *exec;
long ret;
sys_bo = xe_bo_create_user(xe, NULL, NULL, SZ_4M,
sys_bo = xe_bo_create_user(xe, NULL, SZ_4M,
DRM_XE_GEM_CPU_CACHING_WC,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_NEEDS_CPU_ACCESS |
XE_BO_FLAG_PINNED);
XE_BO_FLAG_PINNED, NULL);
if (IS_ERR(sys_bo)) {
KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@ -650,8 +657,9 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
return;
}
exec = XE_VALIDATION_OPT_OUT;
xe_bo_lock(sys_bo, false);
ret = xe_bo_validate(sys_bo, NULL, false);
ret = xe_bo_validate(sys_bo, NULL, false, exec);
if (ret) {
KUNIT_FAIL(test, "Failed to validate system bo for: %li\n", ret);
goto free_sysbo;
@ -664,10 +672,10 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
}
xe_bo_unlock(sys_bo);
ccs_bo = xe_bo_create_user(xe, NULL, NULL, SZ_4M,
ccs_bo = xe_bo_create_user(xe, NULL, SZ_4M,
DRM_XE_GEM_CPU_CACHING_WC,
bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
XE_BO_FLAG_PINNED);
XE_BO_FLAG_PINNED, NULL);
if (IS_ERR(ccs_bo)) {
KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@ -676,7 +684,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
}
xe_bo_lock(ccs_bo, false);
ret = xe_bo_validate(ccs_bo, NULL, false);
ret = xe_bo_validate(ccs_bo, NULL, false, exec);
if (ret) {
KUNIT_FAIL(test, "Failed to validate system bo for: %li\n", ret);
goto free_ccsbo;
@ -689,10 +697,10 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
}
xe_bo_unlock(ccs_bo);
vram_bo = xe_bo_create_user(xe, NULL, NULL, SZ_4M,
vram_bo = xe_bo_create_user(xe, NULL, SZ_4M,
DRM_XE_GEM_CPU_CACHING_WC,
bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
XE_BO_FLAG_PINNED);
XE_BO_FLAG_PINNED, NULL);
if (IS_ERR(vram_bo)) {
KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
PTR_ERR(vram_bo));
@ -700,7 +708,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
}
xe_bo_lock(vram_bo, false);
ret = xe_bo_validate(vram_bo, NULL, false);
ret = xe_bo_validate(vram_bo, NULL, false, exec);
if (ret) {
KUNIT_FAIL(test, "Failed to validate vram bo for: %li\n", ret);
goto free_vrambo;
@ -713,7 +721,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
}
test_clear(xe, tile, sys_bo, vram_bo, test);
test_migrate(xe, tile, sys_bo, vram_bo, ccs_bo, test);
test_migrate(xe, tile, sys_bo, vram_bo, ccs_bo, exec, test);
xe_bo_unlock(vram_bo);
xe_bo_lock(vram_bo, false);

225
drivers/gpu/drm/xe/tests/xe_pci.c
View File

@ -12,12 +12,219 @@
#include <kunit/test-bug.h>
#include <kunit/visibility.h>
#define PLATFORM_CASE(platform__, graphics_step__) \
{ \
.platform = XE_ ## platform__, \
.subplatform = XE_SUBPLATFORM_NONE, \
.step = { .graphics = STEP_ ## graphics_step__ } \
}
#define SUBPLATFORM_CASE(platform__, subplatform__, graphics_step__) \
{ \
.platform = XE_ ## platform__, \
.subplatform = XE_SUBPLATFORM_ ## platform__ ## _ ## subplatform__, \
.step = { .graphics = STEP_ ## graphics_step__ } \
}
#define GMDID_CASE(platform__, graphics_verx100__, graphics_step__, \
media_verx100__, media_step__) \
{ \
.platform = XE_ ## platform__, \
.subplatform = XE_SUBPLATFORM_NONE, \
.graphics_verx100 = graphics_verx100__, \
.media_verx100 = media_verx100__, \
.step = { .graphics = STEP_ ## graphics_step__, \
.media = STEP_ ## media_step__ } \
}
static const struct xe_pci_fake_data cases[] = {
PLATFORM_CASE(TIGERLAKE, B0),
PLATFORM_CASE(DG1, A0),
PLATFORM_CASE(DG1, B0),
PLATFORM_CASE(ALDERLAKE_S, A0),
PLATFORM_CASE(ALDERLAKE_S, B0),
PLATFORM_CASE(ALDERLAKE_S, C0),
PLATFORM_CASE(ALDERLAKE_S, D0),
PLATFORM_CASE(ALDERLAKE_P, A0),
PLATFORM_CASE(ALDERLAKE_P, B0),
PLATFORM_CASE(ALDERLAKE_P, C0),
SUBPLATFORM_CASE(ALDERLAKE_S, RPLS, D0),
SUBPLATFORM_CASE(ALDERLAKE_P, RPLU, E0),
SUBPLATFORM_CASE(DG2, G10, C0),
SUBPLATFORM_CASE(DG2, G11, B1),
SUBPLATFORM_CASE(DG2, G12, A1),
GMDID_CASE(METEORLAKE, 1270, A0, 1300, A0),
GMDID_CASE(METEORLAKE, 1271, A0, 1300, A0),
GMDID_CASE(METEORLAKE, 1274, A0, 1300, A0),
GMDID_CASE(LUNARLAKE, 2004, A0, 2000, A0),
GMDID_CASE(LUNARLAKE, 2004, B0, 2000, A0),
GMDID_CASE(BATTLEMAGE, 2001, A0, 1301, A1),
GMDID_CASE(PANTHERLAKE, 3000, A0, 3000, A0),
};
KUNIT_ARRAY_PARAM(platform, cases, xe_pci_fake_data_desc);
/**
* xe_pci_fake_data_gen_params - Generate struct xe_pci_fake_data parameters
* @prev: the pointer to the previous parameter to iterate from or NULL
* @desc: output buffer with minimum size of KUNIT_PARAM_DESC_SIZE
*
* This function prepares struct xe_pci_fake_data parameter.
*
* To be used only as a parameter generator function in &KUNIT_CASE_PARAM.
*
* Return: pointer to the next parameter or NULL if no more parameters
*/
const void *xe_pci_fake_data_gen_params(const void *prev, char *desc)
{
return platform_gen_params(prev, desc);
}
EXPORT_SYMBOL_IF_KUNIT(xe_pci_fake_data_gen_params);
static const struct xe_device_desc *lookup_desc(enum xe_platform p)
{
const struct xe_device_desc *desc;
const struct pci_device_id *ids;
for (ids = pciidlist; ids->driver_data; ids++) {
desc = (const void *)ids->driver_data;
if (desc->platform == p)
return desc;
}
return NULL;
}
static const struct xe_subplatform_desc *lookup_sub_desc(enum xe_platform p, enum xe_subplatform s)
{
const struct xe_device_desc *desc = lookup_desc(p);
const struct xe_subplatform_desc *spd;
if (desc && desc->subplatforms)
for (spd = desc->subplatforms; spd->subplatform; spd++)
if (spd->subplatform == s)
return spd;
return NULL;
}
static const char *lookup_platform_name(enum xe_platform p)
{
const struct xe_device_desc *desc = lookup_desc(p);
return desc ? desc->platform_name : "INVALID";
}
static const char *__lookup_subplatform_name(enum xe_platform p, enum xe_subplatform s)
{
const struct xe_subplatform_desc *desc = lookup_sub_desc(p, s);
return desc ? desc->name : "INVALID";
}
static const char *lookup_subplatform_name(enum xe_platform p, enum xe_subplatform s)
{
return s == XE_SUBPLATFORM_NONE ? "" : __lookup_subplatform_name(p, s);
}
static const char *subplatform_prefix(enum xe_subplatform s)
{
return s == XE_SUBPLATFORM_NONE ? "" : " ";
}
static const char *step_prefix(enum xe_step step)
{
return step == STEP_NONE ? "" : " ";
}
static const char *step_name(enum xe_step step)
{
return step == STEP_NONE ? "" : xe_step_name(step);
}
static const char *sriov_prefix(enum xe_sriov_mode mode)
{
return mode <= XE_SRIOV_MODE_NONE ? "" : " ";
}
static const char *sriov_name(enum xe_sriov_mode mode)
{
return mode <= XE_SRIOV_MODE_NONE ? "" : xe_sriov_mode_to_string(mode);
}
static const char *lookup_graphics_name(unsigned int verx100)
{
const struct xe_ip *ip = find_graphics_ip(verx100);
return ip ? ip->name : "";
}
static const char *lookup_media_name(unsigned int verx100)
{
const struct xe_ip *ip = find_media_ip(verx100);
return ip ? ip->name : "";
}
/**
* xe_pci_fake_data_desc - Describe struct xe_pci_fake_data parameter
* @param: the &struct xe_pci_fake_data parameter to describe
* @desc: output buffer with minimum size of KUNIT_PARAM_DESC_SIZE
*
* This function prepares description of the struct xe_pci_fake_data parameter.
*
* It is tailored for use in parameterized KUnit tests where parameter generator
* is based on the struct xe_pci_fake_data arrays.
*/
void xe_pci_fake_data_desc(const struct xe_pci_fake_data *param, char *desc)
{
if (param->graphics_verx100 || param->media_verx100)
snprintf(desc, KUNIT_PARAM_DESC_SIZE, "%s%s%s %u.%02u(%s)%s%s %u.%02u(%s)%s%s%s%s",
lookup_platform_name(param->platform),
subplatform_prefix(param->subplatform),
lookup_subplatform_name(param->platform, param->subplatform),
param->graphics_verx100 / 100, param->graphics_verx100 % 100,
lookup_graphics_name(param->graphics_verx100),
step_prefix(param->step.graphics), step_name(param->step.graphics),
param->media_verx100 / 100, param->media_verx100 % 100,
lookup_media_name(param->media_verx100),
step_prefix(param->step.media), step_name(param->step.media),
sriov_prefix(param->sriov_mode), sriov_name(param->sriov_mode));
else
snprintf(desc, KUNIT_PARAM_DESC_SIZE, "%s%s%s%s%s%s%s",
lookup_platform_name(param->platform),
subplatform_prefix(param->subplatform),
lookup_subplatform_name(param->platform, param->subplatform),
step_prefix(param->step.graphics), step_name(param->step.graphics),
sriov_prefix(param->sriov_mode), sriov_name(param->sriov_mode));
}
EXPORT_SYMBOL_IF_KUNIT(xe_pci_fake_data_desc);
static void xe_ip_kunit_desc(const struct xe_ip *param, char *desc)
{
snprintf(desc, KUNIT_PARAM_DESC_SIZE, "%u.%02u %s",
param->verx100 / 100, param->verx100 % 100, param->name);
}
/*
* Pre-GMDID Graphics and Media IPs definitions.
*
* Mimic the way GMDID IPs are declared so the same
* param generator can be used for both
*/
static const struct xe_ip pre_gmdid_graphics_ips[] = {
graphics_ip_xelp,
graphics_ip_xelpp,
graphics_ip_xehpg,
graphics_ip_xehpc,
};
static const struct xe_ip pre_gmdid_media_ips[] = {
media_ip_xem,
media_ip_xehpm,
};
KUNIT_ARRAY_PARAM(pre_gmdid_graphics_ip, pre_gmdid_graphics_ips, xe_ip_kunit_desc);
KUNIT_ARRAY_PARAM(pre_gmdid_media_ip, pre_gmdid_media_ips, xe_ip_kunit_desc);
KUNIT_ARRAY_PARAM(graphics_ip, graphics_ips, xe_ip_kunit_desc);
KUNIT_ARRAY_PARAM(media_ip, media_ips, xe_ip_kunit_desc);
@ -46,6 +253,13 @@ KUNIT_ARRAY_PARAM(pci_id, pciidlist, xe_pci_id_kunit_desc);
*/
const void *xe_pci_graphics_ip_gen_param(const void *prev, char *desc)
{
const void *next = pre_gmdid_graphics_ip_gen_params(prev, desc);
if (next)
return next;
if (is_insidevar(prev, pre_gmdid_graphics_ips))
prev = NULL;
return graphics_ip_gen_params(prev, desc);
}
EXPORT_SYMBOL_IF_KUNIT(xe_pci_graphics_ip_gen_param);
@ -63,6 +277,13 @@ EXPORT_SYMBOL_IF_KUNIT(xe_pci_graphics_ip_gen_param);
*/
const void *xe_pci_media_ip_gen_param(const void *prev, char *desc)
{
const void *next = pre_gmdid_media_ip_gen_params(prev, desc);
if (next)
return next;
if (is_insidevar(prev, pre_gmdid_media_ips))
prev = NULL;
return media_ip_gen_params(prev, desc);
}
EXPORT_SYMBOL_IF_KUNIT(xe_pci_media_ip_gen_param);
@ -94,10 +315,10 @@ static void fake_read_gmdid(struct xe_device *xe, enum xe_gmdid_type type,
if (type == GMDID_MEDIA) {
*ver = data->media_verx100;
*revid = xe_step_to_gmdid(data->media_step);
*revid = xe_step_to_gmdid(data->step.media);
} else {
*ver = data->graphics_verx100;
*revid = xe_step_to_gmdid(data->graphics_step);
*revid = xe_step_to_gmdid(data->step.graphics);
}
}

6
drivers/gpu/drm/xe/tests/xe_pci_test.h
View File

@ -10,6 +10,7 @@
#include "xe_platform_types.h"
#include "xe_sriov_types.h"
#include "xe_step_types.h"
struct xe_device;
@ -17,13 +18,14 @@ struct xe_pci_fake_data {
enum xe_sriov_mode sriov_mode;
enum xe_platform platform;
enum xe_subplatform subplatform;
struct xe_step_info step;
u32 graphics_verx100;
u32 media_verx100;
u32 graphics_step;
u32 media_step;
};
int xe_pci_fake_device_init(struct xe_device *xe);
const void *xe_pci_fake_data_gen_params(const void *prev, char *desc);
void xe_pci_fake_data_desc(const struct xe_pci_fake_data *param, char *desc);
const void *xe_pci_graphics_ip_gen_param(const void *prev, char *desc);
const void *xe_pci_media_ip_gen_param(const void *prev, char *desc);

91
drivers/gpu/drm/xe/tests/xe_wa_test.c
View File

@ -15,87 +15,10 @@
#include "xe_tuning.h"
#include "xe_wa.h"
struct platform_test_case {
const char *name;
enum xe_platform platform;
enum xe_subplatform subplatform;
u32 graphics_verx100;
u32 media_verx100;
struct xe_step_info step;
};
#define PLATFORM_CASE(platform__, graphics_step__) \
{ \
.name = #platform__ " (" #graphics_step__ ")", \
.platform = XE_ ## platform__, \
.subplatform = XE_SUBPLATFORM_NONE, \
.step = { .graphics = STEP_ ## graphics_step__ } \
}
#define SUBPLATFORM_CASE(platform__, subplatform__, graphics_step__) \
{ \
.name = #platform__ "_" #subplatform__ " (" #graphics_step__ ")", \
.platform = XE_ ## platform__, \
.subplatform = XE_SUBPLATFORM_ ## platform__ ## _ ## subplatform__, \
.step = { .graphics = STEP_ ## graphics_step__ } \
}
#define GMDID_CASE(platform__, graphics_verx100__, graphics_step__, \
media_verx100__, media_step__) \
{ \
.name = #platform__ " (g:" #graphics_step__ ", m:" #media_step__ ")",\
.platform = XE_ ## platform__, \
.subplatform = XE_SUBPLATFORM_NONE, \
.graphics_verx100 = graphics_verx100__, \
.media_verx100 = media_verx100__, \
.step = { .graphics = STEP_ ## graphics_step__, \
.media = STEP_ ## media_step__ } \
}
static const struct platform_test_case cases[] = {
PLATFORM_CASE(TIGERLAKE, B0),
PLATFORM_CASE(DG1, A0),
PLATFORM_CASE(DG1, B0),
PLATFORM_CASE(ALDERLAKE_S, A0),
PLATFORM_CASE(ALDERLAKE_S, B0),
PLATFORM_CASE(ALDERLAKE_S, C0),
PLATFORM_CASE(ALDERLAKE_S, D0),
PLATFORM_CASE(ALDERLAKE_P, A0),
PLATFORM_CASE(ALDERLAKE_P, B0),
PLATFORM_CASE(ALDERLAKE_P, C0),
SUBPLATFORM_CASE(ALDERLAKE_S, RPLS, D0),
SUBPLATFORM_CASE(ALDERLAKE_P, RPLU, E0),
SUBPLATFORM_CASE(DG2, G10, C0),
SUBPLATFORM_CASE(DG2, G11, B1),
SUBPLATFORM_CASE(DG2, G12, A1),
GMDID_CASE(METEORLAKE, 1270, A0, 1300, A0),
GMDID_CASE(METEORLAKE, 1271, A0, 1300, A0),
GMDID_CASE(METEORLAKE, 1274, A0, 1300, A0),
GMDID_CASE(LUNARLAKE, 2004, A0, 2000, A0),
GMDID_CASE(LUNARLAKE, 2004, B0, 2000, A0),
GMDID_CASE(BATTLEMAGE, 2001, A0, 1301, A1),
GMDID_CASE(PANTHERLAKE, 3000, A0, 3000, A0),
};
static void platform_desc(const struct platform_test_case *t, char *desc)
{
strscpy(desc, t->name, KUNIT_PARAM_DESC_SIZE);
}
KUNIT_ARRAY_PARAM(platform, cases, platform_desc);
static int xe_wa_test_init(struct kunit *test)
{
const struct platform_test_case *param = test->param_value;
struct xe_pci_fake_data data = {
.platform = param->platform,
.subplatform = param->subplatform,
.graphics_verx100 = param->graphics_verx100,
.media_verx100 = param->media_verx100,
.graphics_step = param->step.graphics,
.media_step = param->step.media,
};
const struct xe_pci_fake_data *param = test->param_value;
struct xe_pci_fake_data data = *param;
struct xe_device *xe;
struct device *dev;
int ret;
@ -120,13 +43,6 @@ static int xe_wa_test_init(struct kunit *test)
return 0;
}
static void xe_wa_test_exit(struct kunit *test)
{
struct xe_device *xe = test->priv;
drm_kunit_helper_free_device(test, xe->drm.dev);
}
static void xe_wa_gt(struct kunit *test)
{
struct xe_device *xe = test->priv;
@ -144,14 +60,13 @@ static void xe_wa_gt(struct kunit *test)
}
static struct kunit_case xe_wa_tests[] = {
KUNIT_CASE_PARAM(xe_wa_gt, platform_gen_params),
KUNIT_CASE_PARAM(xe_wa_gt, xe_pci_fake_data_gen_params),
{}
};
static struct kunit_suite xe_rtp_test_suite = {
.name = "xe_wa",
.init = xe_wa_test_init,
.exit = xe_wa_test_exit,
.test_cases = xe_wa_tests,
};

4
drivers/gpu/drm/xe/xe_bb.c
View File

@ -64,7 +64,7 @@ struct xe_bb *xe_bb_ccs_new(struct xe_gt *gt, u32 dwords,
enum xe_sriov_vf_ccs_rw_ctxs ctx_id)
{
struct xe_bb *bb = kmalloc(sizeof(*bb), GFP_KERNEL);
struct xe_tile *tile = gt_to_tile(gt);
struct xe_device *xe = gt_to_xe(gt);
struct xe_sa_manager *bb_pool;
int err;
@ -78,7 +78,7 @@ struct xe_bb *xe_bb_ccs_new(struct xe_gt *gt, u32 dwords,
* So, this extra DW acts as a guard here.
*/
bb_pool = tile->sriov.vf.ccs[ctx_id].mem.ccs_bb_pool;
bb_pool = xe->sriov.vf.ccs.contexts[ctx_id].mem.ccs_bb_pool;
bb->bo = xe_sa_bo_new(bb_pool, 4 * (dwords + 1));
if (IS_ERR(bb->bo)) {

897
drivers/gpu/drm/xe/xe_bo.c
View File

File diff suppressed because it is too large Load Diff

78
drivers/gpu/drm/xe/xe_bo.h
View File

@ -10,6 +10,7 @@
#include "xe_bo_types.h"
#include "xe_macros.h"
#include "xe_validation.h"
#include "xe_vm_types.h"
#include "xe_vm.h"
#include "xe_vram_types.h"
@ -88,40 +89,34 @@ struct sg_table;
struct xe_bo *xe_bo_alloc(void);
void xe_bo_free(struct xe_bo *bo);
struct xe_bo *___xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo,
struct xe_tile *tile, struct dma_resv *resv,
struct ttm_lru_bulk_move *bulk, size_t size,
u16 cpu_caching, enum ttm_bo_type type,
u32 flags);
struct xe_bo *
xe_bo_create_locked_range(struct xe_device *xe,
struct xe_tile *tile, struct xe_vm *vm,
size_t size, u64 start, u64 end,
enum ttm_bo_type type, u32 flags, u64 alignment);
struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
struct xe_tile *tile, struct dma_resv *resv,
struct ttm_lru_bulk_move *bulk, size_t size,
u16 cpu_caching, enum ttm_bo_type type,
u32 flags, struct drm_exec *exec);
struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile,
struct xe_vm *vm, size_t size,
enum ttm_bo_type type, u32 flags);
struct xe_bo *xe_bo_create(struct xe_device *xe, struct xe_tile *tile,
struct xe_vm *vm, size_t size,
enum ttm_bo_type type, u32 flags);
struct xe_bo *xe_bo_create_user(struct xe_device *xe, struct xe_tile *tile,
struct xe_vm *vm, size_t size,
u16 cpu_caching,
u32 flags);
enum ttm_bo_type type, u32 flags,
struct drm_exec *exec);
struct xe_bo *xe_bo_create_user(struct xe_device *xe, struct xe_vm *vm, size_t size,
u16 cpu_caching, u32 flags, struct drm_exec *exec);
struct xe_bo *xe_bo_create_pin_map(struct xe_device *xe, struct xe_tile *tile,
struct xe_vm *vm, size_t size,
enum ttm_bo_type type, u32 flags);
struct xe_bo *xe_bo_create_pin_map_at(struct xe_device *xe, struct xe_tile *tile,
struct xe_vm *vm, size_t size, u64 offset,
enum ttm_bo_type type, u32 flags);
struct xe_bo *xe_bo_create_pin_map_at_aligned(struct xe_device *xe,
struct xe_tile *tile,
struct xe_vm *vm,
size_t size, u64 offset,
enum ttm_bo_type type, u32 flags,
u64 alignment);
enum ttm_bo_type type, u32 flags,
struct drm_exec *exec);
struct xe_bo *xe_bo_create_pin_map_novm(struct xe_device *xe, struct xe_tile *tile,
size_t size, enum ttm_bo_type type, u32 flags,
bool intr);
struct xe_bo *xe_bo_create_pin_range_novm(struct xe_device *xe, struct xe_tile *tile,
size_t size, u64 start, u64 end,
enum ttm_bo_type type, u32 flags);
struct xe_bo *
xe_bo_create_pin_map_at_novm(struct xe_device *xe, struct xe_tile *tile,
size_t size, u64 offset, enum ttm_bo_type type,
u32 flags, u64 alignment, bool intr);
struct xe_bo *xe_managed_bo_create_pin_map(struct xe_device *xe, struct xe_tile *tile,
size_t size, u32 flags);
void xe_managed_bo_unpin_map_no_vm(struct xe_bo *bo);
struct xe_bo *xe_managed_bo_create_from_data(struct xe_device *xe, struct xe_tile *tile,
const void *data, size_t size, u32 flags);
int xe_managed_bo_reinit_in_vram(struct xe_device *xe, struct xe_tile *tile, struct xe_bo **src);
@ -200,11 +195,12 @@ static inline void xe_bo_unlock_vm_held(struct xe_bo *bo)
}
}
int xe_bo_pin_external(struct xe_bo *bo, bool in_place);
int xe_bo_pin(struct xe_bo *bo);
int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
void xe_bo_unpin_external(struct xe_bo *bo);
void xe_bo_unpin(struct xe_bo *bo);
int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict);
int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
struct drm_exec *exec);
static inline bool xe_bo_is_pinned(struct xe_bo *bo)
{
@ -285,8 +281,9 @@ uint64_t vram_region_gpu_offset(struct ttm_resource *res);
bool xe_bo_can_migrate(struct xe_bo *bo, u32 mem_type);
int xe_bo_migrate(struct xe_bo *bo, u32 mem_type);
int xe_bo_evict(struct xe_bo *bo);
int xe_bo_migrate(struct xe_bo *bo, u32 mem_type, struct ttm_operation_ctx *ctc,
struct drm_exec *exec);
int xe_bo_evict(struct xe_bo *bo, struct drm_exec *exec);
int xe_bo_evict_pinned(struct xe_bo *bo);
int xe_bo_notifier_prepare_pinned(struct xe_bo *bo);
@ -315,6 +312,21 @@ static inline size_t xe_bo_ccs_pages_start(struct xe_bo *bo)
return PAGE_ALIGN(xe_bo_size(bo));
}
/**
* xe_bo_has_valid_ccs_bb - Check if CCS's BBs were setup for the BO.
* @bo: the &xe_bo to check
*
* The CCS's BBs should only be setup by the driver VF, but it is safe
* to call this function also by non-VF driver.
*
* Return: true iff the CCS's BBs are setup, false otherwise.
*/
static inline bool xe_bo_has_valid_ccs_bb(struct xe_bo *bo)
{
return bo->bb_ccs[XE_SRIOV_VF_CCS_READ_CTX] &&
bo->bb_ccs[XE_SRIOV_VF_CCS_WRITE_CTX];
}
static inline bool xe_bo_has_pages(struct xe_bo *bo)
{
if ((bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm)) ||

15
drivers/gpu/drm/xe/xe_bo_types.h
View File

@ -25,7 +25,9 @@ struct xe_vm;
/* TODO: To be selected with VM_MADVISE */
#define XE_BO_PRIORITY_NORMAL 1
/** @xe_bo: XE buffer object */
/**
* struct xe_bo - Xe buffer object
*/
struct xe_bo {
/** @ttm: TTM base buffer object */
struct ttm_buffer_object ttm;
@ -47,7 +49,7 @@ struct xe_bo {
struct xe_ggtt_node *ggtt_node[XE_MAX_TILES_PER_DEVICE];
/** @vmap: iosys map of this buffer */
struct iosys_map vmap;
/** @ttm_kmap: TTM bo kmap object for internal use only. Keep off. */
/** @kmap: TTM bo kmap object for internal use only. Keep off. */
struct ttm_bo_kmap_obj kmap;
/** @pinned_link: link to present / evicted list of pinned BO */
struct list_head pinned_link;
@ -82,10 +84,10 @@ struct xe_bo {
/** @created: Whether the bo has passed initial creation */
bool created;
/** @ccs_cleared */
/** @ccs_cleared: true means that CCS region of BO is already cleared */
bool ccs_cleared;
/** @bb_ccs_rw: BB instructions of CCS read/write. Valid only for VF */
/** @bb_ccs: BB instructions of CCS read/write. Valid only for VF */
struct xe_bb *bb_ccs[XE_SRIOV_VF_CCS_CTX_COUNT];
/**
@ -99,9 +101,10 @@ struct xe_bo {
struct drm_pagemap_devmem devmem_allocation;
/** @vram_userfault_link: Link into @mem_access.vram_userfault.list */
struct list_head vram_userfault_link;
struct list_head vram_userfault_link;
/** @min_align: minimum alignment needed for this BO if different
/**
* @min_align: minimum alignment needed for this BO if different
* from default
*/
u64 min_align;

468
drivers/gpu/drm/xe/xe_configfs.c
View File

@ -4,6 +4,7 @@
*/
#include <linux/bitops.h>
#include <linux/ctype.h>
#include <linux/configfs.h>
#include <linux/cleanup.h>
#include <linux/find.h>
@ -12,6 +13,7 @@
#include <linux/pci.h>
#include <linux/string.h>
#include "instructions/xe_mi_commands.h"
#include "xe_configfs.h"
#include "xe_hw_engine_types.h"
#include "xe_module.h"
@ -21,7 +23,7 @@
* DOC: Xe Configfs
*
* Overview
* =========
* ========
*
* Configfs is a filesystem-based manager of kernel objects. XE KMD registers a
* configfs subsystem called ``xe`` that creates a directory in the mounted
@ -34,7 +36,7 @@
*
* To create a device, the ``xe`` module should already be loaded, but some
* attributes can only be set before binding the device. It can be accomplished
* by blocking the driver autoprobe:
* by blocking the driver autoprobe::
*
* # echo 0 > /sys/bus/pci/drivers_autoprobe
* # modprobe xe
@ -115,6 +117,45 @@
*
* This attribute can only be set before binding to the device.
*
* Context restore BB
* ------------------
*
* Allow to execute a batch buffer during any context switches. When the
* GPU is restoring the context, it executes additional commands. It's useful
* for testing additional workarounds and validating certain HW behaviors: it's
* not intended for normal execution and will taint the kernel with TAINT_TEST
* when used.
*
* Currently this is implemented only for post and mid context restore.
* Examples:
*
* #. Execute a LRI command to write 0xDEADBEEF to register 0x4f10 after the
* normal context restore::
*
* # echo 'rcs cmd 11000001 4F100 DEADBEEF' \
* > /sys/kernel/config/xe/0000:03:00.0/ctx_restore_post_bb
*
* #. Execute a LRI command to write 0xDEADBEEF to register 0x4f10 at the
* beginning of the context restore::
*
* # echo 'rcs cmd 11000001 4F100 DEADBEEF' \
* > /sys/kernel/config/xe/0000:03:00.0/ctx_restore_mid_bb
* #. Load certain values in a couple of registers (it can be used as a simpler
* alternative to the `cmd`) action::
*
* # cat > /sys/kernel/config/xe/0000:03:00.0/ctx_restore_post_bb <<EOF
* rcs reg 4F100 DEADBEEF
* rcs reg 4F104 FFFFFFFF
* EOF
*
* .. note::
*
* When using multiple lines, make sure to use a command that is
* implemented with a single write syscall, like HEREDOC.
*
* These attributes can only be set before binding to the device.
*
* Remove devices
* ==============
*
@ -123,17 +164,27 @@
* # rmdir /sys/kernel/config/xe/0000:03:00.0/
*/
/* Similar to struct xe_bb, but not tied to HW (yet) */
struct wa_bb {
u32 *cs;
u32 len; /* in dwords */
};
struct xe_config_group_device {
struct config_group group;
struct xe_config_device {
u64 engines_allowed;
struct wa_bb ctx_restore_post_bb[XE_ENGINE_CLASS_MAX];
struct wa_bb ctx_restore_mid_bb[XE_ENGINE_CLASS_MAX];
bool survivability_mode;
bool enable_psmi;
} config;
/* protects attributes */
struct mutex lock;
/* matching descriptor */
const struct xe_device_desc *desc;
};
static const struct xe_config_device device_defaults = {
@ -150,6 +201,7 @@ static void set_device_defaults(struct xe_config_device *config)
struct engine_info {
const char *cls;
u64 mask;
enum xe_engine_class engine_class;
};
/* Some helpful macros to aid on the sizing of buffer allocation when parsing */
@ -157,12 +209,12 @@ struct engine_info {
#define MAX_ENGINE_INSTANCE_CHARS 2
static const struct engine_info engine_info[] = {
{ .cls = "rcs", .mask = XE_HW_ENGINE_RCS_MASK },
{ .cls = "bcs", .mask = XE_HW_ENGINE_BCS_MASK },
{ .cls = "vcs", .mask = XE_HW_ENGINE_VCS_MASK },
{ .cls = "vecs", .mask = XE_HW_ENGINE_VECS_MASK },
{ .cls = "ccs", .mask = XE_HW_ENGINE_CCS_MASK },
{ .cls = "gsccs", .mask = XE_HW_ENGINE_GSCCS_MASK },
{ .cls = "rcs", .mask = XE_HW_ENGINE_RCS_MASK, .engine_class = XE_ENGINE_CLASS_RENDER },
{ .cls = "bcs", .mask = XE_HW_ENGINE_BCS_MASK, .engine_class = XE_ENGINE_CLASS_COPY },
{ .cls = "vcs", .mask = XE_HW_ENGINE_VCS_MASK, .engine_class = XE_ENGINE_CLASS_VIDEO_DECODE },
{ .cls = "vecs", .mask = XE_HW_ENGINE_VECS_MASK, .engine_class = XE_ENGINE_CLASS_VIDEO_ENHANCE },
{ .cls = "ccs", .mask = XE_HW_ENGINE_CCS_MASK, .engine_class = XE_ENGINE_CLASS_COMPUTE },
{ .cls = "gsccs", .mask = XE_HW_ENGINE_GSCCS_MASK, .engine_class = XE_ENGINE_CLASS_OTHER },
};
static struct xe_config_group_device *to_xe_config_group_device(struct config_item *item)
@ -251,7 +303,18 @@ static ssize_t engines_allowed_show(struct config_item *item, char *page)
return p - page;
}
static bool lookup_engine_mask(const char *pattern, u64 *mask)
/*
* Lookup engine_info. If @mask is not NULL, reduce the mask according to the
* instance in @pattern.
*
* Examples of inputs:
* - lookup_engine_info("rcs0", &mask): return "rcs" entry from @engine_info and
* mask == BIT_ULL(XE_HW_ENGINE_RCS0)
* - lookup_engine_info("rcs*", &mask): return "rcs" entry from @engine_info and
* mask == XE_HW_ENGINE_RCS_MASK
* - lookup_engine_info("rcs", NULL): return "rcs" entry from @engine_info
*/
static const struct engine_info *lookup_engine_info(const char *pattern, u64 *mask)
{
for (size_t i = 0; i < ARRAY_SIZE(engine_info); i++) {
u8 instance;
@ -261,44 +324,62 @@ static bool lookup_engine_mask(const char *pattern, u64 *mask)
continue;
pattern += strlen(engine_info[i].cls);
if (!mask && !*pattern)
return &engine_info[i];
if (!strcmp(pattern, "*")) {
*mask = engine_info[i].mask;
return true;
return &engine_info[i];
}
if (kstrtou8(pattern, 10, &instance))
return false;
return NULL;
bit = __ffs64(engine_info[i].mask) + instance;
if (bit >= fls64(engine_info[i].mask))
return false;
return NULL;
*mask = BIT_ULL(bit);
return true;
return &engine_info[i];
}
return false;
return NULL;
}
static int parse_engine(const char *s, const char *end_chars, u64 *mask,
const struct engine_info **pinfo)
{
char buf[MAX_ENGINE_CLASS_CHARS + MAX_ENGINE_INSTANCE_CHARS + 1];
const struct engine_info *info;
size_t len;
len = strcspn(s, end_chars);
if (len >= sizeof(buf))
return -EINVAL;
memcpy(buf, s, len);
buf[len] = '\0';
info = lookup_engine_info(buf, mask);
if (!info)
return -ENOENT;
if (pinfo)
*pinfo = info;
return len;
}
static ssize_t engines_allowed_store(struct config_item *item, const char *page,
size_t len)
{
struct xe_config_group_device *dev = to_xe_config_group_device(item);
size_t patternlen, p;
ssize_t patternlen, p;
u64 mask, val = 0;
for (p = 0; p < len; p += patternlen + 1) {
char buf[MAX_ENGINE_CLASS_CHARS + MAX_ENGINE_INSTANCE_CHARS + 1];
patternlen = strcspn(page + p, ",\n");
if (patternlen >= sizeof(buf))
return -EINVAL;
memcpy(buf, page + p, patternlen);
buf[patternlen] = '\0';
if (!lookup_engine_mask(buf, &mask))
patternlen = parse_engine(page + p, ",\n", &mask, NULL);
if (patternlen < 0)
return -EINVAL;
val |= mask;
@ -339,11 +420,250 @@ static ssize_t enable_psmi_store(struct config_item *item, const char *page, siz
return len;
}
static bool wa_bb_read_advance(bool dereference, char **p,
const char *append, size_t len,
size_t *max_size)
{
if (dereference) {
if (len >= *max_size)
return false;
*max_size -= len;
if (append)
memcpy(*p, append, len);
}
*p += len;
return true;
}
static ssize_t wa_bb_show(struct xe_config_group_device *dev,
struct wa_bb wa_bb[static XE_ENGINE_CLASS_MAX],
char *data, size_t sz)
{
char *p = data;
guard(mutex)(&dev->lock);
for (size_t i = 0; i < ARRAY_SIZE(engine_info); i++) {
enum xe_engine_class ec = engine_info[i].engine_class;
size_t len;
if (!wa_bb[ec].len)
continue;
len = snprintf(p, sz, "%s:", engine_info[i].cls);
if (!wa_bb_read_advance(data, &p, NULL, len, &sz))
return -ENOBUFS;
for (size_t j = 0; j < wa_bb[ec].len; j++) {
len = snprintf(p, sz, " %08x", wa_bb[ec].cs[j]);
if (!wa_bb_read_advance(data, &p, NULL, len, &sz))
return -ENOBUFS;
}
if (!wa_bb_read_advance(data, &p, "\n", 1, &sz))
return -ENOBUFS;
}
if (!wa_bb_read_advance(data, &p, "", 1, &sz))
return -ENOBUFS;
/* Reserve one more to match check for '\0' */
if (!data)
p++;
return p - data;
}
static ssize_t ctx_restore_mid_bb_show(struct config_item *item, char *page)
{
struct xe_config_group_device *dev = to_xe_config_group_device(item);
return wa_bb_show(dev, dev->config.ctx_restore_mid_bb, page, SZ_4K);
}
static ssize_t ctx_restore_post_bb_show(struct config_item *item, char *page)
{
struct xe_config_group_device *dev = to_xe_config_group_device(item);
return wa_bb_show(dev, dev->config.ctx_restore_post_bb, page, SZ_4K);
}
static void wa_bb_append(struct wa_bb *wa_bb, u32 val)
{
if (wa_bb->cs)
wa_bb->cs[wa_bb->len] = val;
wa_bb->len++;
}
static ssize_t parse_hex(const char *line, u32 *pval)
{
char numstr[12];
const char *p;
ssize_t numlen;
p = line + strspn(line, " \t");
if (!*p || *p == '\n')
return 0;
numlen = strcspn(p, " \t\n");
if (!numlen || numlen >= sizeof(numstr) - 1)
return -EINVAL;
memcpy(numstr, p, numlen);
numstr[numlen] = '\0';
p += numlen;
if (kstrtou32(numstr, 16, pval))
return -EINVAL;
return p - line;
}
/*
* Parse lines with the format
*
* <engine-class> cmd <u32> <u32...>
* <engine-class> reg <u32_addr> <u32_val>
*
* and optionally save them in @wa_bb[i].cs is non-NULL.
*
* Return the number of dwords parsed.
*/
static ssize_t parse_wa_bb_lines(const char *lines,
struct wa_bb wa_bb[static XE_ENGINE_CLASS_MAX])
{
ssize_t dwords = 0, ret;
const char *p;
for (p = lines; *p; p++) {
const struct engine_info *info = NULL;
u32 val, val2;
/* Also allow empty lines */
p += strspn(p, " \t\n");
if (!*p)
break;
ret = parse_engine(p, " \t\n", NULL, &info);
if (ret < 0)
return ret;
p += ret;
p += strspn(p, " \t");
if (str_has_prefix(p, "cmd")) {
for (p += strlen("cmd"); *p;) {
ret = parse_hex(p, &val);
if (ret < 0)
return -EINVAL;
if (!ret)
break;
p += ret;
dwords++;
wa_bb_append(&wa_bb[info->engine_class], val);
}
} else if (str_has_prefix(p, "reg")) {
p += strlen("reg");
ret = parse_hex(p, &val);
if (ret <= 0)
return -EINVAL;
p += ret;
ret = parse_hex(p, &val2);
if (ret <= 0)
return -EINVAL;
p += ret;
dwords += 3;
wa_bb_append(&wa_bb[info->engine_class],
MI_LOAD_REGISTER_IMM | MI_LRI_NUM_REGS(1));
wa_bb_append(&wa_bb[info->engine_class], val);
wa_bb_append(&wa_bb[info->engine_class], val2);
} else {
return -EINVAL;
}
}
return dwords;
}
static ssize_t wa_bb_store(struct wa_bb wa_bb[static XE_ENGINE_CLASS_MAX],
struct xe_config_group_device *dev,
const char *page, size_t len)
{
/* tmp_wa_bb must match wa_bb's size */
struct wa_bb tmp_wa_bb[XE_ENGINE_CLASS_MAX] = { };
ssize_t count, class;
u32 *tmp;
/* 1. Count dwords - wa_bb[i].cs is NULL for all classes */
count = parse_wa_bb_lines(page, tmp_wa_bb);
if (count < 0)
return count;
guard(mutex)(&dev->lock);
if (is_bound(dev))
return -EBUSY;
/*
* 2. Allocate a u32 array and set the pointers to the right positions
* according to the length of each class' wa_bb
*/
tmp = krealloc(wa_bb[0].cs, count * sizeof(u32), GFP_KERNEL);
if (!tmp)
return -ENOMEM;
if (!count) {
memset(wa_bb, 0, sizeof(tmp_wa_bb));
return len;
}
for (class = 0, count = 0; class < XE_ENGINE_CLASS_MAX; ++class) {
tmp_wa_bb[class].cs = tmp + count;
count += tmp_wa_bb[class].len;
tmp_wa_bb[class].len = 0;
}
/* 3. Parse wa_bb lines again, this time saving the values */
count = parse_wa_bb_lines(page, tmp_wa_bb);
if (count < 0)
return count;
memcpy(wa_bb, tmp_wa_bb, sizeof(tmp_wa_bb));
return len;
}
static ssize_t ctx_restore_mid_bb_store(struct config_item *item,
const char *data, size_t sz)
{
struct xe_config_group_device *dev = to_xe_config_group_device(item);
return wa_bb_store(dev->config.ctx_restore_mid_bb, dev, data, sz);
}
static ssize_t ctx_restore_post_bb_store(struct config_item *item,
const char *data, size_t sz)
{
struct xe_config_group_device *dev = to_xe_config_group_device(item);
return wa_bb_store(dev->config.ctx_restore_post_bb, dev, data, sz);
}
CONFIGFS_ATTR(, ctx_restore_mid_bb);
CONFIGFS_ATTR(, ctx_restore_post_bb);
CONFIGFS_ATTR(, enable_psmi);
CONFIGFS_ATTR(, engines_allowed);
CONFIGFS_ATTR(, survivability_mode);
static struct configfs_attribute *xe_config_device_attrs[] = {
&attr_ctx_restore_mid_bb,
&attr_ctx_restore_post_bb,
&attr_enable_psmi,
&attr_engines_allowed,
&attr_survivability_mode,
@ -355,6 +675,8 @@ static void xe_config_device_release(struct config_item *item)
struct xe_config_group_device *dev = to_xe_config_group_device(item);
mutex_destroy(&dev->lock);
kfree(dev->config.ctx_restore_post_bb[0].cs);
kfree(dev);
}
@ -362,8 +684,26 @@ static struct configfs_item_operations xe_config_device_ops = {
.release = xe_config_device_release,
};
static bool xe_config_device_is_visible(struct config_item *item,
struct configfs_attribute *attr, int n)
{
struct xe_config_group_device *dev = to_xe_config_group_device(item);
if (attr == &attr_survivability_mode) {
if (!dev->desc->is_dgfx || dev->desc->platform < XE_BATTLEMAGE)
return false;
}
return true;
}
static struct configfs_group_operations xe_config_device_group_ops = {
.is_visible = xe_config_device_is_visible,
};
static const struct config_item_type xe_config_device_type = {
.ct_item_ops = &xe_config_device_ops,
.ct_group_ops = &xe_config_device_group_ops,
.ct_attrs = xe_config_device_attrs,
.ct_owner = THIS_MODULE,
};
@ -442,6 +782,7 @@ static struct config_group *xe_config_make_device_group(struct config_group *gro
if (!dev)
return ERR_PTR(-ENOMEM);
dev->desc = match;
set_device_defaults(&dev->config);
config_group_init_type_name(&dev->group, name, &xe_config_device_type);
@ -451,12 +792,12 @@ static struct config_group *xe_config_make_device_group(struct config_group *gro
return &dev->group;
}
static struct configfs_group_operations xe_config_device_group_ops = {
static struct configfs_group_operations xe_config_group_ops = {
.make_group = xe_config_make_device_group,
};
static const struct config_item_type xe_configfs_type = {
.ct_group_ops = &xe_config_device_group_ops,
.ct_group_ops = &xe_config_group_ops,
.ct_owner = THIS_MODULE,
};
@ -542,23 +883,6 @@ bool xe_configfs_get_survivability_mode(struct pci_dev *pdev)
return mode;
}
/**
* xe_configfs_clear_survivability_mode - clear configfs survivability mode
* @pdev: pci device
*/
void xe_configfs_clear_survivability_mode(struct pci_dev *pdev)
{
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
if (!dev)
return;
guard(mutex)(&dev->lock);
dev->config.survivability_mode = 0;
config_group_put(&dev->group);
}
/**
* xe_configfs_get_engines_allowed - get engine allowed mask from configfs
* @pdev: pci device
@ -594,11 +918,63 @@ bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev)
return false;
ret = dev->config.enable_psmi;
config_item_put(&dev->group.cg_item);
config_group_put(&dev->group);
return ret;
}
/**
* xe_configfs_get_ctx_restore_mid_bb - get configfs ctx_restore_mid_bb setting
* @pdev: pci device
* @class: hw engine class
* @cs: pointer to the bb to use - only valid during probe
*
* Return: Number of dwords used in the mid_ctx_restore setting in configfs
*/
u32 xe_configfs_get_ctx_restore_mid_bb(struct pci_dev *pdev,
enum xe_engine_class class,
const u32 **cs)
{
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
u32 len;
if (!dev)
return 0;
if (cs)
*cs = dev->config.ctx_restore_mid_bb[class].cs;
len = dev->config.ctx_restore_mid_bb[class].len;
config_group_put(&dev->group);
return len;
}
/**
* xe_configfs_get_ctx_restore_post_bb - get configfs ctx_restore_post_bb setting
* @pdev: pci device
* @class: hw engine class
* @cs: pointer to the bb to use - only valid during probe
*
* Return: Number of dwords used in the post_ctx_restore setting in configfs
*/
u32 xe_configfs_get_ctx_restore_post_bb(struct pci_dev *pdev,
enum xe_engine_class class,
const u32 **cs)
{
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
u32 len;
if (!dev)
return 0;
*cs = dev->config.ctx_restore_post_bb[class].cs;
len = dev->config.ctx_restore_post_bb[class].len;
config_group_put(&dev->group);
return len;
}
int __init xe_configfs_init(void)
{
int ret;
@ -614,7 +990,7 @@ int __init xe_configfs_init(void)
return 0;
}
void __exit xe_configfs_exit(void)
void xe_configfs_exit(void)
{
configfs_unregister_subsystem(&xe_configfs);
mutex_destroy(&xe_configfs.su_mutex);

12
drivers/gpu/drm/xe/xe_configfs.h
View File

@ -8,6 +8,8 @@
#include <linux/limits.h>
#include <linux/types.h>
#include <xe_hw_engine_types.h>
struct pci_dev;
#if IS_ENABLED(CONFIG_CONFIGFS_FS)
@ -15,17 +17,23 @@ int xe_configfs_init(void);
void xe_configfs_exit(void);
void xe_configfs_check_device(struct pci_dev *pdev);
bool xe_configfs_get_survivability_mode(struct pci_dev *pdev);
void xe_configfs_clear_survivability_mode(struct pci_dev *pdev);
u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev);
bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev);
u32 xe_configfs_get_ctx_restore_mid_bb(struct pci_dev *pdev, enum xe_engine_class,
const u32 **cs);
u32 xe_configfs_get_ctx_restore_post_bb(struct pci_dev *pdev, enum xe_engine_class,
const u32 **cs);
#else
static inline int xe_configfs_init(void) { return 0; }
static inline void xe_configfs_exit(void) { }
static inline void xe_configfs_check_device(struct pci_dev *pdev) { }
static inline bool xe_configfs_get_survivability_mode(struct pci_dev *pdev) { return false; }
static inline void xe_configfs_clear_survivability_mode(struct pci_dev *pdev) { }
static inline u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev) { return U64_MAX; }
static inline bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev) { return false; }
static inline u32 xe_configfs_get_ctx_restore_mid_bb(struct pci_dev *pdev, enum xe_engine_class,
const u32 **cs) { return 0; }
static inline u32 xe_configfs_get_ctx_restore_post_bb(struct pci_dev *pdev, enum xe_engine_class,
const u32 **cs) { return 0; }
#endif
#endif

66
drivers/gpu/drm/xe/xe_debugfs.c
View File

@ -24,7 +24,9 @@
#include "xe_pxp_debugfs.h"
#include "xe_sriov.h"
#include "xe_sriov_pf.h"
#include "xe_sriov_vf.h"
#include "xe_step.h"
#include "xe_tile_debugfs.h"
#include "xe_wa.h"
#include "xe_vsec.h"
@ -38,7 +40,7 @@ DECLARE_FAULT_ATTR(gt_reset_failure);
DECLARE_FAULT_ATTR(inject_csc_hw_error);
static void read_residency_counter(struct xe_device *xe, struct xe_mmio *mmio,
u32 offset, char *name, struct drm_printer *p)
u32 offset, const char *name, struct drm_printer *p)
{
u64 residency = 0;
int ret;
@ -134,9 +136,9 @@ static int dgfx_pkg_residencies_show(struct seq_file *m, void *data)
p = drm_seq_file_printer(m);
xe_pm_runtime_get(xe);
mmio = xe_root_tile_mmio(xe);
struct {
static const struct {
u32 offset;
char *name;
const char *name;
} residencies[] = {
{BMG_G2_RESIDENCY_OFFSET, "Package G2"},
{BMG_G6_RESIDENCY_OFFSET, "Package G6"},
@ -163,9 +165,9 @@ static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
xe_pm_runtime_get(xe);
mmio = xe_root_tile_mmio(xe);
struct {
static const struct {
u32 offset;
char *name;
const char *name;
} residencies[] = {
{BMG_PCIE_LINK_L0_RESIDENCY_OFFSET, "PCIE LINK L0 RESIDENCY"},
{BMG_PCIE_LINK_L1_RESIDENCY_OFFSET, "PCIE LINK L1 RESIDENCY"},
@ -329,23 +331,44 @@ static const struct file_operations atomic_svm_timeslice_ms_fops = {
.write = atomic_svm_timeslice_ms_set,
};
static void create_tile_debugfs(struct xe_tile *tile, struct dentry *root)
static ssize_t disable_late_binding_show(struct file *f, char __user *ubuf,
size_t size, loff_t *pos)
{
char name[8];
struct xe_device *xe = file_inode(f)->i_private;
struct xe_late_bind *late_bind = &xe->late_bind;
char buf[32];
int len;
snprintf(name, sizeof(name), "tile%u", tile->id);
tile->debugfs = debugfs_create_dir(name, root);
if (IS_ERR(tile->debugfs))
return;
len = scnprintf(buf, sizeof(buf), "%d\n", late_bind->disable);
/*
* Store the xe_tile pointer as private data of the tile/ directory
* node so other tile specific attributes under that directory may
* refer to it by looking at its parent node private data.
*/
tile->debugfs->d_inode->i_private = tile;
return simple_read_from_buffer(ubuf, size, pos, buf, len);
}
static ssize_t disable_late_binding_set(struct file *f, const char __user *ubuf,
size_t size, loff_t *pos)
{
struct xe_device *xe = file_inode(f)->i_private;
struct xe_late_bind *late_bind = &xe->late_bind;
u32 uval;
ssize_t ret;
ret = kstrtouint_from_user(ubuf, size, sizeof(uval), &uval);
if (ret)
return ret;
if (uval > 1)
return -EINVAL;
late_bind->disable = !!uval;
return size;
}
static const struct file_operations disable_late_binding_fops = {
.owner = THIS_MODULE,
.read = disable_late_binding_show,
.write = disable_late_binding_set,
};
void xe_debugfs_register(struct xe_device *xe)
{
struct ttm_device *bdev = &xe->ttm;
@ -362,7 +385,7 @@ void xe_debugfs_register(struct xe_device *xe)
ARRAY_SIZE(debugfs_list),
root, minor);
if (xe->info.platform == XE_BATTLEMAGE) {
if (xe->info.platform == XE_BATTLEMAGE && !IS_SRIOV_VF(xe)) {
drm_debugfs_create_files(debugfs_residencies,
ARRAY_SIZE(debugfs_residencies),
root, minor);
@ -379,6 +402,9 @@ void xe_debugfs_register(struct xe_device *xe)
debugfs_create_file("atomic_svm_timeslice_ms", 0600, root, xe,
&atomic_svm_timeslice_ms_fops);
debugfs_create_file("disable_late_binding", 0600, root, xe,
&disable_late_binding_fops);
for (mem_type = XE_PL_VRAM0; mem_type <= XE_PL_VRAM1; ++mem_type) {
man = ttm_manager_type(bdev, mem_type);
@ -398,7 +424,7 @@ void xe_debugfs_register(struct xe_device *xe)
ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
for_each_tile(tile, xe, tile_id)
create_tile_debugfs(tile, root);
xe_tile_debugfs_register(tile);
for_each_gt(gt, xe, id)
xe_gt_debugfs_register(gt);
@ -411,4 +437,6 @@ void xe_debugfs_register(struct xe_device *xe)
if (IS_SRIOV_PF(xe))
xe_sriov_pf_debugfs_register(xe, root);
else if (IS_SRIOV_VF(xe))
xe_sriov_vf_debugfs_register(xe, root);
}

11
drivers/gpu/drm/xe/xe_device.c
View File

@ -45,6 +45,7 @@
#include "xe_hwmon.h"
#include "xe_i2c.h"
#include "xe_irq.h"
#include "xe_late_bind_fw.h"
#include "xe_mmio.h"
#include "xe_module.h"
#include "xe_nvm.h"
@ -457,6 +458,8 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
if (err)
goto err;
xe_validation_device_init(&xe->val);
init_waitqueue_head(&xe->ufence_wq);
init_rwsem(&xe->usm.lock);
@ -530,7 +533,7 @@ static bool xe_driver_flr_disabled(struct xe_device *xe)
* re-init and saving/restoring (or re-populating) the wiped memory. Since we
* perform the FLR as the very last action before releasing access to the HW
* during the driver release flow, we don't attempt recovery at all, because
* if/when a new instance of i915 is bound to the device it will do a full
* if/when a new instance of Xe is bound to the device it will do a full
* re-init anyway.
*/
static void __xe_driver_flr(struct xe_device *xe)
@ -901,6 +904,10 @@ int xe_device_probe(struct xe_device *xe)
if (err)
return err;
err = xe_late_bind_init(&xe->late_bind);
if (err)
return err;
err = xe_oa_init(xe);
if (err)
return err;
@ -950,7 +957,7 @@ int xe_device_probe(struct xe_device *xe)
xe_vsec_init(xe);
err = xe_sriov_late_init(xe);
err = xe_sriov_init_late(xe);
if (err)
goto err_unregister_display;

98
drivers/gpu/drm/xe/xe_device_sysfs.c
View File

@ -71,6 +71,15 @@ vram_d3cold_threshold_store(struct device *dev, struct device_attribute *attr,
static DEVICE_ATTR_RW(vram_d3cold_threshold);
static struct attribute *vram_attrs[] = {
&dev_attr_vram_d3cold_threshold.attr,
NULL
};
static const struct attribute_group vram_attr_group = {
.attrs = vram_attrs,
};
static ssize_t
lb_fan_control_version_show(struct device *dev, struct device_attribute *attr, char *buf)
{
@ -149,8 +158,16 @@ lb_voltage_regulator_version_show(struct device *dev, struct device_attribute *a
}
static DEVICE_ATTR_ADMIN_RO(lb_voltage_regulator_version);
static int late_bind_create_files(struct device *dev)
static struct attribute *late_bind_attrs[] = {
&dev_attr_lb_fan_control_version.attr,
&dev_attr_lb_voltage_regulator_version.attr,
NULL
};
static umode_t late_bind_attr_is_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
struct device *dev = kobj_to_dev(kobj);
struct xe_device *xe = pdev_to_xe_device(to_pci_dev(dev));
struct xe_tile *root = xe_device_get_root_tile(xe);
u32 cap = 0;
@ -160,51 +177,25 @@ static int late_bind_create_files(struct device *dev)
ret = xe_pcode_read(root, PCODE_MBOX(PCODE_LATE_BINDING, GET_CAPABILITY_STATUS, 0),
&cap, NULL);
if (ret) {
if (ret == -ENXIO) {
drm_dbg(&xe->drm, "Late binding not supported by firmware\n");
ret = 0;
}
goto out;
}
if (REG_FIELD_GET(V1_FAN_SUPPORTED, cap)) {
ret = sysfs_create_file(&dev->kobj, &dev_attr_lb_fan_control_version.attr);
if (ret)
goto out;
}
if (REG_FIELD_GET(VR_PARAMS_SUPPORTED, cap))
ret = sysfs_create_file(&dev->kobj, &dev_attr_lb_voltage_regulator_version.attr);
out:
xe_pm_runtime_put(xe);
return ret;
}
static void late_bind_remove_files(struct device *dev)
{
struct xe_device *xe = pdev_to_xe_device(to_pci_dev(dev));
struct xe_tile *root = xe_device_get_root_tile(xe);
u32 cap = 0;
int ret;
xe_pm_runtime_get(xe);
ret = xe_pcode_read(root, PCODE_MBOX(PCODE_LATE_BINDING, GET_CAPABILITY_STATUS, 0),
&cap, NULL);
if (ret)
goto out;
return 0;
if (REG_FIELD_GET(V1_FAN_SUPPORTED, cap))
sysfs_remove_file(&dev->kobj, &dev_attr_lb_fan_control_version.attr);
if (attr == &dev_attr_lb_fan_control_version.attr &&
REG_FIELD_GET(V1_FAN_SUPPORTED, cap))
return attr->mode;
if (attr == &dev_attr_lb_voltage_regulator_version.attr &&
REG_FIELD_GET(VR_PARAMS_SUPPORTED, cap))
return attr->mode;
if (REG_FIELD_GET(VR_PARAMS_SUPPORTED, cap))
sysfs_remove_file(&dev->kobj, &dev_attr_lb_voltage_regulator_version.attr);
out:
xe_pm_runtime_put(xe);
return 0;
}
static const struct attribute_group late_bind_attr_group = {
.attrs = late_bind_attrs,
.is_visible = late_bind_attr_is_visible,
};
/**
* DOC: PCIe Gen5 Limitations
*
@ -278,24 +269,15 @@ auto_link_downgrade_status_show(struct device *dev, struct device_attribute *att
}
static DEVICE_ATTR_ADMIN_RO(auto_link_downgrade_status);
static const struct attribute *auto_link_downgrade_attrs[] = {
static struct attribute *auto_link_downgrade_attrs[] = {
&dev_attr_auto_link_downgrade_capable.attr,
&dev_attr_auto_link_downgrade_status.attr,
NULL
};
static void xe_device_sysfs_fini(void *arg)
{
struct xe_device *xe = arg;
if (xe->d3cold.capable)
sysfs_remove_file(&xe->drm.dev->kobj, &dev_attr_vram_d3cold_threshold.attr);
if (xe->info.platform == XE_BATTLEMAGE) {
sysfs_remove_files(&xe->drm.dev->kobj, auto_link_downgrade_attrs);
late_bind_remove_files(xe->drm.dev);
}
}
static const struct attribute_group auto_link_downgrade_attr_group = {
.attrs = auto_link_downgrade_attrs,
};
int xe_device_sysfs_init(struct xe_device *xe)
{
@ -303,20 +285,20 @@ int xe_device_sysfs_init(struct xe_device *xe)
int ret;
if (xe->d3cold.capable) {
ret = sysfs_create_file(&dev->kobj, &dev_attr_vram_d3cold_threshold.attr);
ret = devm_device_add_group(dev, &vram_attr_group);
if (ret)
return ret;
}
if (xe->info.platform == XE_BATTLEMAGE) {
ret = sysfs_create_files(&dev->kobj, auto_link_downgrade_attrs);
if (xe->info.platform == XE_BATTLEMAGE && !IS_SRIOV_VF(xe)) {
ret = devm_device_add_group(dev, &auto_link_downgrade_attr_group);
if (ret)
return ret;
ret = late_bind_create_files(dev);
ret = devm_device_add_group(dev, &late_bind_attr_group);
if (ret)
return ret;
}
return devm_add_action_or_reset(dev, xe_device_sysfs_fini, xe);
return 0;
}

19
drivers/gpu/drm/xe/xe_device_types.h
View File

@ -14,6 +14,7 @@
#include "xe_devcoredump_types.h"
#include "xe_heci_gsc.h"
#include "xe_late_bind_fw_types.h"
#include "xe_lmtt_types.h"
#include "xe_memirq_types.h"
#include "xe_oa_types.h"
@ -26,6 +27,7 @@
#include "xe_sriov_vf_ccs_types.h"
#include "xe_step_types.h"
#include "xe_survivability_mode_types.h"
#include "xe_validation.h"
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
#define TEST_VM_OPS_ERROR
@ -183,9 +185,6 @@ struct xe_tile {
struct {
/** @sriov.vf.ggtt_balloon: GGTT regions excluded from use. */
struct xe_ggtt_node *ggtt_balloon[2];
/** @sriov.vf.ccs: CCS read and write contexts for VF. */
struct xe_tile_vf_ccs ccs[XE_SRIOV_VF_CCS_CTX_COUNT];
} vf;
} sriov;
@ -282,6 +281,8 @@ struct xe_device {
u8 has_heci_cscfi:1;
/** @info.has_heci_gscfi: device has heci gscfi */
u8 has_heci_gscfi:1;
/** @info.has_late_bind: Device has firmware late binding support */
u8 has_late_bind:1;
/** @info.has_llc: Device has a shared CPU+GPU last level cache */
u8 has_llc:1;
/** @info.has_mbx_power_limits: Device has support to manage power limits using
@ -535,6 +536,9 @@ struct xe_device {
/** @nvm: discrete graphics non-volatile memory */
struct intel_dg_nvm_dev *nvm;
/** @late_bind: xe mei late bind interface */
struct xe_late_bind late_bind;
/** @oa: oa observation subsystem */
struct xe_oa oa;
@ -586,6 +590,8 @@ struct xe_device {
*/
atomic64_t global_total_pages;
#endif
/** @val: The domain for exhaustive eviction, which is currently per device. */
struct xe_validation_device val;
/** @psmi: GPU debugging via additional validation HW */
struct {
@ -595,6 +601,13 @@ struct xe_device {
u8 region_mask;
} psmi;
#if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
/** @g2g_test_array: for testing G2G communications */
u32 *g2g_test_array;
/** @g2g_test_count: for testing G2G communications */
atomic_t g2g_test_count;
#endif
/* private: */
#if IS_ENABLED(CONFIG_DRM_XE_DISPLAY)

70
drivers/gpu/drm/xe/xe_dma_buf.c
View File

@ -51,6 +51,7 @@ static int xe_dma_buf_pin(struct dma_buf_attachment *attach)
struct drm_gem_object *obj = attach->dmabuf->priv;
struct xe_bo *bo = gem_to_xe_bo(obj);
struct xe_device *xe = xe_bo_device(bo);
struct drm_exec *exec = XE_VALIDATION_UNSUPPORTED;
int ret;
/*
@ -63,7 +64,7 @@ static int xe_dma_buf_pin(struct dma_buf_attachment *attach)
return -EINVAL;
}
ret = xe_bo_migrate(bo, XE_PL_TT);
ret = xe_bo_migrate(bo, XE_PL_TT, NULL, exec);
if (ret) {
if (ret != -EINTR && ret != -ERESTARTSYS)
drm_dbg(&xe->drm,
@ -72,7 +73,7 @@ static int xe_dma_buf_pin(struct dma_buf_attachment *attach)
return ret;
}
ret = xe_bo_pin_external(bo, true);
ret = xe_bo_pin_external(bo, true, exec);
xe_assert(xe, !ret);
return 0;
@ -92,6 +93,7 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,
struct dma_buf *dma_buf = attach->dmabuf;
struct drm_gem_object *obj = dma_buf->priv;
struct xe_bo *bo = gem_to_xe_bo(obj);
struct drm_exec *exec = XE_VALIDATION_UNSUPPORTED;
struct sg_table *sgt;
int r = 0;
@ -100,9 +102,9 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,
if (!xe_bo_is_pinned(bo)) {
if (!attach->peer2peer)
r = xe_bo_migrate(bo, XE_PL_TT);
r = xe_bo_migrate(bo, XE_PL_TT, NULL, exec);
else
r = xe_bo_validate(bo, NULL, false);
r = xe_bo_validate(bo, NULL, false, exec);
if (r)
return ERR_PTR(r);
}
@ -161,15 +163,26 @@ static int xe_dma_buf_begin_cpu_access(struct dma_buf *dma_buf,
struct xe_bo *bo = gem_to_xe_bo(obj);
bool reads = (direction == DMA_BIDIRECTIONAL ||
direction == DMA_FROM_DEVICE);
struct xe_validation_ctx ctx;
struct drm_exec exec;
int ret = 0;
if (!reads)
return 0;
/* Can we do interruptible lock here? */
xe_bo_lock(bo, false);
(void)xe_bo_migrate(bo, XE_PL_TT);
xe_bo_unlock(bo);
xe_validation_guard(&ctx, &xe_bo_device(bo)->val, &exec, (struct xe_val_flags) {}, ret) {
ret = drm_exec_lock_obj(&exec, &bo->ttm.base);
drm_exec_retry_on_contention(&exec);
if (ret)
break;
ret = xe_bo_migrate(bo, XE_PL_TT, NULL, &exec);
drm_exec_retry_on_contention(&exec);
xe_validation_retry_on_oom(&ctx, &ret);
}
/* If we failed, cpu-access takes place in current placement. */
return 0;
}
@ -220,32 +233,45 @@ xe_dma_buf_init_obj(struct drm_device *dev, struct xe_bo *storage,
{
struct dma_resv *resv = dma_buf->resv;
struct xe_device *xe = to_xe_device(dev);
struct xe_validation_ctx ctx;
struct drm_gem_object *dummy_obj;
struct drm_exec exec;
struct xe_bo *bo;
int ret;
int ret = 0;
dma_resv_lock(resv, NULL);
bo = ___xe_bo_create_locked(xe, storage, NULL, resv, NULL, dma_buf->size,
0, /* Will require 1way or 2way for vm_bind */
ttm_bo_type_sg, XE_BO_FLAG_SYSTEM);
if (IS_ERR(bo)) {
ret = PTR_ERR(bo);
goto error;
dummy_obj = drm_gpuvm_resv_object_alloc(&xe->drm);
if (!dummy_obj)
return ERR_PTR(-ENOMEM);
dummy_obj->resv = resv;
xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, ret) {
ret = drm_exec_lock_obj(&exec, dummy_obj);
drm_exec_retry_on_contention(&exec);
if (ret)
break;
bo = xe_bo_init_locked(xe, storage, NULL, resv, NULL, dma_buf->size,
0, /* Will require 1way or 2way for vm_bind */
ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, &exec);
drm_exec_retry_on_contention(&exec);
if (IS_ERR(bo)) {
ret = PTR_ERR(bo);
xe_validation_retry_on_oom(&ctx, &ret);
break;
}
}
dma_resv_unlock(resv);
drm_gem_object_put(dummy_obj);
return &bo->ttm.base;
error:
dma_resv_unlock(resv);
return ERR_PTR(ret);
return ret ? ERR_PTR(ret) : &bo->ttm.base;
}
static void xe_dma_buf_move_notify(struct dma_buf_attachment *attach)
{
struct drm_gem_object *obj = attach->importer_priv;
struct xe_bo *bo = gem_to_xe_bo(obj);
struct drm_exec *exec = XE_VALIDATION_UNSUPPORTED;
XE_WARN_ON(xe_bo_evict(bo));
XE_WARN_ON(xe_bo_evict(bo, exec));
}
static const struct dma_buf_attach_ops xe_dma_buf_attach_ops = {

5
drivers/gpu/drm/xe/xe_eu_stall.c
View File

@ -617,9 +617,8 @@ static int xe_eu_stall_data_buf_alloc(struct xe_eu_stall_data_stream *stream,
size = stream->per_xecore_buf_size * last_xecore;
bo = xe_bo_create_pin_map_at_aligned(tile->xe, tile, NULL,
size, ~0ull, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_GGTT, SZ_64);
bo = xe_bo_create_pin_map_at_novm(tile->xe, tile, size, ~0ull, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_GGTT, SZ_64, false);
if (IS_ERR(bo)) {
kfree(stream->xecore_buf);
return PTR_ERR(bo);

31
drivers/gpu/drm/xe/xe_exec.c
View File

@ -19,6 +19,7 @@
#include "xe_ring_ops_types.h"
#include "xe_sched_job.h"
#include "xe_sync.h"
#include "xe_svm.h"
#include "xe_vm.h"
/**
@ -97,9 +98,13 @@
static int xe_exec_fn(struct drm_gpuvm_exec *vm_exec)
{
struct xe_vm *vm = container_of(vm_exec->vm, struct xe_vm, gpuvm);
int ret;
/* The fence slot added here is intended for the exec sched job. */
return xe_vm_validate_rebind(vm, &vm_exec->exec, 1);
xe_vm_set_validation_exec(vm, &vm_exec->exec);
ret = xe_vm_validate_rebind(vm, &vm_exec->exec, 1);
xe_vm_set_validation_exec(vm, NULL);
return ret;
}
int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
@ -115,10 +120,10 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
struct drm_gpuvm_exec vm_exec = {.extra.fn = xe_exec_fn};
struct drm_exec *exec = &vm_exec.exec;
u32 i, num_syncs, num_ufence = 0;
struct xe_validation_ctx ctx;
struct xe_sched_job *job;
struct xe_vm *vm;
bool write_locked, skip_retry = false;
ktime_t end = 0;
int err = 0;
struct xe_hw_engine_group *group;
enum xe_hw_engine_group_execution_mode mode, previous_mode;
@ -246,17 +251,12 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
if (err)
goto err_unlock_list;
vm_exec.vm = &vm->gpuvm;
vm_exec.flags = DRM_EXEC_INTERRUPTIBLE_WAIT;
if (xe_vm_in_lr_mode(vm)) {
drm_exec_init(exec, vm_exec.flags, 0);
} else {
err = drm_gpuvm_exec_lock(&vm_exec);
if (err) {
if (xe_vm_validate_should_retry(exec, err, &end))
err = -EAGAIN;
if (!xe_vm_in_lr_mode(vm)) {
vm_exec.vm = &vm->gpuvm;
vm_exec.flags = DRM_EXEC_INTERRUPTIBLE_WAIT;
err = xe_validation_exec_lock(&ctx, &vm_exec, &xe->val);
if (err)
goto err_unlock_list;
}
}
if (xe_vm_is_closed_or_banned(q->vm)) {
@ -303,7 +303,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
if (err)
goto err_put_job;
err = down_read_interruptible(&vm->userptr.notifier_lock);
err = xe_svm_notifier_lock_interruptible(vm);
if (err)
goto err_put_job;
@ -345,12 +345,13 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
err_repin:
if (!xe_vm_in_lr_mode(vm))
up_read(&vm->userptr.notifier_lock);
xe_svm_notifier_unlock(vm);
err_put_job:
if (err)
xe_sched_job_put(job);
err_exec:
drm_exec_fini(exec);
if (!xe_vm_in_lr_mode(vm))
xe_validation_ctx_fini(&ctx);
err_unlock_list:
up_read(&vm->lock);
if (err == -EAGAIN && !skip_retry)

22
drivers/gpu/drm/xe/xe_exec_queue.c
View File

@ -199,6 +199,16 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q)
return err;
}
static void __xe_exec_queue_fini(struct xe_exec_queue *q)
{
int i;
q->ops->fini(q);
for (i = 0; i < q->width; ++i)
xe_lrc_put(q->lrc[i]);
}
struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *vm,
u32 logical_mask, u16 width,
struct xe_hw_engine *hwe, u32 flags,
@ -229,11 +239,13 @@ struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *v
if (xe_exec_queue_uses_pxp(q)) {
err = xe_pxp_exec_queue_add(xe->pxp, q);
if (err)
goto err_post_alloc;
goto err_post_init;
}
return q;
err_post_init:
__xe_exec_queue_fini(q);
err_post_alloc:
__xe_exec_queue_free(q);
return ERR_PTR(err);
@ -331,13 +343,11 @@ void xe_exec_queue_destroy(struct kref *ref)
xe_exec_queue_put(eq);
}
q->ops->fini(q);
q->ops->destroy(q);
}
void xe_exec_queue_fini(struct xe_exec_queue *q)
{
int i;
/*
* Before releasing our ref to lrc and xef, accumulate our run ticks
* and wakeup any waiters.
@ -346,9 +356,7 @@ void xe_exec_queue_fini(struct xe_exec_queue *q)
if (q->xef && atomic_dec_and_test(&q->xef->exec_queue.pending_removal))
wake_up_var(&q->xef->exec_queue.pending_removal);
for (i = 0; i < q->width; ++i)
xe_lrc_put(q->lrc[i]);
__xe_exec_queue_fini(q);
__xe_exec_queue_free(q);
}

8
drivers/gpu/drm/xe/xe_exec_queue_types.h
View File

@ -181,8 +181,14 @@ struct xe_exec_queue_ops {
int (*init)(struct xe_exec_queue *q);
/** @kill: Kill inflight submissions for backend */
void (*kill)(struct xe_exec_queue *q);
/** @fini: Fini exec queue for submission backend */
/** @fini: Undoes the init() for submission backend */
void (*fini)(struct xe_exec_queue *q);
/**
* @destroy: Destroy exec queue for submission backend. The backend
* function must call xe_exec_queue_fini() (which will in turn call the
* fini() backend function) to ensure the queue is properly cleaned up.
*/
void (*destroy)(struct xe_exec_queue *q);
/** @set_priority: Set priority for exec queue */
int (*set_priority)(struct xe_exec_queue *q,
enum xe_exec_queue_priority priority);

25
drivers/gpu/drm/xe/xe_execlist.c
View File

@ -385,10 +385,20 @@ static int execlist_exec_queue_init(struct xe_exec_queue *q)
return err;
}
static void execlist_exec_queue_fini_async(struct work_struct *w)
static void execlist_exec_queue_fini(struct xe_exec_queue *q)
{
struct xe_execlist_exec_queue *exl = q->execlist;
drm_sched_entity_fini(&exl->entity);
drm_sched_fini(&exl->sched);
kfree(exl);
}
static void execlist_exec_queue_destroy_async(struct work_struct *w)
{
struct xe_execlist_exec_queue *ee =
container_of(w, struct xe_execlist_exec_queue, fini_async);
container_of(w, struct xe_execlist_exec_queue, destroy_async);
struct xe_exec_queue *q = ee->q;
struct xe_execlist_exec_queue *exl = q->execlist;
struct xe_device *xe = gt_to_xe(q->gt);
@ -401,10 +411,6 @@ static void execlist_exec_queue_fini_async(struct work_struct *w)
list_del(&exl->active_link);
spin_unlock_irqrestore(&exl->port->lock, flags);
drm_sched_entity_fini(&exl->entity);
drm_sched_fini(&exl->sched);
kfree(exl);
xe_exec_queue_fini(q);
}
@ -413,10 +419,10 @@ static void execlist_exec_queue_kill(struct xe_exec_queue *q)
/* NIY */
}
static void execlist_exec_queue_fini(struct xe_exec_queue *q)
static void execlist_exec_queue_destroy(struct xe_exec_queue *q)
{
INIT_WORK(&q->execlist->fini_async, execlist_exec_queue_fini_async);
queue_work(system_unbound_wq, &q->execlist->fini_async);
INIT_WORK(&q->execlist->destroy_async, execlist_exec_queue_destroy_async);
queue_work(system_unbound_wq, &q->execlist->destroy_async);
}
static int execlist_exec_queue_set_priority(struct xe_exec_queue *q,
@ -467,6 +473,7 @@ static const struct xe_exec_queue_ops execlist_exec_queue_ops = {
.init = execlist_exec_queue_init,
.kill = execlist_exec_queue_kill,
.fini = execlist_exec_queue_fini,
.destroy = execlist_exec_queue_destroy,
.set_priority = execlist_exec_queue_set_priority,
.set_timeslice = execlist_exec_queue_set_timeslice,
.set_preempt_timeout = execlist_exec_queue_set_preempt_timeout,

2
drivers/gpu/drm/xe/xe_execlist_types.h
View File

@ -42,7 +42,7 @@ struct xe_execlist_exec_queue {
bool has_run;
struct work_struct fini_async;
struct work_struct destroy_async;
enum xe_exec_queue_priority active_priority;
struct list_head active_link;

27
drivers/gpu/drm/xe/xe_ggtt.c
View File

@ -28,6 +28,7 @@
#include "xe_pm.h"
#include "xe_res_cursor.h"
#include "xe_sriov.h"
#include "xe_tile_printk.h"
#include "xe_tile_sriov_vf.h"
#include "xe_tlb_inval.h"
#include "xe_wa.h"
@ -269,7 +270,7 @@ int xe_ggtt_init_early(struct xe_ggtt *ggtt)
gsm_size = probe_gsm_size(pdev);
if (gsm_size == 0) {
drm_err(&xe->drm, "Hardware reported no preallocated GSM\n");
xe_tile_err(ggtt->tile, "Hardware reported no preallocated GSM\n");
return -ENOMEM;
}
@ -466,8 +467,8 @@ static void xe_ggtt_dump_node(struct xe_ggtt *ggtt,
if (IS_ENABLED(CONFIG_DRM_XE_DEBUG)) {
string_get_size(node->size, 1, STRING_UNITS_2, buf, sizeof(buf));
xe_gt_dbg(ggtt->tile->primary_gt, "GGTT %#llx-%#llx (%s) %s\n",
node->start, node->start + node->size, buf, description);
xe_tile_dbg(ggtt->tile, "GGTT %#llx-%#llx (%s) %s\n",
node->start, node->start + node->size, buf, description);
}
}
@ -499,9 +500,8 @@ int xe_ggtt_node_insert_balloon_locked(struct xe_ggtt_node *node, u64 start, u64
err = drm_mm_reserve_node(&ggtt->mm, &node->base);
if (xe_gt_WARN(ggtt->tile->primary_gt, err,
"Failed to balloon GGTT %#llx-%#llx (%pe)\n",
node->base.start, node->base.start + node->base.size, ERR_PTR(err)))
if (xe_tile_WARN(ggtt->tile, err, "Failed to balloon GGTT %#llx-%#llx (%pe)\n",
node->base.start, node->base.start + node->base.size, ERR_PTR(err)))
return err;
xe_ggtt_dump_node(ggtt, &node->base, "balloon");
@ -731,7 +731,7 @@ void xe_ggtt_map_bo_unlocked(struct xe_ggtt *ggtt, struct xe_bo *bo)
}
static int __xe_ggtt_insert_bo_at(struct xe_ggtt *ggtt, struct xe_bo *bo,
u64 start, u64 end)
u64 start, u64 end, struct drm_exec *exec)
{
u64 alignment = bo->min_align > 0 ? bo->min_align : XE_PAGE_SIZE;
u8 tile_id = ggtt->tile->id;
@ -746,7 +746,7 @@ static int __xe_ggtt_insert_bo_at(struct xe_ggtt *ggtt, struct xe_bo *bo,
return 0;
}
err = xe_bo_validate(bo, NULL, false);
err = xe_bo_validate(bo, NULL, false, exec);
if (err)
return err;
@ -788,25 +788,28 @@ static int __xe_ggtt_insert_bo_at(struct xe_ggtt *ggtt, struct xe_bo *bo,
* @bo: the &xe_bo to be inserted
* @start: address where it will be inserted
* @end: end of the range where it will be inserted
* @exec: The drm_exec transaction to use for exhaustive eviction.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_ggtt_insert_bo_at(struct xe_ggtt *ggtt, struct xe_bo *bo,
u64 start, u64 end)
u64 start, u64 end, struct drm_exec *exec)
{
return __xe_ggtt_insert_bo_at(ggtt, bo, start, end);
return __xe_ggtt_insert_bo_at(ggtt, bo, start, end, exec);
}
/**
* xe_ggtt_insert_bo - Insert BO into GGTT
* @ggtt: the &xe_ggtt where bo will be inserted
* @bo: the &xe_bo to be inserted
* @exec: The drm_exec transaction to use for exhaustive eviction.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_ggtt_insert_bo(struct xe_ggtt *ggtt, struct xe_bo *bo)
int xe_ggtt_insert_bo(struct xe_ggtt *ggtt, struct xe_bo *bo,
struct drm_exec *exec)
{
return __xe_ggtt_insert_bo_at(ggtt, bo, 0, U64_MAX);
return __xe_ggtt_insert_bo_at(ggtt, bo, 0, U64_MAX, exec);
}
/**

5
drivers/gpu/drm/xe/xe_ggtt.h
View File

@ -10,6 +10,7 @@
struct drm_printer;
struct xe_tile;
struct drm_exec;
struct xe_ggtt *xe_ggtt_alloc(struct xe_tile *tile);
int xe_ggtt_init_early(struct xe_ggtt *ggtt);
@ -31,9 +32,9 @@ bool xe_ggtt_node_allocated(const struct xe_ggtt_node *node);
void xe_ggtt_map_bo(struct xe_ggtt *ggtt, struct xe_ggtt_node *node,
struct xe_bo *bo, u16 pat_index);
void xe_ggtt_map_bo_unlocked(struct xe_ggtt *ggtt, struct xe_bo *bo);
int xe_ggtt_insert_bo(struct xe_ggtt *ggtt, struct xe_bo *bo);
int xe_ggtt_insert_bo(struct xe_ggtt *ggtt, struct xe_bo *bo, struct drm_exec *exec);
int xe_ggtt_insert_bo_at(struct xe_ggtt *ggtt, struct xe_bo *bo,
u64 start, u64 end);
u64 start, u64 end, struct drm_exec *exec);
void xe_ggtt_remove_bo(struct xe_ggtt *ggtt, struct xe_bo *bo);
u64 xe_ggtt_largest_hole(struct xe_ggtt *ggtt, u64 alignment, u64 *spare);

8
drivers/gpu/drm/xe/xe_gsc.c
View File

@ -136,10 +136,10 @@ static int query_compatibility_version(struct xe_gsc *gsc)
u64 ggtt_offset;
int err;
bo = xe_bo_create_pin_map(xe, tile, NULL, GSC_VER_PKT_SZ * 2,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT);
bo = xe_bo_create_pin_map_novm(xe, tile, GSC_VER_PKT_SZ * 2,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT, false);
if (IS_ERR(bo)) {
xe_gt_err(gt, "failed to allocate bo for GSC version query\n");
return PTR_ERR(bo);

2
drivers/gpu/drm/xe/xe_gt.c
View File

@ -98,7 +98,7 @@ void xe_gt_sanitize(struct xe_gt *gt)
* FIXME: if xe_uc_sanitize is called here, on TGL driver will not
* reload
*/
gt->uc.guc.submission_state.enabled = false;
xe_guc_submit_disable(&gt->uc.guc);
}
static void xe_gt_enable_host_l2_vram(struct xe_gt *gt)

73
drivers/gpu/drm/xe/xe_gt_debugfs.c
View File

@ -31,6 +31,7 @@
#include "xe_reg_whitelist.h"
#include "xe_sa.h"
#include "xe_sriov.h"
#include "xe_sriov_vf_ccs.h"
#include "xe_tuning.h"
#include "xe_uc_debugfs.h"
#include "xe_wa.h"
@ -123,45 +124,6 @@ static int powergate_info(struct xe_gt *gt, struct drm_printer *p)
return ret;
}
static int sa_info(struct xe_gt *gt, struct drm_printer *p)
{
struct xe_tile *tile = gt_to_tile(gt);
xe_pm_runtime_get(gt_to_xe(gt));
drm_suballoc_dump_debug_info(&tile->mem.kernel_bb_pool->base, p,
xe_sa_manager_gpu_addr(tile->mem.kernel_bb_pool));
xe_pm_runtime_put(gt_to_xe(gt));
return 0;
}
static int sa_info_vf_ccs(struct xe_gt *gt, struct drm_printer *p)
{
struct xe_tile *tile = gt_to_tile(gt);
struct xe_sa_manager *bb_pool;
enum xe_sriov_vf_ccs_rw_ctxs ctx_id;
if (!IS_VF_CCS_READY(gt_to_xe(gt)))
return 0;
xe_pm_runtime_get(gt_to_xe(gt));
for_each_ccs_rw_ctx(ctx_id) {
bb_pool = tile->sriov.vf.ccs[ctx_id].mem.ccs_bb_pool;
if (!bb_pool)
break;
drm_printf(p, "ccs %s bb suballoc info\n", ctx_id ? "write" : "read");
drm_printf(p, "-------------------------\n");
drm_suballoc_dump_debug_info(&bb_pool->base, p, xe_sa_manager_gpu_addr(bb_pool));
drm_puts(p, "\n");
}
xe_pm_runtime_put(gt_to_xe(gt));
return 0;
}
static int topology(struct xe_gt *gt, struct drm_printer *p)
{
xe_pm_runtime_get(gt_to_xe(gt));
@ -316,7 +278,6 @@ static int hwconfig(struct xe_gt *gt, struct drm_printer *p)
* - without access to the PF specific data
*/
static const struct drm_info_list vf_safe_debugfs_list[] = {
{"sa_info", .show = xe_gt_debugfs_simple_show, .data = sa_info},
{"topology", .show = xe_gt_debugfs_simple_show, .data = topology},
{"ggtt", .show = xe_gt_debugfs_simple_show, .data = ggtt},
{"register-save-restore", .show = xe_gt_debugfs_simple_show, .data = register_save_restore},
@ -327,17 +288,9 @@ static const struct drm_info_list vf_safe_debugfs_list[] = {
{"default_lrc_bcs", .show = xe_gt_debugfs_simple_show, .data = bcs_default_lrc},
{"default_lrc_vcs", .show = xe_gt_debugfs_simple_show, .data = vcs_default_lrc},
{"default_lrc_vecs", .show = xe_gt_debugfs_simple_show, .data = vecs_default_lrc},
{"stats", .show = xe_gt_debugfs_simple_show, .data = xe_gt_stats_print_info},
{"hwconfig", .show = xe_gt_debugfs_simple_show, .data = hwconfig},
};
/*
* only for GT debugfs files which are valid on VF. Not valid on PF.
*/
static const struct drm_info_list vf_only_debugfs_list[] = {
{"sa_info_vf_ccs", .show = xe_gt_debugfs_simple_show, .data = sa_info_vf_ccs},
};
/* everything else should be added here */
static const struct drm_info_list pf_only_debugfs_list[] = {
{"hw_engines", .show = xe_gt_debugfs_simple_show, .data = hw_engines},
@ -363,6 +316,24 @@ static ssize_t write_to_gt_call(const char __user *userbuf, size_t count, loff_t
return count;
}
static ssize_t stats_write(struct file *file, const char __user *userbuf,
size_t count, loff_t *ppos)
{
struct seq_file *s = file->private_data;
struct xe_gt *gt = s->private;
return write_to_gt_call(userbuf, count, ppos, xe_gt_stats_clear, gt);
}
static int stats_show(struct seq_file *s, void *unused)
{
struct drm_printer p = drm_seq_file_printer(s);
struct xe_gt *gt = s->private;
return xe_gt_stats_print_info(gt, &p);
}
DEFINE_SHOW_STORE_ATTRIBUTE(stats);
static void force_reset(struct xe_gt *gt)
{
struct xe_device *xe = gt_to_xe(gt);
@ -448,6 +419,7 @@ void xe_gt_debugfs_register(struct xe_gt *gt)
root->d_inode->i_private = gt;
/* VF safe */
debugfs_create_file("stats", 0600, root, gt, &stats_fops);
debugfs_create_file("force_reset", 0600, root, gt, &force_reset_fops);
debugfs_create_file("force_reset_sync", 0600, root, gt, &force_reset_sync_fops);
@ -459,11 +431,6 @@ void xe_gt_debugfs_register(struct xe_gt *gt)
drm_debugfs_create_files(pf_only_debugfs_list,
ARRAY_SIZE(pf_only_debugfs_list),
root, minor);
else
drm_debugfs_create_files(vf_only_debugfs_list,
ARRAY_SIZE(vf_only_debugfs_list),
root, minor);
xe_uc_debugfs_register(&gt->uc, root);

28
drivers/gpu/drm/xe/xe_gt_freq.c
View File

@ -227,6 +227,33 @@ static ssize_t max_freq_store(struct kobject *kobj,
}
static struct kobj_attribute attr_max_freq = __ATTR_RW(max_freq);
static ssize_t power_profile_show(struct kobject *kobj,
struct kobj_attribute *attr,
char *buff)
{
struct device *dev = kobj_to_dev(kobj);
xe_guc_pc_get_power_profile(dev_to_pc(dev), buff);
return strlen(buff);
}
static ssize_t power_profile_store(struct kobject *kobj,
struct kobj_attribute *attr,
const char *buff, size_t count)
{
struct device *dev = kobj_to_dev(kobj);
struct xe_guc_pc *pc = dev_to_pc(dev);
int err;
xe_pm_runtime_get(dev_to_xe(dev));
err = xe_guc_pc_set_power_profile(pc, buff);
xe_pm_runtime_put(dev_to_xe(dev));
return err ?: count;
}
static struct kobj_attribute attr_power_profile = __ATTR_RW(power_profile);
static const struct attribute *freq_attrs[] = {
&attr_act_freq.attr,
&attr_cur_freq.attr,
@ -236,6 +263,7 @@ static const struct attribute *freq_attrs[] = {
&attr_rpn_freq.attr,
&attr_min_freq.attr,
&attr_max_freq.attr,
&attr_power_profile.attr,
NULL
};

2
drivers/gpu/drm/xe/xe_gt_mcr.c
View File

@ -362,7 +362,7 @@ static unsigned int dss_per_group(struct xe_gt *gt)
* @group: pointer to storage for steering group ID
* @instance: pointer to storage for steering instance ID
*/
void xe_gt_mcr_get_dss_steering(struct xe_gt *gt, unsigned int dss, u16 *group, u16 *instance)
void xe_gt_mcr_get_dss_steering(const struct xe_gt *gt, unsigned int dss, u16 *group, u16 *instance)
{
xe_gt_assert(gt, dss < XE_MAX_DSS_FUSE_BITS);

3
drivers/gpu/drm/xe/xe_gt_mcr.h
View File

@ -31,7 +31,8 @@ bool xe_gt_mcr_get_nonterminated_steering(struct xe_gt *gt,
u8 *group, u8 *instance);
void xe_gt_mcr_steering_dump(struct xe_gt *gt, struct drm_printer *p);
void xe_gt_mcr_get_dss_steering(struct xe_gt *gt, unsigned int dss, u16 *group, u16 *instance);
void xe_gt_mcr_get_dss_steering(const struct xe_gt *gt,
unsigned int dss, u16 *group, u16 *instance);
u32 xe_gt_mcr_steering_info_to_dss_id(struct xe_gt *gt, u16 group, u16 instance);
/*

28
drivers/gpu/drm/xe/xe_gt_pagefault.c
View File

@ -87,10 +87,8 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,
if (!bo)
return 0;
err = need_vram_move ? xe_bo_migrate(bo, vram->placement) :
xe_bo_validate(bo, vm, true);
return err;
return need_vram_move ? xe_bo_migrate(bo, vram->placement, NULL, exec) :
xe_bo_validate(bo, vm, true, exec);
}
static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,
@ -98,9 +96,9 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,
{
struct xe_vm *vm = xe_vma_vm(vma);
struct xe_tile *tile = gt_to_tile(gt);
struct xe_validation_ctx ctx;
struct drm_exec exec;
struct dma_fence *fence;
ktime_t end = 0;
int err, needs_vram;
lockdep_assert_held_write(&vm->lock);
@ -129,22 +127,22 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,
}
/* Lock VM and BOs dma-resv */
drm_exec_init(&exec, 0, 0);
xe_validation_ctx_init(&ctx, &vm->xe->val, &exec, (struct xe_val_flags) {});
drm_exec_until_all_locked(&exec) {
err = xe_pf_begin(&exec, vma, needs_vram == 1, tile->mem.vram);
drm_exec_retry_on_contention(&exec);
if (xe_vm_validate_should_retry(&exec, err, &end))
err = -EAGAIN;
xe_validation_retry_on_oom(&ctx, &err);
if (err)
goto unlock_dma_resv;
/* Bind VMA only to the GT that has faulted */
trace_xe_vma_pf_bind(vma);
xe_vm_set_validation_exec(vm, &exec);
fence = xe_vma_rebind(vm, vma, BIT(tile->id));
xe_vm_set_validation_exec(vm, NULL);
if (IS_ERR(fence)) {
err = PTR_ERR(fence);
if (xe_vm_validate_should_retry(&exec, err, &end))
err = -EAGAIN;
xe_validation_retry_on_oom(&ctx, &err);
goto unlock_dma_resv;
}
}
@ -153,7 +151,7 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,
dma_fence_put(fence);
unlock_dma_resv:
drm_exec_fini(&exec);
xe_validation_ctx_fini(&ctx);
if (err == -EAGAIN)
goto retry_userptr;
@ -535,6 +533,7 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc)
{
struct xe_device *xe = gt_to_xe(gt);
struct xe_tile *tile = gt_to_tile(gt);
struct xe_validation_ctx ctx;
struct drm_exec exec;
struct xe_vm *vm;
struct xe_vma *vma;
@ -564,15 +563,14 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc)
goto unlock_vm;
/* Lock VM and BOs dma-resv */
drm_exec_init(&exec, 0, 0);
xe_validation_ctx_init(&ctx, &vm->xe->val, &exec, (struct xe_val_flags) {});
drm_exec_until_all_locked(&exec) {
ret = xe_pf_begin(&exec, vma, IS_DGFX(vm->xe), tile->mem.vram);
drm_exec_retry_on_contention(&exec);
if (ret)
break;
xe_validation_retry_on_oom(&ctx, &ret);
}
drm_exec_fini(&exec);
xe_validation_ctx_fini(&ctx);
unlock_vm:
up_read(&vm->lock);
xe_vm_put(vm);

32
drivers/gpu/drm/xe/xe_gt_printk.h
View File

@ -6,18 +6,22 @@
#ifndef _XE_GT_PRINTK_H_
#define _XE_GT_PRINTK_H_
#include <drm/drm_print.h>
#include "xe_gt_types.h"
#include "xe_tile_printk.h"
#define __XE_GT_PRINTK_FMT(_gt, _fmt, _args...) "GT%u: " _fmt, (_gt)->info.id, ##_args
#define xe_gt_printk(_gt, _level, _fmt, ...) \
drm_##_level(&gt_to_xe(_gt)->drm, "GT%u: " _fmt, (_gt)->info.id, ##__VA_ARGS__)
xe_tile_printk((_gt)->tile, _level, __XE_GT_PRINTK_FMT((_gt), _fmt, ##__VA_ARGS__))
#define xe_gt_err(_gt, _fmt, ...) \
xe_gt_printk((_gt), err, _fmt, ##__VA_ARGS__)
#define xe_gt_err_once(_gt, _fmt, ...) \
xe_gt_printk((_gt), err_once, _fmt, ##__VA_ARGS__)
#define xe_gt_err(_gt, _fmt, ...) \
xe_gt_printk((_gt), err, _fmt, ##__VA_ARGS__)
#define xe_gt_err_ratelimited(_gt, _fmt, ...) \
xe_gt_printk((_gt), err_ratelimited, _fmt, ##__VA_ARGS__)
#define xe_gt_warn(_gt, _fmt, ...) \
xe_gt_printk((_gt), warn, _fmt, ##__VA_ARGS__)
@ -31,20 +35,20 @@
#define xe_gt_dbg(_gt, _fmt, ...) \
xe_gt_printk((_gt), dbg, _fmt, ##__VA_ARGS__)
#define xe_gt_err_ratelimited(_gt, _fmt, ...) \
xe_gt_printk((_gt), err_ratelimited, _fmt, ##__VA_ARGS__)
#define xe_gt_WARN_type(_gt, _type, _condition, _fmt, ...) \
xe_tile_WARN##_type((_gt)->tile, _condition, _fmt, ## __VA_ARGS__)
#define xe_gt_WARN(_gt, _condition, _fmt, ...) \
drm_WARN(&gt_to_xe(_gt)->drm, _condition, "GT%u: " _fmt, (_gt)->info.id, ##__VA_ARGS__)
xe_gt_WARN_type((_gt),, _condition, __XE_GT_PRINTK_FMT((_gt), _fmt, ##__VA_ARGS__))
#define xe_gt_WARN_ONCE(_gt, _condition, _fmt, ...) \
drm_WARN_ONCE(&gt_to_xe(_gt)->drm, _condition, "GT%u: " _fmt, (_gt)->info.id, ##__VA_ARGS__)
xe_gt_WARN_type((_gt), _ONCE, _condition, __XE_GT_PRINTK_FMT((_gt), _fmt, ##__VA_ARGS__))
#define xe_gt_WARN_ON(_gt, _condition) \
xe_gt_WARN((_gt), _condition, "%s(%s)", "gt_WARN_ON", __stringify(_condition))
xe_gt_WARN((_gt), _condition, "%s(%s)", "WARN_ON", __stringify(_condition))
#define xe_gt_WARN_ON_ONCE(_gt, _condition) \
xe_gt_WARN_ONCE((_gt), _condition, "%s(%s)", "gt_WARN_ON_ONCE", __stringify(_condition))
xe_gt_WARN_ONCE((_gt), _condition, "%s(%s)", "WARN_ON_ONCE", __stringify(_condition))
static inline void __xe_gt_printfn_err(struct drm_printer *p, struct va_format *vaf)
{
@ -67,12 +71,12 @@ static inline void __xe_gt_printfn_dbg(struct drm_printer *p, struct va_format *
/*
* The original xe_gt_dbg() callsite annotations are useless here,
* redirect to the tweaked drm_dbg_printer() instead.
* redirect to the tweaked xe_tile_dbg_printer() instead.
*/
dbg = drm_dbg_printer(&gt_to_xe(gt)->drm, DRM_UT_DRIVER, NULL);
dbg = xe_tile_dbg_printer((gt)->tile);
dbg.origin = p->origin;
drm_printf(&dbg, "GT%u: %pV", gt->info.id, vaf);
drm_printf(&dbg, __XE_GT_PRINTK_FMT(gt, "%pV", vaf));
}
/**

22
drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c
View File

@ -1478,23 +1478,16 @@ static int pf_provision_vf_lmem(struct xe_gt *gt, unsigned int vfid, u64 size)
return 0;
xe_gt_assert(gt, pf_get_lmem_alignment(gt) == SZ_2M);
bo = xe_bo_create_locked(xe, tile, NULL,
ALIGN(size, PAGE_SIZE),
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_NEEDS_2M |
XE_BO_FLAG_PINNED |
XE_BO_FLAG_PINNED_LATE_RESTORE);
bo = xe_bo_create_pin_range_novm(xe, tile,
ALIGN(size, PAGE_SIZE), 0, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_NEEDS_2M |
XE_BO_FLAG_PINNED |
XE_BO_FLAG_PINNED_LATE_RESTORE);
if (IS_ERR(bo))
return PTR_ERR(bo);
err = xe_bo_pin(bo);
xe_bo_unlock(bo);
if (unlikely(err)) {
xe_bo_put(bo);
return err;
}
config->lmem_obj = bo;
if (xe_device_has_lmtt(xe)) {
@ -1636,7 +1629,6 @@ static u64 pf_estimate_fair_lmem(struct xe_gt *gt, unsigned int num_vfs)
u64 fair;
fair = div_u64(available, num_vfs);
fair = rounddown_pow_of_two(fair); /* XXX: ttm_vram_mgr & drm_buddy limitation */
fair = ALIGN_DOWN(fair, alignment);
#ifdef MAX_FAIR_LMEM
fair = min_t(u64, MAX_FAIR_LMEM, fair);

24
drivers/gpu/drm/xe/xe_gt_sriov_pf_migration.c
View File

@ -55,12 +55,12 @@ static int pf_send_guc_save_vf_state(struct xe_gt *gt, unsigned int vfid,
xe_gt_assert(gt, size % sizeof(u32) == 0);
xe_gt_assert(gt, size == ndwords * sizeof(u32));
bo = xe_bo_create_pin_map(xe, tile, NULL,
ALIGN(size, PAGE_SIZE),
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_INVALIDATE);
bo = xe_bo_create_pin_map_novm(xe, tile,
ALIGN(size, PAGE_SIZE),
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_INVALIDATE, false);
if (IS_ERR(bo))
return PTR_ERR(bo);
@ -91,12 +91,12 @@ static int pf_send_guc_restore_vf_state(struct xe_gt *gt, unsigned int vfid,
xe_gt_assert(gt, size % sizeof(u32) == 0);
xe_gt_assert(gt, size == ndwords * sizeof(u32));
bo = xe_bo_create_pin_map(xe, tile, NULL,
ALIGN(size, PAGE_SIZE),
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_INVALIDATE);
bo = xe_bo_create_pin_map_novm(xe, tile,
ALIGN(size, PAGE_SIZE),
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_INVALIDATE, false);
if (IS_ERR(bo))
return PTR_ERR(bo);

57
drivers/gpu/drm/xe/xe_gt_stats.c
View File

@ -26,11 +26,46 @@ void xe_gt_stats_incr(struct xe_gt *gt, const enum xe_gt_stats_id id, int incr)
atomic64_add(incr, &gt->stats.counters[id]);
}
#define DEF_STAT_STR(ID, name) [XE_GT_STATS_ID_##ID] = name
static const char *const stat_description[__XE_GT_STATS_NUM_IDS] = {
"svm_pagefault_count",
"tlb_inval_count",
"vma_pagefault_count",
"vma_pagefault_kb",
DEF_STAT_STR(SVM_PAGEFAULT_COUNT, "svm_pagefault_count"),
DEF_STAT_STR(TLB_INVAL, "tlb_inval_count"),
DEF_STAT_STR(SVM_TLB_INVAL_COUNT, "svm_tlb_inval_count"),
DEF_STAT_STR(SVM_TLB_INVAL_US, "svm_tlb_inval_us"),
DEF_STAT_STR(VMA_PAGEFAULT_COUNT, "vma_pagefault_count"),
DEF_STAT_STR(VMA_PAGEFAULT_KB, "vma_pagefault_kb"),
DEF_STAT_STR(SVM_4K_PAGEFAULT_COUNT, "svm_4K_pagefault_count"),
DEF_STAT_STR(SVM_64K_PAGEFAULT_COUNT, "svm_64K_pagefault_count"),
DEF_STAT_STR(SVM_2M_PAGEFAULT_COUNT, "svm_2M_pagefault_count"),
DEF_STAT_STR(SVM_4K_VALID_PAGEFAULT_COUNT, "svm_4K_valid_pagefault_count"),
DEF_STAT_STR(SVM_64K_VALID_PAGEFAULT_COUNT, "svm_64K_valid_pagefault_count"),
DEF_STAT_STR(SVM_2M_VALID_PAGEFAULT_COUNT, "svm_2M_valid_pagefault_count"),
DEF_STAT_STR(SVM_4K_PAGEFAULT_US, "svm_4K_pagefault_us"),
DEF_STAT_STR(SVM_64K_PAGEFAULT_US, "svm_64K_pagefault_us"),
DEF_STAT_STR(SVM_2M_PAGEFAULT_US, "svm_2M_pagefault_us"),
DEF_STAT_STR(SVM_4K_MIGRATE_COUNT, "svm_4K_migrate_count"),
DEF_STAT_STR(SVM_64K_MIGRATE_COUNT, "svm_64K_migrate_count"),
DEF_STAT_STR(SVM_2M_MIGRATE_COUNT, "svm_2M_migrate_count"),
DEF_STAT_STR(SVM_4K_MIGRATE_US, "svm_4K_migrate_us"),
DEF_STAT_STR(SVM_64K_MIGRATE_US, "svm_64K_migrate_us"),
DEF_STAT_STR(SVM_2M_MIGRATE_US, "svm_2M_migrate_us"),
DEF_STAT_STR(SVM_DEVICE_COPY_US, "svm_device_copy_us"),
DEF_STAT_STR(SVM_4K_DEVICE_COPY_US, "svm_4K_device_copy_us"),
DEF_STAT_STR(SVM_64K_DEVICE_COPY_US, "svm_64K_device_copy_us"),
DEF_STAT_STR(SVM_2M_DEVICE_COPY_US, "svm_2M_device_copy_us"),
DEF_STAT_STR(SVM_CPU_COPY_US, "svm_cpu_copy_us"),
DEF_STAT_STR(SVM_4K_CPU_COPY_US, "svm_4K_cpu_copy_us"),
DEF_STAT_STR(SVM_64K_CPU_COPY_US, "svm_64K_cpu_copy_us"),
DEF_STAT_STR(SVM_2M_CPU_COPY_US, "svm_2M_cpu_copy_us"),
DEF_STAT_STR(SVM_DEVICE_COPY_KB, "svm_device_copy_kb"),
DEF_STAT_STR(SVM_CPU_COPY_KB, "svm_cpu_copy_kb"),
DEF_STAT_STR(SVM_4K_GET_PAGES_US, "svm_4K_get_pages_us"),
DEF_STAT_STR(SVM_64K_GET_PAGES_US, "svm_64K_get_pages_us"),
DEF_STAT_STR(SVM_2M_GET_PAGES_US, "svm_2M_get_pages_us"),
DEF_STAT_STR(SVM_4K_BIND_US, "svm_4K_bind_us"),
DEF_STAT_STR(SVM_64K_BIND_US, "svm_64K_bind_us"),
DEF_STAT_STR(SVM_2M_BIND_US, "svm_2M_bind_us"),
};
/**
@ -50,3 +85,17 @@ int xe_gt_stats_print_info(struct xe_gt *gt, struct drm_printer *p)
return 0;
}
/**
* xe_gt_stats_clear - Clear the GT stats
* @gt: GT structure
*
* This clear (zeros) all the available GT stats.
*/
void xe_gt_stats_clear(struct xe_gt *gt)
{
int id;
for (id = 0; id < ARRAY_SIZE(gt->stats.counters); ++id)
atomic64_set(&gt->stats.counters[id], 0);
}

1
drivers/gpu/drm/xe/xe_gt_stats.h
View File

@ -13,6 +13,7 @@ struct drm_printer;
#ifdef CONFIG_DEBUG_FS
int xe_gt_stats_print_info(struct xe_gt *gt, struct drm_printer *p);
void xe_gt_stats_clear(struct xe_gt *gt);
void xe_gt_stats_incr(struct xe_gt *gt, const enum xe_gt_stats_id id, int incr);
#else
static inline void

33
drivers/gpu/drm/xe/xe_gt_stats_types.h
View File

@ -9,8 +9,41 @@
enum xe_gt_stats_id {
XE_GT_STATS_ID_SVM_PAGEFAULT_COUNT,
XE_GT_STATS_ID_TLB_INVAL,
XE_GT_STATS_ID_SVM_TLB_INVAL_COUNT,
XE_GT_STATS_ID_SVM_TLB_INVAL_US,
XE_GT_STATS_ID_VMA_PAGEFAULT_COUNT,
XE_GT_STATS_ID_VMA_PAGEFAULT_KB,
XE_GT_STATS_ID_SVM_4K_PAGEFAULT_COUNT,
XE_GT_STATS_ID_SVM_64K_PAGEFAULT_COUNT,
XE_GT_STATS_ID_SVM_2M_PAGEFAULT_COUNT,
XE_GT_STATS_ID_SVM_4K_VALID_PAGEFAULT_COUNT,
XE_GT_STATS_ID_SVM_64K_VALID_PAGEFAULT_COUNT,
XE_GT_STATS_ID_SVM_2M_VALID_PAGEFAULT_COUNT,
XE_GT_STATS_ID_SVM_4K_PAGEFAULT_US,
XE_GT_STATS_ID_SVM_64K_PAGEFAULT_US,
XE_GT_STATS_ID_SVM_2M_PAGEFAULT_US,
XE_GT_STATS_ID_SVM_4K_MIGRATE_COUNT,
XE_GT_STATS_ID_SVM_64K_MIGRATE_COUNT,
XE_GT_STATS_ID_SVM_2M_MIGRATE_COUNT,
XE_GT_STATS_ID_SVM_4K_MIGRATE_US,
XE_GT_STATS_ID_SVM_64K_MIGRATE_US,
XE_GT_STATS_ID_SVM_2M_MIGRATE_US,
XE_GT_STATS_ID_SVM_DEVICE_COPY_US,
XE_GT_STATS_ID_SVM_4K_DEVICE_COPY_US,
XE_GT_STATS_ID_SVM_64K_DEVICE_COPY_US,
XE_GT_STATS_ID_SVM_2M_DEVICE_COPY_US,
XE_GT_STATS_ID_SVM_CPU_COPY_US,
XE_GT_STATS_ID_SVM_4K_CPU_COPY_US,
XE_GT_STATS_ID_SVM_64K_CPU_COPY_US,
XE_GT_STATS_ID_SVM_2M_CPU_COPY_US,
XE_GT_STATS_ID_SVM_DEVICE_COPY_KB,
XE_GT_STATS_ID_SVM_CPU_COPY_KB,
XE_GT_STATS_ID_SVM_4K_GET_PAGES_US,
XE_GT_STATS_ID_SVM_64K_GET_PAGES_US,
XE_GT_STATS_ID_SVM_2M_GET_PAGES_US,
XE_GT_STATS_ID_SVM_4K_BIND_US,
XE_GT_STATS_ID_SVM_64K_BIND_US,
XE_GT_STATS_ID_SVM_2M_BIND_US,
/* must be the last entry */
__XE_GT_STATS_NUM_IDS,
};

48
drivers/gpu/drm/xe/xe_gt_topology.c
View File

@ -12,6 +12,7 @@
#include "regs/xe_gt_regs.h"
#include "xe_assert.h"
#include "xe_gt.h"
#include "xe_gt_mcr.h"
#include "xe_gt_printk.h"
#include "xe_mmio.h"
#include "xe_wa.h"
@ -122,6 +123,21 @@ gen_l3_mask_from_pattern(struct xe_device *xe, xe_l3_bank_mask_t dst,
}
}
bool xe_gt_topology_report_l3(struct xe_gt *gt)
{
/*
* No known userspace needs/uses the L3 bank mask reported by
* the media GT, and the hardware itself is known to report bogus
* values on several platforms. Only report L3 bank mask as part
* of the media GT's topology on pre-Xe3 platforms since that's
* already part of our ABI.
*/
if (xe_gt_is_media_type(gt) && MEDIA_VER(gt_to_xe(gt)) >= 30)
return false;
return true;
}
static void
load_l3_bank_mask(struct xe_gt *gt, xe_l3_bank_mask_t l3_bank_mask)
{
@ -129,16 +145,7 @@ load_l3_bank_mask(struct xe_gt *gt, xe_l3_bank_mask_t l3_bank_mask)
struct xe_mmio *mmio = &gt->mmio;
u32 fuse3 = xe_mmio_read32(mmio, MIRROR_FUSE3);
/*
* PTL platforms with media version 30.00 do not provide proper values
* for the media GT's L3 bank registers. Skip the readout since we
* don't have any way to obtain real values.
*
* This may get re-described as an official workaround in the future,
* but there's no tracking number assigned yet so we use a custom
* OOB workaround descriptor.
*/
if (XE_GT_WA(gt, no_media_l3))
if (!xe_gt_topology_report_l3(gt))
return;
if (GRAPHICS_VER(xe) >= 30) {
@ -275,8 +282,9 @@ xe_gt_topology_dump(struct xe_gt *gt, struct drm_printer *p)
drm_printf(p, "EU type: %s\n",
eu_type_to_str(gt->fuse_topo.eu_type));
drm_printf(p, "L3 bank mask: %*pb\n", XE_MAX_L3_BANK_MASK_BITS,
gt->fuse_topo.l3_bank_mask);
if (xe_gt_topology_report_l3(gt))
drm_printf(p, "L3 bank mask: %*pb\n", XE_MAX_L3_BANK_MASK_BITS,
gt->fuse_topo.l3_bank_mask);
}
/*
@ -328,3 +336,19 @@ bool xe_gt_has_compute_dss(struct xe_gt *gt, unsigned int dss)
{
return test_bit(dss, gt->fuse_topo.c_dss_mask);
}
bool xe_gt_has_discontiguous_dss_groups(const struct xe_gt *gt)
{
unsigned int xecore;
int last_group = -1;
u16 group, instance;
for_each_dss_steering(xecore, gt, group, instance) {
if (last_group != group) {
if (group - last_group > 1)
return true;
last_group = group;
}
}
return false;
}

4
drivers/gpu/drm/xe/xe_gt_topology.h
View File

@ -47,4 +47,8 @@ xe_gt_topology_has_dss_in_quadrant(struct xe_gt *gt, int quad);
bool xe_gt_has_geometry_dss(struct xe_gt *gt, unsigned int dss);
bool xe_gt_has_compute_dss(struct xe_gt *gt, unsigned int dss);
bool xe_gt_has_discontiguous_dss_groups(const struct xe_gt *gt);
bool xe_gt_topology_report_l3(struct xe_gt *gt);
#endif /* _XE_GT_TOPOLOGY_H_ */

52
drivers/gpu/drm/xe/xe_guc.c
View File

@ -74,8 +74,7 @@ static u32 guc_ctl_debug_flags(struct xe_guc *guc)
if (!GUC_LOG_LEVEL_IS_VERBOSE(level))
flags |= GUC_LOG_DISABLED;
else
flags |= GUC_LOG_LEVEL_TO_VERBOSITY(level) <<
GUC_LOG_VERBOSITY_SHIFT;
flags |= FIELD_PREP(GUC_LOG_VERBOSITY, GUC_LOG_LEVEL_TO_VERBOSITY(level));
return flags;
}
@ -122,22 +121,14 @@ static u32 guc_ctl_log_params_flags(struct xe_guc *guc)
BUILD_BUG_ON(!CAPTURE_BUFFER_SIZE);
BUILD_BUG_ON(!IS_ALIGNED(CAPTURE_BUFFER_SIZE, CAPTURE_UNIT));
BUILD_BUG_ON((CRASH_BUFFER_SIZE / LOG_UNIT - 1) >
(GUC_LOG_CRASH_MASK >> GUC_LOG_CRASH_SHIFT));
BUILD_BUG_ON((DEBUG_BUFFER_SIZE / LOG_UNIT - 1) >
(GUC_LOG_DEBUG_MASK >> GUC_LOG_DEBUG_SHIFT));
BUILD_BUG_ON((CAPTURE_BUFFER_SIZE / CAPTURE_UNIT - 1) >
(GUC_LOG_CAPTURE_MASK >> GUC_LOG_CAPTURE_SHIFT));
flags = GUC_LOG_VALID |
GUC_LOG_NOTIFY_ON_HALF_FULL |
CAPTURE_FLAG |
LOG_FLAG |
((CRASH_BUFFER_SIZE / LOG_UNIT - 1) << GUC_LOG_CRASH_SHIFT) |
((DEBUG_BUFFER_SIZE / LOG_UNIT - 1) << GUC_LOG_DEBUG_SHIFT) |
((CAPTURE_BUFFER_SIZE / CAPTURE_UNIT - 1) <<
GUC_LOG_CAPTURE_SHIFT) |
(offset << GUC_LOG_BUF_ADDR_SHIFT);
FIELD_PREP(GUC_LOG_CRASH, CRASH_BUFFER_SIZE / LOG_UNIT - 1) |
FIELD_PREP(GUC_LOG_DEBUG, DEBUG_BUFFER_SIZE / LOG_UNIT - 1) |
FIELD_PREP(GUC_LOG_CAPTURE, CAPTURE_BUFFER_SIZE / CAPTURE_UNIT - 1) |
FIELD_PREP(GUC_LOG_BUF_ADDR, offset);
#undef LOG_UNIT
#undef LOG_FLAG
@ -150,7 +141,7 @@ static u32 guc_ctl_log_params_flags(struct xe_guc *guc)
static u32 guc_ctl_ads_flags(struct xe_guc *guc)
{
u32 ads = guc_bo_ggtt_addr(guc, guc->ads.bo) >> PAGE_SHIFT;
u32 flags = ads << GUC_ADS_ADDR_SHIFT;
u32 flags = FIELD_PREP(GUC_ADS_ADDR, ads);
return flags;
}
@ -709,10 +700,6 @@ static int xe_guc_realloc_post_hwconfig(struct xe_guc *guc)
if (ret)
return ret;
ret = xe_managed_bo_reinit_in_vram(xe, tile, &guc->ct.bo);
if (ret)
return ret;
return 0;
}
@ -847,6 +834,10 @@ int xe_guc_init_post_hwconfig(struct xe_guc *guc)
if (ret)
return ret;
ret = xe_guc_ct_init_post_hwconfig(&guc->ct);
if (ret)
return ret;
guc_init_params_post_hwconfig(guc);
ret = xe_guc_submit_init(guc, ~0);
@ -888,9 +879,7 @@ int xe_guc_post_load_init(struct xe_guc *guc)
return ret;
}
guc->submission_state.enabled = true;
return 0;
return xe_guc_submit_enable(guc);
}
int xe_guc_reset(struct xe_guc *guc)
@ -1066,7 +1055,7 @@ static s32 guc_pc_get_cur_freq(struct xe_guc_pc *guc_pc)
#endif
#define GUC_LOAD_TIME_WARN_MS 200
static void guc_wait_ucode(struct xe_guc *guc)
static int guc_wait_ucode(struct xe_guc *guc)
{
struct xe_gt *gt = guc_to_gt(guc);
struct xe_mmio *mmio = &gt->mmio;
@ -1173,7 +1162,7 @@ static void guc_wait_ucode(struct xe_guc *guc)
break;
}
xe_device_declare_wedged(gt_to_xe(gt));
return -EPROTO;
} else if (delta_ms > GUC_LOAD_TIME_WARN_MS) {
xe_gt_warn(gt, "excessive init time: %lldms! [status = 0x%08X, timeouts = %d]\n",
delta_ms, status, count);
@ -1185,7 +1174,10 @@ static void guc_wait_ucode(struct xe_guc *guc)
delta_ms, xe_guc_pc_get_act_freq(guc_pc), guc_pc_get_cur_freq(guc_pc),
before_freq, status, count);
}
return 0;
}
ALLOW_ERROR_INJECTION(guc_wait_ucode, ERRNO);
static int __xe_guc_upload(struct xe_guc *guc)
{
@ -1217,14 +1209,16 @@ static int __xe_guc_upload(struct xe_guc *guc)
goto out;
/* Wait for authentication */
guc_wait_ucode(guc);
ret = guc_wait_ucode(guc);
if (ret)
goto out;
xe_uc_fw_change_status(&guc->fw, XE_UC_FIRMWARE_RUNNING);
return 0;
out:
xe_uc_fw_change_status(&guc->fw, XE_UC_FIRMWARE_LOAD_FAIL);
return 0 /* FIXME: ret, don't want to stop load currently */;
return ret;
}
static int vf_guc_min_load_for_hwconfig(struct xe_guc *guc)
@ -1602,7 +1596,7 @@ void xe_guc_sanitize(struct xe_guc *guc)
{
xe_uc_fw_sanitize(&guc->fw);
xe_guc_ct_disable(&guc->ct);
guc->submission_state.enabled = false;
xe_guc_submit_disable(guc);
}
int xe_guc_reset_prepare(struct xe_guc *guc)
@ -1695,3 +1689,7 @@ void xe_guc_declare_wedged(struct xe_guc *guc)
xe_guc_ct_stop(&guc->ct);
xe_guc_submit_wedge(guc);
}
#if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
#include "tests/xe_guc_g2g_test.c"
#endif

4
drivers/gpu/drm/xe/xe_guc.h
View File

@ -53,6 +53,10 @@ void xe_guc_stop(struct xe_guc *guc);
int xe_guc_start(struct xe_guc *guc);
void xe_guc_declare_wedged(struct xe_guc *guc);
#if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
int xe_guc_g2g_test_notification(struct xe_guc *guc, u32 *payload, u32 len);
#endif
static inline u16 xe_engine_class_to_guc_class(enum xe_engine_class class)
{
switch (class) {

4
drivers/gpu/drm/xe/xe_guc_ads.c
View File

@ -339,7 +339,7 @@ static void guc_waklv_init(struct xe_guc_ads *ads)
if (XE_GT_WA(gt, 13011645652)) {
u32 data = 0xC40;
guc_waklv_enable(ads, &data, sizeof(data) / sizeof(u32), &offset, &remain,
guc_waklv_enable(ads, &data, 1, &offset, &remain,
GUC_WA_KLV_NP_RD_WRITE_TO_CLEAR_RCSM_AT_CGP_LATE_RESTORE);
}
@ -355,7 +355,7 @@ static void guc_waklv_init(struct xe_guc_ads *ads)
0x0,
0xF,
};
guc_waklv_enable(ads, data, sizeof(data) / sizeof(u32), &offset, &remain,
guc_waklv_enable(ads, data, ARRAY_SIZE(data), &offset, &remain,
GUC_WA_KLV_RESTORE_UNSAVED_MEDIA_CONTROL_REG);
}

55
drivers/gpu/drm/xe/xe_guc_ct.c
View File

@ -39,6 +39,8 @@ static void receive_g2h(struct xe_guc_ct *ct);
static void g2h_worker_func(struct work_struct *w);
static void safe_mode_worker_func(struct work_struct *w);
static void ct_exit_safe_mode(struct xe_guc_ct *ct);
static void guc_ct_change_state(struct xe_guc_ct *ct,
enum xe_guc_ct_state state);
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
enum {
@ -252,6 +254,13 @@ int xe_guc_ct_init_noalloc(struct xe_guc_ct *ct)
}
ALLOW_ERROR_INJECTION(xe_guc_ct_init_noalloc, ERRNO); /* See xe_pci_probe() */
static void guc_action_disable_ct(void *arg)
{
struct xe_guc_ct *ct = arg;
guc_ct_change_state(ct, XE_GUC_CT_STATE_DISABLED);
}
int xe_guc_ct_init(struct xe_guc_ct *ct)
{
struct xe_device *xe = ct_to_xe(ct);
@ -268,10 +277,39 @@ int xe_guc_ct_init(struct xe_guc_ct *ct)
return PTR_ERR(bo);
ct->bo = bo;
return 0;
return devm_add_action_or_reset(xe->drm.dev, guc_action_disable_ct, ct);
}
ALLOW_ERROR_INJECTION(xe_guc_ct_init, ERRNO); /* See xe_pci_probe() */
/**
* xe_guc_ct_init_post_hwconfig - Reinitialize the GuC CTB in VRAM
* @ct: the &xe_guc_ct
*
* Allocate a new BO in VRAM and free the previous BO that was allocated
* in system memory (SMEM). Applicable only for DGFX products.
*
* Return: 0 on success, or a negative errno on failure.
*/
int xe_guc_ct_init_post_hwconfig(struct xe_guc_ct *ct)
{
struct xe_device *xe = ct_to_xe(ct);
struct xe_gt *gt = ct_to_gt(ct);
struct xe_tile *tile = gt_to_tile(gt);
int ret;
xe_assert(xe, !xe_guc_ct_enabled(ct));
if (IS_DGFX(xe)) {
ret = xe_managed_bo_reinit_in_vram(xe, tile, &ct->bo);
if (ret)
return ret;
}
devm_remove_action(xe->drm.dev, guc_action_disable_ct, ct);
return devm_add_action_or_reset(xe->drm.dev, guc_action_disable_ct, ct);
}
#define desc_read(xe_, guc_ctb__, field_) \
xe_map_rd_field(xe_, &guc_ctb__->desc, 0, \
struct guc_ct_buffer_desc, field_)
@ -1040,11 +1078,15 @@ static bool retry_failure(struct xe_guc_ct *ct, int ret)
return true;
}
#define GUC_SEND_RETRY_LIMIT 50
#define GUC_SEND_RETRY_MSLEEP 5
static int guc_ct_send_recv(struct xe_guc_ct *ct, const u32 *action, u32 len,
u32 *response_buffer, bool no_fail)
{
struct xe_gt *gt = ct_to_gt(ct);
struct g2h_fence g2h_fence;
unsigned int retries = 0;
int ret = 0;
/*
@ -1109,6 +1151,12 @@ static int guc_ct_send_recv(struct xe_guc_ct *ct, const u32 *action, u32 len,
xe_gt_dbg(gt, "H2G action %#x retrying: reason %#x\n",
action[0], g2h_fence.reason);
mutex_unlock(&ct->lock);
if (++retries > GUC_SEND_RETRY_LIMIT) {
xe_gt_err(gt, "H2G action %#x reached retry limit=%u, aborting\n",
action[0], GUC_SEND_RETRY_LIMIT);
return -ELOOP;
}
msleep(GUC_SEND_RETRY_MSLEEP * retries);
goto retry;
}
if (g2h_fence.fail) {
@ -1438,6 +1486,11 @@ static int process_g2h_msg(struct xe_guc_ct *ct, u32 *msg, u32 len)
case XE_GUC_ACTION_NOTIFY_EXCEPTION:
ret = guc_crash_process_msg(ct, action);
break;
#if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
case XE_GUC_ACTION_TEST_G2G_RECV:
ret = xe_guc_g2g_test_notification(guc, payload, adj_len);
break;
#endif
default:
xe_gt_err(gt, "unexpected G2H action 0x%04x\n", action);
}

1
drivers/gpu/drm/xe/xe_guc_ct.h
View File

@ -13,6 +13,7 @@ struct xe_device;
int xe_guc_ct_init_noalloc(struct xe_guc_ct *ct);
int xe_guc_ct_init(struct xe_guc_ct *ct);
int xe_guc_ct_init_post_hwconfig(struct xe_guc_ct *ct);
int xe_guc_ct_enable(struct xe_guc_ct *ct);
void xe_guc_ct_disable(struct xe_guc_ct *ct);
void xe_guc_ct_stop(struct xe_guc_ct *ct);

13
drivers/gpu/drm/xe/xe_guc_engine_activity.c
View File

@ -94,16 +94,17 @@ static int allocate_engine_activity_buffers(struct xe_guc *guc,
struct xe_tile *tile = gt_to_tile(gt);
struct xe_bo *bo, *metadata_bo;
metadata_bo = xe_bo_create_pin_map(gt_to_xe(gt), tile, NULL, PAGE_ALIGN(metadata_size),
ttm_bo_type_kernel, XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT | XE_BO_FLAG_GGTT_INVALIDATE);
metadata_bo = xe_bo_create_pin_map_novm(gt_to_xe(gt), tile, PAGE_ALIGN(metadata_size),
ttm_bo_type_kernel, XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT | XE_BO_FLAG_GGTT_INVALIDATE,
false);
if (IS_ERR(metadata_bo))
return PTR_ERR(metadata_bo);
bo = xe_bo_create_pin_map(gt_to_xe(gt), tile, NULL, PAGE_ALIGN(size),
ttm_bo_type_kernel, XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_GGTT | XE_BO_FLAG_GGTT_INVALIDATE);
bo = xe_bo_create_pin_map_novm(gt_to_xe(gt), tile, PAGE_ALIGN(size),
ttm_bo_type_kernel, XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_GGTT | XE_BO_FLAG_GGTT_INVALIDATE, false);
if (IS_ERR(bo)) {
xe_bo_unpin_map_no_vm(metadata_bo);

4
drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
View File

@ -35,8 +35,8 @@ struct xe_guc_exec_queue {
struct xe_sched_msg static_msgs[MAX_STATIC_MSG_TYPE];
/** @lr_tdr: long running TDR worker */
struct work_struct lr_tdr;
/** @fini_async: do final fini async from this worker */
struct work_struct fini_async;
/** @destroy_async: do final destroy async from this worker */
struct work_struct destroy_async;
/** @resume_time: time of last resume */
u64 resume_time;
/** @state: GuC specific state for this xe_exec_queue */

30
drivers/gpu/drm/xe/xe_guc_fwif.h
View File

@ -15,6 +15,7 @@
#define G2H_LEN_DW_SCHED_CONTEXT_MODE_SET 4
#define G2H_LEN_DW_DEREGISTER_CONTEXT 3
#define G2H_LEN_DW_TLB_INVALIDATE 3
#define G2H_LEN_DW_G2G_NOTIFY_MIN 3
#define GUC_ID_MAX 65535
#define GUC_ID_UNKNOWN 0xffffffff
@ -65,6 +66,7 @@ struct guc_ctxt_registration_info {
u32 hwlrca_hi;
};
#define CONTEXT_REGISTRATION_FLAG_KMD BIT(0)
#define CONTEXT_REGISTRATION_FLAG_TYPE GENMASK(2, 1)
/* 32-bit KLV structure as used by policy updates and others */
struct guc_klv_generic_dw_t {
@ -89,13 +91,10 @@ struct guc_update_exec_queue_policy {
#define GUC_LOG_NOTIFY_ON_HALF_FULL BIT(1)
#define GUC_LOG_CAPTURE_ALLOC_UNITS BIT(2)
#define GUC_LOG_LOG_ALLOC_UNITS BIT(3)
#define GUC_LOG_CRASH_SHIFT 4
#define GUC_LOG_CRASH_MASK (0x3 << GUC_LOG_CRASH_SHIFT)
#define GUC_LOG_DEBUG_SHIFT 6
#define GUC_LOG_DEBUG_MASK (0xF << GUC_LOG_DEBUG_SHIFT)
#define GUC_LOG_CAPTURE_SHIFT 10
#define GUC_LOG_CAPTURE_MASK (0x3 << GUC_LOG_CAPTURE_SHIFT)
#define GUC_LOG_BUF_ADDR_SHIFT 12
#define GUC_LOG_CRASH REG_GENMASK(5, 4)
#define GUC_LOG_DEBUG REG_GENMASK(9, 6)
#define GUC_LOG_CAPTURE REG_GENMASK(11, 10)
#define GUC_LOG_BUF_ADDR REG_GENMASK(31, 12)
#define GUC_CTL_WA 1
#define GUC_WA_GAM_CREDITS BIT(10)
@ -117,21 +116,14 @@ struct guc_update_exec_queue_policy {
#define GUC_CTL_DISABLE_SCHEDULER BIT(14)
#define GUC_CTL_DEBUG 3
#define GUC_LOG_VERBOSITY_SHIFT 0
#define GUC_LOG_VERBOSITY_LOW (0 << GUC_LOG_VERBOSITY_SHIFT)
#define GUC_LOG_VERBOSITY_MED (1 << GUC_LOG_VERBOSITY_SHIFT)
#define GUC_LOG_VERBOSITY_HIGH (2 << GUC_LOG_VERBOSITY_SHIFT)
#define GUC_LOG_VERBOSITY_ULTRA (3 << GUC_LOG_VERBOSITY_SHIFT)
#define GUC_LOG_VERBOSITY_MIN 0
#define GUC_LOG_VERBOSITY REG_GENMASK(1, 0)
#define GUC_LOG_VERBOSITY_MAX 3
#define GUC_LOG_VERBOSITY_MASK 0x0000000f
#define GUC_LOG_DESTINATION_MASK (3 << 4)
#define GUC_LOG_DISABLED (1 << 6)
#define GUC_PROFILE_ENABLED (1 << 7)
#define GUC_LOG_DESTINATION REG_GENMASK(5, 4)
#define GUC_LOG_DISABLED BIT(6)
#define GUC_PROFILE_ENABLED BIT(7)
#define GUC_CTL_ADS 4
#define GUC_ADS_ADDR_SHIFT 1
#define GUC_ADS_ADDR_MASK (0xFFFFF << GUC_ADS_ADDR_SHIFT)
#define GUC_ADS_ADDR REG_GENMASK(21, 1)
#define GUC_CTL_DEVID 5

2
drivers/gpu/drm/xe/xe_guc_log.h
View File

@ -17,7 +17,7 @@ struct xe_device;
#define DEBUG_BUFFER_SIZE SZ_8M
#define CAPTURE_BUFFER_SIZE SZ_2M
#else
#define CRASH_BUFFER_SIZE SZ_8K
#define CRASH_BUFFER_SIZE SZ_16K
#define DEBUG_BUFFER_SIZE SZ_64K
#define CAPTURE_BUFFER_SIZE SZ_1M
#endif

67
drivers/gpu/drm/xe/xe_guc_pc.c
View File

@ -79,6 +79,11 @@
* Xe driver enables SLPC with all of its defaults features and frequency
* selection, which varies per platform.
*
* Power profiles add another level of control to SLPC. When power saving
* profile is chosen, SLPC will use conservative thresholds to ramp frequency,
* thus saving power. Base profile is default and ensures balanced performance
* for any workload.
*
* Render-C States:
* ================
*
@ -1171,6 +1176,61 @@ static int pc_action_set_strategy(struct xe_guc_pc *pc, u32 val)
return ret;
}
static const char *power_profile_to_string(struct xe_guc_pc *pc)
{
switch (pc->power_profile) {
case SLPC_POWER_PROFILE_BASE:
return "base";
case SLPC_POWER_PROFILE_POWER_SAVING:
return "power_saving";
default:
return "invalid";
}
}
void xe_guc_pc_get_power_profile(struct xe_guc_pc *pc, char *profile)
{
switch (pc->power_profile) {
case SLPC_POWER_PROFILE_BASE:
sprintf(profile, "[%s] %s\n", "base", "power_saving");
break;
case SLPC_POWER_PROFILE_POWER_SAVING:
sprintf(profile, "%s [%s]\n", "base", "power_saving");
break;
default:
sprintf(profile, "invalid");
}
}
int xe_guc_pc_set_power_profile(struct xe_guc_pc *pc, const char *buf)
{
int ret = 0;
u32 val;
if (strncmp("base", buf, strlen("base")) == 0)
val = SLPC_POWER_PROFILE_BASE;
else if (strncmp("power_saving", buf, strlen("power_saving")) == 0)
val = SLPC_POWER_PROFILE_POWER_SAVING;
else
return -EINVAL;
guard(mutex)(&pc->freq_lock);
xe_pm_runtime_get_noresume(pc_to_xe(pc));
ret = pc_action_set_param(pc,
SLPC_PARAM_POWER_PROFILE,
val);
if (ret)
xe_gt_err_once(pc_to_gt(pc), "Failed to set power profile to %d: %pe\n",
val, ERR_PTR(ret));
else
pc->power_profile = val;
xe_pm_runtime_put(pc_to_xe(pc));
return ret;
}
/**
* xe_guc_pc_start - Start GuC's Power Conservation component
* @pc: Xe_GuC_PC instance
@ -1249,6 +1309,11 @@ int xe_guc_pc_start(struct xe_guc_pc *pc)
/* Enable SLPC Optimized Strategy for compute */
ret = pc_action_set_strategy(pc, SLPC_OPTIMIZED_STRATEGY_COMPUTE);
/* Set cached value of power_profile */
ret = xe_guc_pc_set_power_profile(pc, power_profile_to_string(pc));
if (unlikely(ret))
xe_gt_err(gt, "Failed to set SLPC power profile: %pe\n", ERR_PTR(ret));
out:
xe_force_wake_put(gt_to_fw(gt), fw_ref);
return ret;
@ -1327,6 +1392,8 @@ int xe_guc_pc_init(struct xe_guc_pc *pc)
pc->bo = bo;
pc->power_profile = SLPC_POWER_PROFILE_BASE;
return devm_add_action_or_reset(xe->drm.dev, xe_guc_pc_fini_hw, pc);
}

2
drivers/gpu/drm/xe/xe_guc_pc.h
View File

@ -31,6 +31,8 @@ int xe_guc_pc_get_min_freq(struct xe_guc_pc *pc, u32 *freq);
int xe_guc_pc_set_min_freq(struct xe_guc_pc *pc, u32 freq);
int xe_guc_pc_get_max_freq(struct xe_guc_pc *pc, u32 *freq);
int xe_guc_pc_set_max_freq(struct xe_guc_pc *pc, u32 freq);
int xe_guc_pc_set_power_profile(struct xe_guc_pc *pc, const char *buf);
void xe_guc_pc_get_power_profile(struct xe_guc_pc *pc, char *profile);
enum xe_gt_idle_state xe_guc_pc_c_status(struct xe_guc_pc *pc);
u64 xe_guc_pc_rc6_residency(struct xe_guc_pc *pc);

2
drivers/gpu/drm/xe/xe_guc_pc_types.h
View File

@ -37,6 +37,8 @@ struct xe_guc_pc {
struct mutex freq_lock;
/** @freq_ready: Only handle freq changes, if they are really ready */
bool freq_ready;
/** @power_profile: Base or power_saving profile */
u32 power_profile;
};
#endif /* _XE_GUC_PC_TYPES_H_ */

142
drivers/gpu/drm/xe/xe_guc_submit.c
View File

@ -32,6 +32,7 @@
#include "xe_guc_ct.h"
#include "xe_guc_exec_queue_types.h"
#include "xe_guc_id_mgr.h"
#include "xe_guc_klv_helpers.h"
#include "xe_guc_submit_types.h"
#include "xe_hw_engine.h"
#include "xe_hw_fence.h"
@ -316,6 +317,71 @@ int xe_guc_submit_init(struct xe_guc *guc, unsigned int num_ids)
return drmm_add_action_or_reset(&xe->drm, guc_submit_fini, guc);
}
/*
* Given that we want to guarantee enough RCS throughput to avoid missing
* frames, we set the yield policy to 20% of each 80ms interval.
*/
#define RC_YIELD_DURATION 80 /* in ms */
#define RC_YIELD_RATIO 20 /* in percent */
static u32 *emit_render_compute_yield_klv(u32 *emit)
{
*emit++ = PREP_GUC_KLV_TAG(SCHEDULING_POLICIES_RENDER_COMPUTE_YIELD);
*emit++ = RC_YIELD_DURATION;
*emit++ = RC_YIELD_RATIO;
return emit;
}
#define SCHEDULING_POLICY_MAX_DWORDS 16
static int guc_init_global_schedule_policy(struct xe_guc *guc)
{
u32 data[SCHEDULING_POLICY_MAX_DWORDS];
u32 *emit = data;
u32 count = 0;
int ret;
if (GUC_SUBMIT_VER(guc) < MAKE_GUC_VER(1, 1, 0))
return 0;
*emit++ = XE_GUC_ACTION_UPDATE_SCHEDULING_POLICIES_KLV;
if (CCS_MASK(guc_to_gt(guc)))
emit = emit_render_compute_yield_klv(emit);
count = emit - data;
if (count > 1) {
xe_assert(guc_to_xe(guc), count <= SCHEDULING_POLICY_MAX_DWORDS);
ret = xe_guc_ct_send_block(&guc->ct, data, count);
if (ret < 0) {
xe_gt_err(guc_to_gt(guc),
"failed to enable GuC scheduling policies: %pe\n",
ERR_PTR(ret));
return ret;
}
}
return 0;
}
int xe_guc_submit_enable(struct xe_guc *guc)
{
int ret;
ret = guc_init_global_schedule_policy(guc);
if (ret)
return ret;
guc->submission_state.enabled = true;
return 0;
}
void xe_guc_submit_disable(struct xe_guc *guc)
{
guc->submission_state.enabled = false;
}
static void __release_guc_id(struct xe_guc *guc, struct xe_exec_queue *q, u32 xa_count)
{
int i;
@ -558,10 +624,8 @@ static void register_exec_queue(struct xe_exec_queue *q, int ctx_type)
info.engine_submit_mask = q->logical_mask;
info.hwlrca_lo = lower_32_bits(xe_lrc_descriptor(lrc));
info.hwlrca_hi = upper_32_bits(xe_lrc_descriptor(lrc));
info.flags = CONTEXT_REGISTRATION_FLAG_KMD;
if (ctx_type != GUC_CONTEXT_NORMAL)
info.flags |= BIT(ctx_type);
info.flags = CONTEXT_REGISTRATION_FLAG_KMD |
FIELD_PREP(CONTEXT_REGISTRATION_FLAG_TYPE, ctx_type);
if (xe_exec_queue_is_parallel(q)) {
u64 ggtt_addr = xe_lrc_parallel_ggtt_addr(lrc);
@ -1355,21 +1419,12 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
return DRM_GPU_SCHED_STAT_NO_HANG;
}
static void __guc_exec_queue_fini_async(struct work_struct *w)
static void guc_exec_queue_fini(struct xe_exec_queue *q)
{
struct xe_guc_exec_queue *ge =
container_of(w, struct xe_guc_exec_queue, fini_async);
struct xe_exec_queue *q = ge->q;
struct xe_guc_exec_queue *ge = q->guc;
struct xe_guc *guc = exec_queue_to_guc(q);
xe_pm_runtime_get(guc_to_xe(guc));
trace_xe_exec_queue_destroy(q);
release_guc_id(guc, q);
if (xe_exec_queue_is_lr(q))
cancel_work_sync(&ge->lr_tdr);
/* Confirm no work left behind accessing device structures */
cancel_delayed_work_sync(&ge->sched.base.work_tdr);
xe_sched_entity_fini(&ge->entity);
xe_sched_fini(&ge->sched);
@ -1378,25 +1433,43 @@ static void __guc_exec_queue_fini_async(struct work_struct *w)
* (timeline name).
*/
kfree_rcu(ge, rcu);
}
static void __guc_exec_queue_destroy_async(struct work_struct *w)
{
struct xe_guc_exec_queue *ge =
container_of(w, struct xe_guc_exec_queue, destroy_async);
struct xe_exec_queue *q = ge->q;
struct xe_guc *guc = exec_queue_to_guc(q);
xe_pm_runtime_get(guc_to_xe(guc));
trace_xe_exec_queue_destroy(q);
if (xe_exec_queue_is_lr(q))
cancel_work_sync(&ge->lr_tdr);
/* Confirm no work left behind accessing device structures */
cancel_delayed_work_sync(&ge->sched.base.work_tdr);
xe_exec_queue_fini(q);
xe_pm_runtime_put(guc_to_xe(guc));
}
static void guc_exec_queue_fini_async(struct xe_exec_queue *q)
static void guc_exec_queue_destroy_async(struct xe_exec_queue *q)
{
struct xe_guc *guc = exec_queue_to_guc(q);
struct xe_device *xe = guc_to_xe(guc);
INIT_WORK(&q->guc->fini_async, __guc_exec_queue_fini_async);
INIT_WORK(&q->guc->destroy_async, __guc_exec_queue_destroy_async);
/* We must block on kernel engines so slabs are empty on driver unload */
if (q->flags & EXEC_QUEUE_FLAG_PERMANENT || exec_queue_wedged(q))
__guc_exec_queue_fini_async(&q->guc->fini_async);
__guc_exec_queue_destroy_async(&q->guc->destroy_async);
else
queue_work(xe->destroy_wq, &q->guc->fini_async);
queue_work(xe->destroy_wq, &q->guc->destroy_async);
}
static void __guc_exec_queue_fini(struct xe_guc *guc, struct xe_exec_queue *q)
static void __guc_exec_queue_destroy(struct xe_guc *guc, struct xe_exec_queue *q)
{
/*
* Might be done from within the GPU scheduler, need to do async as we
@ -1405,7 +1478,7 @@ static void __guc_exec_queue_fini(struct xe_guc *guc, struct xe_exec_queue *q)
* this we and don't really care when everything is fini'd, just that it
* is.
*/
guc_exec_queue_fini_async(q);
guc_exec_queue_destroy_async(q);
}
static void __guc_exec_queue_process_msg_cleanup(struct xe_sched_msg *msg)
@ -1419,7 +1492,7 @@ static void __guc_exec_queue_process_msg_cleanup(struct xe_sched_msg *msg)
if (exec_queue_registered(q))
disable_scheduling_deregister(guc, q);
else
__guc_exec_queue_fini(guc, q);
__guc_exec_queue_destroy(guc, q);
}
static bool guc_exec_queue_allowed_to_change_state(struct xe_exec_queue *q)
@ -1652,14 +1725,14 @@ static bool guc_exec_queue_try_add_msg(struct xe_exec_queue *q,
#define STATIC_MSG_CLEANUP 0
#define STATIC_MSG_SUSPEND 1
#define STATIC_MSG_RESUME 2
static void guc_exec_queue_fini(struct xe_exec_queue *q)
static void guc_exec_queue_destroy(struct xe_exec_queue *q)
{
struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_CLEANUP;
if (!(q->flags & EXEC_QUEUE_FLAG_PERMANENT) && !exec_queue_wedged(q))
guc_exec_queue_add_msg(q, msg, CLEANUP);
else
__guc_exec_queue_fini(exec_queue_to_guc(q), q);
__guc_exec_queue_destroy(exec_queue_to_guc(q), q);
}
static int guc_exec_queue_set_priority(struct xe_exec_queue *q,
@ -1789,6 +1862,7 @@ static const struct xe_exec_queue_ops guc_exec_queue_ops = {
.init = guc_exec_queue_init,
.kill = guc_exec_queue_kill,
.fini = guc_exec_queue_fini,
.destroy = guc_exec_queue_destroy,
.set_priority = guc_exec_queue_set_priority,
.set_timeslice = guc_exec_queue_set_timeslice,
.set_preempt_timeout = guc_exec_queue_set_preempt_timeout,
@ -1810,7 +1884,7 @@ static void guc_exec_queue_stop(struct xe_guc *guc, struct xe_exec_queue *q)
if (exec_queue_extra_ref(q) || xe_exec_queue_is_lr(q))
xe_exec_queue_put(q);
else if (exec_queue_destroyed(q))
__guc_exec_queue_fini(guc, q);
__guc_exec_queue_destroy(guc, q);
}
if (q->guc->suspend_pending) {
set_exec_queue_suspended(q);
@ -2029,7 +2103,7 @@ g2h_exec_queue_lookup(struct xe_guc *guc, u32 guc_id)
q = xa_load(&guc->submission_state.exec_queue_lookup, guc_id);
if (unlikely(!q)) {
xe_gt_err(gt, "Not engine present for guc_id %u\n", guc_id);
xe_gt_err(gt, "No exec queue found for guc_id %u\n", guc_id);
return NULL;
}
@ -2139,7 +2213,7 @@ static void handle_deregister_done(struct xe_guc *guc, struct xe_exec_queue *q)
if (exec_queue_extra_ref(q) || xe_exec_queue_is_lr(q))
xe_exec_queue_put(q);
else
__guc_exec_queue_fini(guc, q);
__guc_exec_queue_destroy(guc, q);
}
int xe_guc_deregister_done_handler(struct xe_guc *guc, u32 *msg, u32 len)
@ -2528,7 +2602,7 @@ static void guc_exec_queue_print(struct xe_exec_queue *q, struct drm_printer *p)
}
/**
* xe_guc_register_exec_queue - Register exec queue for a given context type.
* xe_guc_register_vf_exec_queue - Register exec queue for a given context type.
* @q: Execution queue
* @ctx_type: Type of the context
*
@ -2539,15 +2613,17 @@ static void guc_exec_queue_print(struct xe_exec_queue *q, struct drm_printer *p)
*
* Returns - None.
*/
void xe_guc_register_exec_queue(struct xe_exec_queue *q, int ctx_type)
void xe_guc_register_vf_exec_queue(struct xe_exec_queue *q, int ctx_type)
{
struct xe_guc *guc = exec_queue_to_guc(q);
struct xe_device *xe = guc_to_xe(guc);
struct xe_gt *gt = guc_to_gt(guc);
xe_assert(xe, IS_SRIOV_VF(xe));
xe_assert(xe, !IS_DGFX(xe));
xe_assert(xe, (ctx_type > GUC_CONTEXT_NORMAL &&
ctx_type < GUC_CONTEXT_COUNT));
xe_gt_assert(gt, IS_SRIOV_VF(xe));
xe_gt_assert(gt, !IS_DGFX(xe));
xe_gt_assert(gt, ctx_type == GUC_CONTEXT_COMPRESSION_SAVE ||
ctx_type == GUC_CONTEXT_COMPRESSION_RESTORE);
xe_gt_assert(gt, GUC_SUBMIT_VER(guc) >= MAKE_GUC_VER(1, 23, 0));
register_exec_queue(q, ctx_type);
enable_scheduling(q);

4
drivers/gpu/drm/xe/xe_guc_submit.h
View File

@ -13,6 +13,8 @@ struct xe_exec_queue;
struct xe_guc;
int xe_guc_submit_init(struct xe_guc *guc, unsigned int num_ids);
int xe_guc_submit_enable(struct xe_guc *guc);
void xe_guc_submit_disable(struct xe_guc *guc);
int xe_guc_submit_reset_prepare(struct xe_guc *guc);
void xe_guc_submit_reset_wait(struct xe_guc *guc);
@ -46,7 +48,7 @@ xe_guc_exec_queue_snapshot_print(struct xe_guc_submit_exec_queue_snapshot *snaps
void
xe_guc_exec_queue_snapshot_free(struct xe_guc_submit_exec_queue_snapshot *snapshot);
void xe_guc_submit_print(struct xe_guc *guc, struct drm_printer *p);
void xe_guc_register_exec_queue(struct xe_exec_queue *q, int ctx_type);
void xe_guc_register_vf_exec_queue(struct xe_exec_queue *q, int ctx_type);
int xe_guc_contexts_hwsp_rebase(struct xe_guc *guc, void *scratch);

325
drivers/gpu/drm/xe/xe_hmm.c
View File

@ -1,325 +0,0 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2024 Intel Corporation
*/
#include <linux/scatterlist.h>
#include <linux/mmu_notifier.h>
#include <linux/dma-mapping.h>
#include <linux/memremap.h>
#include <linux/swap.h>
#include <linux/hmm.h>
#include <linux/mm.h>
#include "xe_hmm.h"
#include "xe_vm.h"
#include "xe_bo.h"
static u64 xe_npages_in_range(unsigned long start, unsigned long end)
{
return (end - start) >> PAGE_SHIFT;
}
static int xe_alloc_sg(struct xe_device *xe, struct sg_table *st,
struct hmm_range *range, struct rw_semaphore *notifier_sem)
{
unsigned long i, npages, hmm_pfn;
unsigned long num_chunks = 0;
int ret;
/* HMM docs says this is needed. */
ret = down_read_interruptible(notifier_sem);
if (ret)
return ret;
if (mmu_interval_read_retry(range->notifier, range->notifier_seq)) {
up_read(notifier_sem);
return -EAGAIN;
}
npages = xe_npages_in_range(range->start, range->end);
for (i = 0; i < npages;) {
unsigned long len;
hmm_pfn = range->hmm_pfns[i];
xe_assert(xe, hmm_pfn & HMM_PFN_VALID);
len = 1UL << hmm_pfn_to_map_order(hmm_pfn);
/* If order > 0 the page may extend beyond range->start */
len -= (hmm_pfn & ~HMM_PFN_FLAGS) & (len - 1);
i += len;
num_chunks++;
}
up_read(notifier_sem);
return sg_alloc_table(st, num_chunks, GFP_KERNEL);
}
/**
* xe_build_sg() - build a scatter gather table for all the physical pages/pfn
* in a hmm_range. dma-map pages if necessary. dma-address is save in sg table
* and will be used to program GPU page table later.
* @xe: the xe device who will access the dma-address in sg table
* @range: the hmm range that we build the sg table from. range->hmm_pfns[]
* has the pfn numbers of pages that back up this hmm address range.
* @st: pointer to the sg table.
* @notifier_sem: The xe notifier lock.
* @write: whether we write to this range. This decides dma map direction
* for system pages. If write we map it bi-diretional; otherwise
* DMA_TO_DEVICE
*
* All the contiguous pfns will be collapsed into one entry in
* the scatter gather table. This is for the purpose of efficiently
* programming GPU page table.
*
* The dma_address in the sg table will later be used by GPU to
* access memory. So if the memory is system memory, we need to
* do a dma-mapping so it can be accessed by GPU/DMA.
*
* FIXME: This function currently only support pages in system
* memory. If the memory is GPU local memory (of the GPU who
* is going to access memory), we need gpu dpa (device physical
* address), and there is no need of dma-mapping. This is TBD.
*
* FIXME: dma-mapping for peer gpu device to access remote gpu's
* memory. Add this when you support p2p
*
* This function allocates the storage of the sg table. It is
* caller's responsibility to free it calling sg_free_table.
*
* Returns 0 if successful; -ENOMEM if fails to allocate memory
*/
static int xe_build_sg(struct xe_device *xe, struct hmm_range *range,
struct sg_table *st,
struct rw_semaphore *notifier_sem,
bool write)
{
unsigned long npages = xe_npages_in_range(range->start, range->end);
struct device *dev = xe->drm.dev;
struct scatterlist *sgl;
struct page *page;
unsigned long i, j;
lockdep_assert_held(notifier_sem);
i = 0;
for_each_sg(st->sgl, sgl, st->nents, j) {
unsigned long hmm_pfn, size;
hmm_pfn = range->hmm_pfns[i];
page = hmm_pfn_to_page(hmm_pfn);
xe_assert(xe, !is_device_private_page(page));
size = 1UL << hmm_pfn_to_map_order(hmm_pfn);
size -= page_to_pfn(page) & (size - 1);
i += size;
if (unlikely(j == st->nents - 1)) {
xe_assert(xe, i >= npages);
if (i > npages)
size -= (i - npages);
sg_mark_end(sgl);
} else {
xe_assert(xe, i < npages);
}
sg_set_page(sgl, page, size << PAGE_SHIFT, 0);
}
return dma_map_sgtable(dev, st, write ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE,
DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
}
static void xe_hmm_userptr_set_mapped(struct xe_userptr_vma *uvma)
{
struct xe_userptr *userptr = &uvma->userptr;
struct xe_vm *vm = xe_vma_vm(&uvma->vma);
lockdep_assert_held_write(&vm->lock);
lockdep_assert_held(&vm->userptr.notifier_lock);
mutex_lock(&userptr->unmap_mutex);
xe_assert(vm->xe, !userptr->mapped);
userptr->mapped = true;
mutex_unlock(&userptr->unmap_mutex);
}
void xe_hmm_userptr_unmap(struct xe_userptr_vma *uvma)
{
struct xe_userptr *userptr = &uvma->userptr;
struct xe_vma *vma = &uvma->vma;
bool write = !xe_vma_read_only(vma);
struct xe_vm *vm = xe_vma_vm(vma);
struct xe_device *xe = vm->xe;
if (!lockdep_is_held_type(&vm->userptr.notifier_lock, 0) &&
!lockdep_is_held_type(&vm->lock, 0) &&
!(vma->gpuva.flags & XE_VMA_DESTROYED)) {
/* Don't unmap in exec critical section. */
xe_vm_assert_held(vm);
/* Don't unmap while mapping the sg. */
lockdep_assert_held(&vm->lock);
}
mutex_lock(&userptr->unmap_mutex);
if (userptr->sg && userptr->mapped)
dma_unmap_sgtable(xe->drm.dev, userptr->sg,
write ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE, 0);
userptr->mapped = false;
mutex_unlock(&userptr->unmap_mutex);
}
/**
* xe_hmm_userptr_free_sg() - Free the scatter gather table of userptr
* @uvma: the userptr vma which hold the scatter gather table
*
* With function xe_userptr_populate_range, we allocate storage of
* the userptr sg table. This is a helper function to free this
* sg table, and dma unmap the address in the table.
*/
void xe_hmm_userptr_free_sg(struct xe_userptr_vma *uvma)
{
struct xe_userptr *userptr = &uvma->userptr;
xe_assert(xe_vma_vm(&uvma->vma)->xe, userptr->sg);
xe_hmm_userptr_unmap(uvma);
sg_free_table(userptr->sg);
userptr->sg = NULL;
}
/**
* xe_hmm_userptr_populate_range() - Populate physical pages of a virtual
* address range
*
* @uvma: userptr vma which has information of the range to populate.
* @is_mm_mmap_locked: True if mmap_read_lock is already acquired by caller.
*
* This function populate the physical pages of a virtual
* address range. The populated physical pages is saved in
* userptr's sg table. It is similar to get_user_pages but call
* hmm_range_fault.
*
* This function also read mmu notifier sequence # (
* mmu_interval_read_begin), for the purpose of later
* comparison (through mmu_interval_read_retry).
*
* This must be called with mmap read or write lock held.
*
* This function allocates the storage of the userptr sg table.
* It is caller's responsibility to free it calling sg_free_table.
*
* returns: 0 for success; negative error no on failure
*/
int xe_hmm_userptr_populate_range(struct xe_userptr_vma *uvma,
bool is_mm_mmap_locked)
{
unsigned long timeout =
jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
unsigned long *pfns;
struct xe_userptr *userptr;
struct xe_vma *vma = &uvma->vma;
u64 userptr_start = xe_vma_userptr(vma);
u64 userptr_end = userptr_start + xe_vma_size(vma);
struct xe_vm *vm = xe_vma_vm(vma);
struct hmm_range hmm_range = {
.pfn_flags_mask = 0, /* ignore pfns */
.default_flags = HMM_PFN_REQ_FAULT,
.start = userptr_start,
.end = userptr_end,
.notifier = &uvma->userptr.notifier,
.dev_private_owner = vm->xe,
};
bool write = !xe_vma_read_only(vma);
unsigned long notifier_seq;
u64 npages;
int ret;
userptr = &uvma->userptr;
if (is_mm_mmap_locked)
mmap_assert_locked(userptr->notifier.mm);
if (vma->gpuva.flags & XE_VMA_DESTROYED)
return 0;
notifier_seq = mmu_interval_read_begin(&userptr->notifier);
if (notifier_seq == userptr->notifier_seq)
return 0;
if (userptr->sg)
xe_hmm_userptr_free_sg(uvma);
npages = xe_npages_in_range(userptr_start, userptr_end);
pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
if (unlikely(!pfns))
return -ENOMEM;
if (write)
hmm_range.default_flags |= HMM_PFN_REQ_WRITE;
if (!mmget_not_zero(userptr->notifier.mm)) {
ret = -EFAULT;
goto free_pfns;
}
hmm_range.hmm_pfns = pfns;
while (true) {
hmm_range.notifier_seq = mmu_interval_read_begin(&userptr->notifier);
if (!is_mm_mmap_locked)
mmap_read_lock(userptr->notifier.mm);
ret = hmm_range_fault(&hmm_range);
if (!is_mm_mmap_locked)
mmap_read_unlock(userptr->notifier.mm);
if (ret == -EBUSY) {
if (time_after(jiffies, timeout))
break;
continue;
}
break;
}
mmput(userptr->notifier.mm);
if (ret)
goto free_pfns;
ret = xe_alloc_sg(vm->xe, &userptr->sgt, &hmm_range, &vm->userptr.notifier_lock);
if (ret)
goto free_pfns;
ret = down_read_interruptible(&vm->userptr.notifier_lock);
if (ret)
goto free_st;
if (mmu_interval_read_retry(hmm_range.notifier, hmm_range.notifier_seq)) {
ret = -EAGAIN;
goto out_unlock;
}
ret = xe_build_sg(vm->xe, &hmm_range, &userptr->sgt,
&vm->userptr.notifier_lock, write);
if (ret)
goto out_unlock;
userptr->sg = &userptr->sgt;
xe_hmm_userptr_set_mapped(uvma);
userptr->notifier_seq = hmm_range.notifier_seq;
up_read(&vm->userptr.notifier_lock);
kvfree(pfns);
return 0;
out_unlock:
up_read(&vm->userptr.notifier_lock);
free_st:
sg_free_table(&userptr->sgt);
free_pfns:
kvfree(pfns);
return ret;
}

18
drivers/gpu/drm/xe/xe_hmm.h
View File

@ -1,18 +0,0 @@
/* SPDX-License-Identifier: MIT
*
* Copyright © 2024 Intel Corporation
*/
#ifndef _XE_HMM_H_
#define _XE_HMM_H_
#include <linux/types.h>
struct xe_userptr_vma;
int xe_hmm_userptr_populate_range(struct xe_userptr_vma *uvma, bool is_mm_mmap_locked);
void xe_hmm_userptr_free_sg(struct xe_userptr_vma *uvma);
void xe_hmm_userptr_unmap(struct xe_userptr_vma *uvma);
#endif

45
drivers/gpu/drm/xe/xe_hwmon.c
View File

@ -286,7 +286,7 @@ static struct xe_reg xe_hwmon_get_reg(struct xe_hwmon *hwmon, enum xe_hwmon_reg
*/
static void xe_hwmon_power_max_read(struct xe_hwmon *hwmon, u32 attr, int channel, long *value)
{
u64 reg_val = 0, min, max;
u32 reg_val = 0;
struct xe_device *xe = hwmon->xe;
struct xe_reg rapl_limit, pkg_power_sku;
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
@ -294,7 +294,7 @@ static void xe_hwmon_power_max_read(struct xe_hwmon *hwmon, u32 attr, int channe
mutex_lock(&hwmon->hwmon_lock);
if (hwmon->xe->info.has_mbx_power_limits) {
xe_hwmon_pcode_read_power_limit(hwmon, attr, channel, (u32 *)&reg_val);
xe_hwmon_pcode_read_power_limit(hwmon, attr, channel, &reg_val);
} else {
rapl_limit = xe_hwmon_get_reg(hwmon, REG_PKG_RAPL_LIMIT, channel);
pkg_power_sku = xe_hwmon_get_reg(hwmon, REG_PKG_POWER_SKU, channel);
@ -304,19 +304,21 @@ static void xe_hwmon_power_max_read(struct xe_hwmon *hwmon, u32 attr, int channe
/* Check if PL limits are disabled. */
if (!(reg_val & PWR_LIM_EN)) {
*value = PL_DISABLE;
drm_info(&hwmon->xe->drm, "%s disabled for channel %d, val 0x%016llx\n",
drm_info(&hwmon->xe->drm, "%s disabled for channel %d, val 0x%08x\n",
PWR_ATTR_TO_STR(attr), channel, reg_val);
goto unlock;
}
reg_val = REG_FIELD_GET(PWR_LIM_VAL, reg_val);
*value = mul_u64_u32_shr(reg_val, SF_POWER, hwmon->scl_shift_power);
*value = mul_u32_u32(reg_val, SF_POWER) >> hwmon->scl_shift_power;
/* For platforms with mailbox power limit support clamping would be done by pcode. */
if (!hwmon->xe->info.has_mbx_power_limits) {
reg_val = xe_mmio_read64_2x32(mmio, pkg_power_sku);
min = REG_FIELD_GET(PKG_MIN_PWR, reg_val);
max = REG_FIELD_GET(PKG_MAX_PWR, reg_val);
u64 pkg_pwr, min, max;
pkg_pwr = xe_mmio_read64_2x32(mmio, pkg_power_sku);
min = REG_FIELD_GET(PKG_MIN_PWR, pkg_pwr);
max = REG_FIELD_GET(PKG_MAX_PWR, pkg_pwr);
min = mul_u64_u32_shr(min, SF_POWER, hwmon->scl_shift_power);
max = mul_u64_u32_shr(max, SF_POWER, hwmon->scl_shift_power);
if (min && max)
@ -493,8 +495,8 @@ xe_hwmon_power_max_interval_show(struct device *dev, struct device_attribute *at
{
struct xe_hwmon *hwmon = dev_get_drvdata(dev);
struct xe_mmio *mmio = xe_root_tile_mmio(hwmon->xe);
u32 x, y, x_w = 2; /* 2 bits */
u64 r, tau4, out;
u32 reg_val, x, y, x_w = 2; /* 2 bits */
u64 tau4, out;
int channel = (to_sensor_dev_attr(attr)->index % 2) ? CHANNEL_PKG : CHANNEL_CARD;
u32 power_attr = (to_sensor_dev_attr(attr)->index > 1) ? PL2_HWMON_ATTR : PL1_HWMON_ATTR;
@ -505,23 +507,24 @@ xe_hwmon_power_max_interval_show(struct device *dev, struct device_attribute *at
mutex_lock(&hwmon->hwmon_lock);
if (hwmon->xe->info.has_mbx_power_limits) {
ret = xe_hwmon_pcode_read_power_limit(hwmon, power_attr, channel, (u32 *)&r);
ret = xe_hwmon_pcode_read_power_limit(hwmon, power_attr, channel, &reg_val);
if (ret) {
drm_err(&hwmon->xe->drm,
"power interval read fail, ch %d, attr %d, r 0%llx, ret %d\n",
channel, power_attr, r, ret);
r = 0;
"power interval read fail, ch %d, attr %d, val 0x%08x, ret %d\n",
channel, power_attr, reg_val, ret);
reg_val = 0;
}
} else {
r = xe_mmio_read32(mmio, xe_hwmon_get_reg(hwmon, REG_PKG_RAPL_LIMIT, channel));
reg_val = xe_mmio_read32(mmio, xe_hwmon_get_reg(hwmon, REG_PKG_RAPL_LIMIT,
channel));
}
mutex_unlock(&hwmon->hwmon_lock);
xe_pm_runtime_put(hwmon->xe);
x = REG_FIELD_GET(PWR_LIM_TIME_X, r);
y = REG_FIELD_GET(PWR_LIM_TIME_Y, r);
x = REG_FIELD_GET(PWR_LIM_TIME_X, reg_val);
y = REG_FIELD_GET(PWR_LIM_TIME_Y, reg_val);
/*
* tau = (1 + (x / 4)) * power(2,y), x = bits(23:22), y = bits(21:17)
@ -1294,13 +1297,6 @@ xe_hwmon_get_preregistration_info(struct xe_hwmon *hwmon)
xe_hwmon_fan_input_read(hwmon, channel, &fan_speed);
}
static void xe_hwmon_mutex_destroy(void *arg)
{
struct xe_hwmon *hwmon = arg;
mutex_destroy(&hwmon->hwmon_lock);
}
int xe_hwmon_register(struct xe_device *xe)
{
struct device *dev = xe->drm.dev;
@ -1319,8 +1315,7 @@ int xe_hwmon_register(struct xe_device *xe)
if (!hwmon)
return -ENOMEM;
mutex_init(&hwmon->hwmon_lock);
ret = devm_add_action_or_reset(dev, xe_hwmon_mutex_destroy, hwmon);
ret = devm_mutex_init(dev, &hwmon->hwmon_lock);
if (ret)
return ret;

2
drivers/gpu/drm/xe/xe_i2c.c
View File

@ -259,7 +259,7 @@ void xe_i2c_pm_resume(struct xe_device *xe, bool d3cold)
return;
if (d3cold)
xe_mmio_rmw32(mmio, I2C_CONFIG_CMD, 0, PCI_COMMAND_MEMORY);
xe_mmio_rmw32(mmio, I2C_CONFIG_CMD, 0, PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER);
xe_mmio_rmw32(mmio, I2C_CONFIG_PMCSR, PCI_PM_CTRL_STATE_MASK, (__force u32)PCI_D0);
drm_dbg(&xe->drm, "pmcsr: 0x%08x\n", xe_mmio_read32(mmio, I2C_CONFIG_PMCSR));

464
drivers/gpu/drm/xe/xe_late_bind_fw.c Normal file
View File

@ -0,0 +1,464 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2025 Intel Corporation
*/
#include <linux/component.h>
#include <linux/delay.h>
#include <linux/firmware.h>
#include <drm/drm_managed.h>
#include <drm/intel/i915_component.h>
#include <drm/intel/intel_lb_mei_interface.h>
#include <drm/drm_print.h>
#include "xe_device.h"
#include "xe_late_bind_fw.h"
#include "xe_pcode.h"
#include "xe_pcode_api.h"
#include "xe_pm.h"
/*
* The component should load quite quickly in most cases, but it could take
* a bit. Using a very big timeout just to cover the worst case scenario
*/
#define LB_INIT_TIMEOUT_MS 20000
/*
* Retry interval set to 6 seconds, in steps of 200 ms, to allow time for
* other OS components to release the MEI CL handle
*/
#define LB_FW_LOAD_RETRY_MAXCOUNT 30
#define LB_FW_LOAD_RETRY_PAUSE_MS 200
static const u32 fw_id_to_type[] = {
[XE_LB_FW_FAN_CONTROL] = INTEL_LB_TYPE_FAN_CONTROL,
};
static const char * const fw_id_to_name[] = {
[XE_LB_FW_FAN_CONTROL] = "fan_control",
};
static struct xe_device *
late_bind_to_xe(struct xe_late_bind *late_bind)
{
return container_of(late_bind, struct xe_device, late_bind);
}
static struct xe_device *
late_bind_fw_to_xe(struct xe_late_bind_fw *lb_fw)
{
return container_of(lb_fw, struct xe_device, late_bind.late_bind_fw[lb_fw->id]);
}
/* Refer to the "Late Bind based Firmware Layout" documentation entry for details */
static int parse_cpd_header(struct xe_late_bind_fw *lb_fw,
const void *data, size_t size, const char *manifest_entry)
{
struct xe_device *xe = late_bind_fw_to_xe(lb_fw);
const struct gsc_cpd_header_v2 *header = data;
const struct gsc_manifest_header *manifest;
const struct gsc_cpd_entry *entry;
size_t min_size = sizeof(*header);
u32 offset;
int i;
/* manifest_entry is mandatory */
xe_assert(xe, manifest_entry);
if (size < min_size || header->header_marker != GSC_CPD_HEADER_MARKER)
return -ENOENT;
if (header->header_length < sizeof(struct gsc_cpd_header_v2)) {
drm_err(&xe->drm, "%s late binding fw: Invalid CPD header length %u!\n",
fw_id_to_name[lb_fw->id], header->header_length);
return -EINVAL;
}
min_size = header->header_length + sizeof(struct gsc_cpd_entry) * header->num_of_entries;
if (size < min_size) {
drm_err(&xe->drm, "%s late binding fw: too small! %zu < %zu\n",
fw_id_to_name[lb_fw->id], size, min_size);
return -ENODATA;
}
/* Look for the manifest first */
entry = (void *)header + header->header_length;
for (i = 0; i < header->num_of_entries; i++, entry++)
if (strcmp(entry->name, manifest_entry) == 0)
offset = entry->offset & GSC_CPD_ENTRY_OFFSET_MASK;
if (!offset) {
drm_err(&xe->drm, "%s late binding fw: Failed to find manifest_entry\n",
fw_id_to_name[lb_fw->id]);
return -ENODATA;
}
min_size = offset + sizeof(struct gsc_manifest_header);
if (size < min_size) {
drm_err(&xe->drm, "%s late binding fw: too small! %zu < %zu\n",
fw_id_to_name[lb_fw->id], size, min_size);
return -ENODATA;
}
manifest = data + offset;
lb_fw->version = manifest->fw_version;
return 0;
}
/* Refer to the "Late Bind based Firmware Layout" documentation entry for details */
static int parse_lb_layout(struct xe_late_bind_fw *lb_fw,
const void *data, size_t size, const char *fpt_entry)
{
struct xe_device *xe = late_bind_fw_to_xe(lb_fw);
const struct csc_fpt_header *header = data;
const struct csc_fpt_entry *entry;
size_t min_size = sizeof(*header);
u32 offset;
int i;
/* fpt_entry is mandatory */
xe_assert(xe, fpt_entry);
if (size < min_size || header->header_marker != CSC_FPT_HEADER_MARKER)
return -ENOENT;
if (header->header_length < sizeof(struct csc_fpt_header)) {
drm_err(&xe->drm, "%s late binding fw: Invalid FPT header length %u!\n",
fw_id_to_name[lb_fw->id], header->header_length);
return -EINVAL;
}
min_size = header->header_length + sizeof(struct csc_fpt_entry) * header->num_of_entries;
if (size < min_size) {
drm_err(&xe->drm, "%s late binding fw: too small! %zu < %zu\n",
fw_id_to_name[lb_fw->id], size, min_size);
return -ENODATA;
}
/* Look for the cpd header first */
entry = (void *)header + header->header_length;
for (i = 0; i < header->num_of_entries; i++, entry++)
if (strcmp(entry->name, fpt_entry) == 0)
offset = entry->offset;
if (!offset) {
drm_err(&xe->drm, "%s late binding fw: Failed to find fpt_entry\n",
fw_id_to_name[lb_fw->id]);
return -ENODATA;
}
min_size = offset + sizeof(struct gsc_cpd_header_v2);
if (size < min_size) {
drm_err(&xe->drm, "%s late binding fw: too small! %zu < %zu\n",
fw_id_to_name[lb_fw->id], size, min_size);
return -ENODATA;
}
return parse_cpd_header(lb_fw, data + offset, size - offset, "LTES.man");
}
static const char *xe_late_bind_parse_status(uint32_t status)
{
switch (status) {
case INTEL_LB_STATUS_SUCCESS:
return "success";
case INTEL_LB_STATUS_4ID_MISMATCH:
return "4Id Mismatch";
case INTEL_LB_STATUS_ARB_FAILURE:
return "ARB Failure";
case INTEL_LB_STATUS_GENERAL_ERROR:
return "General Error";
case INTEL_LB_STATUS_INVALID_PARAMS:
return "Invalid Params";
case INTEL_LB_STATUS_INVALID_SIGNATURE:
return "Invalid Signature";
case INTEL_LB_STATUS_INVALID_PAYLOAD:
return "Invalid Payload";
case INTEL_LB_STATUS_TIMEOUT:
return "Timeout";
default:
return "Unknown error";
}
}
static int xe_late_bind_fw_num_fans(struct xe_late_bind *late_bind)
{
struct xe_device *xe = late_bind_to_xe(late_bind);
struct xe_tile *root_tile = xe_device_get_root_tile(xe);
u32 uval;
if (!xe_pcode_read(root_tile,
PCODE_MBOX(FAN_SPEED_CONTROL, FSC_READ_NUM_FANS, 0), &uval, NULL))
return uval;
else
return 0;
}
void xe_late_bind_wait_for_worker_completion(struct xe_late_bind *late_bind)
{
struct xe_device *xe = late_bind_to_xe(late_bind);
struct xe_late_bind_fw *lbfw;
int fw_id;
for (fw_id = 0; fw_id < XE_LB_FW_MAX_ID; fw_id++) {
lbfw = &late_bind->late_bind_fw[fw_id];
if (lbfw->payload && late_bind->wq) {
drm_dbg(&xe->drm, "Flush work: load %s firmware\n",
fw_id_to_name[lbfw->id]);
flush_work(&lbfw->work);
}
}
}
static void xe_late_bind_work(struct work_struct *work)
{
struct xe_late_bind_fw *lbfw = container_of(work, struct xe_late_bind_fw, work);
struct xe_late_bind *late_bind = container_of(lbfw, struct xe_late_bind,
late_bind_fw[lbfw->id]);
struct xe_device *xe = late_bind_to_xe(late_bind);
int retry = LB_FW_LOAD_RETRY_MAXCOUNT;
int ret;
int slept;
xe_device_assert_mem_access(xe);
/* we can queue this before the component is bound */
for (slept = 0; slept < LB_INIT_TIMEOUT_MS; slept += 100) {
if (late_bind->component.ops)
break;
msleep(100);
}
if (!late_bind->component.ops) {
drm_err(&xe->drm, "Late bind component not bound\n");
/* Do not re-attempt fw load */
drmm_kfree(&xe->drm, (void *)lbfw->payload);
lbfw->payload = NULL;
goto out;
}
drm_dbg(&xe->drm, "Load %s firmware\n", fw_id_to_name[lbfw->id]);
do {
ret = late_bind->component.ops->push_payload(late_bind->component.mei_dev,
lbfw->type,
lbfw->flags,
lbfw->payload,
lbfw->payload_size);
if (!ret)
break;
msleep(LB_FW_LOAD_RETRY_PAUSE_MS);
} while (--retry && ret == -EBUSY);
if (!ret) {
drm_dbg(&xe->drm, "Load %s firmware successful\n",
fw_id_to_name[lbfw->id]);
goto out;
}
if (ret > 0)
drm_err(&xe->drm, "Load %s firmware failed with err %d, %s\n",
fw_id_to_name[lbfw->id], ret, xe_late_bind_parse_status(ret));
else
drm_err(&xe->drm, "Load %s firmware failed with err %d",
fw_id_to_name[lbfw->id], ret);
/* Do not re-attempt fw load */
drmm_kfree(&xe->drm, (void *)lbfw->payload);
lbfw->payload = NULL;
out:
xe_pm_runtime_put(xe);
}
int xe_late_bind_fw_load(struct xe_late_bind *late_bind)
{
struct xe_device *xe = late_bind_to_xe(late_bind);
struct xe_late_bind_fw *lbfw;
int fw_id;
if (!late_bind->component_added)
return -ENODEV;
if (late_bind->disable)
return 0;
for (fw_id = 0; fw_id < XE_LB_FW_MAX_ID; fw_id++) {
lbfw = &late_bind->late_bind_fw[fw_id];
if (lbfw->payload) {
xe_pm_runtime_get_noresume(xe);
queue_work(late_bind->wq, &lbfw->work);
}
}
return 0;
}
static int __xe_late_bind_fw_init(struct xe_late_bind *late_bind, u32 fw_id)
{
struct xe_device *xe = late_bind_to_xe(late_bind);
struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
struct xe_late_bind_fw *lb_fw;
const struct firmware *fw;
u32 num_fans;
int ret;
if (fw_id >= XE_LB_FW_MAX_ID)
return -EINVAL;
lb_fw = &late_bind->late_bind_fw[fw_id];
lb_fw->id = fw_id;
lb_fw->type = fw_id_to_type[lb_fw->id];
lb_fw->flags &= ~INTEL_LB_FLAG_IS_PERSISTENT;
if (lb_fw->type == INTEL_LB_TYPE_FAN_CONTROL) {
num_fans = xe_late_bind_fw_num_fans(late_bind);
drm_dbg(&xe->drm, "Number of Fans: %d\n", num_fans);
if (!num_fans)
return 0;
}
snprintf(lb_fw->blob_path, sizeof(lb_fw->blob_path), "xe/%s_8086_%04x_%04x_%04x.bin",
fw_id_to_name[lb_fw->id], pdev->device,
pdev->subsystem_vendor, pdev->subsystem_device);
drm_dbg(&xe->drm, "Request late binding firmware %s\n", lb_fw->blob_path);
ret = firmware_request_nowarn(&fw, lb_fw->blob_path, xe->drm.dev);
if (ret) {
drm_dbg(&xe->drm, "%s late binding fw not available for current device",
fw_id_to_name[lb_fw->id]);
return 0;
}
if (fw->size > XE_LB_MAX_PAYLOAD_SIZE) {
drm_err(&xe->drm, "Firmware %s size %zu is larger than max pay load size %u\n",
lb_fw->blob_path, fw->size, XE_LB_MAX_PAYLOAD_SIZE);
release_firmware(fw);
return -ENODATA;
}
ret = parse_lb_layout(lb_fw, fw->data, fw->size, "LTES");
if (ret)
return ret;
lb_fw->payload_size = fw->size;
lb_fw->payload = drmm_kzalloc(&xe->drm, lb_fw->payload_size, GFP_KERNEL);
if (!lb_fw->payload) {
release_firmware(fw);
return -ENOMEM;
}
drm_info(&xe->drm, "Using %s firmware from %s version %u.%u.%u.%u\n",
fw_id_to_name[lb_fw->id], lb_fw->blob_path,
lb_fw->version.major, lb_fw->version.minor,
lb_fw->version.hotfix, lb_fw->version.build);
memcpy((void *)lb_fw->payload, fw->data, lb_fw->payload_size);
release_firmware(fw);
INIT_WORK(&lb_fw->work, xe_late_bind_work);
return 0;
}
static int xe_late_bind_fw_init(struct xe_late_bind *late_bind)
{
int ret;
int fw_id;
late_bind->wq = alloc_ordered_workqueue("late-bind-ordered-wq", 0);
if (!late_bind->wq)
return -ENOMEM;
for (fw_id = 0; fw_id < XE_LB_FW_MAX_ID; fw_id++) {
ret = __xe_late_bind_fw_init(late_bind, fw_id);
if (ret)
return ret;
}
return 0;
}
static int xe_late_bind_component_bind(struct device *xe_kdev,
struct device *mei_kdev, void *data)
{
struct xe_device *xe = kdev_to_xe_device(xe_kdev);
struct xe_late_bind *late_bind = &xe->late_bind;
late_bind->component.ops = data;
late_bind->component.mei_dev = mei_kdev;
return 0;
}
static void xe_late_bind_component_unbind(struct device *xe_kdev,
struct device *mei_kdev, void *data)
{
struct xe_device *xe = kdev_to_xe_device(xe_kdev);
struct xe_late_bind *late_bind = &xe->late_bind;
xe_late_bind_wait_for_worker_completion(late_bind);
late_bind->component.ops = NULL;
}
static const struct component_ops xe_late_bind_component_ops = {
.bind = xe_late_bind_component_bind,
.unbind = xe_late_bind_component_unbind,
};
static void xe_late_bind_remove(void *arg)
{
struct xe_late_bind *late_bind = arg;
struct xe_device *xe = late_bind_to_xe(late_bind);
xe_late_bind_wait_for_worker_completion(late_bind);
late_bind->component_added = false;
component_del(xe->drm.dev, &xe_late_bind_component_ops);
if (late_bind->wq) {
destroy_workqueue(late_bind->wq);
late_bind->wq = NULL;
}
}
/**
* xe_late_bind_init() - add xe mei late binding component
* @late_bind: pointer to late bind structure.
*
* Return: 0 if the initialization was successful, a negative errno otherwise.
*/
int xe_late_bind_init(struct xe_late_bind *late_bind)
{
struct xe_device *xe = late_bind_to_xe(late_bind);
int err;
if (!xe->info.has_late_bind)
return 0;
if (!IS_ENABLED(CONFIG_INTEL_MEI_LB) || !IS_ENABLED(CONFIG_INTEL_MEI_GSC)) {
drm_info(&xe->drm, "Can't init xe mei late bind missing mei component\n");
return 0;
}
err = component_add_typed(xe->drm.dev, &xe_late_bind_component_ops,
INTEL_COMPONENT_LB);
if (err < 0) {
drm_err(&xe->drm, "Failed to add mei late bind component (%pe)\n", ERR_PTR(err));
return err;
}
late_bind->component_added = true;
err = devm_add_action_or_reset(xe->drm.dev, xe_late_bind_remove, late_bind);
if (err)
return err;
err = xe_late_bind_fw_init(late_bind);
if (err)
return err;
return xe_late_bind_fw_load(late_bind);
}

17
drivers/gpu/drm/xe/xe_late_bind_fw.h Normal file
View File

@ -0,0 +1,17 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef _XE_LATE_BIND_FW_H_
#define _XE_LATE_BIND_FW_H_
#include <linux/types.h>
struct xe_late_bind;
int xe_late_bind_init(struct xe_late_bind *late_bind);
int xe_late_bind_fw_load(struct xe_late_bind *late_bind);
void xe_late_bind_wait_for_worker_completion(struct xe_late_bind *late_bind);
#endif

75
drivers/gpu/drm/xe/xe_late_bind_fw_types.h Normal file
View File

@ -0,0 +1,75 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef _XE_LATE_BIND_TYPES_H_
#define _XE_LATE_BIND_TYPES_H_
#include <linux/iosys-map.h>
#include <linux/mutex.h>
#include <linux/types.h>
#include <linux/workqueue.h>
#include "xe_uc_fw_abi.h"
#define XE_LB_MAX_PAYLOAD_SIZE SZ_4K
/**
* xe_late_bind_fw_id - enum to determine late binding fw index
*/
enum xe_late_bind_fw_id {
XE_LB_FW_FAN_CONTROL = 0,
XE_LB_FW_MAX_ID
};
/**
* struct xe_late_bind_fw
*/
struct xe_late_bind_fw {
/** @id: firmware index */
u32 id;
/** @blob_path: firmware binary path */
char blob_path[PATH_MAX];
/** @type: firmware type */
u32 type;
/** @flags: firmware flags */
u32 flags;
/** @payload: to store the late binding blob */
const u8 *payload;
/** @payload_size: late binding blob payload_size */
size_t payload_size;
/** @work: worker to upload latebind blob */
struct work_struct work;
/** @version: late binding blob manifest version */
struct gsc_version version;
};
/**
* struct xe_late_bind_component - Late Binding services component
* @mei_dev: device that provide Late Binding service.
* @ops: Ops implemented by Late Binding driver, used by Xe driver.
*
* Communication between Xe and MEI drivers for Late Binding services
*/
struct xe_late_bind_component {
struct device *mei_dev;
const struct intel_lb_component_ops *ops;
};
/**
* struct xe_late_bind
*/
struct xe_late_bind {
/** @component: struct for communication with mei component */
struct xe_late_bind_component component;
/** @late_bind_fw: late binding firmware array */
struct xe_late_bind_fw late_bind_fw[XE_LB_FW_MAX_ID];
/** @wq: workqueue to submit request to download late bind blob */
struct workqueue_struct *wq;
/** @component_added: whether the component has been added */
bool component_added;
/** @disable: to block late binding reload during pm resume flow*/
bool disable;
};
#endif

12
drivers/gpu/drm/xe/xe_lmtt.c
View File

@ -67,12 +67,12 @@ static struct xe_lmtt_pt *lmtt_pt_alloc(struct xe_lmtt *lmtt, unsigned int level
goto out;
}
bo = xe_bo_create_pin_map(lmtt_to_xe(lmtt), lmtt_to_tile(lmtt), NULL,
PAGE_ALIGN(lmtt->ops->lmtt_pte_size(level) *
lmtt->ops->lmtt_pte_num(level)),
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(lmtt_to_tile(lmtt)) |
XE_BO_FLAG_NEEDS_64K);
bo = xe_bo_create_pin_map_novm(lmtt_to_xe(lmtt), lmtt_to_tile(lmtt),
PAGE_ALIGN(lmtt->ops->lmtt_pte_size(level) *
lmtt->ops->lmtt_pte_num(level)),
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(lmtt_to_tile(lmtt)) |
XE_BO_FLAG_NEEDS_64K, false);
if (IS_ERR(bo)) {
err = PTR_ERR(bo);
goto out_free_pt;

90
drivers/gpu/drm/xe/xe_lrc.c
View File

@ -8,6 +8,7 @@
#include <generated/xe_wa_oob.h>
#include <linux/ascii85.h>
#include <linux/panic.h>
#include "instructions/xe_mi_commands.h"
#include "instructions/xe_gfxpipe_commands.h"
@ -16,6 +17,7 @@
#include "regs/xe_lrc_layout.h"
#include "xe_bb.h"
#include "xe_bo.h"
#include "xe_configfs.h"
#include "xe_device.h"
#include "xe_drm_client.h"
#include "xe_exec_queue_types.h"
@ -75,11 +77,17 @@ lrc_to_xe(struct xe_lrc *lrc)
static bool
gt_engine_needs_indirect_ctx(struct xe_gt *gt, enum xe_engine_class class)
{
struct xe_device *xe = gt_to_xe(gt);
if (XE_GT_WA(gt, 16010904313) &&
(class == XE_ENGINE_CLASS_RENDER ||
class == XE_ENGINE_CLASS_COMPUTE))
return true;
if (xe_configfs_get_ctx_restore_mid_bb(to_pci_dev(xe->drm.dev),
class, NULL))
return true;
return false;
}
@ -1102,6 +1110,64 @@ static ssize_t setup_timestamp_wa(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
return cmd - batch;
}
static ssize_t setup_configfs_post_ctx_restore_bb(struct xe_lrc *lrc,
struct xe_hw_engine *hwe,
u32 *batch, size_t max_len)
{
struct xe_device *xe = gt_to_xe(lrc->gt);
const u32 *user_batch;
u32 *cmd = batch;
u32 count;
count = xe_configfs_get_ctx_restore_post_bb(to_pci_dev(xe->drm.dev),
hwe->class, &user_batch);
if (!count)
return 0;
if (count > max_len)
return -ENOSPC;
/*
* This should be used only for tests and validation. Taint the kernel
* as anything could be submitted directly in context switches
*/
add_taint(TAINT_TEST, LOCKDEP_STILL_OK);
memcpy(cmd, user_batch, count * sizeof(u32));
cmd += count;
return cmd - batch;
}
static ssize_t setup_configfs_mid_ctx_restore_bb(struct xe_lrc *lrc,
struct xe_hw_engine *hwe,
u32 *batch, size_t max_len)
{
struct xe_device *xe = gt_to_xe(lrc->gt);
const u32 *user_batch;
u32 *cmd = batch;
u32 count;
count = xe_configfs_get_ctx_restore_mid_bb(to_pci_dev(xe->drm.dev),
hwe->class, &user_batch);
if (!count)
return 0;
if (count > max_len)
return -ENOSPC;
/*
* This should be used only for tests and validation. Taint the kernel
* as anything could be submitted directly in context switches
*/
add_taint(TAINT_TEST, LOCKDEP_STILL_OK);
memcpy(cmd, user_batch, count * sizeof(u32));
cmd += count;
return cmd - batch;
}
static ssize_t setup_invalidate_state_cache_wa(struct xe_lrc *lrc,
struct xe_hw_engine *hwe,
u32 *batch, size_t max_len)
@ -1203,6 +1269,7 @@ int xe_lrc_setup_wa_bb_with_scratch(struct xe_lrc *lrc, struct xe_hw_engine *hwe
{ .setup = setup_timestamp_wa },
{ .setup = setup_invalidate_state_cache_wa },
{ .setup = setup_utilization_wa },
{ .setup = setup_configfs_post_ctx_restore_bb },
};
struct bo_setup_state state = {
.lrc = lrc,
@ -1249,8 +1316,12 @@ static int setup_wa_bb(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
static int
setup_indirect_ctx(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
{
static struct bo_setup rcs_funcs[] = {
static const struct bo_setup rcs_funcs[] = {
{ .setup = setup_timestamp_wa },
{ .setup = setup_configfs_mid_ctx_restore_bb },
};
static const struct bo_setup xcs_funcs[] = {
{ .setup = setup_configfs_mid_ctx_restore_bb },
};
struct bo_setup_state state = {
.lrc = lrc,
@ -1268,6 +1339,9 @@ setup_indirect_ctx(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
hwe->class == XE_ENGINE_CLASS_COMPUTE) {
state.funcs = rcs_funcs;
state.num_funcs = ARRAY_SIZE(rcs_funcs);
} else {
state.funcs = xcs_funcs;
state.num_funcs = ARRAY_SIZE(xcs_funcs);
}
if (xe_gt_WARN_ON(lrc->gt, !state.funcs))
@ -1294,14 +1368,15 @@ setup_indirect_ctx(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
finish_bo(&state);
kfree(state.buffer);
/*
* Enable INDIRECT_CTX leaving INDIRECT_CTX_OFFSET at its default: it
* varies per engine class, but the default is good enough
*/
xe_lrc_write_ctx_reg(lrc,
CTX_CS_INDIRECT_CTX,
(xe_bo_ggtt_addr(lrc->bo) + state.offset) |
/* Size in CLs. */
(state.written * sizeof(u32) / 64));
xe_lrc_write_ctx_reg(lrc,
CTX_CS_INDIRECT_CTX_OFFSET,
CTX_INDIRECT_CTX_OFFSET_DEFAULT);
return 0;
}
@ -1340,9 +1415,10 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
if (vm && vm->xef) /* userspace */
bo_flags |= XE_BO_FLAG_PINNED_LATE_RESTORE;
lrc->bo = xe_bo_create_pin_map(xe, tile, NULL, bo_size,
ttm_bo_type_kernel,
bo_flags);
lrc->bo = xe_bo_create_pin_map_novm(xe, tile,
bo_size,
ttm_bo_type_kernel,
bo_flags, false);
if (IS_ERR(lrc->bo))
return PTR_ERR(lrc->bo);

47
drivers/gpu/drm/xe/xe_migrate.c
View File

@ -35,6 +35,7 @@
#include "xe_sched_job.h"
#include "xe_sync.h"
#include "xe_trace_bo.h"
#include "xe_validation.h"
#include "xe_vm.h"
#include "xe_vram.h"
@ -173,7 +174,7 @@ static void xe_migrate_program_identity(struct xe_device *xe, struct xe_vm *vm,
}
static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
struct xe_vm *vm)
struct xe_vm *vm, struct drm_exec *exec)
{
struct xe_device *xe = tile_to_xe(tile);
u16 pat_index = xe->pat.idx[XE_CACHE_WB];
@ -200,7 +201,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
num_entries * XE_PAGE_SIZE,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_PAGETABLE);
XE_BO_FLAG_PAGETABLE, exec);
if (IS_ERR(bo))
return PTR_ERR(bo);
@ -393,6 +394,24 @@ struct xe_migrate *xe_migrate_alloc(struct xe_tile *tile)
return m;
}
static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, struct xe_vm *vm)
{
struct xe_device *xe = tile_to_xe(tile);
struct xe_validation_ctx ctx;
struct drm_exec exec;
int err = 0;
xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, err) {
err = xe_vm_drm_exec_lock(vm, &exec);
drm_exec_retry_on_contention(&exec);
err = xe_migrate_prepare_vm(tile, m, vm, &exec);
drm_exec_retry_on_contention(&exec);
xe_validation_retry_on_oom(&ctx, &err);
}
return err;
}
/**
* xe_migrate_init() - Initialize a migrate context
* @m: The migration context
@ -413,11 +432,9 @@ int xe_migrate_init(struct xe_migrate *m)
if (IS_ERR(vm))
return PTR_ERR(vm);
xe_vm_lock(vm, false);
err = xe_migrate_prepare_vm(tile, m, vm);
xe_vm_unlock(vm);
err = xe_migrate_lock_prepare_vm(tile, m, vm);
if (err)
goto err_out;
return err;
if (xe->info.has_usm) {
struct xe_hw_engine *hwe = xe_gt_hw_engine(primary_gt,
@ -842,11 +859,15 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
&src_L0_ofs, &src_L0_pt, 0, 0,
avail_pts);
pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
batch_size += pte_update_size(m, pte_flags, dst, &dst_it, &src_L0,
&dst_L0_ofs, &dst_L0_pt, 0,
avail_pts, avail_pts);
if (copy_only_ccs) {
dst_L0_ofs = src_L0_ofs;
} else {
pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
batch_size += pte_update_size(m, pte_flags, dst,
&dst_it, &src_L0,
&dst_L0_ofs, &dst_L0_pt,
0, avail_pts, avail_pts);
}
if (copy_system_ccs) {
xe_assert(xe, type_device);
@ -876,7 +897,7 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
if (dst_is_vram && xe_migrate_allow_identity(src_L0, &dst_it))
xe_res_next(&dst_it, src_L0);
else
else if (!copy_only_ccs)
emit_pte(m, bb, dst_L0_pt, dst_is_vram, copy_system_ccs,
&dst_it, src_L0, dst);
@ -908,7 +929,7 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
if (!fence) {
err = xe_sched_job_add_deps(job, src_bo->ttm.base.resv,
DMA_RESV_USAGE_BOOKKEEP);
if (!err && src_bo != dst_bo)
if (!err && src_bo->ttm.base.resv != dst_bo->ttm.base.resv)
err = xe_sched_job_add_deps(job, dst_bo->ttm.base.resv,
DMA_RESV_USAGE_BOOKKEEP);
if (err)

5
drivers/gpu/drm/xe/xe_nvm.c
View File

@ -35,6 +35,10 @@ static const struct intel_dg_nvm_region regions[INTEL_DG_NVM_REGIONS] = {
static void xe_nvm_release_dev(struct device *dev)
{
struct auxiliary_device *aux = container_of(dev, struct auxiliary_device, dev);
struct intel_dg_nvm_dev *nvm = container_of(aux, struct intel_dg_nvm_dev, aux_dev);
kfree(nvm);
}
static bool xe_nvm_non_posted_erase(struct xe_device *xe)
@ -162,6 +166,5 @@ void xe_nvm_fini(struct xe_device *xe)
auxiliary_device_delete(&nvm->aux_dev);
auxiliary_device_uninit(&nvm->aux_dev);
kfree(nvm);
xe->nvm = NULL;
}

6
drivers/gpu/drm/xe/xe_oa.c
View File

@ -883,9 +883,9 @@ static int xe_oa_alloc_oa_buffer(struct xe_oa_stream *stream, size_t size)
{
struct xe_bo *bo;
bo = xe_bo_create_pin_map(stream->oa->xe, stream->gt->tile, NULL,
size, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_GGTT);
bo = xe_bo_create_pin_map_novm(stream->oa->xe, stream->gt->tile,
size, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_GGTT, false);
if (IS_ERR(bo))
return PTR_ERR(bo);

40
drivers/gpu/drm/xe/xe_pci.c
View File

@ -334,6 +334,7 @@ static const struct xe_device_desc bmg_desc = {
.has_mbx_power_limits = true,
.has_gsc_nvm = 1,
.has_heci_cscfi = 1,
.has_late_bind = true,
.has_sriov = true,
.max_gt_per_tile = 2,
.needs_scratch = true,
@ -510,6 +511,26 @@ static void read_gmdid(struct xe_device *xe, enum xe_gmdid_type type, u32 *ver,
*revid = REG_FIELD_GET(GMD_ID_REVID, val);
}
static const struct xe_ip *find_graphics_ip(unsigned int verx100)
{
KUNIT_STATIC_STUB_REDIRECT(find_graphics_ip, verx100);
for (int i = 0; i < ARRAY_SIZE(graphics_ips); i++)
if (graphics_ips[i].verx100 == verx100)
return &graphics_ips[i];
return NULL;
}
static const struct xe_ip *find_media_ip(unsigned int verx100)
{
KUNIT_STATIC_STUB_REDIRECT(find_media_ip, verx100);
for (int i = 0; i < ARRAY_SIZE(media_ips); i++)
if (media_ips[i].verx100 == verx100)
return &media_ips[i];
return NULL;
}
/*
* Read IP version from hardware and select graphics/media IP descriptors
* based on the result.
@ -527,14 +548,7 @@ static void handle_gmdid(struct xe_device *xe,
read_gmdid(xe, GMDID_GRAPHICS, &ver, graphics_revid);
for (int i = 0; i < ARRAY_SIZE(graphics_ips); i++) {
if (ver == graphics_ips[i].verx100) {
*graphics_ip = &graphics_ips[i];
break;
}
}
*graphics_ip = find_graphics_ip(ver);
if (!*graphics_ip) {
drm_err(&xe->drm, "Hardware reports unknown graphics version %u.%02u\n",
ver / 100, ver % 100);
@ -545,14 +559,7 @@ static void handle_gmdid(struct xe_device *xe,
if (ver == 0)
return;
for (int i = 0; i < ARRAY_SIZE(media_ips); i++) {
if (ver == media_ips[i].verx100) {
*media_ip = &media_ips[i];
break;
}
}
*media_ip = find_media_ip(ver);
if (!*media_ip) {
drm_err(&xe->drm, "Hardware reports unknown media version %u.%02u\n",
ver / 100, ver % 100);
@ -581,6 +588,7 @@ static int xe_info_init_early(struct xe_device *xe,
xe->info.has_gsc_nvm = desc->has_gsc_nvm;
xe->info.has_heci_gscfi = desc->has_heci_gscfi;
xe->info.has_heci_cscfi = desc->has_heci_cscfi;
xe->info.has_late_bind = desc->has_late_bind;
xe->info.has_llc = desc->has_llc;
xe->info.has_pxp = desc->has_pxp;
xe->info.has_sriov = desc->has_sriov;

1
drivers/gpu/drm/xe/xe_pci_types.h
View File

@ -39,6 +39,7 @@ struct xe_device_desc {
u8 has_gsc_nvm:1;
u8 has_heci_gscfi:1;
u8 has_heci_cscfi:1;
u8 has_late_bind:1;
u8 has_llc:1;
u8 has_mbx_power_limits:1;
u8 has_pxp:1;

12
drivers/gpu/drm/xe/xe_pm.c
View File

@ -21,6 +21,7 @@
#include "xe_gt_idle.h"
#include "xe_i2c.h"
#include "xe_irq.h"
#include "xe_late_bind_fw.h"
#include "xe_pcode.h"
#include "xe_pxp.h"
#include "xe_sriov_vf_ccs.h"
@ -129,6 +130,8 @@ int xe_pm_suspend(struct xe_device *xe)
if (err)
goto err;
xe_late_bind_wait_for_worker_completion(&xe->late_bind);
for_each_gt(gt, xe, id)
xe_gt_suspend_prepare(gt);
@ -213,9 +216,11 @@ int xe_pm_resume(struct xe_device *xe)
xe_pxp_pm_resume(xe->pxp);
if (IS_SRIOV_VF(xe))
if (IS_VF_CCS_READY(xe))
xe_sriov_vf_ccs_register_context(xe);
xe_late_bind_fw_load(&xe->late_bind);
drm_dbg(&xe->drm, "Device resumed\n");
return 0;
err:
@ -598,9 +603,12 @@ int xe_pm_runtime_resume(struct xe_device *xe)
xe_pxp_pm_resume(xe->pxp);
if (IS_SRIOV_VF(xe))
if (IS_VF_CCS_READY(xe))
xe_sriov_vf_ccs_register_context(xe);
if (xe->d3cold.allowed)
xe_late_bind_fw_load(&xe->late_bind);
out:
xe_rpm_lockmap_release(xe);
xe_pm_write_callback_task(xe, NULL);

129
drivers/gpu/drm/xe/xe_printk.h Normal file
View File

@ -0,0 +1,129 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef _XE_PRINTK_H_
#define _XE_PRINTK_H_
#include <drm/drm_print.h>
#include "xe_device_types.h"
#define __XE_PRINTK_FMT(_xe, _fmt, _args...) _fmt, ##_args
#define xe_printk(_xe, _level, _fmt, ...) \
drm_##_level(&(_xe)->drm, __XE_PRINTK_FMT((_xe), _fmt, ## __VA_ARGS__))
#define xe_err(_xe, _fmt, ...) \
xe_printk((_xe), err, _fmt, ##__VA_ARGS__)
#define xe_err_once(_xe, _fmt, ...) \
xe_printk((_xe), err_once, _fmt, ##__VA_ARGS__)
#define xe_err_ratelimited(_xe, _fmt, ...) \
xe_printk((_xe), err_ratelimited, _fmt, ##__VA_ARGS__)
#define xe_warn(_xe, _fmt, ...) \
xe_printk((_xe), warn, _fmt, ##__VA_ARGS__)
#define xe_notice(_xe, _fmt, ...) \
xe_printk((_xe), notice, _fmt, ##__VA_ARGS__)
#define xe_info(_xe, _fmt, ...) \
xe_printk((_xe), info, _fmt, ##__VA_ARGS__)
#define xe_dbg(_xe, _fmt, ...) \
xe_printk((_xe), dbg, _fmt, ##__VA_ARGS__)
#define xe_WARN_type(_xe, _type, _condition, _fmt, ...) \
drm_WARN##_type(&(_xe)->drm, _condition, _fmt, ## __VA_ARGS__)
#define xe_WARN(_xe, _condition, _fmt, ...) \
xe_WARN_type((_xe),, _condition, __XE_PRINTK_FMT((_xe), _fmt, ## __VA_ARGS__))
#define xe_WARN_ONCE(_xe, _condition, _fmt, ...) \
xe_WARN_type((_xe), _ONCE, _condition, __XE_PRINTK_FMT((_xe), _fmt, ## __VA_ARGS__))
#define xe_WARN_ON(_xe, _condition) \
xe_WARN((_xe), _condition, "%s(%s)", "WARN_ON", __stringify(_condition))
#define xe_WARN_ON_ONCE(_xe, _condition) \
xe_WARN_ONCE((_xe), _condition, "%s(%s)", "WARN_ON_ONCE", __stringify(_condition))
static inline void __xe_printfn_err(struct drm_printer *p, struct va_format *vaf)
{
struct xe_device *xe = p->arg;
xe_err(xe, "%pV", vaf);
}
static inline void __xe_printfn_info(struct drm_printer *p, struct va_format *vaf)
{
struct xe_device *xe = p->arg;
xe_info(xe, "%pV", vaf);
}
static inline void __xe_printfn_dbg(struct drm_printer *p, struct va_format *vaf)
{
struct xe_device *xe = p->arg;
struct drm_printer ddp;
/*
* The original xe_dbg() callsite annotations are useless here,
* redirect to the tweaked drm_dbg_printer() instead.
*/
ddp = drm_dbg_printer(&xe->drm, DRM_UT_DRIVER, NULL);
ddp.origin = p->origin;
drm_printf(&ddp, __XE_PRINTK_FMT(xe, "%pV", vaf));
}
/**
* xe_err_printer - Construct a &drm_printer that outputs to xe_err()
* @xe: the &xe_device pointer to use in xe_err()
*
* Return: The &drm_printer object.
*/
static inline struct drm_printer xe_err_printer(struct xe_device *xe)
{
struct drm_printer p = {
.printfn = __xe_printfn_err,
.arg = xe,
};
return p;
}
/**
* xe_info_printer - Construct a &drm_printer that outputs to xe_info()
* @xe: the &xe_device pointer to use in xe_info()
*
* Return: The &drm_printer object.
*/
static inline struct drm_printer xe_info_printer(struct xe_device *xe)
{
struct drm_printer p = {
.printfn = __xe_printfn_info,
.arg = xe,
};
return p;
}
/**
* xe_dbg_printer - Construct a &drm_printer that outputs like xe_dbg()
* @xe: the &xe_device pointer to use in xe_dbg()
*
* Return: The &drm_printer object.
*/
static inline struct drm_printer xe_dbg_printer(struct xe_device *xe)
{
struct drm_printer p = {
.printfn = __xe_printfn_dbg,
.arg = xe,
.origin = (const void *)_THIS_IP_,
};
return p;
}
#endif

24
drivers/gpu/drm/xe/xe_psmi.c
View File

@ -68,9 +68,7 @@ static void psmi_cleanup(struct xe_device *xe)
static struct xe_bo *psmi_alloc_object(struct xe_device *xe,
unsigned int id, size_t bo_size)
{
struct xe_bo *bo = NULL;
struct xe_tile *tile;
int err;
if (!id || !bo_size)
return NULL;
@ -78,22 +76,12 @@ static struct xe_bo *psmi_alloc_object(struct xe_device *xe,
tile = &xe->tiles[id - 1];
/* VRAM: Allocate GEM object for the capture buffer */
bo = xe_bo_create_locked(xe, tile, NULL, bo_size,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_PINNED |
XE_BO_FLAG_PINNED_LATE_RESTORE |
XE_BO_FLAG_NEEDS_CPU_ACCESS);
if (!IS_ERR(bo)) {
/* Buffer written by HW, ensure stays resident */
err = xe_bo_pin(bo);
if (err)
bo = ERR_PTR(err);
xe_bo_unlock(bo);
}
return bo;
return xe_bo_create_pin_range_novm(xe, tile, bo_size, 0, ~0ull,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_PINNED |
XE_BO_FLAG_PINNED_LATE_RESTORE |
XE_BO_FLAG_NEEDS_CPU_ACCESS);
}
/*

157
drivers/gpu/drm/xe/xe_pt.c
View File

@ -13,17 +13,17 @@
#include "xe_drm_client.h"
#include "xe_exec_queue.h"
#include "xe_gt.h"
#include "xe_tlb_inval_job.h"
#include "xe_migrate.h"
#include "xe_pt_types.h"
#include "xe_pt_walk.h"
#include "xe_res_cursor.h"
#include "xe_sched_job.h"
#include "xe_sync.h"
#include "xe_svm.h"
#include "xe_sync.h"
#include "xe_tlb_inval_job.h"
#include "xe_trace.h"
#include "xe_ttm_stolen_mgr.h"
#include "xe_userptr.h"
#include "xe_vm.h"
struct xe_pt_dir {
@ -89,6 +89,7 @@ static void xe_pt_free(struct xe_pt *pt)
* @vm: The vm to create for.
* @tile: The tile to create for.
* @level: The page-table level.
* @exec: The drm_exec object used to lock the vm.
*
* Allocate and initialize a single struct xe_pt metadata structure. Also
* create the corresponding page-table bo, but don't initialize it. If the
@ -100,7 +101,7 @@ static void xe_pt_free(struct xe_pt *pt)
* error.
*/
struct xe_pt *xe_pt_create(struct xe_vm *vm, struct xe_tile *tile,
unsigned int level)
unsigned int level, struct drm_exec *exec)
{
struct xe_pt *pt;
struct xe_bo *bo;
@ -124,9 +125,11 @@ struct xe_pt *xe_pt_create(struct xe_vm *vm, struct xe_tile *tile,
bo_flags |= XE_BO_FLAG_PINNED_LATE_RESTORE;
pt->level = level;
drm_WARN_ON(&vm->xe->drm, IS_ERR_OR_NULL(exec));
bo = xe_bo_create_pin_map(vm->xe, tile, vm, SZ_4K,
ttm_bo_type_kernel,
bo_flags);
bo_flags, exec);
if (IS_ERR(bo)) {
err = PTR_ERR(bo);
goto err_kfree;
@ -590,7 +593,8 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,
if (covers || !*child) {
u64 flags = 0;
xe_child = xe_pt_create(xe_walk->vm, xe_walk->tile, level - 1);
xe_child = xe_pt_create(xe_walk->vm, xe_walk->tile, level - 1,
xe_vm_validation_exec(vm));
if (IS_ERR(xe_child))
return PTR_ERR(xe_child);
@ -729,7 +733,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,
return -EAGAIN;
}
if (xe_svm_range_has_dma_mapping(range)) {
xe_res_first_dma(range->base.dma_addr, 0,
xe_res_first_dma(range->base.pages.dma_addr, 0,
range->base.itree.last + 1 - range->base.itree.start,
&curs);
xe_svm_range_debug(range, "BIND PREPARE - MIXED");
@ -760,8 +764,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,
if (!xe_vma_is_null(vma) && !range) {
if (xe_vma_is_userptr(vma))
xe_res_first_sg(to_userptr_vma(vma)->userptr.sg, 0,
xe_vma_size(vma), &curs);
xe_res_first_dma(to_userptr_vma(vma)->userptr.pages.dma_addr, 0,
xe_vma_size(vma), &curs);
else if (xe_bo_is_vram(bo) || xe_bo_is_stolen(bo))
xe_res_first(bo->ttm.resource, xe_vma_bo_offset(vma),
xe_vma_size(vma), &curs);
@ -914,7 +918,7 @@ bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma)
if (xe_vma_bo(vma))
xe_bo_assert_held(xe_vma_bo(vma));
else if (xe_vma_is_userptr(vma))
lockdep_assert_held(&xe_vma_vm(vma)->userptr.notifier_lock);
lockdep_assert_held(&xe_vma_vm(vma)->svm.gpusvm.notifier_lock);
if (!(pt_mask & BIT(tile->id)))
return false;
@ -1049,7 +1053,7 @@ static void xe_pt_commit_locks_assert(struct xe_vma *vma)
xe_pt_commit_prepare_locks_assert(vma);
if (xe_vma_is_userptr(vma))
lockdep_assert_held_read(&vm->userptr.notifier_lock);
xe_svm_assert_held_read(vm);
}
static void xe_pt_commit(struct xe_vma *vma,
@ -1376,6 +1380,7 @@ static int xe_pt_pre_commit(struct xe_migrate_pt_update *pt_update)
pt_update_ops, rftree);
}
#if IS_ENABLED(CONFIG_DRM_GPUSVM)
#ifdef CONFIG_DRM_XE_USERPTR_INVAL_INJECT
static bool xe_pt_userptr_inject_eagain(struct xe_userptr_vma *uvma)
@ -1406,7 +1411,7 @@ static int vma_check_userptr(struct xe_vm *vm, struct xe_vma *vma,
struct xe_userptr_vma *uvma;
unsigned long notifier_seq;
lockdep_assert_held_read(&vm->userptr.notifier_lock);
xe_svm_assert_held_read(vm);
if (!xe_vma_is_userptr(vma))
return 0;
@ -1415,7 +1420,7 @@ static int vma_check_userptr(struct xe_vm *vm, struct xe_vma *vma,
if (xe_pt_userptr_inject_eagain(uvma))
xe_vma_userptr_force_invalidate(uvma);
notifier_seq = uvma->userptr.notifier_seq;
notifier_seq = uvma->userptr.pages.notifier_seq;
if (!mmu_interval_read_retry(&uvma->userptr.notifier,
notifier_seq))
@ -1431,12 +1436,12 @@ static int vma_check_userptr(struct xe_vm *vm, struct xe_vma *vma,
return 0;
}
static int op_check_userptr(struct xe_vm *vm, struct xe_vma_op *op,
struct xe_vm_pgtable_update_ops *pt_update)
static int op_check_svm_userptr(struct xe_vm *vm, struct xe_vma_op *op,
struct xe_vm_pgtable_update_ops *pt_update)
{
int err = 0;
lockdep_assert_held_read(&vm->userptr.notifier_lock);
xe_svm_assert_held_read(vm);
switch (op->base.op) {
case DRM_GPUVA_OP_MAP:
@ -1454,9 +1459,40 @@ static int op_check_userptr(struct xe_vm *vm, struct xe_vma_op *op,
case DRM_GPUVA_OP_UNMAP:
break;
case DRM_GPUVA_OP_PREFETCH:
err = vma_check_userptr(vm, gpuva_to_vma(op->base.prefetch.va),
pt_update);
if (xe_vma_is_cpu_addr_mirror(gpuva_to_vma(op->base.prefetch.va))) {
struct xe_svm_range *range = op->map_range.range;
unsigned long i;
xe_assert(vm->xe,
xe_vma_is_cpu_addr_mirror(gpuva_to_vma(op->base.prefetch.va)));
xa_for_each(&op->prefetch_range.range, i, range) {
xe_svm_range_debug(range, "PRE-COMMIT");
if (!xe_svm_range_pages_valid(range)) {
xe_svm_range_debug(range, "PRE-COMMIT - RETRY");
return -ENODATA;
}
}
} else {
err = vma_check_userptr(vm, gpuva_to_vma(op->base.prefetch.va), pt_update);
}
break;
#if IS_ENABLED(CONFIG_DRM_XE_GPUSVM)
case DRM_GPUVA_OP_DRIVER:
if (op->subop == XE_VMA_SUBOP_MAP_RANGE) {
struct xe_svm_range *range = op->map_range.range;
xe_assert(vm->xe, xe_vma_is_cpu_addr_mirror(op->map_range.vma));
xe_svm_range_debug(range, "PRE-COMMIT");
if (!xe_svm_range_pages_valid(range)) {
xe_svm_range_debug(range, "PRE-COMMIT - RETRY");
return -EAGAIN;
}
}
break;
#endif
default:
drm_warn(&vm->xe->drm, "NOT POSSIBLE");
}
@ -1464,7 +1500,7 @@ static int op_check_userptr(struct xe_vm *vm, struct xe_vma_op *op,
return err;
}
static int xe_pt_userptr_pre_commit(struct xe_migrate_pt_update *pt_update)
static int xe_pt_svm_userptr_pre_commit(struct xe_migrate_pt_update *pt_update)
{
struct xe_vm *vm = pt_update->vops->vm;
struct xe_vma_ops *vops = pt_update->vops;
@ -1477,69 +1513,18 @@ static int xe_pt_userptr_pre_commit(struct xe_migrate_pt_update *pt_update)
if (err)
return err;
down_read(&vm->userptr.notifier_lock);
xe_svm_notifier_lock(vm);
list_for_each_entry(op, &vops->list, link) {
err = op_check_userptr(vm, op, pt_update_ops);
err = op_check_svm_userptr(vm, op, pt_update_ops);
if (err) {
up_read(&vm->userptr.notifier_lock);
xe_svm_notifier_unlock(vm);
break;
}
}
return err;
}
#if IS_ENABLED(CONFIG_DRM_XE_GPUSVM)
static int xe_pt_svm_pre_commit(struct xe_migrate_pt_update *pt_update)
{
struct xe_vm *vm = pt_update->vops->vm;
struct xe_vma_ops *vops = pt_update->vops;
struct xe_vma_op *op;
unsigned long i;
int err;
err = xe_pt_pre_commit(pt_update);
if (err)
return err;
xe_svm_notifier_lock(vm);
list_for_each_entry(op, &vops->list, link) {
struct xe_svm_range *range = NULL;
if (op->subop == XE_VMA_SUBOP_UNMAP_RANGE)
continue;
if (op->base.op == DRM_GPUVA_OP_PREFETCH) {
xe_assert(vm->xe,
xe_vma_is_cpu_addr_mirror(gpuva_to_vma(op->base.prefetch.va)));
xa_for_each(&op->prefetch_range.range, i, range) {
xe_svm_range_debug(range, "PRE-COMMIT");
if (!xe_svm_range_pages_valid(range)) {
xe_svm_range_debug(range, "PRE-COMMIT - RETRY");
xe_svm_notifier_unlock(vm);
return -ENODATA;
}
}
} else {
xe_assert(vm->xe, xe_vma_is_cpu_addr_mirror(op->map_range.vma));
xe_assert(vm->xe, op->subop == XE_VMA_SUBOP_MAP_RANGE);
range = op->map_range.range;
xe_svm_range_debug(range, "PRE-COMMIT");
if (!xe_svm_range_pages_valid(range)) {
xe_svm_range_debug(range, "PRE-COMMIT - RETRY");
xe_svm_notifier_unlock(vm);
return -EAGAIN;
}
}
}
return 0;
}
#endif
struct xe_pt_stage_unbind_walk {
@ -1843,7 +1828,7 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile,
xe_vma_start(vma),
xe_vma_end(vma));
++pt_update_ops->current_op;
pt_update_ops->needs_userptr_lock |= xe_vma_is_userptr(vma);
pt_update_ops->needs_svm_lock |= xe_vma_is_userptr(vma);
/*
* If rebind, we have to invalidate TLB on !LR vms to invalidate
@ -1951,7 +1936,7 @@ static int unbind_op_prepare(struct xe_tile *tile,
xe_pt_update_ops_rfence_interval(pt_update_ops, xe_vma_start(vma),
xe_vma_end(vma));
++pt_update_ops->current_op;
pt_update_ops->needs_userptr_lock |= xe_vma_is_userptr(vma);
pt_update_ops->needs_svm_lock |= xe_vma_is_userptr(vma);
pt_update_ops->needs_invalidation = true;
xe_pt_commit_prepare_unbind(vma, pt_op->entries, pt_op->num_entries);
@ -2199,7 +2184,7 @@ static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
vma->tile_invalidated & ~BIT(tile->id));
vma->tile_staged &= ~BIT(tile->id);
if (xe_vma_is_userptr(vma)) {
lockdep_assert_held_read(&vm->userptr.notifier_lock);
xe_svm_assert_held_read(vm);
to_userptr_vma(vma)->userptr.initial_bind = true;
}
@ -2235,7 +2220,7 @@ static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
if (!vma->tile_present) {
list_del_init(&vma->combined_links.rebind);
if (xe_vma_is_userptr(vma)) {
lockdep_assert_held_read(&vm->userptr.notifier_lock);
xe_svm_assert_held_read(vm);
spin_lock(&vm->userptr.invalidated_lock);
list_del_init(&to_userptr_vma(vma)->userptr.invalidate_link);
@ -2338,20 +2323,14 @@ static const struct xe_migrate_pt_update_ops migrate_ops = {
.pre_commit = xe_pt_pre_commit,
};
static const struct xe_migrate_pt_update_ops userptr_migrate_ops = {
#if IS_ENABLED(CONFIG_DRM_GPUSVM)
static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops = {
.populate = xe_vm_populate_pgtable,
.clear = xe_migrate_clear_pgtable_callback,
.pre_commit = xe_pt_userptr_pre_commit,
};
#if IS_ENABLED(CONFIG_DRM_XE_GPUSVM)
static const struct xe_migrate_pt_update_ops svm_migrate_ops = {
.populate = xe_vm_populate_pgtable,
.clear = xe_migrate_clear_pgtable_callback,
.pre_commit = xe_pt_svm_pre_commit,
.pre_commit = xe_pt_svm_userptr_pre_commit,
};
#else
static const struct xe_migrate_pt_update_ops svm_migrate_ops;
static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops;
#endif
static struct xe_dep_scheduler *to_dep_scheduler(struct xe_exec_queue *q,
@ -2389,9 +2368,7 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
int err = 0, i;
struct xe_migrate_pt_update update = {
.ops = pt_update_ops->needs_svm_lock ?
&svm_migrate_ops :
pt_update_ops->needs_userptr_lock ?
&userptr_migrate_ops :
&svm_userptr_migrate_ops :
&migrate_ops,
.vops = vops,
.tile_id = tile->id,
@ -2533,8 +2510,6 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
if (pt_update_ops->needs_svm_lock)
xe_svm_notifier_unlock(vm);
if (pt_update_ops->needs_userptr_lock)
up_read(&vm->userptr.notifier_lock);
xe_tlb_inval_job_put(mjob);
xe_tlb_inval_job_put(ijob);

3
drivers/gpu/drm/xe/xe_pt.h
View File

@ -10,6 +10,7 @@
#include "xe_pt_types.h"
struct dma_fence;
struct drm_exec;
struct xe_bo;
struct xe_device;
struct xe_exec_queue;
@ -29,7 +30,7 @@ struct xe_vma_ops;
unsigned int xe_pt_shift(unsigned int level);
struct xe_pt *xe_pt_create(struct xe_vm *vm, struct xe_tile *tile,
unsigned int level);
unsigned int level, struct drm_exec *exec);
void xe_pt_populate_empty(struct xe_tile *tile, struct xe_vm *vm,
struct xe_pt *pt);

2
drivers/gpu/drm/xe/xe_pt_types.h
View File

@ -105,8 +105,6 @@ struct xe_vm_pgtable_update_ops {
u32 current_op;
/** @needs_svm_lock: Needs SVM lock */
bool needs_svm_lock;
/** @needs_userptr_lock: Needs userptr lock */
bool needs_userptr_lock;
/** @needs_invalidation: Needs invalidation */
bool needs_invalidation;
/**

1
drivers/gpu/drm/xe/xe_pxp.c
View File

@ -688,6 +688,7 @@ int xe_pxp_exec_queue_add(struct xe_pxp *pxp, struct xe_exec_queue *q)
return ret;
}
ALLOW_ERROR_INJECTION(xe_pxp_exec_queue_add, ERRNO);
static void __pxp_exec_queue_remove(struct xe_pxp *pxp, struct xe_exec_queue *q, bool lock)
{

34
drivers/gpu/drm/xe/xe_pxp_submit.c
View File

@ -54,8 +54,9 @@ static int allocate_vcs_execution_resources(struct xe_pxp *pxp)
* Each termination is 16 DWORDS, so 4K is enough to contain a
* termination for each sessions.
*/
bo = xe_bo_create_pin_map(xe, tile, NULL, SZ_4K, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED | XE_BO_FLAG_GGTT);
bo = xe_bo_create_pin_map_novm(xe, tile, SZ_4K, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED | XE_BO_FLAG_GGTT,
false);
if (IS_ERR(bo)) {
err = PTR_ERR(bo);
goto out_queue;
@ -87,7 +88,9 @@ static int allocate_gsc_client_resources(struct xe_gt *gt,
{
struct xe_tile *tile = gt_to_tile(gt);
struct xe_device *xe = tile_to_xe(tile);
struct xe_validation_ctx ctx;
struct xe_hw_engine *hwe;
struct drm_exec exec;
struct xe_vm *vm;
struct xe_bo *bo;
struct xe_exec_queue *q;
@ -106,15 +109,26 @@ static int allocate_gsc_client_resources(struct xe_gt *gt,
return PTR_ERR(vm);
/* We allocate a single object for the batch and the in/out memory */
xe_vm_lock(vm, false);
bo = xe_bo_create_pin_map(xe, tile, vm, PXP_BB_SIZE + inout_size * 2,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED | XE_BO_FLAG_NEEDS_UC);
xe_vm_unlock(vm);
if (IS_ERR(bo)) {
err = PTR_ERR(bo);
goto vm_out;
xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags){}, err) {
err = xe_vm_drm_exec_lock(vm, &exec);
drm_exec_retry_on_contention(&exec);
if (err)
break;
bo = xe_bo_create_pin_map(xe, tile, vm, PXP_BB_SIZE + inout_size * 2,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED |
XE_BO_FLAG_NEEDS_UC, &exec);
drm_exec_retry_on_contention(&exec);
if (IS_ERR(bo)) {
err = PTR_ERR(bo);
xe_validation_retry_on_oom(&ctx, &err);
break;
}
}
if (err)
goto vm_out;
fence = xe_vm_bind_kernel_bo(vm, bo, NULL, 0, XE_CACHE_WB);
if (IS_ERR(fence)) {

5
drivers/gpu/drm/xe/xe_query.c
View File

@ -21,6 +21,7 @@
#include "xe_force_wake.h"
#include "xe_ggtt.h"
#include "xe_gt.h"
#include "xe_gt_topology.h"
#include "xe_guc_hwconfig.h"
#include "xe_macros.h"
#include "xe_mmio.h"
@ -477,7 +478,7 @@ static size_t calc_topo_query_size(struct xe_device *xe)
sizeof_field(struct xe_gt, fuse_topo.eu_mask_per_dss);
/* L3bank mask may not be available for some GTs */
if (!XE_GT_WA(gt, no_media_l3))
if (xe_gt_topology_report_l3(gt))
query_size += sizeof(struct drm_xe_query_topology_mask) +
sizeof_field(struct xe_gt, fuse_topo.l3_bank_mask);
}
@ -540,7 +541,7 @@ static int query_gt_topology(struct xe_device *xe,
* mask, then it's better to omit L3 from the query rather than
* reporting bogus or zeroed information to userspace.
*/
if (!XE_GT_WA(gt, no_media_l3)) {
if (xe_gt_topology_report_l3(gt)) {
topo.type = DRM_XE_TOPO_L3_BANK;
err = copy_mask(&query_ptr, &topo, gt->fuse_topo.l3_bank_mask,
sizeof(gt->fuse_topo.l3_bank_mask));

6
drivers/gpu/drm/xe/xe_rtp.c
View File

@ -370,3 +370,9 @@ bool xe_rtp_match_psmi_enabled(const struct xe_gt *gt,
{
return xe_configfs_get_psmi_enabled(to_pci_dev(gt_to_xe(gt)->drm.dev));
}
bool xe_rtp_match_gt_has_discontiguous_dss_groups(const struct xe_gt *gt,
const struct xe_hw_engine *hwe)
{
return xe_gt_has_discontiguous_dss_groups(gt);
}

3
drivers/gpu/drm/xe/xe_rtp.h
View File

@ -480,4 +480,7 @@ bool xe_rtp_match_not_sriov_vf(const struct xe_gt *gt,
bool xe_rtp_match_psmi_enabled(const struct xe_gt *gt,
const struct xe_hw_engine *hwe);
bool xe_rtp_match_gt_has_discontiguous_dss_groups(const struct xe_gt *gt,
const struct xe_hw_engine *hwe);
#endif

14
drivers/gpu/drm/xe/xe_sriov.c
View File

@ -160,19 +160,15 @@ const char *xe_sriov_function_name(unsigned int n, char *buf, size_t size)
}
/**
* xe_sriov_late_init() - SR-IOV late initialization functions.
* xe_sriov_init_late() - SR-IOV late initialization functions.
* @xe: the &xe_device to initialize
*
* On VF this function will initialize code for CCS migration.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_sriov_late_init(struct xe_device *xe)
int xe_sriov_init_late(struct xe_device *xe)
{
int err = 0;
if (IS_SRIOV_VF(xe))
return xe_sriov_vf_init_late(xe);
if (IS_VF_CCS_INIT_NEEDED(xe))
err = xe_sriov_vf_ccs_init(xe);
return err;
return 0;
}

2
drivers/gpu/drm/xe/xe_sriov.h
View File

@ -18,7 +18,7 @@ const char *xe_sriov_function_name(unsigned int n, char *buf, size_t len);
void xe_sriov_probe_early(struct xe_device *xe);
void xe_sriov_print_info(struct xe_device *xe, struct drm_printer *p);
int xe_sriov_init(struct xe_device *xe);
int xe_sriov_late_init(struct xe_device *xe);
int xe_sriov_init_late(struct xe_device *xe);
static inline enum xe_sriov_mode xe_device_sriov_mode(const struct xe_device *xe)
{

115
drivers/gpu/drm/xe/xe_sriov_vf.c
View File

@ -3,6 +3,7 @@
* Copyright © 2023-2024 Intel Corporation
*/
#include <drm/drm_debugfs.h>
#include <drm/drm_managed.h>
#include "xe_assert.h"
@ -10,6 +11,7 @@
#include "xe_gt.h"
#include "xe_gt_sriov_printk.h"
#include "xe_gt_sriov_vf.h"
#include "xe_guc.h"
#include "xe_guc_ct.h"
#include "xe_guc_submit.h"
#include "xe_irq.h"
@ -18,6 +20,7 @@
#include "xe_sriov.h"
#include "xe_sriov_printk.h"
#include "xe_sriov_vf.h"
#include "xe_sriov_vf_ccs.h"
#include "xe_tile_sriov_vf.h"
/**
@ -127,16 +130,66 @@
* | | |
*/
static bool vf_migration_supported(struct xe_device *xe)
/**
* xe_sriov_vf_migration_supported - Report whether SR-IOV VF migration is
* supported or not.
* @xe: the &xe_device to check
*
* Returns: true if VF migration is supported, false otherwise.
*/
bool xe_sriov_vf_migration_supported(struct xe_device *xe)
{
xe_assert(xe, IS_SRIOV_VF(xe));
return xe->sriov.vf.migration.enabled;
}
static void vf_disable_migration(struct xe_device *xe, const char *fmt, ...)
{
struct va_format vaf;
va_list va_args;
xe_assert(xe, IS_SRIOV_VF(xe));
va_start(va_args, fmt);
vaf.fmt = fmt;
vaf.va = &va_args;
xe_sriov_notice(xe, "migration disabled: %pV\n", &vaf);
va_end(va_args);
xe->sriov.vf.migration.enabled = false;
}
static void migration_worker_func(struct work_struct *w);
static void vf_migration_init_early(struct xe_device *xe)
{
/*
* TODO: Add conditions to allow specific platforms, when they're
* supported at production quality.
*/
return IS_ENABLED(CONFIG_DRM_XE_DEBUG);
}
if (!IS_ENABLED(CONFIG_DRM_XE_DEBUG))
return vf_disable_migration(xe,
"experimental feature not available on production builds");
static void migration_worker_func(struct work_struct *w);
if (GRAPHICS_VER(xe) < 20)
return vf_disable_migration(xe, "requires gfx version >= 20, but only %u found",
GRAPHICS_VER(xe));
if (!IS_DGFX(xe)) {
struct xe_uc_fw_version guc_version;
xe_gt_sriov_vf_guc_versions(xe_device_get_gt(xe, 0), NULL, &guc_version);
if (MAKE_GUC_VER_STRUCT(guc_version) < MAKE_GUC_VER(1, 23, 0))
return vf_disable_migration(xe,
"CCS migration requires GuC ABI >= 1.23 but only %u.%u found",
guc_version.major, guc_version.minor);
}
INIT_WORK(&xe->sriov.vf.migration.worker, migration_worker_func);
xe->sriov.vf.migration.enabled = true;
xe_sriov_dbg(xe, "migration support enabled\n");
}
/**
* xe_sriov_vf_init_early - Initialize SR-IOV VF specific data.
@ -144,10 +197,7 @@ static void migration_worker_func(struct work_struct *w);
*/
void xe_sriov_vf_init_early(struct xe_device *xe)
{
INIT_WORK(&xe->sriov.vf.migration.worker, migration_worker_func);
if (!vf_migration_supported(xe))
xe_sriov_info(xe, "migration not supported by this module version\n");
vf_migration_init_early(xe);
}
/**
@ -302,8 +352,8 @@ static void vf_post_migration_recovery(struct xe_device *xe)
xe_pm_runtime_get(xe);
vf_post_migration_shutdown(xe);
if (!vf_migration_supported(xe)) {
xe_sriov_err(xe, "migration not supported by this module version\n");
if (!xe_sriov_vf_migration_supported(xe)) {
xe_sriov_err(xe, "migration is not supported\n");
err = -ENOTRECOVERABLE;
goto fail;
}
@ -378,3 +428,48 @@ void xe_sriov_vf_start_migration_recovery(struct xe_device *xe)
drm_info(&xe->drm, "VF migration recovery %s\n", started ?
"scheduled" : "already in progress");
}
/**
* xe_sriov_vf_init_late() - SR-IOV VF late initialization functions.
* @xe: the &xe_device to initialize
*
* This function initializes code for CCS migration.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_sriov_vf_init_late(struct xe_device *xe)
{
int err = 0;
if (xe_sriov_vf_migration_supported(xe))
err = xe_sriov_vf_ccs_init(xe);
return err;
}
static int sa_info_vf_ccs(struct seq_file *m, void *data)
{
struct drm_info_node *node = m->private;
struct xe_device *xe = to_xe_device(node->minor->dev);
struct drm_printer p = drm_seq_file_printer(m);
xe_sriov_vf_ccs_print(xe, &p);
return 0;
}
static const struct drm_info_list debugfs_list[] = {
{ .name = "sa_info_vf_ccs", .show = sa_info_vf_ccs },
};
/**
* xe_sriov_vf_debugfs_register - Register VF debugfs attributes.
* @xe: the &xe_device
* @root: the root &dentry
*
* Prepare debugfs attributes exposed by the VF.
*/
void xe_sriov_vf_debugfs_register(struct xe_device *xe, struct dentry *root)
{
drm_debugfs_create_files(debugfs_list, ARRAY_SIZE(debugfs_list),
root, xe->drm.primary);
}

6
drivers/gpu/drm/xe/xe_sriov_vf.h
View File

@ -6,9 +6,15 @@
#ifndef _XE_SRIOV_VF_H_
#define _XE_SRIOV_VF_H_
#include <linux/types.h>
struct dentry;
struct xe_device;
void xe_sriov_vf_init_early(struct xe_device *xe);
int xe_sriov_vf_init_late(struct xe_device *xe);
void xe_sriov_vf_start_migration_recovery(struct xe_device *xe);
bool xe_sriov_vf_migration_supported(struct xe_device *xe);
void xe_sriov_vf_debugfs_register(struct xe_device *xe, struct dentry *root);
#endif

Some files were not shown because too many files have changed in this diff Show More