github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-05-29 17:43:52 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Cross-subsystem Changes:

Browse Source 
- Add drm_line_printer (Michal)
 
 Driver Changes:
 - Fix an UAF (Matt Auld)
 - Sanity check compression and coherency mode (Matt Auld)
 - Some PIC-ID work (Jani)
 - Use IS_ENABLED() instead of defined() on config options.
 - gt powergating work (Riana)
 - Suppress missing out ter rpm protection warning (Rodrigo)
 - Fix a vm leak  (Dafna)
 - Clean up and update 'has_flat_ccs' handling (Lucas)
 - Fix arg to pci_iomap (Lucas)
 - Mark reserved engines in shapshot (Lucas)
 - Don't keep stale pointer (Michal)
 - Fix build warning with CONFIG_PM=n (Arnd)
 - Add a xe_bo subtest for shrinking / swapping (Thomas)
 - Add a warkaround (Tejas)
 - Some display PM work (Maarten)
 - Enable Xe2 + PES disaggregation (Ashutosh)
 - Large xe_mmio rework / cleanup (Matt Roper)
 - A couple of fixes / cleanups in the xe client code (Matt Auld)
 - Fix page-fault handling on closed VMs  (Matt Brost)
 - Fix overflow in OA batch buffer (José)
 - Style fixes (Lucas, Jiapeng, Nitin)
 - Fixes and new development around SRIOV (Michal)
 - Use devm_add_action_or_reset() in gt code (He)
 - Fix CCS offset calculation (Matt Auld)
 - Remove i915_drv.h include (Rodrigo)
 - Restore PCI state on resume (Rodrigo)
 - Fix DSB buffer coherency / Revert DSB disabling (Maarten / Animesh)
 - Convert USM lock to rwsem (Matt Brost)
 - Defer gt-mmio intialization (Matt Roper)
 - meemirq changes (Ilia)
 - Move some PVC related code out of xe-for-CI and to the driver (Rodrigo / Jani)
 - Use a helper for ASID->VM lookup (Matt Brost)
 - Add new PCI id for ARL (Dnyaneshwar)
 - Use Xe2_LPM steering tables for Xe2_HPM (Gustavo)
 - Performance tuning work for media GT and L3 cache flushing (Gustavo)
 - Clean up VM- and exec queue file lock usage (Matt Brost)
 - GuC locking fix (Matt Auld)
 - Fix UAF around queue destruction (Matt Auld)
 - Move IRQ-related registers to dedicated header (Matt Roper)
 - Resume TDR after GT reset (Matt Brost)
 - Move xa_alloc to prevent UAF (Matt Auld)
 - Fix OA stream close (José)
 - Remove unused i915_gpu_error.h (Jani)
 - Prevent null pointer access in xe_migrate_copy (Zhanjun)
 - Fix memory leak when aborting binds (Matt Brost)
 - Prevent UAF in send_recv() (Matt Auld)
 - Fix xa_store() error checking (Matt Auld)
 - drop irq disabling around xa_erase in guc code (Matt Auld)
 - Use fault injection infrastructure to find issues as probe time (Francois)
 - Fix a workaround implementation. (Vinay)
 - Mark wedged_mode debugfs writable (Matt Roper)
 - Fix for prviewous memirq work (Michal)
 - More SRIOV work (Michal)
 - Devcoredump work (John)
 - GuC logging + devcoredump support (John)
 - Don't report L3 bank availability on PTL (Shekhar)
 - Replicate Xe2 PAT settings on Xe2 (Matt Roper)
 - Define Xe3 feature flags (Haridhar)
 - Reuse Xe2 MOCS table on on PTL (Haridhar)
 - Add PTL platform definition (Haridhar)
 - Add MCR steering for Xe3 (Matt)
 - More work around GuC capture for devcoredump (Zhanjun)
 - Improve cache flushing behaviour on bmg (Matt Auld)
 - Fix shrinker test compiler warnings on 32-bit (Thomas)
 - Initial set of workarounds for Xe3 (Gustavo)
 - Extend workaround for xe2lpg (Aradhya)
 - Fix unbalanced rpm put x 2 (Matt Auld)
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQRskUM7w1oG5rx2IZO4FpNVCsYGvwUCZwekBwAKCRC4FpNVCsYG
 v32oAQDnIKVwjZecI1V3oUsy2ZE3TKWx8HH4FweT6S5L6tqZwQD/b0vkeA3UaojO
 5FIkPEqyHFbrj+Sw7bLonLb3LHv4WAE=
 =FtY6
 -----END PGP SIGNATURE-----

Merge tag 'drm-xe-next-2024-10-10' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-next

Cross-subsystem Changes:
- Add drm_line_printer (Michal)

Driver Changes:
- Fix an UAF (Matt Auld)
- Sanity check compression and coherency mode (Matt Auld)
- Some PIC-ID work (Jani)
- Use IS_ENABLED() instead of defined() on config options.
- gt powergating work (Riana)
- Suppress missing out ter rpm protection warning (Rodrigo)
- Fix a vm leak  (Dafna)
- Clean up and update 'has_flat_ccs' handling (Lucas)
- Fix arg to pci_iomap (Lucas)
- Mark reserved engines in shapshot (Lucas)
- Don't keep stale pointer (Michal)
- Fix build warning with CONFIG_PM=n (Arnd)
- Add a xe_bo subtest for shrinking / swapping (Thomas)
- Add a warkaround (Tejas)
- Some display PM work (Maarten)
- Enable Xe2 + PES disaggregation (Ashutosh)
- Large xe_mmio rework / cleanup (Matt Roper)
- A couple of fixes / cleanups in the xe client code (Matt Auld)
- Fix page-fault handling on closed VMs  (Matt Brost)
- Fix overflow in OA batch buffer (José)
- Style fixes (Lucas, Jiapeng, Nitin)
- Fixes and new development around SRIOV (Michal)
- Use devm_add_action_or_reset() in gt code (He)
- Fix CCS offset calculation (Matt Auld)
- Remove i915_drv.h include (Rodrigo)
- Restore PCI state on resume (Rodrigo)
- Fix DSB buffer coherency / Revert DSB disabling (Maarten / Animesh)
- Convert USM lock to rwsem (Matt Brost)
- Defer gt-mmio intialization (Matt Roper)
- meemirq changes (Ilia)
- Move some PVC related code out of xe-for-CI and to the driver (Rodrigo / Jani)
- Use a helper for ASID->VM lookup (Matt Brost)
- Add new PCI id for ARL (Dnyaneshwar)
- Use Xe2_LPM steering tables for Xe2_HPM (Gustavo)
- Performance tuning work for media GT and L3 cache flushing (Gustavo)
- Clean up VM- and exec queue file lock usage (Matt Brost)
- GuC locking fix (Matt Auld)
- Fix UAF around queue destruction (Matt Auld)
- Move IRQ-related registers to dedicated header (Matt Roper)
- Resume TDR after GT reset (Matt Brost)
- Move xa_alloc to prevent UAF (Matt Auld)
- Fix OA stream close (José)
- Remove unused i915_gpu_error.h (Jani)
- Prevent null pointer access in xe_migrate_copy (Zhanjun)
- Fix memory leak when aborting binds (Matt Brost)
- Prevent UAF in send_recv() (Matt Auld)
- Fix xa_store() error checking (Matt Auld)
- drop irq disabling around xa_erase in guc code (Matt Auld)
- Use fault injection infrastructure to find issues as probe time (Francois)
- Fix a workaround implementation. (Vinay)
- Mark wedged_mode debugfs writable (Matt Roper)
- Fix for prviewous memirq work (Michal)
- More SRIOV work (Michal)
- Devcoredump work (John)
- GuC logging + devcoredump support (John)
- Don't report L3 bank availability on PTL (Shekhar)
- Replicate Xe2 PAT settings on Xe2 (Matt Roper)
- Define Xe3 feature flags (Haridhar)
- Reuse Xe2 MOCS table on on PTL (Haridhar)
- Add PTL platform definition (Haridhar)
- Add MCR steering for Xe3 (Matt)
- More work around GuC capture for devcoredump (Zhanjun)
- Improve cache flushing behaviour on bmg (Matt Auld)
- Fix shrinker test compiler warnings on 32-bit (Thomas)
- Initial set of workarounds for Xe3 (Gustavo)
- Extend workaround for xe2lpg (Aradhya)
- Fix unbalanced rpm put x 2 (Matt Auld)

Signed-off-by: Dave Airlie <airlied@redhat.com>

# -----BEGIN PGP SIGNATURE-----
#
# iHUEABYKAB0WIQRskUM7w1oG5rx2IZO4FpNVCsYGvwUCZwekBwAKCRC4FpNVCsYG
# v32oAQDnIKVwjZecI1V3oUsy2ZE3TKWx8HH4FweT6S5L6tqZwQD/b0vkeA3UaojO
# 5FIkPEqyHFbrj+Sw7bLonLb3LHv4WAE=
# =FtY6
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 10 Oct 2024 19:53:11 AEST
# gpg:                using EDDSA key 6C91433BC35A06E6BC762193B81693550AC606BF
# gpg: Can't check signature: No public key

# Conflicts:
#	drivers/gpu/drm/xe/xe_gt_mcr.c
#	drivers/gpu/drm/xe/xe_tuning.c
From: Thomas Hellstrom <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/Zwekwrak12c5SSgo@fedora
This commit is contained in:
Dave Airlie 2024-10-11 08:01:16 +10:00
commit 26bb2dc102
127 changed files with 6254 additions and 1263 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
drivers/gpu/drm/drm_print.c
View File

@ -235,6 +235,20 @@ void __drm_printfn_err(struct drm_printer *p, struct va_format *vaf)
}
EXPORT_SYMBOL(__drm_printfn_err);
void __drm_printfn_line(struct drm_printer *p, struct va_format *vaf)
{
unsigned int counter = ++p->line.counter;
const char *prefix = p->prefix ?: "";
const char *pad = p->prefix ? " " : "";
if (p->line.series)
drm_printf(p->arg, "%s%s%u.%u: %pV",
prefix, pad, p->line.series, counter, vaf);
else
drm_printf(p->arg, "%s%s%u: %pV", prefix, pad, counter, vaf);
}
EXPORT_SYMBOL(__drm_printfn_line);
/**
* drm_puts - print a const string to a &drm_printer stream
* @p: the &drm printer

4
drivers/gpu/drm/i915/display/intel_dsb.c
View File

@ -706,10 +706,6 @@ struct intel_dsb *intel_dsb_prepare(struct intel_atomic_state *state,
if (!i915->display.params.enable_dsb)
return NULL;
/* TODO: DSB is broken in Xe KMD, so disabling it until fixed */
if (!IS_ENABLED(I915))
return NULL;
dsb = kzalloc(sizeof(*dsb), GFP_KERNEL);
if (!dsb)
goto out;

12
drivers/gpu/drm/xe/Kconfig.debug
View File

@ -40,9 +40,21 @@ config DRM_XE_DEBUG_VM
If in doubt, say "N".
config DRM_XE_DEBUG_MEMIRQ
bool "Enable extra memirq debugging"
default n
help
Choose this option to enable additional debugging info for
memory based interrupts.
Recommended for driver developers only.
If in doubt, say "N".
config DRM_XE_DEBUG_SRIOV
bool "Enable extra SR-IOV debugging"
default n
select DRM_XE_DEBUG_MEMIRQ
help
Enable extra SR-IOV debugging info.

2
drivers/gpu/drm/xe/Makefile
View File

@ -56,6 +56,7 @@ xe-y += xe_bb.o \
xe_gt_topology.o \
xe_guc.o \
xe_guc_ads.o \
xe_guc_capture.o \
xe_guc_ct.o \
xe_guc_db_mgr.o \
xe_guc_hwconfig.o \
@ -129,6 +130,7 @@ xe-$(CONFIG_PCI_IOV) += \
xe_gt_sriov_pf.o \
xe_gt_sriov_pf_config.o \
xe_gt_sriov_pf_control.o \
xe_gt_sriov_pf_migration.o \
xe_gt_sriov_pf_monitor.o \
xe_gt_sriov_pf_policy.o \
xe_gt_sriov_pf_service.o \

8
drivers/gpu/drm/xe/abi/guc_actions_abi.h
View File

@ -176,6 +176,14 @@ enum xe_guc_sleep_state_status {
#define GUC_LOG_CONTROL_VERBOSITY_MASK (0xF << GUC_LOG_CONTROL_VERBOSITY_SHIFT)
#define GUC_LOG_CONTROL_DEFAULT_LOGGING (1 << 8)
enum xe_guc_state_capture_event_status {
XE_GUC_STATE_CAPTURE_EVENT_STATUS_SUCCESS = 0x0,
XE_GUC_STATE_CAPTURE_EVENT_STATUS_NOSPACE = 0x1,
};
#define XE_GUC_STATE_CAPTURE_EVENT_STATUS_MASK 0x000000FF
#define XE_GUC_ACTION_STATE_CAPTURE_NOTIFICATION_DATA_LEN 1
#define XE_GUC_TLB_INVAL_TYPE_SHIFT 0
#define XE_GUC_TLB_INVAL_MODE_SHIFT 8
/* Flush PPC or SMRO caches along with TLB invalidation request */

61
drivers/gpu/drm/xe/abi/guc_actions_sriov_abi.h
View File

@ -557,4 +557,65 @@
#define VF2GUC_QUERY_SINGLE_KLV_RESPONSE_MSG_2_VALUE64 GUC_HXG_REQUEST_MSG_n_DATAn
#define VF2GUC_QUERY_SINGLE_KLV_RESPONSE_MSG_3_VALUE96 GUC_HXG_REQUEST_MSG_n_DATAn
/**
* DOC: PF2GUC_SAVE_RESTORE_VF
*
* This message is used by the PF to migrate VF info state maintained by the GuC.
*
* This message must be sent as `CTB HXG Message`_.
*
* Available since GuC version 70.25.0
*
* +---+-------+--------------------------------------------------------------+
* | | Bits | Description |
* +===+=======+==============================================================+
* | 0 | 31 | ORIGIN = GUC_HXG_ORIGIN_HOST_ |
* | +-------+--------------------------------------------------------------+
* | | 30:28 | TYPE = GUC_HXG_TYPE_REQUEST_ |
* | +-------+--------------------------------------------------------------+
* | | 27:16 | DATA0 = **OPCODE** - operation to take: |
* | | | |
* | | | - _`GUC_PF_OPCODE_VF_SAVE` = 0 |
* | | | - _`GUC_PF_OPCODE_VF_RESTORE` = 1 |
* | +-------+--------------------------------------------------------------+
* | | 15:0 | ACTION = _`GUC_ACTION_PF2GUC_SAVE_RESTORE_VF` = 0x550B |
* +---+-------+--------------------------------------------------------------+
* | 1 | 31:0 | **VFID** - VF identifier |
* +---+-------+--------------------------------------------------------------+
* | 2 | 31:0 | **ADDR_LO** - lower 32-bits of GGTT offset to the buffer |
* | | | where the VF info will be save to or restored from. |
* +---+-------+--------------------------------------------------------------+
* | 3 | 31:0 | **ADDR_HI** - upper 32-bits of GGTT offset to the buffer |
* | | | where the VF info will be save to or restored from. |
* +---+-------+--------------------------------------------------------------+
* | 4 | 27:0 | **SIZE** - size of the buffer (in dwords) |
* | +-------+--------------------------------------------------------------+
* | | 31:28 | MBZ |
* +---+-------+--------------------------------------------------------------+
*
* +---+-------+--------------------------------------------------------------+
* | | Bits | Description |
* +===+=======+==============================================================+
* | 0 | 31 | ORIGIN = GUC_HXG_ORIGIN_GUC_ |
* | +-------+--------------------------------------------------------------+
* | | 30:28 | TYPE = GUC_HXG_TYPE_RESPONSE_SUCCESS_ |
* | +-------+--------------------------------------------------------------+
* | | 27:0 | DATA0 = **USED** - size of used buffer space (in dwords) |
* +---+-------+--------------------------------------------------------------+
*/
#define GUC_ACTION_PF2GUC_SAVE_RESTORE_VF 0x550Bu
#define PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_LEN (GUC_HXG_EVENT_MSG_MIN_LEN + 4u)
#define PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_0_OPCODE GUC_HXG_EVENT_MSG_0_DATA0
#define GUC_PF_OPCODE_VF_SAVE 0u
#define GUC_PF_OPCODE_VF_RESTORE 1u
#define PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_1_VFID GUC_HXG_EVENT_MSG_n_DATAn
#define PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_2_ADDR_LO GUC_HXG_EVENT_MSG_n_DATAn
#define PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_3_ADDR_HI GUC_HXG_EVENT_MSG_n_DATAn
#define PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_4_SIZE (0xfffffffu << 0)
#define PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_4_MBZ (0xfu << 28)
#define PF2GUC_SAVE_RESTORE_VF_RESPONSE_MSG_LEN GUC_HXG_RESPONSE_MSG_MIN_LEN
#define PF2GUC_SAVE_RESTORE_VF_RESPONSE_MSG_0_USED GUC_HXG_RESPONSE_MSG_0_DATA0
#endif

186
drivers/gpu/drm/xe/abi/guc_capture_abi.h Normal file
View File

@ -0,0 +1,186 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2024 Intel Corporation
*/
#ifndef _ABI_GUC_CAPTURE_ABI_H
#define _ABI_GUC_CAPTURE_ABI_H
#include <linux/types.h>
/* Capture List Index */
enum guc_capture_list_index_type {
GUC_CAPTURE_LIST_INDEX_PF = 0,
GUC_CAPTURE_LIST_INDEX_VF = 1,
};
#define GUC_CAPTURE_LIST_INDEX_MAX (GUC_CAPTURE_LIST_INDEX_VF + 1)
/* Register-types of GuC capture register lists */
enum guc_state_capture_type {
GUC_STATE_CAPTURE_TYPE_GLOBAL = 0,
GUC_STATE_CAPTURE_TYPE_ENGINE_CLASS,
GUC_STATE_CAPTURE_TYPE_ENGINE_INSTANCE
};
#define GUC_STATE_CAPTURE_TYPE_MAX (GUC_STATE_CAPTURE_TYPE_ENGINE_INSTANCE + 1)
/* Class indecies for capture_class and capture_instance arrays */
enum guc_capture_list_class_type {
GUC_CAPTURE_LIST_CLASS_RENDER_COMPUTE = 0,
GUC_CAPTURE_LIST_CLASS_VIDEO = 1,
GUC_CAPTURE_LIST_CLASS_VIDEOENHANCE = 2,
GUC_CAPTURE_LIST_CLASS_BLITTER = 3,
GUC_CAPTURE_LIST_CLASS_GSC_OTHER = 4,
};
#define GUC_CAPTURE_LIST_CLASS_MAX (GUC_CAPTURE_LIST_CLASS_GSC_OTHER + 1)
/**
* struct guc_mmio_reg - GuC MMIO reg state struct
*
* GuC MMIO reg state struct
*/
struct guc_mmio_reg {
/** @offset: MMIO Offset - filled in by Host */
u32 offset;
/** @value: MMIO Value - Used by Firmware to store value */
u32 value;
/** @flags: Flags for accessing the MMIO */
u32 flags;
/** @mask: Value of a mask to apply if mask with value is set */
u32 mask;
#define GUC_REGSET_MASKED BIT(0)
#define GUC_REGSET_STEERING_NEEDED BIT(1)
#define GUC_REGSET_MASKED_WITH_VALUE BIT(2)
#define GUC_REGSET_RESTORE_ONLY BIT(3)
#define GUC_REGSET_STEERING_GROUP GENMASK(16, 12)
#define GUC_REGSET_STEERING_INSTANCE GENMASK(23, 20)
} __packed;
/**
* struct guc_mmio_reg_set - GuC register sets
*
* GuC register sets
*/
struct guc_mmio_reg_set {
/** @address: register address */
u32 address;
/** @count: register count */
u16 count;
/** @reserved: reserved */
u16 reserved;
} __packed;
/**
* struct guc_debug_capture_list_header - Debug capture list header.
*
* Debug capture list header.
*/
struct guc_debug_capture_list_header {
/** @info: contains number of MMIO descriptors in the capture list. */
u32 info;
#define GUC_CAPTURELISTHDR_NUMDESCR GENMASK(15, 0)
} __packed;
/**
* struct guc_debug_capture_list - Debug capture list
*
* As part of ADS registration, these header structures (followed by
* an array of 'struct guc_mmio_reg' entries) are used to register with
* GuC microkernel the list of registers we want it to dump out prior
* to a engine reset.
*/
struct guc_debug_capture_list {
/** @header: Debug capture list header. */
struct guc_debug_capture_list_header header;
/** @regs: MMIO descriptors in the capture list. */
struct guc_mmio_reg regs[];
} __packed;
/**
* struct guc_state_capture_header_t - State capture header.
*
* Prior to resetting engines that have hung or faulted, GuC microkernel
* reports the engine error-state (register values that was read) by
* logging them into the shared GuC log buffer using these hierarchy
* of structures.
*/
struct guc_state_capture_header_t {
/**
* @owner: VFID
* BR[ 7: 0] MBZ when SRIOV is disabled. When SRIOV is enabled
* VFID is an integer in range [0, 63] where 0 means the state capture
* is corresponding to the PF and an integer N in range [1, 63] means
* the state capture is for VF N.
*/
u32 owner;
#define GUC_STATE_CAPTURE_HEADER_VFID GENMASK(7, 0)
/** @info: Engine class/instance and capture type info */
u32 info;
#define GUC_STATE_CAPTURE_HEADER_CAPTURE_TYPE GENMASK(3, 0) /* see guc_state_capture_type */
#define GUC_STATE_CAPTURE_HEADER_ENGINE_CLASS GENMASK(7, 4) /* see guc_capture_list_class_type */
#define GUC_STATE_CAPTURE_HEADER_ENGINE_INSTANCE GENMASK(11, 8)
/**
* @lrca: logical ring context address.
* if type-instance, LRCA (address) that hung, else set to ~0
*/
u32 lrca;
/**
* @guc_id: context_index.
* if type-instance, context index of hung context, else set to ~0
*/
u32 guc_id;
/** @num_mmio_entries: Number of captured MMIO entries. */
u32 num_mmio_entries;
#define GUC_STATE_CAPTURE_HEADER_NUM_MMIO_ENTRIES GENMASK(9, 0)
} __packed;
/**
* struct guc_state_capture_t - State capture.
*
* State capture
*/
struct guc_state_capture_t {
/** @header: State capture header. */
struct guc_state_capture_header_t header;
/** @mmio_entries: Array of captured guc_mmio_reg entries. */
struct guc_mmio_reg mmio_entries[];
} __packed;
/* State Capture Group Type */
enum guc_state_capture_group_type {
GUC_STATE_CAPTURE_GROUP_TYPE_FULL = 0,
GUC_STATE_CAPTURE_GROUP_TYPE_PARTIAL
};
#define GUC_STATE_CAPTURE_GROUP_TYPE_MAX (GUC_STATE_CAPTURE_GROUP_TYPE_PARTIAL + 1)
/**
* struct guc_state_capture_group_header_t - State capture group header
*
* State capture group header.
*/
struct guc_state_capture_group_header_t {
/** @owner: VFID */
u32 owner;
#define GUC_STATE_CAPTURE_GROUP_HEADER_VFID GENMASK(7, 0)
/** @info: Engine class/instance and capture type info */
u32 info;
#define GUC_STATE_CAPTURE_GROUP_HEADER_NUM_CAPTURES GENMASK(7, 0)
#define GUC_STATE_CAPTURE_GROUP_HEADER_CAPTURE_GROUP_TYPE GENMASK(15, 8)
} __packed;
/**
* struct guc_state_capture_group_t - State capture group.
*
* this is the top level structure where an error-capture dump starts
*/
struct guc_state_capture_group_t {
/** @grp_header: State capture group header. */
struct guc_state_capture_group_header_t grp_header;
/** @capture_entries: Array of state captures */
struct guc_state_capture_t capture_entries[];
} __packed;
#endif

1
drivers/gpu/drm/xe/abi/guc_communication_ctb_abi.h
View File

@ -52,6 +52,7 @@ struct guc_ct_buffer_desc {
#define GUC_CTB_STATUS_OVERFLOW (1 << 0)
#define GUC_CTB_STATUS_UNDERFLOW (1 << 1)
#define GUC_CTB_STATUS_MISMATCH (1 << 2)
#define GUC_CTB_STATUS_DISABLED (1 << 3)
u32 reserved[13];
} __packed;
static_assert(sizeof(struct guc_ct_buffer_desc) == 64);

75
drivers/gpu/drm/xe/abi/guc_log_abi.h Normal file
View File

@ -0,0 +1,75 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2024 Intel Corporation
*/
#ifndef _ABI_GUC_LOG_ABI_H
#define _ABI_GUC_LOG_ABI_H
#include <linux/types.h>
/* GuC logging buffer types */
enum guc_log_buffer_type {
GUC_LOG_BUFFER_CRASH_DUMP,
GUC_LOG_BUFFER_DEBUG,
GUC_LOG_BUFFER_CAPTURE,
};
#define GUC_LOG_BUFFER_TYPE_MAX 3
/**
* struct guc_log_buffer_state - GuC log buffer state
*
* Below state structure is used for coordination of retrieval of GuC firmware
* logs. Separate state is maintained for each log buffer type.
* read_ptr points to the location where Xe read last in log buffer and
* is read only for GuC firmware. write_ptr is incremented by GuC with number
* of bytes written for each log entry and is read only for Xe.
* When any type of log buffer becomes half full, GuC sends a flush interrupt.
* GuC firmware expects that while it is writing to 2nd half of the buffer,
* first half would get consumed by Host and then get a flush completed
* acknowledgment from Host, so that it does not end up doing any overwrite
* causing loss of logs. So when buffer gets half filled & Xe has requested
* for interrupt, GuC will set flush_to_file field, set the sampled_write_ptr
* to the value of write_ptr and raise the interrupt.
* On receiving the interrupt Xe should read the buffer, clear flush_to_file
* field and also update read_ptr with the value of sample_write_ptr, before
* sending an acknowledgment to GuC. marker & version fields are for internal
* usage of GuC and opaque to Xe. buffer_full_cnt field is incremented every
* time GuC detects the log buffer overflow.
*/
struct guc_log_buffer_state {
/** @marker: buffer state start marker */
u32 marker[2];
/** @read_ptr: the last byte offset that was read by KMD previously */
u32 read_ptr;
/**
* @write_ptr: the next byte offset location that will be written by
* GuC
*/
u32 write_ptr;
/** @size: Log buffer size */
u32 size;
/**
* @sampled_write_ptr: Log buffer write pointer
* This is written by GuC to the byte offset of the next free entry in
* the buffer on log buffer half full or state capture notification
*/
u32 sampled_write_ptr;
/**
* @wrap_offset: wraparound offset
* This is the byte offset of location 1 byte after last valid guc log
* event entry written by Guc firmware before there was a wraparound.
* This field is updated by guc firmware and should be used by Host
* when copying buffer contents to file.
*/
u32 wrap_offset;
/** @flags: Flush to file flag and buffer full count */
u32 flags;
#define GUC_LOG_BUFFER_STATE_FLUSH_TO_FILE GENMASK(0, 0)
#define GUC_LOG_BUFFER_STATE_BUFFER_FULL_CNT GENMASK(4, 1)
/** @version: The Guc-Log-Entry format version */
u32 version;
} __packed;
#endif

17
drivers/gpu/drm/xe/compat-i915-headers/i915_gpu_error.h
View File

@ -1,17 +0,0 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2023 Intel Corporation
*/
#ifndef _I915_GPU_ERROR_H_
#define _I915_GPU_ERROR_H_
struct drm_i915_error_state_buf;
__printf(2, 3)
static inline void
i915_error_printf(struct drm_i915_error_state_buf *e, const char *f, ...)
{
}
#endif

36
drivers/gpu/drm/xe/compat-i915-headers/intel_uncore.h
View File

@ -10,11 +10,11 @@
#include "xe_device_types.h"
#include "xe_mmio.h"
static inline struct xe_gt *__compat_uncore_to_gt(struct intel_uncore *uncore)
static inline struct xe_mmio *__compat_uncore_to_mmio(struct intel_uncore *uncore)
{
struct xe_device *xe = container_of(uncore, struct xe_device, uncore);
return xe_root_mmio_gt(xe);
return xe_root_tile_mmio(xe);
}
static inline struct xe_tile *__compat_uncore_to_tile(struct intel_uncore *uncore)
@ -29,7 +29,7 @@ static inline u32 intel_uncore_read(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_read32(__compat_uncore_to_gt(uncore), reg);
return xe_mmio_read32(__compat_uncore_to_mmio(uncore), reg);
}
static inline u8 intel_uncore_read8(struct intel_uncore *uncore,
@ -37,7 +37,7 @@ static inline u8 intel_uncore_read8(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_read8(__compat_uncore_to_gt(uncore), reg);
return xe_mmio_read8(__compat_uncore_to_mmio(uncore), reg);
}
static inline u16 intel_uncore_read16(struct intel_uncore *uncore,
@ -45,7 +45,7 @@ static inline u16 intel_uncore_read16(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_read16(__compat_uncore_to_gt(uncore), reg);
return xe_mmio_read16(__compat_uncore_to_mmio(uncore), reg);
}
static inline u64
@ -57,11 +57,11 @@ intel_uncore_read64_2x32(struct intel_uncore *uncore,
u32 upper, lower, old_upper;
int loop = 0;
upper = xe_mmio_read32(__compat_uncore_to_gt(uncore), upper_reg);
upper = xe_mmio_read32(__compat_uncore_to_mmio(uncore), upper_reg);
do {
old_upper = upper;
lower = xe_mmio_read32(__compat_uncore_to_gt(uncore), lower_reg);
upper = xe_mmio_read32(__compat_uncore_to_gt(uncore), upper_reg);
lower = xe_mmio_read32(__compat_uncore_to_mmio(uncore), lower_reg);
upper = xe_mmio_read32(__compat_uncore_to_mmio(uncore), upper_reg);
} while (upper != old_upper && loop++ < 2);
return (u64)upper << 32 | lower;
@ -72,7 +72,7 @@ static inline void intel_uncore_posting_read(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
xe_mmio_read32(__compat_uncore_to_gt(uncore), reg);
xe_mmio_read32(__compat_uncore_to_mmio(uncore), reg);
}
static inline void intel_uncore_write(struct intel_uncore *uncore,
@ -80,7 +80,7 @@ static inline void intel_uncore_write(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
xe_mmio_write32(__compat_uncore_to_gt(uncore), reg, val);
xe_mmio_write32(__compat_uncore_to_mmio(uncore), reg, val);
}
static inline u32 intel_uncore_rmw(struct intel_uncore *uncore,
@ -88,7 +88,7 @@ static inline u32 intel_uncore_rmw(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_rmw32(__compat_uncore_to_gt(uncore), reg, clear, set);
return xe_mmio_rmw32(__compat_uncore_to_mmio(uncore), reg, clear, set);
}
static inline int intel_wait_for_register(struct intel_uncore *uncore,
@ -97,7 +97,7 @@ static inline int intel_wait_for_register(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_wait32(__compat_uncore_to_gt(uncore), reg, mask, value,
return xe_mmio_wait32(__compat_uncore_to_mmio(uncore), reg, mask, value,
timeout * USEC_PER_MSEC, NULL, false);
}
@ -107,7 +107,7 @@ static inline int intel_wait_for_register_fw(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_wait32(__compat_uncore_to_gt(uncore), reg, mask, value,
return xe_mmio_wait32(__compat_uncore_to_mmio(uncore), reg, mask, value,
timeout * USEC_PER_MSEC, NULL, false);
}
@ -118,7 +118,7 @@ __intel_wait_for_register(struct intel_uncore *uncore, i915_reg_t i915_reg,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_wait32(__compat_uncore_to_gt(uncore), reg, mask, value,
return xe_mmio_wait32(__compat_uncore_to_mmio(uncore), reg, mask, value,
fast_timeout_us + 1000 * slow_timeout_ms,
out_value, false);
}
@ -128,7 +128,7 @@ static inline u32 intel_uncore_read_fw(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_read32(__compat_uncore_to_gt(uncore), reg);
return xe_mmio_read32(__compat_uncore_to_mmio(uncore), reg);
}
static inline void intel_uncore_write_fw(struct intel_uncore *uncore,
@ -136,7 +136,7 @@ static inline void intel_uncore_write_fw(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
xe_mmio_write32(__compat_uncore_to_gt(uncore), reg, val);
xe_mmio_write32(__compat_uncore_to_mmio(uncore), reg, val);
}
static inline u32 intel_uncore_read_notrace(struct intel_uncore *uncore,
@ -144,7 +144,7 @@ static inline u32 intel_uncore_read_notrace(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
return xe_mmio_read32(__compat_uncore_to_gt(uncore), reg);
return xe_mmio_read32(__compat_uncore_to_mmio(uncore), reg);
}
static inline void intel_uncore_write_notrace(struct intel_uncore *uncore,
@ -152,7 +152,7 @@ static inline void intel_uncore_write_notrace(struct intel_uncore *uncore,
{
struct xe_reg reg = XE_REG(i915_mmio_reg_offset(i915_reg));
xe_mmio_write32(__compat_uncore_to_gt(uncore), reg, val);
xe_mmio_write32(__compat_uncore_to_mmio(uncore), reg, val);
}
static inline void __iomem *intel_uncore_regs(struct intel_uncore *uncore)

95
drivers/gpu/drm/xe/display/xe_display.c
View File

@ -4,7 +4,7 @@
*/
#include "xe_display.h"
#include "regs/xe_regs.h"
#include "regs/xe_irq_regs.h"
#include <linux/fb.h>
@ -13,7 +13,6 @@
#include <uapi/drm/xe_drm.h>
#include "soc/intel_dram.h"
#include "i915_drv.h" /* FIXME: HAS_DISPLAY() depends on this */
#include "intel_acpi.h"
#include "intel_audio.h"
#include "intel_bw.h"
@ -34,7 +33,7 @@
static bool has_display(struct xe_device *xe)
{
return HAS_DISPLAY(xe);
return HAS_DISPLAY(&xe->display);
}
/**
@ -309,18 +308,7 @@ static void xe_display_flush_cleanup_work(struct xe_device *xe)
}
/* TODO: System and runtime suspend/resume sequences will be sanitized as a follow-up. */
void xe_display_pm_runtime_suspend(struct xe_device *xe)
{
if (!xe->info.probe_display)
return;
if (xe->d3cold.allowed)
xe_display_pm_suspend(xe, true);
intel_hpd_poll_enable(xe);
}
void xe_display_pm_suspend(struct xe_device *xe, bool runtime)
static void __xe_display_pm_suspend(struct xe_device *xe, bool runtime)
{
struct intel_display *display = &xe->display;
bool s2idle = suspend_to_idle();
@ -355,6 +343,52 @@ void xe_display_pm_suspend(struct xe_device *xe, bool runtime)
intel_dmc_suspend(xe);
}
void xe_display_pm_suspend(struct xe_device *xe)
{
__xe_display_pm_suspend(xe, false);
}
void xe_display_pm_shutdown(struct xe_device *xe)
{
struct intel_display *display = &xe->display;
if (!xe->info.probe_display)
return;
intel_power_domains_disable(xe);
intel_fbdev_set_suspend(&xe->drm, FBINFO_STATE_SUSPENDED, true);
if (has_display(xe)) {
drm_kms_helper_poll_disable(&xe->drm);
intel_display_driver_disable_user_access(xe);
intel_display_driver_suspend(xe);
}
xe_display_flush_cleanup_work(xe);
intel_dp_mst_suspend(xe);
intel_hpd_cancel_work(xe);
if (has_display(xe))
intel_display_driver_suspend_access(xe);
intel_encoder_suspend_all(display);
intel_encoder_shutdown_all(display);
intel_opregion_suspend(display, PCI_D3cold);
intel_dmc_suspend(xe);
}
void xe_display_pm_runtime_suspend(struct xe_device *xe)
{
if (!xe->info.probe_display)
return;
if (xe->d3cold.allowed)
__xe_display_pm_suspend(xe, true);
intel_hpd_poll_enable(xe);
}
void xe_display_pm_suspend_late(struct xe_device *xe)
{
bool s2idle = suspend_to_idle();
@ -366,15 +400,17 @@ void xe_display_pm_suspend_late(struct xe_device *xe)
intel_display_power_suspend_late(xe);
}
void xe_display_pm_runtime_resume(struct xe_device *xe)
void xe_display_pm_shutdown_late(struct xe_device *xe)
{
if (!xe->info.probe_display)
return;
intel_hpd_poll_disable(xe);
if (xe->d3cold.allowed)
xe_display_pm_resume(xe, true);
/*
* The only requirement is to reboot with display DC states disabled,
* for now leaving all display power wells in the INIT power domain
* enabled.
*/
intel_power_domains_driver_remove(xe);
}
void xe_display_pm_resume_early(struct xe_device *xe)
@ -387,7 +423,7 @@ void xe_display_pm_resume_early(struct xe_device *xe)
intel_power_domains_resume(xe);
}
void xe_display_pm_resume(struct xe_device *xe, bool runtime)
static void __xe_display_pm_resume(struct xe_device *xe, bool runtime)
{
struct intel_display *display = &xe->display;
@ -421,6 +457,23 @@ void xe_display_pm_resume(struct xe_device *xe, bool runtime)
intel_power_domains_enable(xe);
}
void xe_display_pm_resume(struct xe_device *xe)
{
__xe_display_pm_resume(xe, false);
}
void xe_display_pm_runtime_resume(struct xe_device *xe)
{
if (!xe->info.probe_display)
return;
intel_hpd_poll_disable(xe);
if (xe->d3cold.allowed)
__xe_display_pm_resume(xe, true);
}
static void display_device_remove(struct drm_device *dev, void *arg)
{
struct xe_device *xe = arg;

12
drivers/gpu/drm/xe/display/xe_display.h
View File

@ -34,10 +34,12 @@ void xe_display_irq_enable(struct xe_device *xe, u32 gu_misc_iir);
void xe_display_irq_reset(struct xe_device *xe);
void xe_display_irq_postinstall(struct xe_device *xe, struct xe_gt *gt);
void xe_display_pm_suspend(struct xe_device *xe, bool runtime);
void xe_display_pm_suspend(struct xe_device *xe);
void xe_display_pm_shutdown(struct xe_device *xe);
void xe_display_pm_suspend_late(struct xe_device *xe);
void xe_display_pm_shutdown_late(struct xe_device *xe);
void xe_display_pm_resume_early(struct xe_device *xe);
void xe_display_pm_resume(struct xe_device *xe, bool runtime);
void xe_display_pm_resume(struct xe_device *xe);
void xe_display_pm_runtime_suspend(struct xe_device *xe);
void xe_display_pm_runtime_resume(struct xe_device *xe);
@ -65,10 +67,12 @@ static inline void xe_display_irq_enable(struct xe_device *xe, u32 gu_misc_iir)
static inline void xe_display_irq_reset(struct xe_device *xe) {}
static inline void xe_display_irq_postinstall(struct xe_device *xe, struct xe_gt *gt) {}
static inline void xe_display_pm_suspend(struct xe_device *xe, bool runtime) {}
static inline void xe_display_pm_suspend(struct xe_device *xe) {}
static inline void xe_display_pm_shutdown(struct xe_device *xe) {}
static inline void xe_display_pm_suspend_late(struct xe_device *xe) {}
static inline void xe_display_pm_shutdown_late(struct xe_device *xe) {}
static inline void xe_display_pm_resume_early(struct xe_device *xe) {}
static inline void xe_display_pm_resume(struct xe_device *xe, bool runtime) {}
static inline void xe_display_pm_resume(struct xe_device *xe) {}
static inline void xe_display_pm_runtime_suspend(struct xe_device *xe) {}
static inline void xe_display_pm_runtime_resume(struct xe_device *xe) {}

9
drivers/gpu/drm/xe/display/xe_dsb_buffer.c
View File

@ -48,11 +48,12 @@ bool intel_dsb_buffer_create(struct intel_crtc *crtc, struct intel_dsb_buffer *d
if (!vma)
return false;
/* Set scanout flag for WC mapping */
obj = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe),
NULL, PAGE_ALIGN(size),
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
XE_BO_FLAG_GGTT);
XE_BO_FLAG_SCANOUT | XE_BO_FLAG_GGTT);
if (IS_ERR(obj)) {
kfree(vma);
return false;
@ -73,5 +74,9 @@ void intel_dsb_buffer_cleanup(struct intel_dsb_buffer *dsb_buf)
void intel_dsb_buffer_flush_map(struct intel_dsb_buffer *dsb_buf)
{
/* TODO: add xe specific flush_map() for dsb buffer object. */
/*
* The memory barrier here is to ensure coherency of DSB vs MMIO,
* both for weak ordering archs and discrete cards.
*/
xe_device_wmb(dsb_buf->vma->bo->tile->xe);
}

1
drivers/gpu/drm/xe/regs/xe_engine_regs.h
View File

@ -186,6 +186,7 @@
#define VDBOX_CGCTL3F10(base) XE_REG((base) + 0x3f10)
#define IECPUNIT_CLKGATE_DIS REG_BIT(22)
#define RAMDFTUNIT_CLKGATE_DIS REG_BIT(9)
#define VDBOX_CGCTL3F18(base) XE_REG((base) + 0x3f18)
#define ALNUNIT_CLKGATE_DIS REG_BIT(13)

76
drivers/gpu/drm/xe/regs/xe_gt_regs.h
View File

@ -286,6 +286,9 @@
#define GAMTLBVEBOX0_CLKGATE_DIS REG_BIT(16)
#define LTCDD_CLKGATE_DIS REG_BIT(10)
#define UNSLCGCTL9454 XE_REG(0x9454)
#define LSCFE_CLKGATE_DIS REG_BIT(4)
#define XEHP_SLICE_UNIT_LEVEL_CLKGATE XE_REG_MCR(0x94d4)
#define L3_CR2X_CLKGATE_DIS REG_BIT(17)
#define L3_CLKGATE_DIS REG_BIT(16)
@ -344,6 +347,14 @@
#define CTC_SOURCE_DIVIDE_LOGIC REG_BIT(0)
#define FORCEWAKE_RENDER XE_REG(0xa278)
#define POWERGATE_DOMAIN_STATUS XE_REG(0xa2a0)
#define MEDIA_SLICE3_AWAKE_STATUS REG_BIT(4)
#define MEDIA_SLICE2_AWAKE_STATUS REG_BIT(3)
#define MEDIA_SLICE1_AWAKE_STATUS REG_BIT(2)
#define RENDER_AWAKE_STATUS REG_BIT(1)
#define MEDIA_SLICE0_AWAKE_STATUS REG_BIT(0)
#define FORCEWAKE_MEDIA_VDBOX(n) XE_REG(0xa540 + (n) * 4)
#define FORCEWAKE_MEDIA_VEBOX(n) XE_REG(0xa560 + (n) * 4)
#define FORCEWAKE_GSC XE_REG(0xa618)
@ -393,8 +404,11 @@
#define XE2_GLOBAL_INVAL XE_REG(0xb404)
#define SCRATCH1LPFC XE_REG(0xb474)
#define EN_L3_RW_CCS_CACHE_FLUSH REG_BIT(0)
#define XE2LPM_L3SQCREG2 XE_REG_MCR(0xb604)
#define XE2LPM_L3SQCREG3 XE_REG_MCR(0xb608)
#define XE2LPM_SCRATCH3_LBCF XE_REG_MCR(0xb654)
#define XE2LPM_L3SQCREG2 XE_REG_MCR(0xb604)
@ -559,62 +573,6 @@
#define GT_PERF_STATUS XE_REG(0x1381b4)
#define VOLTAGE_MASK REG_GENMASK(10, 0)
/*
* Note: Interrupt registers 1900xx are VF accessible only until version 12.50.
* On newer platforms, VFs are using memory-based interrupts instead.
* However, for simplicity we keep this XE_REG_OPTION_VF tag intact.
*/
#define GT_INTR_DW(x) XE_REG(0x190018 + ((x) * 4), XE_REG_OPTION_VF)
#define INTR_GSC REG_BIT(31)
#define INTR_GUC REG_BIT(25)
#define INTR_MGUC REG_BIT(24)
#define INTR_BCS8 REG_BIT(23)
#define INTR_BCS(x) REG_BIT(15 - (x))
#define INTR_CCS(x) REG_BIT(4 + (x))
#define INTR_RCS0 REG_BIT(0)
#define INTR_VECS(x) REG_BIT(31 - (x))
#define INTR_VCS(x) REG_BIT(x)
#define RENDER_COPY_INTR_ENABLE XE_REG(0x190030, XE_REG_OPTION_VF)
#define VCS_VECS_INTR_ENABLE XE_REG(0x190034, XE_REG_OPTION_VF)
#define GUC_SG_INTR_ENABLE XE_REG(0x190038, XE_REG_OPTION_VF)
#define ENGINE1_MASK REG_GENMASK(31, 16)
#define ENGINE0_MASK REG_GENMASK(15, 0)
#define GPM_WGBOXPERF_INTR_ENABLE XE_REG(0x19003c, XE_REG_OPTION_VF)
#define GUNIT_GSC_INTR_ENABLE XE_REG(0x190044, XE_REG_OPTION_VF)
#define CCS_RSVD_INTR_ENABLE XE_REG(0x190048, XE_REG_OPTION_VF)
#define INTR_IDENTITY_REG(x) XE_REG(0x190060 + ((x) * 4), XE_REG_OPTION_VF)
#define INTR_DATA_VALID REG_BIT(31)
#define INTR_ENGINE_INSTANCE(x) REG_FIELD_GET(GENMASK(25, 20), x)
#define INTR_ENGINE_CLASS(x) REG_FIELD_GET(GENMASK(18, 16), x)
#define INTR_ENGINE_INTR(x) REG_FIELD_GET(GENMASK(15, 0), x)
#define OTHER_GUC_INSTANCE 0
#define OTHER_GSC_HECI2_INSTANCE 3
#define OTHER_GSC_INSTANCE 6
#define IIR_REG_SELECTOR(x) XE_REG(0x190070 + ((x) * 4), XE_REG_OPTION_VF)
#define RCS0_RSVD_INTR_MASK XE_REG(0x190090, XE_REG_OPTION_VF)
#define BCS_RSVD_INTR_MASK XE_REG(0x1900a0, XE_REG_OPTION_VF)
#define VCS0_VCS1_INTR_MASK XE_REG(0x1900a8, XE_REG_OPTION_VF)
#define VCS2_VCS3_INTR_MASK XE_REG(0x1900ac, XE_REG_OPTION_VF)
#define VECS0_VECS1_INTR_MASK XE_REG(0x1900d0, XE_REG_OPTION_VF)
#define HECI2_RSVD_INTR_MASK XE_REG(0x1900e4)
#define GUC_SG_INTR_MASK XE_REG(0x1900e8, XE_REG_OPTION_VF)
#define GPM_WGBOXPERF_INTR_MASK XE_REG(0x1900ec, XE_REG_OPTION_VF)
#define GUNIT_GSC_INTR_MASK XE_REG(0x1900f4, XE_REG_OPTION_VF)
#define CCS0_CCS1_INTR_MASK XE_REG(0x190100)
#define CCS2_CCS3_INTR_MASK XE_REG(0x190104)
#define XEHPC_BCS1_BCS2_INTR_MASK XE_REG(0x190110)
#define XEHPC_BCS3_BCS4_INTR_MASK XE_REG(0x190114)
#define XEHPC_BCS5_BCS6_INTR_MASK XE_REG(0x190118)
#define XEHPC_BCS7_BCS8_INTR_MASK XE_REG(0x19011c)
#define GT_WAIT_SEMAPHORE_INTERRUPT REG_BIT(11)
#define GT_CONTEXT_SWITCH_INTERRUPT REG_BIT(8)
#define GSC_ER_COMPLETE REG_BIT(5)
#define GT_RENDER_PIPECTL_NOTIFY_INTERRUPT REG_BIT(4)
#define GT_CS_MASTER_ERROR_INTERRUPT REG_BIT(3)
#define GT_RENDER_USER_INTERRUPT REG_BIT(0)
#define SFC_DONE(n) XE_REG(0x1cc000 + (n) * 0x1000)
#endif

1
drivers/gpu/drm/xe/regs/xe_guc_regs.h
View File

@ -84,6 +84,7 @@
#define HUC_LOADING_AGENT_GUC REG_BIT(1)
#define GUC_WOPCM_OFFSET_VALID REG_BIT(0)
#define GUC_MAX_IDLE_COUNT XE_REG(0xc3e4)
#define GUC_PMTIMESTAMP XE_REG(0xc3e8)
#define GUC_SEND_INTERRUPT XE_REG(0xc4c8)
#define GUC_SEND_TRIGGER REG_BIT(0)

82
drivers/gpu/drm/xe/regs/xe_irq_regs.h Normal file
View File

@ -0,0 +1,82 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2024 Intel Corporation
*/
#ifndef _XE_IRQ_REGS_H_
#define _XE_IRQ_REGS_H_
#include "regs/xe_reg_defs.h"
#define PCU_IRQ_OFFSET 0x444e0
#define GU_MISC_IRQ_OFFSET 0x444f0
#define GU_MISC_GSE REG_BIT(27)
#define DG1_MSTR_TILE_INTR XE_REG(0x190008)
#define DG1_MSTR_IRQ REG_BIT(31)
#define DG1_MSTR_TILE(t) REG_BIT(t)
#define GFX_MSTR_IRQ XE_REG(0x190010, XE_REG_OPTION_VF)
#define MASTER_IRQ REG_BIT(31)
#define GU_MISC_IRQ REG_BIT(29)
#define DISPLAY_IRQ REG_BIT(16)
#define GT_DW_IRQ(x) REG_BIT(x)
/*
* Note: Interrupt registers 1900xx are VF accessible only until version 12.50.
* On newer platforms, VFs are using memory-based interrupts instead.
* However, for simplicity we keep this XE_REG_OPTION_VF tag intact.
*/
#define GT_INTR_DW(x) XE_REG(0x190018 + ((x) * 4), XE_REG_OPTION_VF)
#define INTR_GSC REG_BIT(31)
#define INTR_GUC REG_BIT(25)
#define INTR_MGUC REG_BIT(24)
#define INTR_BCS8 REG_BIT(23)
#define INTR_BCS(x) REG_BIT(15 - (x))
#define INTR_CCS(x) REG_BIT(4 + (x))
#define INTR_RCS0 REG_BIT(0)
#define INTR_VECS(x) REG_BIT(31 - (x))
#define INTR_VCS(x) REG_BIT(x)
#define RENDER_COPY_INTR_ENABLE XE_REG(0x190030, XE_REG_OPTION_VF)
#define VCS_VECS_INTR_ENABLE XE_REG(0x190034, XE_REG_OPTION_VF)
#define GUC_SG_INTR_ENABLE XE_REG(0x190038, XE_REG_OPTION_VF)
#define ENGINE1_MASK REG_GENMASK(31, 16)
#define ENGINE0_MASK REG_GENMASK(15, 0)
#define GPM_WGBOXPERF_INTR_ENABLE XE_REG(0x19003c, XE_REG_OPTION_VF)
#define GUNIT_GSC_INTR_ENABLE XE_REG(0x190044, XE_REG_OPTION_VF)
#define CCS_RSVD_INTR_ENABLE XE_REG(0x190048, XE_REG_OPTION_VF)
#define INTR_IDENTITY_REG(x) XE_REG(0x190060 + ((x) * 4), XE_REG_OPTION_VF)
#define INTR_DATA_VALID REG_BIT(31)
#define INTR_ENGINE_INSTANCE(x) REG_FIELD_GET(GENMASK(25, 20), x)
#define INTR_ENGINE_CLASS(x) REG_FIELD_GET(GENMASK(18, 16), x)
#define INTR_ENGINE_INTR(x) REG_FIELD_GET(GENMASK(15, 0), x)
#define OTHER_GUC_INSTANCE 0
#define OTHER_GSC_HECI2_INSTANCE 3
#define OTHER_GSC_INSTANCE 6
#define IIR_REG_SELECTOR(x) XE_REG(0x190070 + ((x) * 4), XE_REG_OPTION_VF)
#define RCS0_RSVD_INTR_MASK XE_REG(0x190090, XE_REG_OPTION_VF)
#define BCS_RSVD_INTR_MASK XE_REG(0x1900a0, XE_REG_OPTION_VF)
#define VCS0_VCS1_INTR_MASK XE_REG(0x1900a8, XE_REG_OPTION_VF)
#define VCS2_VCS3_INTR_MASK XE_REG(0x1900ac, XE_REG_OPTION_VF)
#define VECS0_VECS1_INTR_MASK XE_REG(0x1900d0, XE_REG_OPTION_VF)
#define HECI2_RSVD_INTR_MASK XE_REG(0x1900e4)
#define GUC_SG_INTR_MASK XE_REG(0x1900e8, XE_REG_OPTION_VF)
#define GPM_WGBOXPERF_INTR_MASK XE_REG(0x1900ec, XE_REG_OPTION_VF)
#define GUNIT_GSC_INTR_MASK XE_REG(0x1900f4, XE_REG_OPTION_VF)
#define CCS0_CCS1_INTR_MASK XE_REG(0x190100)
#define CCS2_CCS3_INTR_MASK XE_REG(0x190104)
#define XEHPC_BCS1_BCS2_INTR_MASK XE_REG(0x190110)
#define XEHPC_BCS3_BCS4_INTR_MASK XE_REG(0x190114)
#define XEHPC_BCS5_BCS6_INTR_MASK XE_REG(0x190118)
#define XEHPC_BCS7_BCS8_INTR_MASK XE_REG(0x19011c)
#define GT_WAIT_SEMAPHORE_INTERRUPT REG_BIT(11)
#define GT_CONTEXT_SWITCH_INTERRUPT REG_BIT(8)
#define GSC_ER_COMPLETE REG_BIT(5)
#define GT_RENDER_PIPECTL_NOTIFY_INTERRUPT REG_BIT(4)
#define GT_CS_MASTER_ERROR_INTERRUPT REG_BIT(3)
#define GT_RENDER_USER_INTERRUPT REG_BIT(0)
#endif

2
drivers/gpu/drm/xe/regs/xe_reg_defs.h
View File

@ -128,7 +128,7 @@ struct xe_reg_mcr {
* options.
*/
#define XE_REG_MCR(r_, ...) ((const struct xe_reg_mcr){ \
.__reg = XE_REG_INITIALIZER(r_, ##__VA_ARGS__, .mcr = 1) \
.__reg = XE_REG_INITIALIZER(r_, ##__VA_ARGS__, .mcr = 1) \
})
static inline bool xe_reg_is_valid(struct xe_reg r)

14
drivers/gpu/drm/xe/regs/xe_regs.h
View File

@ -11,10 +11,6 @@
#define TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_MASK REG_GENMASK(15, 12)
#define TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_MASK REG_GENMASK(9, 0)
#define PCU_IRQ_OFFSET 0x444e0
#define GU_MISC_IRQ_OFFSET 0x444f0
#define GU_MISC_GSE REG_BIT(27)
#define GU_CNTL_PROTECTED XE_REG(0x10100C)
#define DRIVERINT_FLR_DIS REG_BIT(31)
@ -57,16 +53,6 @@
#define MTL_MPE_FREQUENCY XE_REG(0x13802c)
#define MTL_RPE_MASK REG_GENMASK(8, 0)
#define DG1_MSTR_TILE_INTR XE_REG(0x190008)
#define DG1_MSTR_IRQ REG_BIT(31)
#define DG1_MSTR_TILE(t) REG_BIT(t)
#define GFX_MSTR_IRQ XE_REG(0x190010, XE_REG_OPTION_VF)
#define MASTER_IRQ REG_BIT(31)
#define GU_MISC_IRQ REG_BIT(29)
#define DISPLAY_IRQ REG_BIT(16)
#define GT_DW_IRQ(x) REG_BIT(x)
#define VF_CAP_REG XE_REG(0x1901f8, XE_REG_OPTION_VF)
#define VF_CAP REG_BIT(0)

240
drivers/gpu/drm/xe/tests/xe_bo.c
View File

@ -6,6 +6,13 @@
#include <kunit/test.h>
#include <kunit/visibility.h>
#include <linux/iosys-map.h>
#include <linux/math64.h>
#include <linux/random.h>
#include <linux/swap.h>
#include <uapi/linux/sysinfo.h>
#include "tests/xe_kunit_helpers.h"
#include "tests/xe_pci_test.h"
#include "tests/xe_test.h"
@ -358,9 +365,242 @@ static void xe_bo_evict_kunit(struct kunit *test)
evict_test_run_device(xe);
}
struct xe_bo_link {
struct list_head link;
struct xe_bo *bo;
u32 val;
};
#define XE_BO_SHRINK_SIZE ((unsigned long)SZ_64M)
static int shrink_test_fill_random(struct xe_bo *bo, struct rnd_state *state,
struct xe_bo_link *link)
{
struct iosys_map map;
int ret = ttm_bo_vmap(&bo->ttm, &map);
size_t __maybe_unused i;
if (ret)
return ret;
for (i = 0; i < bo->ttm.base.size; i += sizeof(u32)) {
u32 val = prandom_u32_state(state);
iosys_map_wr(&map, i, u32, val);
if (i == 0)
link->val = val;
}
ttm_bo_vunmap(&bo->ttm, &map);
return 0;
}
static bool shrink_test_verify(struct kunit *test, struct xe_bo *bo,
unsigned int bo_nr, struct rnd_state *state,
struct xe_bo_link *link)
{
struct iosys_map map;
int ret = ttm_bo_vmap(&bo->ttm, &map);
size_t i;
bool failed = false;
if (ret) {
KUNIT_FAIL(test, "Error mapping bo %u for content check.\n", bo_nr);
return true;
}
for (i = 0; i < bo->ttm.base.size; i += sizeof(u32)) {
u32 val = prandom_u32_state(state);
if (iosys_map_rd(&map, i, u32) != val) {
KUNIT_FAIL(test, "Content not preserved, bo %u offset 0x%016llx",
bo_nr, (unsigned long long)i);
kunit_info(test, "Failed value is 0x%08x, recorded 0x%08x\n",
(unsigned int)iosys_map_rd(&map, i, u32), val);
if (i == 0 && val != link->val)
kunit_info(test, "Looks like PRNG is out of sync.\n");
failed = true;
break;
}
}
ttm_bo_vunmap(&bo->ttm, &map);
return failed;
}
/*
* Try to create system bos corresponding to twice the amount
* of available system memory to test shrinker functionality.
* If no swap space is available to accommodate the
* memory overcommit, mark bos purgeable.
*/
static int shrink_test_run_device(struct xe_device *xe)
{
struct kunit *test = kunit_get_current_test();
LIST_HEAD(bos);
struct xe_bo_link *link, *next;
struct sysinfo si;
u64 ram, ram_and_swap, purgeable = 0, alloced, to_alloc, limit;
unsigned int interrupted = 0, successful = 0, count = 0;
struct rnd_state prng;
u64 rand_seed;
bool failed = false;
rand_seed = get_random_u64();
prandom_seed_state(&prng, rand_seed);
kunit_info(test, "Random seed is 0x%016llx.\n",
(unsigned long long)rand_seed);
/* Skip if execution time is expected to be too long. */
limit = SZ_32G;
/* IGFX with flat CCS needs to copy when swapping / shrinking */
if (!IS_DGFX(xe) && xe_device_has_flat_ccs(xe))
limit = SZ_16G;
si_meminfo(&si);
ram = (size_t)si.freeram * si.mem_unit;
if (ram > limit) {
kunit_skip(test, "Too long expected execution time.\n");
return 0;
}
to_alloc = ram * 2;
ram_and_swap = ram + get_nr_swap_pages() * PAGE_SIZE;
if (to_alloc > ram_and_swap)
purgeable = to_alloc - ram_and_swap;
purgeable += div64_u64(purgeable, 5);
kunit_info(test, "Free ram is %lu bytes. Will allocate twice of that.\n",
(unsigned long)ram);
for (alloced = 0; alloced < to_alloc; alloced += XE_BO_SHRINK_SIZE) {
struct xe_bo *bo;
unsigned int mem_type;
struct xe_ttm_tt *xe_tt;
link = kzalloc(sizeof(*link), GFP_KERNEL);
if (!link) {
KUNIT_FAIL(test, "Unexpected link allocation failure\n");
failed = true;
break;
}
INIT_LIST_HEAD(&link->link);
/* We can create bos using WC caching here. But it is slower. */
bo = xe_bo_create_user(xe, NULL, NULL, XE_BO_SHRINK_SIZE,
DRM_XE_GEM_CPU_CACHING_WB,
XE_BO_FLAG_SYSTEM);
if (IS_ERR(bo)) {
if (bo != ERR_PTR(-ENOMEM) && bo != ERR_PTR(-ENOSPC) &&
bo != ERR_PTR(-EINTR) && bo != ERR_PTR(-ERESTARTSYS))
KUNIT_FAIL(test, "Error creating bo: %pe\n", bo);
kfree(link);
failed = true;
break;
}
xe_bo_lock(bo, false);
xe_tt = container_of(bo->ttm.ttm, typeof(*xe_tt), ttm);
/*
* Allocate purgeable bos first, because if we do it the
* other way around, they may not be subject to swapping...
*/
if (alloced < purgeable) {
xe_tt->purgeable = true;
bo->ttm.priority = 0;
} else {
int ret = shrink_test_fill_random(bo, &prng, link);
if (ret) {
xe_bo_unlock(bo);
xe_bo_put(bo);
KUNIT_FAIL(test, "Error filling bo with random data: %pe\n",
ERR_PTR(ret));
kfree(link);
failed = true;
break;
}
}
mem_type = bo->ttm.resource->mem_type;
xe_bo_unlock(bo);
link->bo = bo;
list_add_tail(&link->link, &bos);
if (mem_type != XE_PL_TT) {
KUNIT_FAIL(test, "Bo in incorrect memory type: %u\n",
bo->ttm.resource->mem_type);
failed = true;
}
cond_resched();
if (signal_pending(current))
break;
}
/*
* Read back and destroy bos. Reset the pseudo-random seed to get an
* identical pseudo-random number sequence for readback.
*/
prandom_seed_state(&prng, rand_seed);
list_for_each_entry_safe(link, next, &bos, link) {
static struct ttm_operation_ctx ctx = {.interruptible = true};
struct xe_bo *bo = link->bo;
struct xe_ttm_tt *xe_tt;
int ret;
count++;
if (!signal_pending(current) && !failed) {
bool purgeable, intr = false;
xe_bo_lock(bo, NULL);
/* xe_tt->purgeable is cleared on validate. */
xe_tt = container_of(bo->ttm.ttm, typeof(*xe_tt), ttm);
purgeable = xe_tt->purgeable;
do {
ret = ttm_bo_validate(&bo->ttm, &tt_placement, &ctx);
if (ret == -EINTR)
intr = true;
} while (ret == -EINTR && !signal_pending(current));
if (!ret && !purgeable)
failed = shrink_test_verify(test, bo, count, &prng, link);
xe_bo_unlock(bo);
if (ret) {
KUNIT_FAIL(test, "Validation failed: %pe\n",
ERR_PTR(ret));
failed = true;
} else if (intr) {
interrupted++;
} else {
successful++;
}
}
xe_bo_put(link->bo);
list_del(&link->link);
kfree(link);
}
kunit_info(test, "Readbacks interrupted: %u successful: %u\n",
interrupted, successful);
return 0;
}
static void xe_bo_shrink_kunit(struct kunit *test)
{
struct xe_device *xe = test->priv;
shrink_test_run_device(xe);
}
static struct kunit_case xe_bo_tests[] = {
KUNIT_CASE_PARAM(xe_ccs_migrate_kunit, xe_pci_live_device_gen_param),
KUNIT_CASE_PARAM(xe_bo_evict_kunit, xe_pci_live_device_gen_param),
KUNIT_CASE_PARAM_ATTR(xe_bo_shrink_kunit, xe_pci_live_device_gen_param,
{.speed = KUNIT_SPEED_SLOW}),
{}
};

4
drivers/gpu/drm/xe/tests/xe_mocs.c
View File

@ -55,7 +55,7 @@ static void read_l3cc_table(struct xe_gt *gt,
if (regs_are_mcr(gt))
reg_val = xe_gt_mcr_unicast_read_any(gt, XEHP_LNCFCMOCS(i >> 1));
else
reg_val = xe_mmio_read32(gt, XELP_LNCFCMOCS(i >> 1));
reg_val = xe_mmio_read32(&gt->mmio, XELP_LNCFCMOCS(i >> 1));
mocs_dbg(gt, "reg_val=0x%x\n", reg_val);
} else {
@ -94,7 +94,7 @@ static void read_mocs_table(struct xe_gt *gt,
if (regs_are_mcr(gt))
reg_val = xe_gt_mcr_unicast_read_any(gt, XEHP_GLOBAL_MOCS(i));
else
reg_val = xe_mmio_read32(gt, XELP_GLOBAL_MOCS(i));
reg_val = xe_mmio_read32(&gt->mmio, XELP_GLOBAL_MOCS(i));
mocs_expected = get_entry_control(info, i);
mocs = reg_val;

2
drivers/gpu/drm/xe/xe_assert.h
View File

@ -10,7 +10,7 @@
#include <drm/drm_print.h>
#include "xe_device_types.h"
#include "xe_gt_types.h"
#include "xe_step.h"
/**

34
drivers/gpu/drm/xe/xe_bo.c
View File

@ -283,6 +283,8 @@ struct xe_ttm_tt {
struct device *dev;
struct sg_table sgt;
struct sg_table *sg;
/** @purgeable: Whether the content of the pages of @ttm is purgeable. */
bool purgeable;
};
static int xe_tt_map_sg(struct ttm_tt *tt)
@ -468,7 +470,7 @@ static int xe_ttm_io_mem_reserve(struct ttm_device *bdev,
mem->bus.offset += vram->io_start;
mem->bus.is_iomem = true;
#if !defined(CONFIG_X86)
#if !IS_ENABLED(CONFIG_X86)
mem->bus.caching = ttm_write_combined;
#endif
return 0;
@ -761,7 +763,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
if (xe_rpm_reclaim_safe(xe)) {
/*
* We might be called through swapout in the validation path of
* another TTM device, so unconditionally acquire rpm here.
* another TTM device, so acquire rpm here.
*/
xe_pm_runtime_get(xe);
} else {
@ -1082,6 +1084,33 @@ static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo)
}
}
static void xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operation_ctx *ctx)
{
struct xe_device *xe = ttm_to_xe_device(ttm_bo->bdev);
if (ttm_bo->ttm) {
struct ttm_placement place = {};
int ret = ttm_bo_validate(ttm_bo, &place, ctx);
drm_WARN_ON(&xe->drm, ret);
}
}
static void xe_ttm_bo_swap_notify(struct ttm_buffer_object *ttm_bo)
{
struct ttm_operation_ctx ctx = {
.interruptible = false
};
if (ttm_bo->ttm) {
struct xe_ttm_tt *xe_tt =
container_of(ttm_bo->ttm, struct xe_ttm_tt, ttm);
if (xe_tt->purgeable)
xe_ttm_bo_purge(ttm_bo, &ctx);
}
}
const struct ttm_device_funcs xe_ttm_funcs = {
.ttm_tt_create = xe_ttm_tt_create,
.ttm_tt_populate = xe_ttm_tt_populate,
@ -1094,6 +1123,7 @@ const struct ttm_device_funcs xe_ttm_funcs = {
.release_notify = xe_ttm_bo_release_notify,
.eviction_valuable = ttm_bo_eviction_valuable,
.delete_mem_notify = xe_ttm_bo_delete_mem_notify,
.swap_notify = xe_ttm_bo_swap_notify,
};
static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo)

2
drivers/gpu/drm/xe/xe_debugfs.c
View File

@ -187,7 +187,7 @@ void xe_debugfs_register(struct xe_device *xe)
debugfs_create_file("forcewake_all", 0400, root, xe,
&forcewake_all_fops);
debugfs_create_file("wedged_mode", 0400, root, xe,
debugfs_create_file("wedged_mode", 0600, root, xe,
&wedged_mode_fops);
for (mem_type = XE_PL_VRAM0; mem_type <= XE_PL_VRAM1; ++mem_type) {

157
drivers/gpu/drm/xe/xe_devcoredump.c
View File

@ -6,6 +6,7 @@
#include "xe_devcoredump.h"
#include "xe_devcoredump_types.h"
#include <linux/ascii85.h>
#include <linux/devcoredump.h>
#include <generated/utsrelease.h>
@ -16,9 +17,12 @@
#include "xe_force_wake.h"
#include "xe_gt.h"
#include "xe_gt_printk.h"
#include "xe_guc_capture.h"
#include "xe_guc_ct.h"
#include "xe_guc_log.h"
#include "xe_guc_submit.h"
#include "xe_hw_engine.h"
#include "xe_module.h"
#include "xe_sched_job.h"
#include "xe_vm.h"
@ -85,9 +89,9 @@ static ssize_t __xe_devcoredump_read(char *buffer, size_t count,
p = drm_coredump_printer(&iter);
drm_printf(&p, "**** Xe Device Coredump ****\n");
drm_printf(&p, "kernel: " UTS_RELEASE "\n");
drm_printf(&p, "module: " KBUILD_MODNAME "\n");
drm_puts(&p, "**** Xe Device Coredump ****\n");
drm_puts(&p, "kernel: " UTS_RELEASE "\n");
drm_puts(&p, "module: " KBUILD_MODNAME "\n");
ts = ktime_to_timespec64(ss->snapshot_time);
drm_printf(&p, "Snapshot time: %lld.%09ld\n", ts.tv_sec, ts.tv_nsec);
@ -96,20 +100,27 @@ static ssize_t __xe_devcoredump_read(char *buffer, size_t count,
drm_printf(&p, "Process: %s\n", ss->process_name);
xe_device_snapshot_print(xe, &p);
drm_printf(&p, "\n**** GuC CT ****\n");
xe_guc_ct_snapshot_print(coredump->snapshot.ct, &p);
xe_guc_exec_queue_snapshot_print(coredump->snapshot.ge, &p);
drm_printf(&p, "\n**** GT #%d ****\n", ss->gt->info.id);
drm_printf(&p, "\tTile: %d\n", ss->gt->tile->id);
drm_printf(&p, "\n**** Job ****\n");
xe_sched_job_snapshot_print(coredump->snapshot.job, &p);
drm_puts(&p, "\n**** GuC Log ****\n");
xe_guc_log_snapshot_print(ss->guc.log, &p);
drm_puts(&p, "\n**** GuC CT ****\n");
xe_guc_ct_snapshot_print(ss->guc.ct, &p);
drm_printf(&p, "\n**** HW Engines ****\n");
drm_puts(&p, "\n**** Contexts ****\n");
xe_guc_exec_queue_snapshot_print(ss->ge, &p);
drm_puts(&p, "\n**** Job ****\n");
xe_sched_job_snapshot_print(ss->job, &p);
drm_puts(&p, "\n**** HW Engines ****\n");
for (i = 0; i < XE_NUM_HW_ENGINES; i++)
if (coredump->snapshot.hwe[i])
xe_hw_engine_snapshot_print(coredump->snapshot.hwe[i],
&p);
drm_printf(&p, "\n**** VM state ****\n");
xe_vm_snapshot_print(coredump->snapshot.vm, &p);
if (ss->hwe[i])
xe_engine_snapshot_print(ss->hwe[i], &p);
drm_puts(&p, "\n**** VM state ****\n");
xe_vm_snapshot_print(ss->vm, &p);
return count - iter.remain;
}
@ -118,8 +129,14 @@ static void xe_devcoredump_snapshot_free(struct xe_devcoredump_snapshot *ss)
{
int i;
xe_guc_ct_snapshot_free(ss->ct);
ss->ct = NULL;
xe_guc_log_snapshot_free(ss->guc.log);
ss->guc.log = NULL;
xe_guc_ct_snapshot_free(ss->guc.ct);
ss->guc.ct = NULL;
xe_guc_capture_put_matched_nodes(&ss->gt->uc.guc);
ss->matched_node = NULL;
xe_guc_exec_queue_snapshot_free(ss->ge);
ss->ge = NULL;
@ -204,6 +221,7 @@ static void xe_devcoredump_free(void *data)
/* To prevent stale data on next snapshot, clear everything */
memset(&coredump->snapshot, 0, sizeof(coredump->snapshot));
coredump->captured = false;
coredump->job = NULL;
drm_info(&coredump_to_xe(coredump)->drm,
"Xe device coredump has been deleted.\n");
}
@ -214,8 +232,6 @@ static void devcoredump_snapshot(struct xe_devcoredump *coredump,
struct xe_devcoredump_snapshot *ss = &coredump->snapshot;
struct xe_exec_queue *q = job->q;
struct xe_guc *guc = exec_queue_to_guc(q);
struct xe_hw_engine *hwe;
enum xe_hw_engine_id id;
u32 adj_logical_mask = q->logical_mask;
u32 width_mask = (0x1 << q->width) - 1;
const char *process_name = "no process";
@ -231,6 +247,7 @@ static void devcoredump_snapshot(struct xe_devcoredump *coredump,
strscpy(ss->process_name, process_name);
ss->gt = q->gt;
coredump->job = job;
INIT_WORK(&ss->work, xe_devcoredump_deferred_snap_work);
cookie = dma_fence_begin_signalling();
@ -247,19 +264,13 @@ static void devcoredump_snapshot(struct xe_devcoredump *coredump,
if (xe_force_wake_get(gt_to_fw(q->gt), XE_FORCEWAKE_ALL))
xe_gt_info(ss->gt, "failed to get forcewake for coredump capture\n");
coredump->snapshot.ct = xe_guc_ct_snapshot_capture(&guc->ct, true);
coredump->snapshot.ge = xe_guc_exec_queue_snapshot_capture(q);
coredump->snapshot.job = xe_sched_job_snapshot_capture(job);
coredump->snapshot.vm = xe_vm_snapshot_capture(q->vm);
ss->guc.log = xe_guc_log_snapshot_capture(&guc->log, true);
ss->guc.ct = xe_guc_ct_snapshot_capture(&guc->ct, true);
ss->ge = xe_guc_exec_queue_snapshot_capture(q);
ss->job = xe_sched_job_snapshot_capture(job);
ss->vm = xe_vm_snapshot_capture(q->vm);
for_each_hw_engine(hwe, q->gt, id) {
if (hwe->class != q->hwe->class ||
!(BIT(hwe->logical_instance) & adj_logical_mask)) {
coredump->snapshot.hwe[id] = NULL;
continue;
}
coredump->snapshot.hwe[id] = xe_hw_engine_snapshot_capture(hwe);
}
xe_engine_snapshot_capture_for_job(job);
queue_work(system_unbound_wq, &ss->work);
@ -310,3 +321,89 @@ int xe_devcoredump_init(struct xe_device *xe)
}
#endif
/**
* xe_print_blob_ascii85 - print a BLOB to some useful location in ASCII85
*
* The output is split to multiple lines because some print targets, e.g. dmesg
* cannot handle arbitrarily long lines. Note also that printing to dmesg in
* piece-meal fashion is not possible, each separate call to drm_puts() has a
* line-feed automatically added! Therefore, the entire output line must be
* constructed in a local buffer first, then printed in one atomic output call.
*
* There is also a scheduler yield call to prevent the 'task has been stuck for
* 120s' kernel hang check feature from firing when printing to a slow target
* such as dmesg over a serial port.
*
* TODO: Add compression prior to the ASCII85 encoding to shrink huge buffers down.
*
* @p: the printer object to output to
* @prefix: optional prefix to add to output string
* @blob: the Binary Large OBject to dump out
* @offset: offset in bytes to skip from the front of the BLOB, must be a multiple of sizeof(u32)
* @size: the size in bytes of the BLOB, must be a multiple of sizeof(u32)
*/
void xe_print_blob_ascii85(struct drm_printer *p, const char *prefix,
const void *blob, size_t offset, size_t size)
{
const u32 *blob32 = (const u32 *)blob;
char buff[ASCII85_BUFSZ], *line_buff;
size_t line_pos = 0;
#define DMESG_MAX_LINE_LEN 800
#define MIN_SPACE (ASCII85_BUFSZ + 2) /* 85 + "\n\0" */
if (size & 3)
drm_printf(p, "Size not word aligned: %zu", size);
if (offset & 3)
drm_printf(p, "Offset not word aligned: %zu", size);
line_buff = kzalloc(DMESG_MAX_LINE_LEN, GFP_KERNEL);
if (IS_ERR_OR_NULL(line_buff)) {
drm_printf(p, "Failed to allocate line buffer: %pe", line_buff);
return;
}
blob32 += offset / sizeof(*blob32);
size /= sizeof(*blob32);
if (prefix) {
strscpy(line_buff, prefix, DMESG_MAX_LINE_LEN - MIN_SPACE - 2);
line_pos = strlen(line_buff);
line_buff[line_pos++] = ':';
line_buff[line_pos++] = ' ';
}
while (size--) {
u32 val = *(blob32++);
strscpy(line_buff + line_pos, ascii85_encode(val, buff),
DMESG_MAX_LINE_LEN - line_pos);
line_pos += strlen(line_buff + line_pos);
if ((line_pos + MIN_SPACE) >= DMESG_MAX_LINE_LEN) {
line_buff[line_pos++] = '\n';
line_buff[line_pos++] = 0;
drm_puts(p, line_buff);
line_pos = 0;
/* Prevent 'stuck thread' time out errors */
cond_resched();
}
}
if (line_pos) {
line_buff[line_pos++] = '\n';
line_buff[line_pos++] = 0;
drm_puts(p, line_buff);
}
kfree(line_buff);
#undef MIN_SPACE
#undef DMESG_MAX_LINE_LEN
}

6
drivers/gpu/drm/xe/xe_devcoredump.h
View File

@ -6,6 +6,9 @@
#ifndef _XE_DEVCOREDUMP_H_
#define _XE_DEVCOREDUMP_H_
#include <linux/types.h>
struct drm_printer;
struct xe_device;
struct xe_sched_job;
@ -23,4 +26,7 @@ static inline int xe_devcoredump_init(struct xe_device *xe)
}
#endif
void xe_print_blob_ascii85(struct drm_printer *p, const char *prefix,
const void *blob, size_t offset, size_t size);
#endif

21
drivers/gpu/drm/xe/xe_devcoredump_types.h
View File

@ -34,16 +34,27 @@ struct xe_devcoredump_snapshot {
/** @work: Workqueue for deferred capture outside of signaling context */
struct work_struct work;
/* GuC snapshots */
/** @ct: GuC CT snapshot */
struct xe_guc_ct_snapshot *ct;
/** @ge: Guc Engine snapshot */
/** @guc: GuC snapshots */
struct {
/** @guc.ct: GuC CT snapshot */
struct xe_guc_ct_snapshot *ct;
/** @guc.log: GuC log snapshot */
struct xe_guc_log_snapshot *log;
} guc;
/** @ge: GuC Submission Engine snapshot */
struct xe_guc_submit_exec_queue_snapshot *ge;
/** @hwe: HW Engine snapshot array */
struct xe_hw_engine_snapshot *hwe[XE_NUM_HW_ENGINES];
/** @job: Snapshot of job state */
struct xe_sched_job_snapshot *job;
/**
* @matched_node: The matched capture node for timedout job
* this single-node tracker works because devcoredump will always only
* produce one hw-engine capture per devcoredump event
*/
struct __guc_capture_parsed_output *matched_node;
/** @vm: Snapshot of VM state */
struct xe_vm_snapshot *vm;
@ -69,6 +80,8 @@ struct xe_devcoredump {
bool captured;
/** @snapshot: Snapshot is captured at time of the first crash */
struct xe_devcoredump_snapshot snapshot;
/** @job: Point to the faulting job */
struct xe_sched_job *job;
};
#endif

106
drivers/gpu/drm/xe/xe_device.c
View File

@ -6,6 +6,7 @@
#include "xe_device.h"
#include <linux/delay.h>
#include <linux/fault-inject.h>
#include <linux/units.h>
#include <drm/drm_aperture.h>
@ -383,6 +384,12 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
err:
return ERR_PTR(err);
}
ALLOW_ERROR_INJECTION(xe_device_create, ERRNO); /* See xe_pci_probe() */
static bool xe_driver_flr_disabled(struct xe_device *xe)
{
return xe_mmio_read32(xe_root_tile_mmio(xe), GU_CNTL_PROTECTED) & DRIVERINT_FLR_DIS;
}
/*
* The driver-initiated FLR is the highest level of reset that we can trigger
@ -397,17 +404,12 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
* if/when a new instance of i915 is bound to the device it will do a full
* re-init anyway.
*/
static void xe_driver_flr(struct xe_device *xe)
static void __xe_driver_flr(struct xe_device *xe)
{
const unsigned int flr_timeout = 3 * MICRO; /* specs recommend a 3s wait */
struct xe_gt *gt = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
int ret;
if (xe_mmio_read32(gt, GU_CNTL_PROTECTED) & DRIVERINT_FLR_DIS) {
drm_info_once(&xe->drm, "BIOS Disabled Driver-FLR\n");
return;
}
drm_dbg(&xe->drm, "Triggering Driver-FLR\n");
/*
@ -419,25 +421,25 @@ static void xe_driver_flr(struct xe_device *xe)
* is still pending (unless the HW is totally dead), but better to be
* safe in case something unexpected happens
*/
ret = xe_mmio_wait32(gt, GU_CNTL, DRIVERFLR, 0, flr_timeout, NULL, false);
ret = xe_mmio_wait32(mmio, GU_CNTL, DRIVERFLR, 0, flr_timeout, NULL, false);
if (ret) {
drm_err(&xe->drm, "Driver-FLR-prepare wait for ready failed! %d\n", ret);
return;
}
xe_mmio_write32(gt, GU_DEBUG, DRIVERFLR_STATUS);
xe_mmio_write32(mmio, GU_DEBUG, DRIVERFLR_STATUS);
/* Trigger the actual Driver-FLR */
xe_mmio_rmw32(gt, GU_CNTL, 0, DRIVERFLR);
xe_mmio_rmw32(mmio, GU_CNTL, 0, DRIVERFLR);
/* Wait for hardware teardown to complete */
ret = xe_mmio_wait32(gt, GU_CNTL, DRIVERFLR, 0, flr_timeout, NULL, false);
ret = xe_mmio_wait32(mmio, GU_CNTL, DRIVERFLR, 0, flr_timeout, NULL, false);
if (ret) {
drm_err(&xe->drm, "Driver-FLR-teardown wait completion failed! %d\n", ret);
return;
}
/* Wait for hardware/firmware re-init to complete */
ret = xe_mmio_wait32(gt, GU_DEBUG, DRIVERFLR_STATUS, DRIVERFLR_STATUS,
ret = xe_mmio_wait32(mmio, GU_DEBUG, DRIVERFLR_STATUS, DRIVERFLR_STATUS,
flr_timeout, NULL, false);
if (ret) {
drm_err(&xe->drm, "Driver-FLR-reinit wait completion failed! %d\n", ret);
@ -445,7 +447,17 @@ static void xe_driver_flr(struct xe_device *xe)
}
/* Clear sticky completion status */
xe_mmio_write32(gt, GU_DEBUG, DRIVERFLR_STATUS);
xe_mmio_write32(mmio, GU_DEBUG, DRIVERFLR_STATUS);
}
static void xe_driver_flr(struct xe_device *xe)
{
if (xe_driver_flr_disabled(xe)) {
drm_info_once(&xe->drm, "BIOS Disabled Driver-FLR\n");
return;
}
__xe_driver_flr(xe);
}
static void xe_driver_flr_fini(void *arg)
@ -488,16 +500,15 @@ static int xe_set_dma_info(struct xe_device *xe)
return err;
}
static bool verify_lmem_ready(struct xe_gt *gt)
static bool verify_lmem_ready(struct xe_device *xe)
{
u32 val = xe_mmio_read32(gt, GU_CNTL) & LMEM_INIT;
u32 val = xe_mmio_read32(xe_root_tile_mmio(xe), GU_CNTL) & LMEM_INIT;
return !!val;
}
static int wait_for_lmem_ready(struct xe_device *xe)
{
struct xe_gt *gt = xe_root_mmio_gt(xe);
unsigned long timeout, start;
if (!IS_DGFX(xe))
@ -506,7 +517,7 @@ static int wait_for_lmem_ready(struct xe_device *xe)
if (IS_SRIOV_VF(xe))
return 0;
if (verify_lmem_ready(gt))
if (verify_lmem_ready(xe))
return 0;
drm_dbg(&xe->drm, "Waiting for lmem initialization\n");
@ -535,13 +546,14 @@ static int wait_for_lmem_ready(struct xe_device *xe)
msleep(20);
} while (!verify_lmem_ready(gt));
} while (!verify_lmem_ready(xe));
drm_dbg(&xe->drm, "lmem ready after %ums",
jiffies_to_msecs(jiffies - start));
return 0;
}
ALLOW_ERROR_INJECTION(wait_for_lmem_ready, ERRNO); /* See xe_pci_probe() */
static void update_device_info(struct xe_device *xe)
{
@ -589,15 +601,17 @@ int xe_device_probe_early(struct xe_device *xe)
return 0;
}
static int xe_device_set_has_flat_ccs(struct xe_device *xe)
static int probe_has_flat_ccs(struct xe_device *xe)
{
struct xe_gt *gt;
u32 reg;
int err;
/* Always enabled/disabled, no runtime check to do */
if (GRAPHICS_VER(xe) < 20 || !xe->info.has_flat_ccs)
return 0;
struct xe_gt *gt = xe_root_mmio_gt(xe);
gt = xe_root_mmio_gt(xe);
err = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
if (err)
@ -646,6 +660,13 @@ int xe_device_probe(struct xe_device *xe)
err = xe_gt_init_early(gt);
if (err)
return err;
/*
* Only after this point can GT-specific MMIO operations
* (including things like communication with the GuC)
* be performed.
*/
xe_gt_mmio_init(gt);
}
for_each_tile(tile, xe, id) {
@ -661,11 +682,9 @@ int xe_device_probe(struct xe_device *xe)
err = xe_ggtt_init_early(tile->mem.ggtt);
if (err)
return err;
if (IS_SRIOV_VF(xe)) {
err = xe_memirq_init(&tile->sriov.vf.memirq);
if (err)
return err;
}
err = xe_memirq_init(&tile->memirq);
if (err)
return err;
}
for_each_gt(gt, xe, id) {
@ -689,7 +708,7 @@ int xe_device_probe(struct xe_device *xe)
if (err)
goto err;
err = xe_device_set_has_flat_ccs(xe);
err = probe_has_flat_ccs(xe);
if (err)
goto err;
@ -799,6 +818,24 @@ void xe_device_remove(struct xe_device *xe)
void xe_device_shutdown(struct xe_device *xe)
{
struct xe_gt *gt;
u8 id;
drm_dbg(&xe->drm, "Shutting down device\n");
if (xe_driver_flr_disabled(xe)) {
xe_display_pm_shutdown(xe);
xe_irq_suspend(xe);
for_each_gt(gt, xe, id)
xe_gt_shutdown(gt);
xe_display_pm_shutdown_late(xe);
} else {
/* BOOM! */
__xe_driver_flr(xe);
}
}
/**
@ -812,11 +849,9 @@ void xe_device_shutdown(struct xe_device *xe)
*/
void xe_device_wmb(struct xe_device *xe)
{
struct xe_gt *gt = xe_root_mmio_gt(xe);
wmb();
if (IS_DGFX(xe))
xe_mmio_write32(gt, VF_CAP_REG, 0);
xe_mmio_write32(xe_root_tile_mmio(xe), VF_CAP_REG, 0);
}
/**
@ -857,7 +892,7 @@ void xe_device_td_flush(struct xe_device *xe)
if (xe_force_wake_get(gt_to_fw(gt), XE_FW_GT))
return;
xe_mmio_write32(gt, XE2_TDF_CTRL, TRANSIENT_FLUSH_REQUEST);
xe_mmio_write32(&gt->mmio, XE2_TDF_CTRL, TRANSIENT_FLUSH_REQUEST);
/*
* FIXME: We can likely do better here with our choice of
* timeout. Currently we just assume the worst case, i.e. 150us,
@ -865,7 +900,7 @@ void xe_device_td_flush(struct xe_device *xe)
* scenario on current platforms if all cache entries are
* transient and need to be flushed..
*/
if (xe_mmio_wait32(gt, XE2_TDF_CTRL, TRANSIENT_FLUSH_REQUEST, 0,
if (xe_mmio_wait32(&gt->mmio, XE2_TDF_CTRL, TRANSIENT_FLUSH_REQUEST, 0,
150, NULL, false))
xe_gt_err_once(gt, "TD flush timeout\n");
@ -888,9 +923,9 @@ void xe_device_l2_flush(struct xe_device *xe)
return;
spin_lock(&gt->global_invl_lock);
xe_mmio_write32(gt, XE2_GLOBAL_INVAL, 0x1);
xe_mmio_write32(&gt->mmio, XE2_GLOBAL_INVAL, 0x1);
if (xe_mmio_wait32(gt, XE2_GLOBAL_INVAL, 0x1, 0x0, 150, NULL, true))
if (xe_mmio_wait32(&gt->mmio, XE2_GLOBAL_INVAL, 0x1, 0x0, 150, NULL, true))
xe_gt_err_once(gt, "Global invalidation timeout\n");
spin_unlock(&gt->global_invl_lock);
@ -929,6 +964,7 @@ void xe_device_snapshot_print(struct xe_device *xe, struct drm_printer *p)
for_each_gt(gt, xe, id) {
drm_printf(p, "GT id: %u\n", id);
drm_printf(p, "\tTile: %u\n", gt->tile->id);
drm_printf(p, "\tType: %s\n",
gt->info.type == XE_GT_TYPE_MAIN ? "main" : "media");
drm_printf(p, "\tIP ver: %u.%u.%u\n",
@ -980,13 +1016,13 @@ void xe_device_declare_wedged(struct xe_device *xe)
return;
}
xe_pm_runtime_get_noresume(xe);
if (drmm_add_action_or_reset(&xe->drm, xe_device_wedged_fini, xe)) {
drm_err(&xe->drm, "Failed to register xe_device_wedged_fini clean-up. Although device is wedged.\n");
return;
}
xe_pm_runtime_get_noresume(xe);
if (!atomic_xchg(&xe->wedged.flag, 1)) {
xe->needs_flr_on_fini = true;
drm_err(&xe->drm,

15
drivers/gpu/drm/xe/xe_device.h
View File

@ -9,6 +9,8 @@
#include <drm/drm_util.h>
#include "xe_device_types.h"
#include "xe_gt_types.h"
#include "xe_sriov.h"
static inline struct xe_device *to_xe_device(const struct drm_device *dev)
{
@ -138,7 +140,7 @@ static inline bool xe_device_uc_enabled(struct xe_device *xe)
static inline struct xe_force_wake *gt_to_fw(struct xe_gt *gt)
{
return &gt->mmio.fw;
return &gt->pm.fw;
}
void xe_device_assert_mem_access(struct xe_device *xe);
@ -153,11 +155,22 @@ static inline bool xe_device_has_sriov(struct xe_device *xe)
return xe->info.has_sriov;
}
static inline bool xe_device_has_msix(struct xe_device *xe)
{
/* TODO: change this when MSI-X support is fully integrated */
return false;
}
static inline bool xe_device_has_memirq(struct xe_device *xe)
{
return GRAPHICS_VERx100(xe) >= 1250;
}
static inline bool xe_device_uses_memirq(struct xe_device *xe)
{
return xe_device_has_memirq(xe) && (IS_SRIOV_VF(xe) || xe_device_has_msix(xe));
}
u32 xe_device_ccs_bytes(struct xe_device *xe, u64 size);
void xe_device_snapshot_print(struct xe_device *xe, struct drm_printer *p);

62
drivers/gpu/drm/xe/xe_device_types.h
View File

@ -14,7 +14,6 @@
#include "xe_devcoredump_types.h"
#include "xe_heci_gsc.h"
#include "xe_gt_types.h"
#include "xe_lmtt_types.h"
#include "xe_memirq_types.h"
#include "xe_oa.h"
@ -107,6 +106,45 @@ struct xe_mem_region {
void __iomem *mapping;
};
/**
* struct xe_mmio - register mmio structure
*
* Represents an MMIO region that the CPU may use to access registers. A
* region may share its IO map with other regions (e.g., all GTs within a
* tile share the same map with their parent tile, but represent different
* subregions of the overall IO space).
*/
struct xe_mmio {
/** @tile: Backpointer to tile, used for tracing */
struct xe_tile *tile;
/** @regs: Map used to access registers. */
void __iomem *regs;
/**
* @sriov_vf_gt: Backpointer to GT.
*
* This pointer is only set for GT MMIO regions and only when running
* as an SRIOV VF structure
*/
struct xe_gt *sriov_vf_gt;
/**
* @regs_size: Length of the register region within the map.
*
* The size of the iomap set in *regs is generally larger than the
* register mmio space since it includes unused regions and/or
* non-register regions such as the GGTT PTEs.
*/
size_t regs_size;
/** @adj_limit: adjust MMIO address if address is below this value */
u32 adj_limit;
/** @adj_offset: offset to add to MMIO address when adjusting */
u32 adj_offset;
};
/**
* struct xe_tile - hardware tile structure
*
@ -148,26 +186,14 @@ struct xe_tile {
* * 4MB-8MB: reserved
* * 8MB-16MB: global GTT
*/
struct {
/** @mmio.size: size of tile's MMIO space */
size_t size;
/** @mmio.regs: pointer to tile's MMIO space (starting with registers) */
void __iomem *regs;
} mmio;
struct xe_mmio mmio;
/**
* @mmio_ext: MMIO-extension info for a tile.
*
* Each tile has its own additional 256MB (28-bit) MMIO-extension space.
*/
struct {
/** @mmio_ext.size: size of tile's additional MMIO-extension space */
size_t size;
/** @mmio_ext.regs: pointer to tile's additional MMIO-extension space */
void __iomem *regs;
} mmio_ext;
struct xe_mmio mmio_ext;
/** @mem: memory management info for tile */
struct {
@ -200,14 +226,14 @@ struct xe_tile {
struct xe_lmtt lmtt;
} pf;
struct {
/** @sriov.vf.memirq: Memory Based Interrupts. */
struct xe_memirq memirq;
/** @sriov.vf.ggtt_balloon: GGTT regions excluded from use. */
struct xe_ggtt_node *ggtt_balloon[2];
} vf;
} sriov;
/** @memirq: Memory Based Interrupts. */
struct xe_memirq memirq;
/** @pcode: tile's PCODE */
struct {
/** @pcode.lock: protecting tile's PCODE mailbox data */

19
drivers/gpu/drm/xe/xe_execlist.c
View File

@ -44,6 +44,7 @@ static void __start_lrc(struct xe_hw_engine *hwe, struct xe_lrc *lrc,
u32 ctx_id)
{
struct xe_gt *gt = hwe->gt;
struct xe_mmio *mmio = &gt->mmio;
struct xe_device *xe = gt_to_xe(gt);
u64 lrc_desc;
@ -58,7 +59,7 @@ static void __start_lrc(struct xe_hw_engine *hwe, struct xe_lrc *lrc,
}
if (hwe->class == XE_ENGINE_CLASS_COMPUTE)
xe_mmio_write32(hwe->gt, RCU_MODE,
xe_mmio_write32(mmio, RCU_MODE,
_MASKED_BIT_ENABLE(RCU_MODE_CCS_ENABLE));
xe_lrc_write_ctx_reg(lrc, CTX_RING_TAIL, lrc->ring.tail);
@ -76,17 +77,17 @@ static void __start_lrc(struct xe_hw_engine *hwe, struct xe_lrc *lrc,
*/
wmb();
xe_mmio_write32(gt, RING_HWS_PGA(hwe->mmio_base),
xe_mmio_write32(mmio, RING_HWS_PGA(hwe->mmio_base),
xe_bo_ggtt_addr(hwe->hwsp));
xe_mmio_read32(gt, RING_HWS_PGA(hwe->mmio_base));
xe_mmio_write32(gt, RING_MODE(hwe->mmio_base),
xe_mmio_read32(mmio, RING_HWS_PGA(hwe->mmio_base));
xe_mmio_write32(mmio, RING_MODE(hwe->mmio_base),
_MASKED_BIT_ENABLE(GFX_DISABLE_LEGACY_MODE));
xe_mmio_write32(gt, RING_EXECLIST_SQ_CONTENTS_LO(hwe->mmio_base),
xe_mmio_write32(mmio, RING_EXECLIST_SQ_CONTENTS_LO(hwe->mmio_base),
lower_32_bits(lrc_desc));
xe_mmio_write32(gt, RING_EXECLIST_SQ_CONTENTS_HI(hwe->mmio_base),
xe_mmio_write32(mmio, RING_EXECLIST_SQ_CONTENTS_HI(hwe->mmio_base),
upper_32_bits(lrc_desc));
xe_mmio_write32(gt, RING_EXECLIST_CONTROL(hwe->mmio_base),
xe_mmio_write32(mmio, RING_EXECLIST_CONTROL(hwe->mmio_base),
EL_CTRL_LOAD);
}
@ -168,8 +169,8 @@ static u64 read_execlist_status(struct xe_hw_engine *hwe)
struct xe_gt *gt = hwe->gt;
u32 hi, lo;
lo = xe_mmio_read32(gt, RING_EXECLIST_STATUS_LO(hwe->mmio_base));
hi = xe_mmio_read32(gt, RING_EXECLIST_STATUS_HI(hwe->mmio_base));
lo = xe_mmio_read32(&gt->mmio, RING_EXECLIST_STATUS_LO(hwe->mmio_base));
hi = xe_mmio_read32(&gt->mmio, RING_EXECLIST_STATUS_HI(hwe->mmio_base));
return lo | (u64)hi << 32;
}

4
drivers/gpu/drm/xe/xe_force_wake.c
View File

@ -100,7 +100,7 @@ static void __domain_ctl(struct xe_gt *gt, struct xe_force_wake_domain *domain,
if (IS_SRIOV_VF(gt_to_xe(gt)))
return;
xe_mmio_write32(gt, domain->reg_ctl, domain->mask | (wake ? domain->val : 0));
xe_mmio_write32(&gt->mmio, domain->reg_ctl, domain->mask | (wake ? domain->val : 0));
}
static int __domain_wait(struct xe_gt *gt, struct xe_force_wake_domain *domain, bool wake)
@ -111,7 +111,7 @@ static int __domain_wait(struct xe_gt *gt, struct xe_force_wake_domain *domain,
if (IS_SRIOV_VF(gt_to_xe(gt)))
return 0;
ret = xe_mmio_wait32(gt, domain->reg_ack, domain->val, wake ? domain->val : 0,
ret = xe_mmio_wait32(&gt->mmio, domain->reg_ack, domain->val, wake ? domain->val : 0,
XE_FORCE_WAKE_ACK_TIMEOUT_MS * USEC_PER_MSEC,
&value, true);
if (ret)

10
drivers/gpu/drm/xe/xe_ggtt.c
View File

@ -5,6 +5,7 @@
#include "xe_ggtt.h"
#include <linux/fault-inject.h>
#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/sizes.h>
@ -107,8 +108,10 @@ static unsigned int probe_gsm_size(struct pci_dev *pdev)
static void ggtt_update_access_counter(struct xe_ggtt *ggtt)
{
struct xe_gt *gt = XE_WA(ggtt->tile->primary_gt, 22019338487) ? ggtt->tile->primary_gt :
ggtt->tile->media_gt;
struct xe_tile *tile = ggtt->tile;
struct xe_gt *affected_gt = XE_WA(tile->primary_gt, 22019338487) ?
tile->primary_gt : tile->media_gt;
struct xe_mmio *mmio = &affected_gt->mmio;
u32 max_gtt_writes = XE_WA(ggtt->tile->primary_gt, 22019338487) ? 1100 : 63;
/*
* Wa_22019338487: GMD_ID is a RO register, a dummy write forces gunit
@ -118,7 +121,7 @@ static void ggtt_update_access_counter(struct xe_ggtt *ggtt)
lockdep_assert_held(&ggtt->lock);
if ((++ggtt->access_count % max_gtt_writes) == 0) {
xe_mmio_write32(gt, GMD_ID, 0x0);
xe_mmio_write32(mmio, GMD_ID, 0x0);
ggtt->access_count = 0;
}
}
@ -262,6 +265,7 @@ int xe_ggtt_init_early(struct xe_ggtt *ggtt)
return 0;
}
ALLOW_ERROR_INJECTION(xe_ggtt_init_early, ERRNO); /* See xe_pci_probe() */
static void xe_ggtt_invalidate(struct xe_ggtt *ggtt);

24
drivers/gpu/drm/xe/xe_gsc.c
View File

@ -34,6 +34,7 @@
#include "instructions/xe_gsc_commands.h"
#include "regs/xe_gsc_regs.h"
#include "regs/xe_gt_regs.h"
#include "regs/xe_irq_regs.h"
static struct xe_gt *
gsc_to_gt(struct xe_gsc *gsc)
@ -179,7 +180,7 @@ static int query_compatibility_version(struct xe_gsc *gsc)
static int gsc_fw_is_loaded(struct xe_gt *gt)
{
return xe_mmio_read32(gt, HECI_FWSTS1(MTL_GSC_HECI1_BASE)) &
return xe_mmio_read32(&gt->mmio, HECI_FWSTS1(MTL_GSC_HECI1_BASE)) &
HECI1_FWSTS1_INIT_COMPLETE;
}
@ -190,7 +191,7 @@ static int gsc_fw_wait(struct xe_gt *gt)
* executed by the GSCCS. To account for possible submission delays or
* other issues, we use a 500ms timeout in the wait here.
*/
return xe_mmio_wait32(gt, HECI_FWSTS1(MTL_GSC_HECI1_BASE),
return xe_mmio_wait32(&gt->mmio, HECI_FWSTS1(MTL_GSC_HECI1_BASE),
HECI1_FWSTS1_INIT_COMPLETE,
HECI1_FWSTS1_INIT_COMPLETE,
500 * USEC_PER_MSEC, NULL, false);
@ -330,7 +331,7 @@ static int gsc_er_complete(struct xe_gt *gt)
* so in that scenario we're always guaranteed to find the correct
* value.
*/
er_status = xe_mmio_read32(gt, GSCI_TIMER_STATUS) & GSCI_TIMER_STATUS_VALUE;
er_status = xe_mmio_read32(&gt->mmio, GSCI_TIMER_STATUS) & GSCI_TIMER_STATUS_VALUE;
if (er_status == GSCI_TIMER_STATUS_TIMER_EXPIRED) {
/*
@ -581,11 +582,11 @@ void xe_gsc_wa_14015076503(struct xe_gt *gt, bool prep)
if (!XE_WA(gt, 14015076503) || !gsc_fw_is_loaded(gt))
return;
xe_mmio_rmw32(gt, HECI_H_GS1(MTL_GSC_HECI2_BASE), gs1_clr, gs1_set);
xe_mmio_rmw32(&gt->mmio, HECI_H_GS1(MTL_GSC_HECI2_BASE), gs1_clr, gs1_set);
if (prep) {
/* make sure the reset bit is clear when writing the CSR reg */
xe_mmio_rmw32(gt, HECI_H_CSR(MTL_GSC_HECI2_BASE),
xe_mmio_rmw32(&gt->mmio, HECI_H_CSR(MTL_GSC_HECI2_BASE),
HECI_H_CSR_RST, HECI_H_CSR_IG);
msleep(200);
}
@ -599,6 +600,7 @@ void xe_gsc_wa_14015076503(struct xe_gt *gt, bool prep)
void xe_gsc_print_info(struct xe_gsc *gsc, struct drm_printer *p)
{
struct xe_gt *gt = gsc_to_gt(gsc);
struct xe_mmio *mmio = &gt->mmio;
int err;
xe_uc_fw_print(&gsc->fw, p);
@ -613,12 +615,12 @@ void xe_gsc_print_info(struct xe_gsc *gsc, struct drm_printer *p)
return;
drm_printf(p, "\nHECI1 FWSTS: 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n",
xe_mmio_read32(gt, HECI_FWSTS1(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(gt, HECI_FWSTS2(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(gt, HECI_FWSTS3(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(gt, HECI_FWSTS4(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(gt, HECI_FWSTS5(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(gt, HECI_FWSTS6(MTL_GSC_HECI1_BASE)));
xe_mmio_read32(mmio, HECI_FWSTS1(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(mmio, HECI_FWSTS2(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(mmio, HECI_FWSTS3(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(mmio, HECI_FWSTS4(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(mmio, HECI_FWSTS5(MTL_GSC_HECI1_BASE)),
xe_mmio_read32(mmio, HECI_FWSTS6(MTL_GSC_HECI1_BASE)));
xe_force_wake_put(gt_to_fw(gt), XE_FW_GSC);
}

4
drivers/gpu/drm/xe/xe_gsc_proxy.c
View File

@ -65,7 +65,7 @@ gsc_to_gt(struct xe_gsc *gsc)
bool xe_gsc_proxy_init_done(struct xe_gsc *gsc)
{
struct xe_gt *gt = gsc_to_gt(gsc);
u32 fwsts1 = xe_mmio_read32(gt, HECI_FWSTS1(MTL_GSC_HECI1_BASE));
u32 fwsts1 = xe_mmio_read32(&gt->mmio, HECI_FWSTS1(MTL_GSC_HECI1_BASE));
return REG_FIELD_GET(HECI1_FWSTS1_CURRENT_STATE, fwsts1) ==
HECI1_FWSTS1_PROXY_STATE_NORMAL;
@ -78,7 +78,7 @@ static void __gsc_proxy_irq_rmw(struct xe_gsc *gsc, u32 clr, u32 set)
/* make sure we never accidentally write the RST bit */
clr |= HECI_H_CSR_RST;
xe_mmio_rmw32(gt, HECI_H_CSR(MTL_GSC_HECI2_BASE), clr, set);
xe_mmio_rmw32(&gt->mmio, HECI_H_CSR(MTL_GSC_HECI2_BASE), clr, set);
}
static void gsc_proxy_irq_clear(struct xe_gsc *gsc)

44
drivers/gpu/drm/xe/xe_gt.c
View File

@ -108,7 +108,6 @@ static void xe_gt_enable_host_l2_vram(struct xe_gt *gt)
return;
if (!xe_gt_is_media_type(gt)) {
xe_mmio_write32(gt, SCRATCH1LPFC, EN_L3_RW_CCS_CACHE_FLUSH);
reg = xe_gt_mcr_unicast_read_any(gt, XE2_GAMREQSTRM_CTRL);
reg |= CG_DIS_CNTLBUS;
xe_gt_mcr_multicast_write(gt, XE2_GAMREQSTRM_CTRL, reg);
@ -245,7 +244,7 @@ static int emit_wa_job(struct xe_gt *gt, struct xe_exec_queue *q)
else if (entry->clr_bits + 1)
val = (reg.mcr ?
xe_gt_mcr_unicast_read_any(gt, reg_mcr) :
xe_mmio_read32(gt, reg)) & (~entry->clr_bits);
xe_mmio_read32(&gt->mmio, reg)) & (~entry->clr_bits);
else
val = 0;
@ -440,7 +439,7 @@ static int gt_fw_domain_init(struct xe_gt *gt)
* Stash hardware-reported version. Since this register does not exist
* on pre-MTL platforms, reading it there will (correctly) return 0.
*/
gt->info.gmdid = xe_mmio_read32(gt, GMD_ID);
gt->info.gmdid = xe_mmio_read32(&gt->mmio, GMD_ID);
err = xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
XE_WARN_ON(err);
@ -623,6 +622,30 @@ int xe_gt_init(struct xe_gt *gt)
return 0;
}
/**
* xe_gt_mmio_init() - Initialize GT's MMIO access
* @gt: the GT object
*
* Initialize GT's MMIO accessor, which will be used to access registers inside
* this GT.
*/
void xe_gt_mmio_init(struct xe_gt *gt)
{
struct xe_tile *tile = gt_to_tile(gt);
gt->mmio.regs = tile->mmio.regs;
gt->mmio.regs_size = tile->mmio.regs_size;
gt->mmio.tile = tile;
if (gt->info.type == XE_GT_TYPE_MEDIA) {
gt->mmio.adj_offset = MEDIA_GT_GSI_OFFSET;
gt->mmio.adj_limit = MEDIA_GT_GSI_LENGTH;
}
if (IS_SRIOV_VF(gt_to_xe(gt)))
gt->mmio.sriov_vf_gt = gt;
}
void xe_gt_record_user_engines(struct xe_gt *gt)
{
struct xe_hw_engine *hwe;
@ -650,8 +673,8 @@ static int do_gt_reset(struct xe_gt *gt)
xe_gsc_wa_14015076503(gt, true);
xe_mmio_write32(gt, GDRST, GRDOM_FULL);
err = xe_mmio_wait32(gt, GDRST, GRDOM_FULL, 0, 5000, NULL, false);
xe_mmio_write32(&gt->mmio, GDRST, GRDOM_FULL);
err = xe_mmio_wait32(&gt->mmio, GDRST, GRDOM_FULL, 0, 5000, NULL, false);
if (err)
xe_gt_err(gt, "failed to clear GRDOM_FULL (%pe)\n",
ERR_PTR(err));
@ -862,6 +885,13 @@ int xe_gt_suspend(struct xe_gt *gt)
return err;
}
void xe_gt_shutdown(struct xe_gt *gt)
{
xe_force_wake_get(gt_to_fw(gt), XE_FORCEWAKE_ALL);
do_gt_reset(gt);
xe_force_wake_put(gt_to_fw(gt), XE_FORCEWAKE_ALL);
}
/**
* xe_gt_sanitize_freq() - Restore saved frequencies if necessary.
* @gt: the GT object
@ -874,7 +904,9 @@ int xe_gt_sanitize_freq(struct xe_gt *gt)
int ret = 0;
if ((!xe_uc_fw_is_available(&gt->uc.gsc.fw) ||
xe_uc_fw_is_loaded(&gt->uc.gsc.fw)) && XE_WA(gt, 22019338487))
xe_uc_fw_is_loaded(&gt->uc.gsc.fw) ||
xe_uc_fw_is_in_error_state(&gt->uc.gsc.fw)) &&
XE_WA(gt, 22019338487))
ret = xe_guc_pc_restore_stashed_freq(&gt->uc.guc.pc);
return ret;

2
drivers/gpu/drm/xe/xe_gt.h
View File

@ -31,6 +31,7 @@ struct xe_gt *xe_gt_alloc(struct xe_tile *tile);
int xe_gt_init_hwconfig(struct xe_gt *gt);
int xe_gt_init_early(struct xe_gt *gt);
int xe_gt_init(struct xe_gt *gt);
void xe_gt_mmio_init(struct xe_gt *gt);
void xe_gt_declare_wedged(struct xe_gt *gt);
int xe_gt_record_default_lrcs(struct xe_gt *gt);
@ -48,6 +49,7 @@ void xe_gt_record_user_engines(struct xe_gt *gt);
void xe_gt_suspend_prepare(struct xe_gt *gt);
int xe_gt_suspend(struct xe_gt *gt);
void xe_gt_shutdown(struct xe_gt *gt);
int xe_gt_resume(struct xe_gt *gt);
void xe_gt_reset_async(struct xe_gt *gt);
void xe_gt_sanitize(struct xe_gt *gt);

2
drivers/gpu/drm/xe/xe_gt_ccs_mode.c
View File

@ -68,7 +68,7 @@ static void __xe_gt_apply_ccs_mode(struct xe_gt *gt, u32 num_engines)
}
}
xe_mmio_write32(gt, CCS_MODE, mode);
xe_mmio_write32(&gt->mmio, CCS_MODE, mode);
xe_gt_dbg(gt, "CCS_MODE=%x config:%08x, num_engines:%d, num_slices:%d\n",
mode, config, num_engines, num_slices);

6
drivers/gpu/drm/xe/xe_gt_clock.c
View File

@ -17,7 +17,7 @@
static u32 read_reference_ts_freq(struct xe_gt *gt)
{
u32 ts_override = xe_mmio_read32(gt, TIMESTAMP_OVERRIDE);
u32 ts_override = xe_mmio_read32(&gt->mmio, TIMESTAMP_OVERRIDE);
u32 base_freq, frac_freq;
base_freq = REG_FIELD_GET(TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_MASK,
@ -57,7 +57,7 @@ static u32 get_crystal_clock_freq(u32 rpm_config_reg)
int xe_gt_clock_init(struct xe_gt *gt)
{
u32 ctc_reg = xe_mmio_read32(gt, CTC_MODE);
u32 ctc_reg = xe_mmio_read32(&gt->mmio, CTC_MODE);
u32 freq = 0;
/* Assuming gen11+ so assert this assumption is correct */
@ -66,7 +66,7 @@ int xe_gt_clock_init(struct xe_gt *gt)
if (ctc_reg & CTC_SOURCE_DIVIDE_LOGIC) {
freq = read_reference_ts_freq(gt);
} else {
u32 c0 = xe_mmio_read32(gt, RPM_CONFIG0);
u32 c0 = xe_mmio_read32(&gt->mmio, RPM_CONFIG0);
freq = get_crystal_clock_freq(c0);

13
drivers/gpu/drm/xe/xe_gt_debugfs.c
View File

@ -15,6 +15,7 @@
#include "xe_ggtt.h"
#include "xe_gt.h"
#include "xe_gt_mcr.h"
#include "xe_gt_idle.h"
#include "xe_gt_sriov_pf_debugfs.h"
#include "xe_gt_sriov_vf_debugfs.h"
#include "xe_gt_stats.h"
@ -109,6 +110,17 @@ static int hw_engines(struct xe_gt *gt, struct drm_printer *p)
return 0;
}
static int powergate_info(struct xe_gt *gt, struct drm_printer *p)
{
int ret;
xe_pm_runtime_get(gt_to_xe(gt));
ret = xe_gt_idle_pg_print(gt, p);
xe_pm_runtime_put(gt_to_xe(gt));
return ret;
}
static int force_reset(struct xe_gt *gt, struct drm_printer *p)
{
xe_pm_runtime_get(gt_to_xe(gt));
@ -288,6 +300,7 @@ static const struct drm_info_list debugfs_list[] = {
{"topology", .show = xe_gt_debugfs_simple_show, .data = topology},
{"steering", .show = xe_gt_debugfs_simple_show, .data = steering},
{"ggtt", .show = xe_gt_debugfs_simple_show, .data = ggtt},
{"powergate_info", .show = xe_gt_debugfs_simple_show, .data = powergate_info},
{"register-save-restore", .show = xe_gt_debugfs_simple_show, .data = register_save_restore},
{"workarounds", .show = xe_gt_debugfs_simple_show, .data = workarounds},
{"pat", .show = xe_gt_debugfs_simple_show, .data = pat},

2
drivers/gpu/drm/xe/xe_gt_freq.c
View File

@ -11,9 +11,9 @@
#include <drm/drm_managed.h>
#include <drm/drm_print.h>
#include "xe_device_types.h"
#include "xe_gt_sysfs.h"
#include "xe_gt_throttle.h"
#include "xe_gt_types.h"
#include "xe_guc_pc.h"
#include "xe_pm.h"

125
drivers/gpu/drm/xe/xe_gt_idle.c
View File

@ -98,7 +98,9 @@ static u64 get_residency_ms(struct xe_gt_idle *gtidle, u64 cur_residency)
void xe_gt_idle_enable_pg(struct xe_gt *gt)
{
struct xe_device *xe = gt_to_xe(gt);
u32 pg_enable;
struct xe_gt_idle *gtidle = &gt->gtidle;
struct xe_mmio *mmio = &gt->mmio;
u32 vcs_mask, vecs_mask;
int i, j;
if (IS_SRIOV_VF(xe))
@ -110,12 +112,19 @@ void xe_gt_idle_enable_pg(struct xe_gt *gt)
xe_device_assert_mem_access(gt_to_xe(gt));
pg_enable = RENDER_POWERGATE_ENABLE | MEDIA_POWERGATE_ENABLE;
vcs_mask = xe_hw_engine_mask_per_class(gt, XE_ENGINE_CLASS_VIDEO_DECODE);
vecs_mask = xe_hw_engine_mask_per_class(gt, XE_ENGINE_CLASS_VIDEO_ENHANCE);
if (vcs_mask || vecs_mask)
gtidle->powergate_enable = MEDIA_POWERGATE_ENABLE;
if (!xe_gt_is_media_type(gt))
gtidle->powergate_enable |= RENDER_POWERGATE_ENABLE;
for (i = XE_HW_ENGINE_VCS0, j = 0; i <= XE_HW_ENGINE_VCS7; ++i, ++j) {
if ((gt->info.engine_mask & BIT(i)))
pg_enable |= (VDN_HCP_POWERGATE_ENABLE(j) |
VDN_MFXVDENC_POWERGATE_ENABLE(j));
gtidle->powergate_enable |= (VDN_HCP_POWERGATE_ENABLE(j) |
VDN_MFXVDENC_POWERGATE_ENABLE(j));
}
XE_WARN_ON(xe_force_wake_get(gt_to_fw(gt), XE_FW_GT));
@ -124,27 +133,115 @@ void xe_gt_idle_enable_pg(struct xe_gt *gt)
* GuC sets the hysteresis value when GuC PC is enabled
* else set it to 25 (25 * 1.28us)
*/
xe_mmio_write32(gt, MEDIA_POWERGATE_IDLE_HYSTERESIS, 25);
xe_mmio_write32(gt, RENDER_POWERGATE_IDLE_HYSTERESIS, 25);
xe_mmio_write32(mmio, MEDIA_POWERGATE_IDLE_HYSTERESIS, 25);
xe_mmio_write32(mmio, RENDER_POWERGATE_IDLE_HYSTERESIS, 25);
}
xe_mmio_write32(gt, POWERGATE_ENABLE, pg_enable);
xe_mmio_write32(mmio, POWERGATE_ENABLE, gtidle->powergate_enable);
XE_WARN_ON(xe_force_wake_put(gt_to_fw(gt), XE_FW_GT));
}
void xe_gt_idle_disable_pg(struct xe_gt *gt)
{
struct xe_gt_idle *gtidle = &gt->gtidle;
if (IS_SRIOV_VF(gt_to_xe(gt)))
return;
xe_device_assert_mem_access(gt_to_xe(gt));
gtidle->powergate_enable = 0;
XE_WARN_ON(xe_force_wake_get(gt_to_fw(gt), XE_FW_GT));
xe_mmio_write32(gt, POWERGATE_ENABLE, 0);
xe_mmio_write32(&gt->mmio, POWERGATE_ENABLE, gtidle->powergate_enable);
XE_WARN_ON(xe_force_wake_put(gt_to_fw(gt), XE_FW_GT));
}
/**
* xe_gt_idle_pg_print - Xe powergating info
* @gt: GT object
* @p: drm_printer.
*
* This function prints the powergating information
*
* Return: 0 on success, negative error code otherwise
*/
int xe_gt_idle_pg_print(struct xe_gt *gt, struct drm_printer *p)
{
struct xe_gt_idle *gtidle = &gt->gtidle;
struct xe_device *xe = gt_to_xe(gt);
enum xe_gt_idle_state state;
u32 pg_enabled, pg_status = 0;
u32 vcs_mask, vecs_mask;
int err, n;
/*
* Media Slices
*
* Slice 0: VCS0, VCS1, VECS0
* Slice 1: VCS2, VCS3, VECS1
* Slice 2: VCS4, VCS5, VECS2
* Slice 3: VCS6, VCS7, VECS3
*/
static const struct {
u64 engines;
u32 status_bit;
} media_slices[] = {
{(BIT(XE_HW_ENGINE_VCS0) | BIT(XE_HW_ENGINE_VCS1) |
BIT(XE_HW_ENGINE_VECS0)), MEDIA_SLICE0_AWAKE_STATUS},
{(BIT(XE_HW_ENGINE_VCS2) | BIT(XE_HW_ENGINE_VCS3) |
BIT(XE_HW_ENGINE_VECS1)), MEDIA_SLICE1_AWAKE_STATUS},
{(BIT(XE_HW_ENGINE_VCS4) | BIT(XE_HW_ENGINE_VCS5) |
BIT(XE_HW_ENGINE_VECS2)), MEDIA_SLICE2_AWAKE_STATUS},
{(BIT(XE_HW_ENGINE_VCS6) | BIT(XE_HW_ENGINE_VCS7) |
BIT(XE_HW_ENGINE_VECS3)), MEDIA_SLICE3_AWAKE_STATUS},
};
if (xe->info.platform == XE_PVC) {
drm_printf(p, "Power Gating not supported\n");
return 0;
}
state = gtidle->idle_status(gtidle_to_pc(gtidle));
pg_enabled = gtidle->powergate_enable;
/* Do not wake the GT to read powergating status */
if (state != GT_IDLE_C6) {
err = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
if (err)
return err;
pg_enabled = xe_mmio_read32(&gt->mmio, POWERGATE_ENABLE);
pg_status = xe_mmio_read32(&gt->mmio, POWERGATE_DOMAIN_STATUS);
XE_WARN_ON(xe_force_wake_put(gt_to_fw(gt), XE_FW_GT));
}
if (gt->info.engine_mask & XE_HW_ENGINE_RCS_MASK) {
drm_printf(p, "Render Power Gating Enabled: %s\n",
str_yes_no(pg_enabled & RENDER_POWERGATE_ENABLE));
drm_printf(p, "Render Power Gate Status: %s\n",
str_up_down(pg_status & RENDER_AWAKE_STATUS));
}
vcs_mask = xe_hw_engine_mask_per_class(gt, XE_ENGINE_CLASS_VIDEO_DECODE);
vecs_mask = xe_hw_engine_mask_per_class(gt, XE_ENGINE_CLASS_VIDEO_ENHANCE);
/* Print media CPG status only if media is present */
if (vcs_mask || vecs_mask) {
drm_printf(p, "Media Power Gating Enabled: %s\n",
str_yes_no(pg_enabled & MEDIA_POWERGATE_ENABLE));
for (n = 0; n < ARRAY_SIZE(media_slices); n++)
if (gt->info.engine_mask & media_slices[n].engines)
drm_printf(p, "Media Slice%d Power Gate Status: %s\n", n,
str_up_down(pg_status & media_slices[n].status_bit));
}
return 0;
}
static ssize_t name_show(struct device *dev,
struct device_attribute *attr, char *buff)
{
@ -260,9 +357,9 @@ void xe_gt_idle_enable_c6(struct xe_gt *gt)
return;
/* Units of 1280 ns for a total of 5s */
xe_mmio_write32(gt, RC_IDLE_HYSTERSIS, 0x3B9ACA);
xe_mmio_write32(&gt->mmio, RC_IDLE_HYSTERSIS, 0x3B9ACA);
/* Enable RC6 */
xe_mmio_write32(gt, RC_CONTROL,
xe_mmio_write32(&gt->mmio, RC_CONTROL,
RC_CTL_HW_ENABLE | RC_CTL_TO_MODE | RC_CTL_RC6_ENABLE);
}
@ -274,6 +371,6 @@ void xe_gt_idle_disable_c6(struct xe_gt *gt)
if (IS_SRIOV_VF(gt_to_xe(gt)))
return;
xe_mmio_write32(gt, RC_CONTROL, 0);
xe_mmio_write32(gt, RC_STATE, 0);
xe_mmio_write32(&gt->mmio, RC_CONTROL, 0);
xe_mmio_write32(&gt->mmio, RC_STATE, 0);
}

2
drivers/gpu/drm/xe/xe_gt_idle.h
View File

@ -8,6 +8,7 @@
#include "xe_gt_idle_types.h"
struct drm_printer;
struct xe_gt;
int xe_gt_idle_init(struct xe_gt_idle *gtidle);
@ -15,5 +16,6 @@ void xe_gt_idle_enable_c6(struct xe_gt *gt);
void xe_gt_idle_disable_c6(struct xe_gt *gt);
void xe_gt_idle_enable_pg(struct xe_gt *gt);
void xe_gt_idle_disable_pg(struct xe_gt *gt);
int xe_gt_idle_pg_print(struct xe_gt *gt, struct drm_printer *p);
#endif /* _XE_GT_IDLE_H_ */

2
drivers/gpu/drm/xe/xe_gt_idle_types.h
View File

@ -23,6 +23,8 @@ enum xe_gt_idle_state {
struct xe_gt_idle {
/** @name: name */
char name[16];
/** @powergate_enable: copy of powergate enable bits */
u32 powergate_enable;
/** @residency_multiplier: residency multiplier in ns */
u32 residency_multiplier;
/** @cur_residency: raw driver copy of idle residency */

68
drivers/gpu/drm/xe/xe_gt_mcr.c
View File

@ -237,13 +237,26 @@ static const struct xe_mmio_range xe2lpm_instance0_steering_table[] = {
{},
};
static const struct xe_mmio_range xe3lpm_instance0_steering_table[] = {
{ 0x384000, 0x3847DF }, /* GAM, rsvd, GAM */
{ 0x384900, 0x384AFF }, /* GAM */
{ 0x389560, 0x3895FF }, /* MEDIAINF */
{ 0x38B600, 0x38B8FF }, /* L3BANK */
{ 0x38C800, 0x38D07F }, /* GAM, MEDIAINF */
{ 0x38D0D0, 0x38F0FF }, /* MEDIAINF, GAM */
{ 0x393C00, 0x393C7F }, /* MEDIAINF */
{},
};
static void init_steering_l3bank(struct xe_gt *gt)
{
struct xe_mmio *mmio = &gt->mmio;
if (GRAPHICS_VERx100(gt_to_xe(gt)) >= 1270) {
u32 mslice_mask = REG_FIELD_GET(MEML3_EN_MASK,
xe_mmio_read32(gt, MIRROR_FUSE3));
xe_mmio_read32(mmio, MIRROR_FUSE3));
u32 bank_mask = REG_FIELD_GET(GT_L3_EXC_MASK,
xe_mmio_read32(gt, XEHP_FUSE4));
xe_mmio_read32(mmio, XEHP_FUSE4));
/*
* Group selects mslice, instance selects bank within mslice.
@ -254,7 +267,7 @@ static void init_steering_l3bank(struct xe_gt *gt)
bank_mask & BIT(0) ? 0 : 2;
} else if (gt_to_xe(gt)->info.platform == XE_DG2) {
u32 mslice_mask = REG_FIELD_GET(MEML3_EN_MASK,
xe_mmio_read32(gt, MIRROR_FUSE3));
xe_mmio_read32(mmio, MIRROR_FUSE3));
u32 bank = __ffs(mslice_mask) * 8;
/*
@ -266,7 +279,7 @@ static void init_steering_l3bank(struct xe_gt *gt)
gt->steering[L3BANK].instance_target = bank & 0x3;
} else {
u32 fuse = REG_FIELD_GET(L3BANK_MASK,
~xe_mmio_read32(gt, MIRROR_FUSE3));
~xe_mmio_read32(mmio, MIRROR_FUSE3));
gt->steering[L3BANK].group_target = 0; /* unused */
gt->steering[L3BANK].instance_target = __ffs(fuse);
@ -276,7 +289,7 @@ static void init_steering_l3bank(struct xe_gt *gt)
static void init_steering_mslice(struct xe_gt *gt)
{
u32 mask = REG_FIELD_GET(MEML3_EN_MASK,
xe_mmio_read32(gt, MIRROR_FUSE3));
xe_mmio_read32(&gt->mmio, MIRROR_FUSE3));
/*
* mslice registers are valid (not terminated) if either the meml3
@ -352,6 +365,19 @@ void xe_gt_mcr_get_dss_steering(struct xe_gt *gt, unsigned int dss, u16 *group,
*instance = dss % gt->steering_dss_per_grp;
}
/**
* xe_gt_mcr_steering_info_to_dss_id - Get DSS ID from group/instance steering
* @gt: GT structure
* @group: steering group ID
* @instance: steering instance ID
*
* Return: the coverted DSS id.
*/
u32 xe_gt_mcr_steering_info_to_dss_id(struct xe_gt *gt, u16 group, u16 instance)
{
return group * dss_per_group(gt) + instance;
}
static void init_steering_dss(struct xe_gt *gt)
{
gt->steering_dss_per_grp = dss_per_group(gt);
@ -380,7 +406,7 @@ static void init_steering_oaddrm(struct xe_gt *gt)
static void init_steering_sqidi_psmi(struct xe_gt *gt)
{
u32 mask = REG_FIELD_GET(XE2_NODE_ENABLE_MASK,
xe_mmio_read32(gt, MIRROR_FUSE3));
xe_mmio_read32(&gt->mmio, MIRROR_FUSE3));
u32 select = __ffs(mask);
gt->steering[SQIDI_PSMI].group_target = select >> 1;
@ -439,7 +465,10 @@ void xe_gt_mcr_init(struct xe_gt *gt)
if (gt->info.type == XE_GT_TYPE_MEDIA) {
drm_WARN_ON(&xe->drm, MEDIA_VER(xe) < 13);
if (MEDIA_VERx100(xe) >= 1301) {
if (MEDIA_VER(xe) >= 30) {
gt->steering[OADDRM].ranges = xe2lpm_gpmxmt_steering_table;
gt->steering[INSTANCE0].ranges = xe3lpm_instance0_steering_table;
} else if (MEDIA_VERx100(xe) >= 1301) {
gt->steering[OADDRM].ranges = xe2lpm_gpmxmt_steering_table;
gt->steering[INSTANCE0].ranges = xe2lpm_instance0_steering_table;
} else {
@ -494,8 +523,8 @@ void xe_gt_mcr_set_implicit_defaults(struct xe_gt *gt)
u32 steer_val = REG_FIELD_PREP(MCR_SLICE_MASK, 0) |
REG_FIELD_PREP(MCR_SUBSLICE_MASK, 2);
xe_mmio_write32(gt, MCFG_MCR_SELECTOR, steer_val);
xe_mmio_write32(gt, SF_MCR_SELECTOR, steer_val);
xe_mmio_write32(&gt->mmio, MCFG_MCR_SELECTOR, steer_val);
xe_mmio_write32(&gt->mmio, SF_MCR_SELECTOR, steer_val);
/*
* For GAM registers, all reads should be directed to instance 1
* (unicast reads against other instances are not allowed),
@ -533,7 +562,7 @@ static bool xe_gt_mcr_get_nonterminated_steering(struct xe_gt *gt,
continue;
for (int i = 0; gt->steering[type].ranges[i].end > 0; i++) {
if (xe_mmio_in_range(gt, &gt->steering[type].ranges[i], reg)) {
if (xe_mmio_in_range(&gt->mmio, &gt->steering[type].ranges[i], reg)) {
*group = gt->steering[type].group_target;
*instance = gt->steering[type].instance_target;
return true;
@ -544,7 +573,7 @@ static bool xe_gt_mcr_get_nonterminated_steering(struct xe_gt *gt,
implicit_ranges = gt->steering[IMPLICIT_STEERING].ranges;
if (implicit_ranges)
for (int i = 0; implicit_ranges[i].end > 0; i++)
if (xe_mmio_in_range(gt, &implicit_ranges[i], reg))
if (xe_mmio_in_range(&gt->mmio, &implicit_ranges[i], reg))
return false;
/*
@ -579,7 +608,7 @@ static void mcr_lock(struct xe_gt *gt) __acquires(&gt->mcr_lock)
* when a read to the relevant register returns 1.
*/
if (GRAPHICS_VERx100(xe) >= 1270)
ret = xe_mmio_wait32(gt, STEER_SEMAPHORE, 0x1, 0x1, 10, NULL,
ret = xe_mmio_wait32(&gt->mmio, STEER_SEMAPHORE, 0x1, 0x1, 10, NULL,
true);
drm_WARN_ON_ONCE(&xe->drm, ret == -ETIMEDOUT);
@ -589,7 +618,7 @@ static void mcr_unlock(struct xe_gt *gt) __releases(&gt->mcr_lock)
{
/* Release hardware semaphore - this is done by writing 1 to the register */
if (GRAPHICS_VERx100(gt_to_xe(gt)) >= 1270)
xe_mmio_write32(gt, STEER_SEMAPHORE, 0x1);
xe_mmio_write32(&gt->mmio, STEER_SEMAPHORE, 0x1);
spin_unlock(&gt->mcr_lock);
}
@ -603,6 +632,7 @@ static u32 rw_with_mcr_steering(struct xe_gt *gt, struct xe_reg_mcr reg_mcr,
u8 rw_flag, int group, int instance, u32 value)
{
const struct xe_reg reg = to_xe_reg(reg_mcr);
struct xe_mmio *mmio = &gt->mmio;
struct xe_reg steer_reg;
u32 steer_val, val = 0;
@ -635,12 +665,12 @@ static u32 rw_with_mcr_steering(struct xe_gt *gt, struct xe_reg_mcr reg_mcr,
if (rw_flag == MCR_OP_READ)
steer_val |= MCR_MULTICAST;
xe_mmio_write32(gt, steer_reg, steer_val);
xe_mmio_write32(mmio, steer_reg, steer_val);
if (rw_flag == MCR_OP_READ)
val = xe_mmio_read32(gt, reg);
val = xe_mmio_read32(mmio, reg);
else
xe_mmio_write32(gt, reg, value);
xe_mmio_write32(mmio, reg, value);
/*
* If we turned off the multicast bit (during a write) we're required
@ -649,7 +679,7 @@ static u32 rw_with_mcr_steering(struct xe_gt *gt, struct xe_reg_mcr reg_mcr,
* operation.
*/
if (rw_flag == MCR_OP_WRITE)
xe_mmio_write32(gt, steer_reg, MCR_MULTICAST);
xe_mmio_write32(mmio, steer_reg, MCR_MULTICAST);
return val;
}
@ -684,7 +714,7 @@ u32 xe_gt_mcr_unicast_read_any(struct xe_gt *gt, struct xe_reg_mcr reg_mcr)
group, instance, 0);
mcr_unlock(gt);
} else {
val = xe_mmio_read32(gt, reg);
val = xe_mmio_read32(&gt->mmio, reg);
}
return val;
@ -757,7 +787,7 @@ void xe_gt_mcr_multicast_write(struct xe_gt *gt, struct xe_reg_mcr reg_mcr,
* to touch the steering register.
*/
mcr_lock(gt);
xe_mmio_write32(gt, reg, value);
xe_mmio_write32(&gt->mmio, reg, value);
mcr_unlock(gt);
}

1
drivers/gpu/drm/xe/xe_gt_mcr.h
View File

@ -28,6 +28,7 @@ void xe_gt_mcr_multicast_write(struct xe_gt *gt, struct xe_reg_mcr mcr_reg,
void xe_gt_mcr_steering_dump(struct xe_gt *gt, struct drm_printer *p);
void xe_gt_mcr_get_dss_steering(struct xe_gt *gt, unsigned int dss, u16 *group, u16 *instance);
u32 xe_gt_mcr_steering_info_to_dss_id(struct xe_gt *gt, u16 group, u16 instance);
/*
* Loop over each DSS and determine the group and instance IDs that

2
drivers/gpu/drm/xe/xe_gt_printk.h
View File

@ -8,7 +8,7 @@
#include <drm/drm_print.h>
#include "xe_device_types.h"
#include "xe_gt_types.h"
#define xe_gt_printk(_gt, _level, _fmt, ...) \
drm_##_level(&gt_to_xe(_gt)->drm, "GT%u: " _fmt, (_gt)->info.id, ##__VA_ARGS__)

56
drivers/gpu/drm/xe/xe_gt_sriov_pf.c
View File

@ -5,12 +5,15 @@
#include <drm/drm_managed.h>
#include "regs/xe_guc_regs.h"
#include "regs/xe_regs.h"
#include "xe_gt.h"
#include "xe_gt_sriov_pf.h"
#include "xe_gt_sriov_pf_config.h"
#include "xe_gt_sriov_pf_control.h"
#include "xe_gt_sriov_pf_helpers.h"
#include "xe_gt_sriov_pf_migration.h"
#include "xe_gt_sriov_pf_service.h"
#include "xe_mmio.h"
@ -72,7 +75,7 @@ static bool pf_needs_enable_ggtt_guest_update(struct xe_device *xe)
static void pf_enable_ggtt_guest_update(struct xe_gt *gt)
{
xe_mmio_write32(gt, VIRTUAL_CTRL_REG, GUEST_GTT_UPDATE_EN);
xe_mmio_write32(&gt->mmio, VIRTUAL_CTRL_REG, GUEST_GTT_UPDATE_EN);
}
/**
@ -87,6 +90,57 @@ void xe_gt_sriov_pf_init_hw(struct xe_gt *gt)
pf_enable_ggtt_guest_update(gt);
xe_gt_sriov_pf_service_update(gt);
xe_gt_sriov_pf_migration_init(gt);
}
static u32 pf_get_vf_regs_stride(struct xe_device *xe)
{
return GRAPHICS_VERx100(xe) > 1200 ? 0x400 : 0x1000;
}
static struct xe_reg xe_reg_vf_to_pf(struct xe_reg vf_reg, unsigned int vfid, u32 stride)
{
struct xe_reg pf_reg = vf_reg;
pf_reg.vf = 0;
pf_reg.addr += stride * vfid;
return pf_reg;
}
static void pf_clear_vf_scratch_regs(struct xe_gt *gt, unsigned int vfid)
{
u32 stride = pf_get_vf_regs_stride(gt_to_xe(gt));
struct xe_reg scratch;
int n, count;
if (xe_gt_is_media_type(gt)) {
count = MED_VF_SW_FLAG_COUNT;
for (n = 0; n < count; n++) {
scratch = xe_reg_vf_to_pf(MED_VF_SW_FLAG(n), vfid, stride);
xe_mmio_write32(&gt->mmio, scratch, 0);
}
} else {
count = VF_SW_FLAG_COUNT;
for (n = 0; n < count; n++) {
scratch = xe_reg_vf_to_pf(VF_SW_FLAG(n), vfid, stride);
xe_mmio_write32(&gt->mmio, scratch, 0);
}
}
}
/**
* xe_gt_sriov_pf_sanitize_hw() - Reset hardware state related to a VF.
* @gt: the &xe_gt
* @vfid: the VF identifier
*
* This function can only be called on PF.
*/
void xe_gt_sriov_pf_sanitize_hw(struct xe_gt *gt, unsigned int vfid)
{
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
pf_clear_vf_scratch_regs(gt, vfid);
}
/**

1
drivers/gpu/drm/xe/xe_gt_sriov_pf.h
View File

@ -11,6 +11,7 @@ struct xe_gt;
#ifdef CONFIG_PCI_IOV
int xe_gt_sriov_pf_init_early(struct xe_gt *gt);
void xe_gt_sriov_pf_init_hw(struct xe_gt *gt);
void xe_gt_sriov_pf_sanitize_hw(struct xe_gt *gt, unsigned int vfid);
void xe_gt_sriov_pf_restart(struct xe_gt *gt);
#else
static inline int xe_gt_sriov_pf_init_early(struct xe_gt *gt)

204
drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c
View File

@ -34,6 +34,8 @@
#include "xe_ttm_vram_mgr.h"
#include "xe_wopcm.h"
#define make_u64_from_u32(hi, lo) ((u64)((u64)(u32)(hi) << 32 | (u32)(lo)))
/*
* Return: number of KLVs that were successfully parsed and saved,
* negative error code on failure.
@ -229,14 +231,16 @@ static struct xe_gt_sriov_config *pf_pick_vf_config(struct xe_gt *gt, unsigned i
}
/* Return: number of configuration dwords written */
static u32 encode_config_ggtt(u32 *cfg, const struct xe_gt_sriov_config *config)
static u32 encode_config_ggtt(u32 *cfg, const struct xe_gt_sriov_config *config, bool details)
{
u32 n = 0;
if (xe_ggtt_node_allocated(config->ggtt_region)) {
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_GGTT_START);
cfg[n++] = lower_32_bits(config->ggtt_region->base.start);
cfg[n++] = upper_32_bits(config->ggtt_region->base.start);
if (details) {
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_GGTT_START);
cfg[n++] = lower_32_bits(config->ggtt_region->base.start);
cfg[n++] = upper_32_bits(config->ggtt_region->base.start);
}
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_GGTT_SIZE);
cfg[n++] = lower_32_bits(config->ggtt_region->base.size);
@ -247,20 +251,24 @@ static u32 encode_config_ggtt(u32 *cfg, const struct xe_gt_sriov_config *config)
}
/* Return: number of configuration dwords written */
static u32 encode_config(u32 *cfg, const struct xe_gt_sriov_config *config)
static u32 encode_config(u32 *cfg, const struct xe_gt_sriov_config *config, bool details)
{
u32 n = 0;
n += encode_config_ggtt(cfg, config);
n += encode_config_ggtt(cfg, config, details);
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_BEGIN_CONTEXT_ID);
cfg[n++] = config->begin_ctx;
if (details) {
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_BEGIN_CONTEXT_ID);
cfg[n++] = config->begin_ctx;
}
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_NUM_CONTEXTS);
cfg[n++] = config->num_ctxs;
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_BEGIN_DOORBELL_ID);
cfg[n++] = config->begin_db;
if (details) {
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_BEGIN_DOORBELL_ID);
cfg[n++] = config->begin_db;
}
cfg[n++] = PREP_GUC_KLV_TAG(VF_CFG_NUM_DOORBELLS);
cfg[n++] = config->num_dbs;
@ -301,7 +309,7 @@ static int pf_push_full_vf_config(struct xe_gt *gt, unsigned int vfid)
if (!cfg)
return -ENOMEM;
num_dwords = encode_config(cfg, config);
num_dwords = encode_config(cfg, config, true);
xe_gt_assert(gt, num_dwords <= max_cfg_dwords);
if (xe_gt_is_media_type(gt)) {
@ -309,10 +317,10 @@ static int pf_push_full_vf_config(struct xe_gt *gt, unsigned int vfid)
struct xe_gt_sriov_config *other = pf_pick_vf_config(primary, vfid);
/* media-GT will never include a GGTT config */
xe_gt_assert(gt, !encode_config_ggtt(cfg + num_dwords, config));
xe_gt_assert(gt, !encode_config_ggtt(cfg + num_dwords, config, true));
/* the GGTT config must be taken from the primary-GT instead */
num_dwords += encode_config_ggtt(cfg + num_dwords, other);
num_dwords += encode_config_ggtt(cfg + num_dwords, other, true);
}
xe_gt_assert(gt, num_dwords <= max_cfg_dwords);
@ -2042,7 +2050,7 @@ static int pf_validate_vf_config(struct xe_gt *gt, unsigned int vfid)
valid_all = valid_all && valid_lmem;
}
return valid_all ? 1 : valid_any ? -ENOKEY : -ENODATA;
return valid_all ? 0 : valid_any ? -ENOKEY : -ENODATA;
}
/**
@ -2068,6 +2076,174 @@ bool xe_gt_sriov_pf_config_is_empty(struct xe_gt *gt, unsigned int vfid)
return empty;
}
/**
* xe_gt_sriov_pf_config_save - Save a VF provisioning config as binary blob.
* @gt: the &xe_gt
* @vfid: the VF identifier (can't be PF)
* @buf: the buffer to save a config to (or NULL if query the buf size)
* @size: the size of the buffer (or 0 if query the buf size)
*
* This function can only be called on PF.
*
* Return: mininum size of the buffer or the number of bytes saved,
* or a negative error code on failure.
*/
ssize_t xe_gt_sriov_pf_config_save(struct xe_gt *gt, unsigned int vfid, void *buf, size_t size)
{
struct xe_gt_sriov_config *config;
ssize_t ret;
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
xe_gt_assert(gt, vfid);
xe_gt_assert(gt, !(!buf ^ !size));
mutex_lock(xe_gt_sriov_pf_master_mutex(gt));
ret = pf_validate_vf_config(gt, vfid);
if (!size) {
ret = ret ? 0 : SZ_4K;
} else if (!ret) {
if (size < SZ_4K) {
ret = -ENOBUFS;
} else {
config = pf_pick_vf_config(gt, vfid);
ret = encode_config(buf, config, false) * sizeof(u32);
}
}
mutex_unlock(xe_gt_sriov_pf_master_mutex(gt));
return ret;
}
static int pf_restore_vf_config_klv(struct xe_gt *gt, unsigned int vfid,
u32 key, u32 len, const u32 *value)
{
switch (key) {
case GUC_KLV_VF_CFG_NUM_CONTEXTS_KEY:
if (len != GUC_KLV_VF_CFG_NUM_CONTEXTS_LEN)
return -EBADMSG;
return pf_provision_vf_ctxs(gt, vfid, value[0]);
case GUC_KLV_VF_CFG_NUM_DOORBELLS_KEY:
if (len != GUC_KLV_VF_CFG_NUM_DOORBELLS_LEN)
return -EBADMSG;
return pf_provision_vf_dbs(gt, vfid, value[0]);
case GUC_KLV_VF_CFG_EXEC_QUANTUM_KEY:
if (len != GUC_KLV_VF_CFG_EXEC_QUANTUM_LEN)
return -EBADMSG;
return pf_provision_exec_quantum(gt, vfid, value[0]);
case GUC_KLV_VF_CFG_PREEMPT_TIMEOUT_KEY:
if (len != GUC_KLV_VF_CFG_PREEMPT_TIMEOUT_LEN)
return -EBADMSG;
return pf_provision_preempt_timeout(gt, vfid, value[0]);
/* auto-generate case statements */
#define define_threshold_key_to_provision_case(TAG, ...) \
case MAKE_GUC_KLV_VF_CFG_THRESHOLD_KEY(TAG): \
BUILD_BUG_ON(MAKE_GUC_KLV_VF_CFG_THRESHOLD_LEN(TAG) != 1u); \
if (len != MAKE_GUC_KLV_VF_CFG_THRESHOLD_LEN(TAG)) \
return -EBADMSG; \
return pf_provision_threshold(gt, vfid, \
MAKE_XE_GUC_KLV_THRESHOLD_INDEX(TAG), \
value[0]);
MAKE_XE_GUC_KLV_THRESHOLDS_SET(define_threshold_key_to_provision_case)
#undef define_threshold_key_to_provision_case
}
if (xe_gt_is_media_type(gt))
return -EKEYREJECTED;
switch (key) {
case GUC_KLV_VF_CFG_GGTT_SIZE_KEY:
if (len != GUC_KLV_VF_CFG_GGTT_SIZE_LEN)
return -EBADMSG;
return pf_provision_vf_ggtt(gt, vfid, make_u64_from_u32(value[1], value[0]));
case GUC_KLV_VF_CFG_LMEM_SIZE_KEY:
if (!IS_DGFX(gt_to_xe(gt)))
return -EKEYREJECTED;
if (len != GUC_KLV_VF_CFG_LMEM_SIZE_LEN)
return -EBADMSG;
return pf_provision_vf_lmem(gt, vfid, make_u64_from_u32(value[1], value[0]));
}
return -EKEYREJECTED;
}
static int pf_restore_vf_config(struct xe_gt *gt, unsigned int vfid,
const u32 *klvs, size_t num_dwords)
{
int err;
while (num_dwords >= GUC_KLV_LEN_MIN) {
u32 key = FIELD_GET(GUC_KLV_0_KEY, klvs[0]);
u32 len = FIELD_GET(GUC_KLV_0_LEN, klvs[0]);
klvs += GUC_KLV_LEN_MIN;
num_dwords -= GUC_KLV_LEN_MIN;
if (num_dwords < len)
err = -EBADMSG;
else
err = pf_restore_vf_config_klv(gt, vfid, key, len, klvs);
if (err) {
xe_gt_sriov_dbg(gt, "restore failed on key %#x (%pe)\n", key, ERR_PTR(err));
return err;
}
klvs += len;
num_dwords -= len;
}
return pf_validate_vf_config(gt, vfid);
}
/**
* xe_gt_sriov_pf_config_restore - Restore a VF provisioning config from binary blob.
* @gt: the &xe_gt
* @vfid: the VF identifier (can't be PF)
* @buf: the buffer with config data
* @size: the size of the config data
*
* This function can only be called on PF.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_gt_sriov_pf_config_restore(struct xe_gt *gt, unsigned int vfid,
const void *buf, size_t size)
{
int err;
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
xe_gt_assert(gt, vfid);
if (!size)
return -ENODATA;
if (size % sizeof(u32))
return -EINVAL;
if (IS_ENABLED(CONFIG_DRM_XE_DEBUG_SRIOV)) {
struct drm_printer p = xe_gt_info_printer(gt);
drm_printf(&p, "restoring VF%u config:\n", vfid);
xe_guc_klv_print(buf, size / sizeof(u32), &p);
}
mutex_lock(xe_gt_sriov_pf_master_mutex(gt));
err = pf_send_vf_cfg_reset(gt, vfid);
if (!err) {
pf_release_vf_config(gt, vfid);
err = pf_restore_vf_config(gt, vfid, buf, size / sizeof(u32));
}
mutex_unlock(xe_gt_sriov_pf_master_mutex(gt));
return err;
}
/**
* xe_gt_sriov_pf_config_restart - Restart SR-IOV configurations after a GT reset.
* @gt: the &xe_gt

4
drivers/gpu/drm/xe/xe_gt_sriov_pf_config.h
View File

@ -54,6 +54,10 @@ int xe_gt_sriov_pf_config_sanitize(struct xe_gt *gt, unsigned int vfid, long tim
int xe_gt_sriov_pf_config_release(struct xe_gt *gt, unsigned int vfid, bool force);
int xe_gt_sriov_pf_config_push(struct xe_gt *gt, unsigned int vfid, bool refresh);
ssize_t xe_gt_sriov_pf_config_save(struct xe_gt *gt, unsigned int vfid, void *buf, size_t size);
int xe_gt_sriov_pf_config_restore(struct xe_gt *gt, unsigned int vfid,
const void *buf, size_t size);
bool xe_gt_sriov_pf_config_is_empty(struct xe_gt *gt, unsigned int vfid);
void xe_gt_sriov_pf_config_restart(struct xe_gt *gt);

44
drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c
View File

@ -9,9 +9,11 @@
#include "xe_device.h"
#include "xe_gt.h"
#include "xe_gt_sriov_pf.h"
#include "xe_gt_sriov_pf_config.h"
#include "xe_gt_sriov_pf_control.h"
#include "xe_gt_sriov_pf_helpers.h"
#include "xe_gt_sriov_pf_migration.h"
#include "xe_gt_sriov_pf_monitor.h"
#include "xe_gt_sriov_pf_service.h"
#include "xe_gt_sriov_printk.h"
@ -176,6 +178,7 @@ static const char *control_bit_to_string(enum xe_gt_sriov_control_bits bit)
CASE2STR(PAUSE_SEND_PAUSE);
CASE2STR(PAUSE_WAIT_GUC);
CASE2STR(PAUSE_GUC_DONE);
CASE2STR(PAUSE_SAVE_GUC);
CASE2STR(PAUSE_FAILED);
CASE2STR(PAUSED);
CASE2STR(RESUME_WIP);
@ -415,6 +418,10 @@ static void pf_enter_vf_ready(struct xe_gt *gt, unsigned int vfid)
* : | : /
* : v : /
* : PAUSE_GUC_DONE o-----restart
* : | :
* : | o---<--busy :
* : v / / :
* : PAUSE_SAVE_GUC :
* : / :
* : / :
* :....o..............o...............o...........:
@ -434,6 +441,7 @@ static void pf_exit_vf_pause_wip(struct xe_gt *gt, unsigned int vfid)
pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_PAUSE_SEND_PAUSE);
pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_PAUSE_WAIT_GUC);
pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_PAUSE_GUC_DONE);
pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_PAUSE_SAVE_GUC);
}
}
@ -464,12 +472,41 @@ static void pf_enter_vf_pause_rejected(struct xe_gt *gt, unsigned int vfid)
pf_enter_vf_pause_failed(gt, vfid);
}
static void pf_enter_vf_pause_save_guc(struct xe_gt *gt, unsigned int vfid)
{
if (!pf_enter_vf_state(gt, vfid, XE_GT_SRIOV_STATE_PAUSE_SAVE_GUC))
pf_enter_vf_state_machine_bug(gt, vfid);
}
static bool pf_exit_vf_pause_save_guc(struct xe_gt *gt, unsigned int vfid)
{
int err;
if (!pf_exit_vf_state(gt, vfid, XE_GT_SRIOV_STATE_PAUSE_SAVE_GUC))
return false;
err = xe_gt_sriov_pf_migration_save_guc_state(gt, vfid);
if (err) {
/* retry if busy */
if (err == -EBUSY) {
pf_enter_vf_pause_save_guc(gt, vfid);
return true;
}
/* give up on error */
if (err == -EIO)
pf_enter_vf_mismatch(gt, vfid);
}
pf_enter_vf_pause_completed(gt, vfid);
return true;
}
static bool pf_exit_vf_pause_guc_done(struct xe_gt *gt, unsigned int vfid)
{
if (!pf_exit_vf_state(gt, vfid, XE_GT_SRIOV_STATE_PAUSE_GUC_DONE))
return false;
pf_enter_vf_pause_completed(gt, vfid);
pf_enter_vf_pause_save_guc(gt, vfid);
return true;
}
@ -1008,7 +1045,7 @@ static bool pf_exit_vf_flr_reset_mmio(struct xe_gt *gt, unsigned int vfid)
if (!pf_exit_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_RESET_MMIO))
return false;
/* XXX: placeholder */
xe_gt_sriov_pf_sanitize_hw(gt, vfid);
pf_enter_vf_flr_send_finish(gt, vfid);
return true;
@ -1338,6 +1375,9 @@ static bool pf_process_vf_state_machine(struct xe_gt *gt, unsigned int vfid)
if (pf_exit_vf_pause_guc_done(gt, vfid))
return true;
if (pf_exit_vf_pause_save_guc(gt, vfid))
return true;
if (pf_exit_vf_resume_send_resume(gt, vfid))
return true;

2
drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h
View File

@ -27,6 +27,7 @@
* @XE_GT_SRIOV_STATE_PAUSE_SEND_PAUSE: indicates that the PF is about to send a PAUSE command.
* @XE_GT_SRIOV_STATE_PAUSE_WAIT_GUC: indicates that the PF awaits for a response from the GuC.
* @XE_GT_SRIOV_STATE_PAUSE_GUC_DONE: indicates that the PF has received a response from the GuC.
* @XE_GT_SRIOV_STATE_PAUSE_SAVE_GUC: indicates that the PF needs to save the VF GuC state.
* @XE_GT_SRIOV_STATE_PAUSE_FAILED: indicates that a VF pause operation has failed.
* @XE_GT_SRIOV_STATE_PAUSED: indicates that the VF is paused.
* @XE_GT_SRIOV_STATE_RESUME_WIP: indicates the a VF resume operation is in progress.
@ -56,6 +57,7 @@ enum xe_gt_sriov_control_bits {
XE_GT_SRIOV_STATE_PAUSE_SEND_PAUSE,
XE_GT_SRIOV_STATE_PAUSE_WAIT_GUC,
XE_GT_SRIOV_STATE_PAUSE_GUC_DONE,
XE_GT_SRIOV_STATE_PAUSE_SAVE_GUC,
XE_GT_SRIOV_STATE_PAUSE_FAILED,
XE_GT_SRIOV_STATE_PAUSED,

127
drivers/gpu/drm/xe/xe_gt_sriov_pf_debugfs.c
View File

@ -17,6 +17,7 @@
#include "xe_gt_sriov_pf_control.h"
#include "xe_gt_sriov_pf_debugfs.h"
#include "xe_gt_sriov_pf_helpers.h"
#include "xe_gt_sriov_pf_migration.h"
#include "xe_gt_sriov_pf_monitor.h"
#include "xe_gt_sriov_pf_policy.h"
#include "xe_gt_sriov_pf_service.h"
@ -312,6 +313,9 @@ static const struct {
{ "stop", xe_gt_sriov_pf_control_stop_vf },
{ "pause", xe_gt_sriov_pf_control_pause_vf },
{ "resume", xe_gt_sriov_pf_control_resume_vf },
#ifdef CONFIG_DRM_XE_DEBUG_SRIOV
{ "restore!", xe_gt_sriov_pf_migration_restore_guc_state },
#endif
};
static ssize_t control_write(struct file *file, const char __user *buf, size_t count, loff_t *pos)
@ -375,6 +379,119 @@ static const struct file_operations control_ops = {
.llseek = default_llseek,
};
/*
* /sys/kernel/debug/dri/0/
* ├── gt0
* │   ├── vf1
* │   │   ├── guc_state
*/
static ssize_t guc_state_read(struct file *file, char __user *buf,
size_t count, loff_t *pos)
{
struct dentry *dent = file_dentry(file);
struct dentry *parent = dent->d_parent;
struct xe_gt *gt = extract_gt(parent);
unsigned int vfid = extract_vfid(parent);
return xe_gt_sriov_pf_migration_read_guc_state(gt, vfid, buf, count, pos);
}
static ssize_t guc_state_write(struct file *file, const char __user *buf,
size_t count, loff_t *pos)
{
struct dentry *dent = file_dentry(file);
struct dentry *parent = dent->d_parent;
struct xe_gt *gt = extract_gt(parent);
unsigned int vfid = extract_vfid(parent);
if (*pos)
return -EINVAL;
return xe_gt_sriov_pf_migration_write_guc_state(gt, vfid, buf, count);
}
static const struct file_operations guc_state_ops = {
.owner = THIS_MODULE,
.read = guc_state_read,
.write = guc_state_write,
.llseek = default_llseek,
};
/*
* /sys/kernel/debug/dri/0/
* ├── gt0
* │   ├── vf1
* │   │   ├── config_blob
*/
static ssize_t config_blob_read(struct file *file, char __user *buf,
size_t count, loff_t *pos)
{
struct dentry *dent = file_dentry(file);
struct dentry *parent = dent->d_parent;
struct xe_gt *gt = extract_gt(parent);
unsigned int vfid = extract_vfid(parent);
ssize_t ret;
void *tmp;
ret = xe_gt_sriov_pf_config_save(gt, vfid, NULL, 0);
if (!ret)
return -ENODATA;
if (ret < 0)
return ret;
tmp = kzalloc(ret, GFP_KERNEL);
if (!tmp)
return -ENOMEM;
ret = xe_gt_sriov_pf_config_save(gt, vfid, tmp, ret);
if (ret > 0)
ret = simple_read_from_buffer(buf, count, pos, tmp, ret);
kfree(tmp);
return ret;
}
static ssize_t config_blob_write(struct file *file, const char __user *buf,
size_t count, loff_t *pos)
{
struct dentry *dent = file_dentry(file);
struct dentry *parent = dent->d_parent;
struct xe_gt *gt = extract_gt(parent);
unsigned int vfid = extract_vfid(parent);
ssize_t ret;
void *tmp;
if (*pos)
return -EINVAL;
if (!count)
return -ENODATA;
if (count > SZ_4K)
return -EINVAL;
tmp = kzalloc(count, GFP_KERNEL);
if (!tmp)
return -ENOMEM;
if (copy_from_user(tmp, buf, count)) {
ret = -EFAULT;
} else {
ret = xe_gt_sriov_pf_config_restore(gt, vfid, tmp, count);
if (!ret)
ret = count;
}
kfree(tmp);
return ret;
}
static const struct file_operations config_blob_ops = {
.owner = THIS_MODULE,
.read = config_blob_read,
.write = config_blob_write,
.llseek = default_llseek,
};
/**
* xe_gt_sriov_pf_debugfs_register - Register SR-IOV PF specific entries in GT debugfs.
* @gt: the &xe_gt to register
@ -423,5 +540,15 @@ void xe_gt_sriov_pf_debugfs_register(struct xe_gt *gt, struct dentry *root)
pf_add_config_attrs(gt, vfdentry, VFID(n));
debugfs_create_file("control", 0600, vfdentry, NULL, &control_ops);
/* for testing/debugging purposes only! */
if (IS_ENABLED(CONFIG_DRM_XE_DEBUG)) {
debugfs_create_file("guc_state",
IS_ENABLED(CONFIG_DRM_XE_DEBUG_SRIOV) ? 0600 : 0400,
vfdentry, NULL, &guc_state_ops);
debugfs_create_file("config_blob",
IS_ENABLED(CONFIG_DRM_XE_DEBUG_SRIOV) ? 0600 : 0400,
vfdentry, NULL, &config_blob_ops);
}
}
}

419
drivers/gpu/drm/xe/xe_gt_sriov_pf_migration.c Normal file
View File

@ -0,0 +1,419 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2024 Intel Corporation
*/
#include <drm/drm_managed.h>
#include "abi/guc_actions_sriov_abi.h"
#include "xe_bo.h"
#include "xe_gt_sriov_pf_helpers.h"
#include "xe_gt_sriov_pf_migration.h"
#include "xe_gt_sriov_printk.h"
#include "xe_guc.h"
#include "xe_guc_ct.h"
#include "xe_sriov.h"
/* Return: number of dwords saved/restored/required or a negative error code on failure */
static int guc_action_vf_save_restore(struct xe_guc *guc, u32 vfid, u32 opcode,
u64 addr, u32 ndwords)
{
u32 request[PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_LEN] = {
FIELD_PREP(GUC_HXG_MSG_0_ORIGIN, GUC_HXG_ORIGIN_HOST) |
FIELD_PREP(GUC_HXG_MSG_0_TYPE, GUC_HXG_TYPE_REQUEST) |
FIELD_PREP(GUC_HXG_REQUEST_MSG_0_ACTION, GUC_ACTION_PF2GUC_SAVE_RESTORE_VF) |
FIELD_PREP(PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_0_OPCODE, opcode),
FIELD_PREP(PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_1_VFID, vfid),
FIELD_PREP(PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_2_ADDR_LO, lower_32_bits(addr)),
FIELD_PREP(PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_3_ADDR_HI, upper_32_bits(addr)),
FIELD_PREP(PF2GUC_SAVE_RESTORE_VF_REQUEST_MSG_4_SIZE, ndwords),
};
return xe_guc_ct_send_block(&guc->ct, request, ARRAY_SIZE(request));
}
/* Return: size of the state in dwords or a negative error code on failure */
static int pf_send_guc_query_vf_state_size(struct xe_gt *gt, unsigned int vfid)
{
int ret;
ret = guc_action_vf_save_restore(&gt->uc.guc, vfid, GUC_PF_OPCODE_VF_SAVE, 0, 0);
return ret ?: -ENODATA;
}
/* Return: number of state dwords saved or a negative error code on failure */
static int pf_send_guc_save_vf_state(struct xe_gt *gt, unsigned int vfid,
void *buff, size_t size)
{
const int ndwords = size / sizeof(u32);
struct xe_tile *tile = gt_to_tile(gt);
struct xe_device *xe = tile_to_xe(tile);
struct xe_guc *guc = &gt->uc.guc;
struct xe_bo *bo;
int ret;
xe_gt_assert(gt, size % sizeof(u32) == 0);
xe_gt_assert(gt, size == ndwords * sizeof(u32));
bo = xe_bo_create_pin_map(xe, tile, NULL,
ALIGN(size, PAGE_SIZE),
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_INVALIDATE);
if (IS_ERR(bo))
return PTR_ERR(bo);
ret = guc_action_vf_save_restore(guc, vfid, GUC_PF_OPCODE_VF_SAVE,
xe_bo_ggtt_addr(bo), ndwords);
if (!ret)
ret = -ENODATA;
else if (ret > ndwords)
ret = -EPROTO;
else if (ret > 0)
xe_map_memcpy_from(xe, buff, &bo->vmap, 0, ret * sizeof(u32));
xe_bo_unpin_map_no_vm(bo);
return ret;
}
/* Return: number of state dwords restored or a negative error code on failure */
static int pf_send_guc_restore_vf_state(struct xe_gt *gt, unsigned int vfid,
const void *buff, size_t size)
{
const int ndwords = size / sizeof(u32);
struct xe_tile *tile = gt_to_tile(gt);
struct xe_device *xe = tile_to_xe(tile);
struct xe_guc *guc = &gt->uc.guc;
struct xe_bo *bo;
int ret;
xe_gt_assert(gt, size % sizeof(u32) == 0);
xe_gt_assert(gt, size == ndwords * sizeof(u32));
bo = xe_bo_create_pin_map(xe, tile, NULL,
ALIGN(size, PAGE_SIZE),
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
XE_BO_FLAG_GGTT_INVALIDATE);
if (IS_ERR(bo))
return PTR_ERR(bo);
xe_map_memcpy_to(xe, &bo->vmap, 0, buff, size);
ret = guc_action_vf_save_restore(guc, vfid, GUC_PF_OPCODE_VF_RESTORE,
xe_bo_ggtt_addr(bo), ndwords);
if (!ret)
ret = -ENODATA;
else if (ret > ndwords)
ret = -EPROTO;
xe_bo_unpin_map_no_vm(bo);
return ret;
}
static bool pf_migration_supported(struct xe_gt *gt)
{
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
return gt->sriov.pf.migration.supported;
}
static struct mutex *pf_migration_mutex(struct xe_gt *gt)
{
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
return &gt->sriov.pf.migration.snapshot_lock;
}
static struct xe_gt_sriov_state_snapshot *pf_pick_vf_snapshot(struct xe_gt *gt,
unsigned int vfid)
{
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
xe_gt_assert(gt, vfid <= xe_sriov_pf_get_totalvfs(gt_to_xe(gt)));
lockdep_assert_held(pf_migration_mutex(gt));
return &gt->sriov.pf.vfs[vfid].snapshot;
}
static unsigned int pf_snapshot_index(struct xe_gt *gt, struct xe_gt_sriov_state_snapshot *snapshot)
{
return container_of(snapshot, struct xe_gt_sriov_metadata, snapshot) - gt->sriov.pf.vfs;
}
static void pf_free_guc_state(struct xe_gt *gt, struct xe_gt_sriov_state_snapshot *snapshot)
{
struct xe_device *xe = gt_to_xe(gt);
drmm_kfree(&xe->drm, snapshot->guc.buff);
snapshot->guc.buff = NULL;
snapshot->guc.size = 0;
}
static int pf_alloc_guc_state(struct xe_gt *gt,
struct xe_gt_sriov_state_snapshot *snapshot,
size_t size)
{
struct xe_device *xe = gt_to_xe(gt);
void *p;
pf_free_guc_state(gt, snapshot);
if (!size)
return -ENODATA;
if (size % sizeof(u32))
return -EINVAL;
if (size > SZ_2M)
return -EFBIG;
p = drmm_kzalloc(&xe->drm, size, GFP_KERNEL);
if (!p)
return -ENOMEM;
snapshot->guc.buff = p;
snapshot->guc.size = size;
return 0;
}
static void pf_dump_guc_state(struct xe_gt *gt, struct xe_gt_sriov_state_snapshot *snapshot)
{
if (IS_ENABLED(CONFIG_DRM_XE_DEBUG_SRIOV)) {
unsigned int vfid __maybe_unused = pf_snapshot_index(gt, snapshot);
xe_gt_sriov_dbg_verbose(gt, "VF%u GuC state is %zu dwords:\n",
vfid, snapshot->guc.size / sizeof(u32));
print_hex_dump_bytes("state: ", DUMP_PREFIX_OFFSET,
snapshot->guc.buff, min(SZ_64, snapshot->guc.size));
}
}
static int pf_save_vf_guc_state(struct xe_gt *gt, unsigned int vfid)
{
struct xe_gt_sriov_state_snapshot *snapshot = pf_pick_vf_snapshot(gt, vfid);
size_t size;
int ret;
ret = pf_send_guc_query_vf_state_size(gt, vfid);
if (ret < 0)
goto fail;
size = ret * sizeof(u32);
xe_gt_sriov_dbg_verbose(gt, "VF%u state size is %d dwords (%zu bytes)\n", vfid, ret, size);
ret = pf_alloc_guc_state(gt, snapshot, size);
if (ret < 0)
goto fail;
ret = pf_send_guc_save_vf_state(gt, vfid, snapshot->guc.buff, size);
if (ret < 0)
goto fail;
size = ret * sizeof(u32);
xe_gt_assert(gt, size);
xe_gt_assert(gt, size <= snapshot->guc.size);
snapshot->guc.size = size;
pf_dump_guc_state(gt, snapshot);
return 0;
fail:
xe_gt_sriov_dbg(gt, "Unable to save VF%u state (%pe)\n", vfid, ERR_PTR(ret));
pf_free_guc_state(gt, snapshot);
return ret;
}
/**
* xe_gt_sriov_pf_migration_save_guc_state() - Take a GuC VF state snapshot.
* @gt: the &xe_gt
* @vfid: the VF identifier
*
* This function is for PF only.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_gt_sriov_pf_migration_save_guc_state(struct xe_gt *gt, unsigned int vfid)
{
int err;
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
xe_gt_assert(gt, vfid != PFID);
xe_gt_assert(gt, vfid <= xe_sriov_pf_get_totalvfs(gt_to_xe(gt)));
if (!pf_migration_supported(gt))
return -ENOPKG;
mutex_lock(pf_migration_mutex(gt));
err = pf_save_vf_guc_state(gt, vfid);
mutex_unlock(pf_migration_mutex(gt));
return err;
}
static int pf_restore_vf_guc_state(struct xe_gt *gt, unsigned int vfid)
{
struct xe_gt_sriov_state_snapshot *snapshot = pf_pick_vf_snapshot(gt, vfid);
int ret;
if (!snapshot->guc.size)
return -ENODATA;
xe_gt_sriov_dbg_verbose(gt, "restoring %zu dwords of VF%u GuC state\n",
snapshot->guc.size / sizeof(u32), vfid);
ret = pf_send_guc_restore_vf_state(gt, vfid, snapshot->guc.buff, snapshot->guc.size);
if (ret < 0)
goto fail;
xe_gt_sriov_dbg_verbose(gt, "restored %d dwords of VF%u GuC state\n", ret, vfid);
return 0;
fail:
xe_gt_sriov_dbg(gt, "Failed to restore VF%u GuC state (%pe)\n", vfid, ERR_PTR(ret));
return ret;
}
/**
* xe_gt_sriov_pf_migration_restore_guc_state() - Restore a GuC VF state.
* @gt: the &xe_gt
* @vfid: the VF identifier
*
* This function is for PF only.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_gt_sriov_pf_migration_restore_guc_state(struct xe_gt *gt, unsigned int vfid)
{
int ret;
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
xe_gt_assert(gt, vfid != PFID);
xe_gt_assert(gt, vfid <= xe_sriov_pf_get_totalvfs(gt_to_xe(gt)));
if (!pf_migration_supported(gt))
return -ENOPKG;
mutex_lock(pf_migration_mutex(gt));
ret = pf_restore_vf_guc_state(gt, vfid);
mutex_unlock(pf_migration_mutex(gt));
return ret;
}
#ifdef CONFIG_DEBUG_FS
/**
* xe_gt_sriov_pf_migration_read_guc_state() - Read a GuC VF state.
* @gt: the &xe_gt
* @vfid: the VF identifier
* @buf: the user space buffer to read to
* @count: the maximum number of bytes to read
* @pos: the current position in the buffer
*
* This function is for PF only.
*
* This function reads up to @count bytes from the saved VF GuC state buffer
* at offset @pos into the user space address starting at @buf.
*
* Return: the number of bytes read or a negative error code on failure.
*/
ssize_t xe_gt_sriov_pf_migration_read_guc_state(struct xe_gt *gt, unsigned int vfid,
char __user *buf, size_t count, loff_t *pos)
{
struct xe_gt_sriov_state_snapshot *snapshot;
ssize_t ret;
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
xe_gt_assert(gt, vfid != PFID);
xe_gt_assert(gt, vfid <= xe_sriov_pf_get_totalvfs(gt_to_xe(gt)));
if (!pf_migration_supported(gt))
return -ENOPKG;
mutex_lock(pf_migration_mutex(gt));
snapshot = pf_pick_vf_snapshot(gt, vfid);
if (snapshot->guc.size)
ret = simple_read_from_buffer(buf, count, pos, snapshot->guc.buff,
snapshot->guc.size);
else
ret = -ENODATA;
mutex_unlock(pf_migration_mutex(gt));
return ret;
}
/**
* xe_gt_sriov_pf_migration_write_guc_state() - Write a GuC VF state.
* @gt: the &xe_gt
* @vfid: the VF identifier
* @buf: the user space buffer with GuC VF state
* @size: the size of GuC VF state (in bytes)
*
* This function is for PF only.
*
* This function reads @size bytes of the VF GuC state stored at user space
* address @buf and writes it into a internal VF state buffer.
*
* Return: the number of bytes used or a negative error code on failure.
*/
ssize_t xe_gt_sriov_pf_migration_write_guc_state(struct xe_gt *gt, unsigned int vfid,
const char __user *buf, size_t size)
{
struct xe_gt_sriov_state_snapshot *snapshot;
loff_t pos = 0;
ssize_t ret;
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
xe_gt_assert(gt, vfid != PFID);
xe_gt_assert(gt, vfid <= xe_sriov_pf_get_totalvfs(gt_to_xe(gt)));
if (!pf_migration_supported(gt))
return -ENOPKG;
mutex_lock(pf_migration_mutex(gt));
snapshot = pf_pick_vf_snapshot(gt, vfid);
ret = pf_alloc_guc_state(gt, snapshot, size);
if (!ret) {
ret = simple_write_to_buffer(snapshot->guc.buff, size, &pos, buf, size);
if (ret < 0)
pf_free_guc_state(gt, snapshot);
else
pf_dump_guc_state(gt, snapshot);
}
mutex_unlock(pf_migration_mutex(gt));
return ret;
}
#endif /* CONFIG_DEBUG_FS */
static bool pf_check_migration_support(struct xe_gt *gt)
{
/* GuC 70.25 with save/restore v2 is required */
xe_gt_assert(gt, GUC_FIRMWARE_VER(&gt->uc.guc) >= MAKE_GUC_VER(70, 25, 0));
/* XXX: for now this is for feature enabling only */
return IS_ENABLED(CONFIG_DRM_XE_DEBUG);
}
/**
* xe_gt_sriov_pf_migration_init() - Initialize support for VF migration.
* @gt: the &xe_gt
*
* This function is for PF only.
*
* Return: 0 on success or a negative error code on failure.
*/
int xe_gt_sriov_pf_migration_init(struct xe_gt *gt)
{
struct xe_device *xe = gt_to_xe(gt);
int err;
xe_gt_assert(gt, IS_SRIOV_PF(xe));
gt->sriov.pf.migration.supported = pf_check_migration_support(gt);
if (!pf_migration_supported(gt))
return 0;
err = drmm_mutex_init(&xe->drm, &gt->sriov.pf.migration.snapshot_lock);
if (err)
return err;
return 0;
}

24
drivers/gpu/drm/xe/xe_gt_sriov_pf_migration.h Normal file
View File

@ -0,0 +1,24 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2024 Intel Corporation
*/
#ifndef _XE_GT_SRIOV_PF_MIGRATION_H_
#define _XE_GT_SRIOV_PF_MIGRATION_H_
#include <linux/types.h>
struct xe_gt;
int xe_gt_sriov_pf_migration_init(struct xe_gt *gt);
int xe_gt_sriov_pf_migration_save_guc_state(struct xe_gt *gt, unsigned int vfid);
int xe_gt_sriov_pf_migration_restore_guc_state(struct xe_gt *gt, unsigned int vfid);
#ifdef CONFIG_DEBUG_FS
ssize_t xe_gt_sriov_pf_migration_read_guc_state(struct xe_gt *gt, unsigned int vfid,
char __user *buf, size_t count, loff_t *pos);
ssize_t xe_gt_sriov_pf_migration_write_guc_state(struct xe_gt *gt, unsigned int vfid,
const char __user *buf, size_t count);
#endif
#endif

40
drivers/gpu/drm/xe/xe_gt_sriov_pf_migration_types.h Normal file
View File

@ -0,0 +1,40 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2024 Intel Corporation
*/
#ifndef _XE_GT_SRIOV_PF_MIGRATION_TYPES_H_
#define _XE_GT_SRIOV_PF_MIGRATION_TYPES_H_
#include <linux/mutex.h>
#include <linux/types.h>
/**
* struct xe_gt_sriov_state_snapshot - GT-level per-VF state snapshot data.
*
* Used by the PF driver to maintain per-VF migration data.
*/
struct xe_gt_sriov_state_snapshot {
/** @guc: GuC VF state snapshot */
struct {
/** @guc.buff: buffer with the VF state */
u32 *buff;
/** @guc.size: size of the buffer (must be dwords aligned) */
u32 size;
} guc;
};
/**
* struct xe_gt_sriov_pf_migration - GT-level data.
*
* Used by the PF driver to maintain non-VF specific per-GT data.
*/
struct xe_gt_sriov_pf_migration {
/** @supported: indicates whether the feature is supported */
bool supported;
/** @snapshot_lock: protects all VFs snapshots */
struct mutex snapshot_lock;
};
#endif

6
drivers/gpu/drm/xe/xe_gt_sriov_pf_service.c
View File

@ -237,7 +237,7 @@ static void read_many(struct xe_gt *gt, unsigned int count,
const struct xe_reg *regs, u32 *values)
{
while (count--)
*values++ = xe_mmio_read32(gt, *regs++);
*values++ = xe_mmio_read32(&gt->mmio, *regs++);
}
static void pf_prepare_runtime_info(struct xe_gt *gt)
@ -402,7 +402,7 @@ static int pf_service_runtime_query(struct xe_gt *gt, u32 start, u32 limit,
for (i = 0; i < count; ++i, ++data) {
addr = runtime->regs[start + i].addr;
data->offset = xe_mmio_adjusted_addr(gt, addr);
data->offset = xe_mmio_adjusted_addr(&gt->mmio, addr);
data->value = runtime->values[start + i];
}
@ -513,7 +513,7 @@ int xe_gt_sriov_pf_service_print_runtime(struct xe_gt *gt, struct drm_printer *p
for (; size--; regs++, values++) {
drm_printf(p, "reg[%#x] = %#x\n",
xe_mmio_adjusted_addr(gt, regs->addr), *values);
xe_mmio_adjusted_addr(&gt->mmio, regs->addr), *values);
}
return 0;

6
drivers/gpu/drm/xe/xe_gt_sriov_pf_types.h
View File

@ -10,6 +10,7 @@
#include "xe_gt_sriov_pf_config_types.h"
#include "xe_gt_sriov_pf_control_types.h"
#include "xe_gt_sriov_pf_migration_types.h"
#include "xe_gt_sriov_pf_monitor_types.h"
#include "xe_gt_sriov_pf_policy_types.h"
#include "xe_gt_sriov_pf_service_types.h"
@ -29,6 +30,9 @@ struct xe_gt_sriov_metadata {
/** @version: negotiated VF/PF ABI version */
struct xe_gt_sriov_pf_service_version version;
/** @snapshot: snapshot of the VF state data */
struct xe_gt_sriov_state_snapshot snapshot;
};
/**
@ -36,6 +40,7 @@ struct xe_gt_sriov_metadata {
* @service: service data.
* @control: control data.
* @policy: policy data.
* @migration: migration data.
* @spare: PF-only provisioning configuration.
* @vfs: metadata for all VFs.
*/
@ -43,6 +48,7 @@ struct xe_gt_sriov_pf {
struct xe_gt_sriov_pf_service service;
struct xe_gt_sriov_pf_control control;
struct xe_gt_sriov_pf_policy policy;
struct xe_gt_sriov_pf_migration migration;
struct xe_gt_sriov_spare_config spare;
struct xe_gt_sriov_metadata *vfs;
};

4
drivers/gpu/drm/xe/xe_gt_sriov_vf.c
View File

@ -881,7 +881,7 @@ static struct vf_runtime_reg *vf_lookup_reg(struct xe_gt *gt, u32 addr)
*/
u32 xe_gt_sriov_vf_read32(struct xe_gt *gt, struct xe_reg reg)
{
u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
u32 addr = xe_mmio_adjusted_addr(&gt->mmio, reg.addr);
struct vf_runtime_reg *rr;
xe_gt_assert(gt, IS_SRIOV_VF(gt_to_xe(gt)));
@ -917,7 +917,7 @@ u32 xe_gt_sriov_vf_read32(struct xe_gt *gt, struct xe_reg reg)
*/
void xe_gt_sriov_vf_write32(struct xe_gt *gt, struct xe_reg reg, u32 val)
{
u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
u32 addr = xe_mmio_adjusted_addr(&gt->mmio, reg.addr);
xe_gt_assert(gt, IS_SRIOV_VF(gt_to_xe(gt)));
xe_gt_assert(gt, !reg.vf);

2
drivers/gpu/drm/xe/xe_gt_sriov_vf_debugfs.c
View File

@ -33,7 +33,7 @@ static const struct drm_info_list vf_info[] = {
.show = xe_gt_debugfs_simple_show,
.data = xe_gt_sriov_vf_print_version,
},
#if defined(CONFIG_DRM_XE_DEBUG) || defined(CONFIG_DRM_XE_DEBUG_SRIOV)
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG) || IS_ENABLED(CONFIG_DRM_XE_DEBUG_SRIOV)
{
"runtime_regs",
.show = xe_gt_debugfs_simple_show,

4
drivers/gpu/drm/xe/xe_gt_throttle.c
View File

@ -41,9 +41,9 @@ u32 xe_gt_throttle_get_limit_reasons(struct xe_gt *gt)
xe_pm_runtime_get(gt_to_xe(gt));
if (xe_gt_is_media_type(gt))
reg = xe_mmio_read32(gt, MTL_MEDIA_PERF_LIMIT_REASONS);
reg = xe_mmio_read32(&gt->mmio, MTL_MEDIA_PERF_LIMIT_REASONS);
else
reg = xe_mmio_read32(gt, GT0_PERF_LIMIT_REASONS);
reg = xe_mmio_read32(&gt->mmio, GT0_PERF_LIMIT_REASONS);
xe_pm_runtime_put(gt_to_xe(gt));
return reg;

37
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c
View File

@ -37,6 +37,15 @@ static long tlb_timeout_jiffies(struct xe_gt *gt)
return hw_tlb_timeout + 2 * delay;
}
static void xe_gt_tlb_invalidation_fence_fini(struct xe_gt_tlb_invalidation_fence *fence)
{
if (WARN_ON_ONCE(!fence->gt))
return;
xe_pm_runtime_put(gt_to_xe(fence->gt));
fence->gt = NULL; /* fini() should be called once */
}
static void
__invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fence *fence)
{
@ -204,7 +213,7 @@ static int send_tlb_invalidation(struct xe_guc *guc,
tlb_timeout_jiffies(gt));
}
spin_unlock_irq(&gt->tlb_invalidation.pending_lock);
} else if (ret < 0) {
} else {
__invalidation_fence_signal(xe, fence);
}
if (!ret) {
@ -267,24 +276,24 @@ int xe_gt_tlb_invalidation_ggtt(struct xe_gt *gt)
xe_gt_tlb_invalidation_fence_init(gt, &fence, true);
ret = xe_gt_tlb_invalidation_guc(gt, &fence);
if (ret < 0) {
xe_gt_tlb_invalidation_fence_fini(&fence);
if (ret)
return ret;
}
xe_gt_tlb_invalidation_fence_wait(&fence);
} else if (xe_device_uc_enabled(xe) && !xe_device_wedged(xe)) {
struct xe_mmio *mmio = &gt->mmio;
if (IS_SRIOV_VF(xe))
return 0;
xe_gt_WARN_ON(gt, xe_force_wake_get(gt_to_fw(gt), XE_FW_GT));
if (xe->info.platform == XE_PVC || GRAPHICS_VER(xe) >= 20) {
xe_mmio_write32(gt, PVC_GUC_TLB_INV_DESC1,
xe_mmio_write32(mmio, PVC_GUC_TLB_INV_DESC1,
PVC_GUC_TLB_INV_DESC1_INVALIDATE);
xe_mmio_write32(gt, PVC_GUC_TLB_INV_DESC0,
xe_mmio_write32(mmio, PVC_GUC_TLB_INV_DESC0,
PVC_GUC_TLB_INV_DESC0_VALID);
} else {
xe_mmio_write32(gt, GUC_TLB_INV_CR,
xe_mmio_write32(mmio, GUC_TLB_INV_CR,
GUC_TLB_INV_CR_INVALIDATE);
}
xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
@ -496,7 +505,8 @@ static const struct dma_fence_ops invalidation_fence_ops = {
* @stack: fence is stack variable
*
* Initialize TLB invalidation fence for use. xe_gt_tlb_invalidation_fence_fini
* must be called if fence is not signaled.
* will be automatically called when fence is signalled (all fences must signal),
* even on error.
*/
void xe_gt_tlb_invalidation_fence_init(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence,
@ -516,14 +526,3 @@ void xe_gt_tlb_invalidation_fence_init(struct xe_gt *gt,
dma_fence_get(&fence->base);
fence->gt = gt;
}
/**
* xe_gt_tlb_invalidation_fence_fini - Finalize TLB invalidation fence
* @fence: TLB invalidation fence to finalize
*
* Drop PM ref which fence took durinig init.
*/
void xe_gt_tlb_invalidation_fence_fini(struct xe_gt_tlb_invalidation_fence *fence)
{
xe_pm_runtime_put(gt_to_xe(fence->gt));
}

1
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h
View File

@ -28,7 +28,6 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len);
void xe_gt_tlb_invalidation_fence_init(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence,
bool stack);
void xe_gt_tlb_invalidation_fence_fini(struct xe_gt_tlb_invalidation_fence *fence);
static inline void
xe_gt_tlb_invalidation_fence_wait(struct xe_gt_tlb_invalidation_fence *fence)

22
drivers/gpu/drm/xe/xe_gt_topology.c
View File

@ -5,6 +5,7 @@
#include "xe_gt_topology.h"
#include <generated/xe_wa_oob.h>
#include <linux/bitmap.h>
#include <linux/compiler.h>
@ -12,6 +13,7 @@
#include "xe_assert.h"
#include "xe_gt.h"
#include "xe_mmio.h"
#include "xe_wa.h"
static void
load_dss_mask(struct xe_gt *gt, xe_dss_mask_t mask, int numregs, ...)
@ -25,7 +27,7 @@ load_dss_mask(struct xe_gt *gt, xe_dss_mask_t mask, int numregs, ...)
va_start(argp, numregs);
for (i = 0; i < numregs; i++)
fuse_val[i] = xe_mmio_read32(gt, va_arg(argp, struct xe_reg));
fuse_val[i] = xe_mmio_read32(&gt->mmio, va_arg(argp, struct xe_reg));
va_end(argp);
bitmap_from_arr32(mask, fuse_val, numregs * 32);
@ -35,7 +37,7 @@ static void
load_eu_mask(struct xe_gt *gt, xe_eu_mask_t mask, enum xe_gt_eu_type *eu_type)
{
struct xe_device *xe = gt_to_xe(gt);
u32 reg_val = xe_mmio_read32(gt, XELP_EU_ENABLE);
u32 reg_val = xe_mmio_read32(&gt->mmio, XELP_EU_ENABLE);
u32 val = 0;
int i;
@ -127,7 +129,19 @@ static void
load_l3_bank_mask(struct xe_gt *gt, xe_l3_bank_mask_t l3_bank_mask)
{
struct xe_device *xe = gt_to_xe(gt);
u32 fuse3 = xe_mmio_read32(gt, MIRROR_FUSE3);
u32 fuse3 = xe_mmio_read32(&gt->mmio, MIRROR_FUSE3);
/*
* PTL platforms with media version 30.00 do not provide proper values
* for the media GT's L3 bank registers. Skip the readout since we
* don't have any way to obtain real values.
*
* This may get re-described as an official workaround in the future,
* but there's no tracking number assigned yet so we use a custom
* OOB workaround descriptor.
*/
if (XE_WA(gt, no_media_l3))
return;
if (GRAPHICS_VER(xe) >= 20) {
xe_l3_bank_mask_t per_node = {};
@ -141,7 +155,7 @@ load_l3_bank_mask(struct xe_gt *gt, xe_l3_bank_mask_t l3_bank_mask)
xe_l3_bank_mask_t per_node = {};
xe_l3_bank_mask_t per_mask_bit = {};
u32 meml3_en = REG_FIELD_GET(MEML3_EN_MASK, fuse3);
u32 fuse4 = xe_mmio_read32(gt, XEHP_FUSE4);
u32 fuse4 = xe_mmio_read32(&gt->mmio, XEHP_FUSE4);
u32 bank_val = REG_FIELD_GET(GT_L3_EXC_MASK, fuse4);
bitmap_set_value8(per_mask_bit, 0x3, 0);

22
drivers/gpu/drm/xe/xe_gt_types.h
View File

@ -6,6 +6,7 @@
#ifndef _XE_GT_TYPES_H_
#define _XE_GT_TYPES_H_
#include "xe_device_types.h"
#include "xe_force_wake_types.h"
#include "xe_gt_idle_types.h"
#include "xe_gt_sriov_pf_types.h"
@ -145,19 +146,20 @@ struct xe_gt {
/**
* @mmio: mmio info for GT. All GTs within a tile share the same
* register space, but have their own copy of GSI registers at a
* specific offset, as well as their own forcewake handling.
* specific offset.
*/
struct xe_mmio mmio;
/**
* @pm: power management info for GT. The driver uses the GT's
* "force wake" interface to wake up specific parts of the GT hardware
* from C6 sleep states and ensure the hardware remains awake while it
* is being actively used.
*/
struct {
/** @mmio.fw: force wake for GT */
/** @pm.fw: force wake for GT */
struct xe_force_wake fw;
/**
* @mmio.adj_limit: adjust MMIO address if address is below this
* value
*/
u32 adj_limit;
/** @mmio.adj_offset: offect to add to MMIO address when adjusting */
u32 adj_offset;
} mmio;
} pm;
/** @sriov: virtualization data related to GT */
union {

72
drivers/gpu/drm/xe/xe_guc.c
View File

@ -14,6 +14,7 @@
#include "regs/xe_gt_regs.h"
#include "regs/xe_gtt_defs.h"
#include "regs/xe_guc_regs.h"
#include "regs/xe_irq_regs.h"
#include "xe_bo.h"
#include "xe_device.h"
#include "xe_force_wake.h"
@ -22,6 +23,7 @@
#include "xe_gt_sriov_vf.h"
#include "xe_gt_throttle.h"
#include "xe_guc_ads.h"
#include "xe_guc_capture.h"
#include "xe_guc_ct.h"
#include "xe_guc_db_mgr.h"
#include "xe_guc_hwconfig.h"
@ -236,10 +238,10 @@ static void guc_write_params(struct xe_guc *guc)
xe_force_wake_assert_held(gt_to_fw(gt), XE_FW_GT);
xe_mmio_write32(gt, SOFT_SCRATCH(0), 0);
xe_mmio_write32(&gt->mmio, SOFT_SCRATCH(0), 0);
for (i = 0; i < GUC_CTL_MAX_DWORDS; i++)
xe_mmio_write32(gt, SOFT_SCRATCH(1 + i), guc->params[i]);
xe_mmio_write32(&gt->mmio, SOFT_SCRATCH(1 + i), guc->params[i]);
}
static void guc_fini_hw(void *arg)
@ -338,6 +340,10 @@ int xe_guc_init(struct xe_guc *guc)
if (ret)
goto out;
ret = xe_guc_capture_init(guc);
if (ret)
goto out;
ret = xe_guc_ads_init(&guc->ads);
if (ret)
goto out;
@ -425,6 +431,7 @@ int xe_guc_post_load_init(struct xe_guc *guc)
int xe_guc_reset(struct xe_guc *guc)
{
struct xe_gt *gt = guc_to_gt(guc);
struct xe_mmio *mmio = &gt->mmio;
u32 guc_status, gdrst;
int ret;
@ -433,15 +440,15 @@ int xe_guc_reset(struct xe_guc *guc)
if (IS_SRIOV_VF(gt_to_xe(gt)))
return xe_gt_sriov_vf_bootstrap(gt);
xe_mmio_write32(gt, GDRST, GRDOM_GUC);
xe_mmio_write32(mmio, GDRST, GRDOM_GUC);
ret = xe_mmio_wait32(gt, GDRST, GRDOM_GUC, 0, 5000, &gdrst, false);
ret = xe_mmio_wait32(mmio, GDRST, GRDOM_GUC, 0, 5000, &gdrst, false);
if (ret) {
xe_gt_err(gt, "GuC reset timed out, GDRST=%#x\n", gdrst);
goto err_out;
}
guc_status = xe_mmio_read32(gt, GUC_STATUS);
guc_status = xe_mmio_read32(mmio, GUC_STATUS);
if (!(guc_status & GS_MIA_IN_RESET)) {
xe_gt_err(gt, "GuC status: %#x, MIA core expected to be in reset\n",
guc_status);
@ -459,6 +466,7 @@ int xe_guc_reset(struct xe_guc *guc)
static void guc_prepare_xfer(struct xe_guc *guc)
{
struct xe_gt *gt = guc_to_gt(guc);
struct xe_mmio *mmio = &gt->mmio;
struct xe_device *xe = guc_to_xe(guc);
u32 shim_flags = GUC_ENABLE_READ_CACHE_LOGIC |
GUC_ENABLE_READ_CACHE_FOR_SRAM_DATA |
@ -473,12 +481,12 @@ static void guc_prepare_xfer(struct xe_guc *guc)
shim_flags |= REG_FIELD_PREP(GUC_MOCS_INDEX_MASK, gt->mocs.uc_index);
/* Must program this register before loading the ucode with DMA */
xe_mmio_write32(gt, GUC_SHIM_CONTROL, shim_flags);
xe_mmio_write32(mmio, GUC_SHIM_CONTROL, shim_flags);
xe_mmio_write32(gt, GT_PM_CONFIG, GT_DOORBELL_ENABLE);
xe_mmio_write32(mmio, GT_PM_CONFIG, GT_DOORBELL_ENABLE);
/* Make sure GuC receives ARAT interrupts */
xe_mmio_rmw32(gt, PMINTRMSK, ARAT_EXPIRED_INTRMSK, 0);
xe_mmio_rmw32(mmio, PMINTRMSK, ARAT_EXPIRED_INTRMSK, 0);
}
/*
@ -494,7 +502,7 @@ static int guc_xfer_rsa(struct xe_guc *guc)
if (guc->fw.rsa_size > 256) {
u32 rsa_ggtt_addr = xe_bo_ggtt_addr(guc->fw.bo) +
xe_uc_fw_rsa_offset(&guc->fw);
xe_mmio_write32(gt, UOS_RSA_SCRATCH(0), rsa_ggtt_addr);
xe_mmio_write32(&gt->mmio, UOS_RSA_SCRATCH(0), rsa_ggtt_addr);
return 0;
}
@ -503,7 +511,7 @@ static int guc_xfer_rsa(struct xe_guc *guc)
return -ENOMEM;
for (i = 0; i < UOS_RSA_SCRATCH_COUNT; i++)
xe_mmio_write32(gt, UOS_RSA_SCRATCH(i), rsa[i]);
xe_mmio_write32(&gt->mmio, UOS_RSA_SCRATCH(i), rsa[i]);
return 0;
}
@ -583,7 +591,7 @@ static s32 guc_pc_get_cur_freq(struct xe_guc_pc *guc_pc)
* extreme thermal throttling. And a system that is that hot during boot is probably
* dead anyway!
*/
#if defined(CONFIG_DRM_XE_DEBUG)
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
#define GUC_LOAD_RETRY_LIMIT 20
#else
#define GUC_LOAD_RETRY_LIMIT 3
@ -593,6 +601,7 @@ static s32 guc_pc_get_cur_freq(struct xe_guc_pc *guc_pc)
static void guc_wait_ucode(struct xe_guc *guc)
{
struct xe_gt *gt = guc_to_gt(guc);
struct xe_mmio *mmio = &gt->mmio;
struct xe_guc_pc *guc_pc = &gt->uc.guc.pc;
ktime_t before, after, delta;
int load_done;
@ -619,7 +628,7 @@ static void guc_wait_ucode(struct xe_guc *guc)
* timeouts rather than allowing a huge timeout each time. So basically, need
* to treat a timeout no different to a value change.
*/
ret = xe_mmio_wait32_not(gt, GUC_STATUS, GS_UKERNEL_MASK | GS_BOOTROM_MASK,
ret = xe_mmio_wait32_not(mmio, GUC_STATUS, GS_UKERNEL_MASK | GS_BOOTROM_MASK,
last_status, 1000 * 1000, &status, false);
if (ret < 0)
count++;
@ -657,7 +666,7 @@ static void guc_wait_ucode(struct xe_guc *guc)
switch (bootrom) {
case XE_BOOTROM_STATUS_NO_KEY_FOUND:
xe_gt_err(gt, "invalid key requested, header = 0x%08X\n",
xe_mmio_read32(gt, GUC_HEADER_INFO));
xe_mmio_read32(mmio, GUC_HEADER_INFO));
break;
case XE_BOOTROM_STATUS_RSA_FAILED:
@ -672,7 +681,7 @@ static void guc_wait_ucode(struct xe_guc *guc)
switch (ukernel) {
case XE_GUC_LOAD_STATUS_EXCEPTION:
xe_gt_err(gt, "firmware exception. EIP: %#x\n",
xe_mmio_read32(gt, SOFT_SCRATCH(13)));
xe_mmio_read32(mmio, SOFT_SCRATCH(13)));
break;
case XE_GUC_LOAD_STATUS_INIT_MMIO_SAVE_RESTORE_INVALID:
@ -824,10 +833,10 @@ static void guc_handle_mmio_msg(struct xe_guc *guc)
xe_force_wake_assert_held(gt_to_fw(gt), XE_FW_GT);
msg = xe_mmio_read32(gt, SOFT_SCRATCH(15));
msg = xe_mmio_read32(&gt->mmio, SOFT_SCRATCH(15));
msg &= XE_GUC_RECV_MSG_EXCEPTION |
XE_GUC_RECV_MSG_CRASH_DUMP_POSTED;
xe_mmio_write32(gt, SOFT_SCRATCH(15), 0);
xe_mmio_write32(&gt->mmio, SOFT_SCRATCH(15), 0);
if (msg & XE_GUC_RECV_MSG_CRASH_DUMP_POSTED)
xe_gt_err(gt, "Received early GuC crash dump notification!\n");
@ -844,14 +853,14 @@ static void guc_enable_irq(struct xe_guc *guc)
REG_FIELD_PREP(ENGINE1_MASK, GUC_INTR_GUC2HOST);
/* Primary GuC and media GuC share a single enable bit */
xe_mmio_write32(gt, GUC_SG_INTR_ENABLE,
xe_mmio_write32(&gt->mmio, GUC_SG_INTR_ENABLE,
REG_FIELD_PREP(ENGINE1_MASK, GUC_INTR_GUC2HOST));
/*
* There are separate mask bits for primary and media GuCs, so use
* a RMW operation to avoid clobbering the other GuC's setting.
*/
xe_mmio_rmw32(gt, GUC_SG_INTR_MASK, events, 0);
xe_mmio_rmw32(&gt->mmio, GUC_SG_INTR_MASK, events, 0);
}
int xe_guc_enable_communication(struct xe_guc *guc)
@ -863,7 +872,7 @@ int xe_guc_enable_communication(struct xe_guc *guc)
struct xe_gt *gt = guc_to_gt(guc);
struct xe_tile *tile = gt_to_tile(gt);
err = xe_memirq_init_guc(&tile->sriov.vf.memirq, guc);
err = xe_memirq_init_guc(&tile->memirq, guc);
if (err)
return err;
} else {
@ -907,7 +916,7 @@ void xe_guc_notify(struct xe_guc *guc)
* additional payload data to the GuC but this capability is not
* used by the firmware yet. Use default value in the meantime.
*/
xe_mmio_write32(gt, guc->notify_reg, default_notify_data);
xe_mmio_write32(&gt->mmio, guc->notify_reg, default_notify_data);
}
int xe_guc_auth_huc(struct xe_guc *guc, u32 rsa_addr)
@ -925,6 +934,7 @@ int xe_guc_mmio_send_recv(struct xe_guc *guc, const u32 *request,
{
struct xe_device *xe = guc_to_xe(guc);
struct xe_gt *gt = guc_to_gt(guc);
struct xe_mmio *mmio = &gt->mmio;
u32 header, reply;
struct xe_reg reply_reg = xe_gt_is_media_type(gt) ?
MED_VF_SW_FLAG(0) : VF_SW_FLAG(0);
@ -947,19 +957,19 @@ int xe_guc_mmio_send_recv(struct xe_guc *guc, const u32 *request,
/* Not in critical data-path, just do if else for GT type */
if (xe_gt_is_media_type(gt)) {
for (i = 0; i < len; ++i)
xe_mmio_write32(gt, MED_VF_SW_FLAG(i),
xe_mmio_write32(mmio, MED_VF_SW_FLAG(i),
request[i]);
xe_mmio_read32(gt, MED_VF_SW_FLAG(LAST_INDEX));
xe_mmio_read32(mmio, MED_VF_SW_FLAG(LAST_INDEX));
} else {
for (i = 0; i < len; ++i)
xe_mmio_write32(gt, VF_SW_FLAG(i),
xe_mmio_write32(mmio, VF_SW_FLAG(i),
request[i]);
xe_mmio_read32(gt, VF_SW_FLAG(LAST_INDEX));
xe_mmio_read32(mmio, VF_SW_FLAG(LAST_INDEX));
}
xe_guc_notify(guc);
ret = xe_mmio_wait32(gt, reply_reg, GUC_HXG_MSG_0_ORIGIN,
ret = xe_mmio_wait32(mmio, reply_reg, GUC_HXG_MSG_0_ORIGIN,
FIELD_PREP(GUC_HXG_MSG_0_ORIGIN, GUC_HXG_ORIGIN_GUC),
50000, &reply, false);
if (ret) {
@ -969,7 +979,7 @@ int xe_guc_mmio_send_recv(struct xe_guc *guc, const u32 *request,
return ret;
}
header = xe_mmio_read32(gt, reply_reg);
header = xe_mmio_read32(mmio, reply_reg);
if (FIELD_GET(GUC_HXG_MSG_0_TYPE, header) ==
GUC_HXG_TYPE_NO_RESPONSE_BUSY) {
/*
@ -985,7 +995,7 @@ int xe_guc_mmio_send_recv(struct xe_guc *guc, const u32 *request,
BUILD_BUG_ON(FIELD_MAX(GUC_HXG_MSG_0_TYPE) != GUC_HXG_TYPE_RESPONSE_SUCCESS);
BUILD_BUG_ON((GUC_HXG_TYPE_RESPONSE_SUCCESS ^ GUC_HXG_TYPE_RESPONSE_FAILURE) != 1);
ret = xe_mmio_wait32(gt, reply_reg, resp_mask, resp_mask,
ret = xe_mmio_wait32(mmio, reply_reg, resp_mask, resp_mask,
1000000, &header, false);
if (unlikely(FIELD_GET(GUC_HXG_MSG_0_ORIGIN, header) !=
@ -1032,7 +1042,7 @@ int xe_guc_mmio_send_recv(struct xe_guc *guc, const u32 *request,
for (i = 1; i < VF_SW_FLAG_COUNT; i++) {
reply_reg.addr += sizeof(u32);
response_buf[i] = xe_mmio_read32(gt, reply_reg);
response_buf[i] = xe_mmio_read32(mmio, reply_reg);
}
}
@ -1155,7 +1165,7 @@ void xe_guc_print_info(struct xe_guc *guc, struct drm_printer *p)
if (err)
return;
status = xe_mmio_read32(gt, GUC_STATUS);
status = xe_mmio_read32(&gt->mmio, GUC_STATUS);
drm_printf(p, "\nGuC status 0x%08x:\n", status);
drm_printf(p, "\tBootrom status = 0x%x\n",
@ -1170,12 +1180,12 @@ void xe_guc_print_info(struct xe_guc *guc, struct drm_printer *p)
drm_puts(p, "\nScratch registers:\n");
for (i = 0; i < SOFT_SCRATCH_COUNT; i++) {
drm_printf(p, "\t%2d: \t0x%x\n",
i, xe_mmio_read32(gt, SOFT_SCRATCH(i)));
i, xe_mmio_read32(&gt->mmio, SOFT_SCRATCH(i)));
}
xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
xe_guc_ct_print(&guc->ct, p, false);
xe_guc_ct_print(&guc->ct, p);
xe_guc_submit_print(guc, p);
}

5
drivers/gpu/drm/xe/xe_guc.h
View File

@ -82,4 +82,9 @@ static inline struct xe_device *guc_to_xe(struct xe_guc *guc)
return gt_to_xe(guc_to_gt(guc));
}
static inline struct drm_device *guc_to_drm(struct xe_guc *guc)
{
return &guc_to_xe(guc)->drm;
}
#endif

164
drivers/gpu/drm/xe/xe_guc_ads.c
View File

@ -5,6 +5,8 @@
#include "xe_guc_ads.h"
#include <linux/fault-inject.h>
#include <drm/drm_managed.h>
#include <generated/xe_wa_oob.h>
@ -18,6 +20,7 @@
#include "xe_gt_ccs_mode.h"
#include "xe_gt_printk.h"
#include "xe_guc.h"
#include "xe_guc_capture.h"
#include "xe_guc_ct.h"
#include "xe_hw_engine.h"
#include "xe_lrc.h"
@ -149,8 +152,7 @@ static u32 guc_ads_waklv_size(struct xe_guc_ads *ads)
static size_t guc_ads_capture_size(struct xe_guc_ads *ads)
{
/* FIXME: Allocate a proper capture list */
return PAGE_ALIGN(PAGE_SIZE);
return PAGE_ALIGN(ads->capture_size);
}
static size_t guc_ads_um_queues_size(struct xe_guc_ads *ads)
@ -404,6 +406,7 @@ int xe_guc_ads_init(struct xe_guc_ads *ads)
struct xe_bo *bo;
ads->golden_lrc_size = calculate_golden_lrc_size(ads);
ads->capture_size = xe_guc_capture_ads_input_worst_size(ads_to_guc(ads));
ads->regset_size = calculate_regset_size(gt);
ads->ads_waklv_size = calculate_waklv_size(ads);
@ -418,14 +421,15 @@ int xe_guc_ads_init(struct xe_guc_ads *ads)
return 0;
}
ALLOW_ERROR_INJECTION(xe_guc_ads_init, ERRNO); /* See xe_pci_probe() */
/**
* xe_guc_ads_init_post_hwconfig - initialize ADS post hwconfig load
* @ads: Additional data structures object
*
* Recalcuate golden_lrc_size & regset_size as the number hardware engines may
* have changed after the hwconfig was loaded. Also verify the new sizes fit in
* the already allocated ADS buffer object.
* Recalculate golden_lrc_size, capture_size and regset_size as the number
* hardware engines may have changed after the hwconfig was loaded. Also verify
* the new sizes fit in the already allocated ADS buffer object.
*
* Return: 0 on success, negative error code on error.
*/
@ -437,6 +441,8 @@ int xe_guc_ads_init_post_hwconfig(struct xe_guc_ads *ads)
xe_gt_assert(gt, ads->bo);
ads->golden_lrc_size = calculate_golden_lrc_size(ads);
/* Calculate Capture size with worst size */
ads->capture_size = xe_guc_capture_ads_input_worst_size(ads_to_guc(ads));
ads->regset_size = calculate_regset_size(gt);
xe_gt_assert(gt, ads->golden_lrc_size +
@ -536,20 +542,148 @@ static void guc_mapping_table_init(struct xe_gt *gt,
}
}
static void guc_capture_list_init(struct xe_guc_ads *ads)
static u32 guc_get_capture_engine_mask(struct xe_gt *gt, struct iosys_map *info_map,
enum guc_capture_list_class_type capture_class)
{
int i, j;
u32 addr = xe_bo_ggtt_addr(ads->bo) + guc_ads_capture_offset(ads);
struct xe_device *xe = gt_to_xe(gt);
u32 mask;
/* FIXME: Populate a proper capture list */
switch (capture_class) {
case GUC_CAPTURE_LIST_CLASS_RENDER_COMPUTE:
mask = info_map_read(xe, info_map, engine_enabled_masks[GUC_RENDER_CLASS]);
mask |= info_map_read(xe, info_map, engine_enabled_masks[GUC_COMPUTE_CLASS]);
break;
case GUC_CAPTURE_LIST_CLASS_VIDEO:
mask = info_map_read(xe, info_map, engine_enabled_masks[GUC_VIDEO_CLASS]);
break;
case GUC_CAPTURE_LIST_CLASS_VIDEOENHANCE:
mask = info_map_read(xe, info_map, engine_enabled_masks[GUC_VIDEOENHANCE_CLASS]);
break;
case GUC_CAPTURE_LIST_CLASS_BLITTER:
mask = info_map_read(xe, info_map, engine_enabled_masks[GUC_BLITTER_CLASS]);
break;
case GUC_CAPTURE_LIST_CLASS_GSC_OTHER:
mask = info_map_read(xe, info_map, engine_enabled_masks[GUC_GSC_OTHER_CLASS]);
break;
default:
mask = 0;
}
return mask;
}
static inline bool get_capture_list(struct xe_guc_ads *ads, struct xe_guc *guc, struct xe_gt *gt,
int owner, int type, int class, u32 *total_size, size_t *size,
void **pptr)
{
*size = 0;
if (!xe_guc_capture_getlistsize(guc, owner, type, class, size)) {
if (*total_size + *size > ads->capture_size)
xe_gt_dbg(gt, "Capture size overflow :%zu vs %d\n",
*total_size + *size, ads->capture_size);
else if (!xe_guc_capture_getlist(guc, owner, type, class, pptr))
return false;
}
return true;
}
static int guc_capture_prep_lists(struct xe_guc_ads *ads)
{
struct xe_guc *guc = ads_to_guc(ads);
struct xe_gt *gt = ads_to_gt(ads);
u32 ads_ggtt, capture_offset, null_ggtt, total_size = 0;
struct iosys_map info_map;
size_t size = 0;
void *ptr;
int i, j;
/*
* GuC Capture's steered reg-list needs to be allocated and initialized
* after the GuC-hwconfig is available which guaranteed from here.
*/
xe_guc_capture_steered_list_init(ads_to_guc(ads));
capture_offset = guc_ads_capture_offset(ads);
ads_ggtt = xe_bo_ggtt_addr(ads->bo);
info_map = IOSYS_MAP_INIT_OFFSET(ads_to_map(ads),
offsetof(struct __guc_ads_blob, system_info));
/* first, set aside the first page for a capture_list with zero descriptors */
total_size = PAGE_SIZE;
if (!xe_guc_capture_getnullheader(guc, &ptr, &size))
xe_map_memcpy_to(ads_to_xe(ads), ads_to_map(ads), capture_offset, ptr, size);
null_ggtt = ads_ggtt + capture_offset;
capture_offset += PAGE_SIZE;
/*
* Populate capture list : at this point adps is already allocated and
* mapped to worst case size
*/
for (i = 0; i < GUC_CAPTURE_LIST_INDEX_MAX; i++) {
for (j = 0; j < GUC_MAX_ENGINE_CLASSES; j++) {
ads_blob_write(ads, ads.capture_instance[i][j], addr);
ads_blob_write(ads, ads.capture_class[i][j], addr);
bool write_empty_list;
for (j = 0; j < GUC_CAPTURE_LIST_CLASS_MAX; j++) {
u32 engine_mask = guc_get_capture_engine_mask(gt, &info_map, j);
/* null list if we dont have said engine or list */
if (!engine_mask) {
ads_blob_write(ads, ads.capture_class[i][j], null_ggtt);
ads_blob_write(ads, ads.capture_instance[i][j], null_ggtt);
continue;
}
/* engine exists: start with engine-class registers */
write_empty_list = get_capture_list(ads, guc, gt, i,
GUC_STATE_CAPTURE_TYPE_ENGINE_CLASS,
j, &total_size, &size, &ptr);
if (!write_empty_list) {
ads_blob_write(ads, ads.capture_class[i][j],
ads_ggtt + capture_offset);
xe_map_memcpy_to(ads_to_xe(ads), ads_to_map(ads), capture_offset,
ptr, size);
total_size += size;
capture_offset += size;
} else {
ads_blob_write(ads, ads.capture_class[i][j], null_ggtt);
}
/* engine exists: next, engine-instance registers */
write_empty_list = get_capture_list(ads, guc, gt, i,
GUC_STATE_CAPTURE_TYPE_ENGINE_INSTANCE,
j, &total_size, &size, &ptr);
if (!write_empty_list) {
ads_blob_write(ads, ads.capture_instance[i][j],
ads_ggtt + capture_offset);
xe_map_memcpy_to(ads_to_xe(ads), ads_to_map(ads), capture_offset,
ptr, size);
total_size += size;
capture_offset += size;
} else {
ads_blob_write(ads, ads.capture_instance[i][j], null_ggtt);
}
}
ads_blob_write(ads, ads.capture_global[i], addr);
/* global registers is last in our PF/VF loops */
write_empty_list = get_capture_list(ads, guc, gt, i,
GUC_STATE_CAPTURE_TYPE_GLOBAL,
0, &total_size, &size, &ptr);
if (!write_empty_list) {
ads_blob_write(ads, ads.capture_global[i], ads_ggtt + capture_offset);
xe_map_memcpy_to(ads_to_xe(ads), ads_to_map(ads), capture_offset, ptr,
size);
total_size += size;
capture_offset += size;
} else {
ads_blob_write(ads, ads.capture_global[i], null_ggtt);
}
}
if (ads->capture_size != PAGE_ALIGN(total_size))
xe_gt_dbg(gt, "ADS capture alloc size changed from %d to %d\n",
ads->capture_size, PAGE_ALIGN(total_size));
return PAGE_ALIGN(total_size);
}
static void guc_mmio_regset_write_one(struct xe_guc_ads *ads,
@ -684,7 +818,7 @@ static void guc_doorbell_init(struct xe_guc_ads *ads)
if (GRAPHICS_VER(xe) >= 12 && !IS_DGFX(xe)) {
u32 distdbreg =
xe_mmio_read32(gt, DIST_DBS_POPULATED);
xe_mmio_read32(&gt->mmio, DIST_DBS_POPULATED);
ads_blob_write(ads,
system_info.generic_gt_sysinfo[GUC_GENERIC_GT_SYSINFO_DOORBELL_COUNT_PER_SQIDI],
@ -738,7 +872,7 @@ void xe_guc_ads_populate(struct xe_guc_ads *ads)
guc_mmio_reg_state_init(ads);
guc_prep_golden_lrc_null(ads);
guc_mapping_table_init(gt, &info_map);
guc_capture_list_init(ads);
guc_capture_prep_lists(ads);
guc_doorbell_init(ads);
guc_waklv_init(ads);

2
drivers/gpu/drm/xe/xe_guc_ads_types.h
View File

@ -22,6 +22,8 @@ struct xe_guc_ads {
u32 regset_size;
/** @ads_waklv_size: total waklv size supported by platform */
u32 ads_waklv_size;
/** @capture_size: size of register set passed to GuC for capture */
u32 capture_size;
};
#endif

1972
drivers/gpu/drm/xe/xe_guc_capture.c Normal file
View File

File diff suppressed because it is too large Load Diff

61
drivers/gpu/drm/xe/xe_guc_capture.h Normal file
View File

@ -0,0 +1,61 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2021-2024 Intel Corporation
*/
#ifndef _XE_GUC_CAPTURE_H
#define _XE_GUC_CAPTURE_H
#include <linux/types.h>
#include "abi/guc_capture_abi.h"
#include "xe_guc.h"
#include "xe_guc_fwif.h"
struct xe_guc;
struct xe_hw_engine;
struct xe_hw_engine_snapshot;
struct xe_sched_job;
static inline enum guc_capture_list_class_type xe_guc_class_to_capture_class(u16 class)
{
switch (class) {
case GUC_RENDER_CLASS:
case GUC_COMPUTE_CLASS:
return GUC_CAPTURE_LIST_CLASS_RENDER_COMPUTE;
case GUC_GSC_OTHER_CLASS:
return GUC_CAPTURE_LIST_CLASS_GSC_OTHER;
case GUC_VIDEO_CLASS:
case GUC_VIDEOENHANCE_CLASS:
case GUC_BLITTER_CLASS:
return class;
default:
XE_WARN_ON(class);
return GUC_CAPTURE_LIST_CLASS_MAX;
}
}
static inline enum guc_capture_list_class_type
xe_engine_class_to_guc_capture_class(enum xe_engine_class class)
{
return xe_guc_class_to_capture_class(xe_engine_class_to_guc_class(class));
}
void xe_guc_capture_process(struct xe_guc *guc);
int xe_guc_capture_getlist(struct xe_guc *guc, u32 owner, u32 type,
enum guc_capture_list_class_type capture_class, void **outptr);
int xe_guc_capture_getlistsize(struct xe_guc *guc, u32 owner, u32 type,
enum guc_capture_list_class_type capture_class, size_t *size);
int xe_guc_capture_getnullheader(struct xe_guc *guc, void **outptr, size_t *size);
size_t xe_guc_capture_ads_input_worst_size(struct xe_guc *guc);
const struct __guc_mmio_reg_descr_group *
xe_guc_capture_get_reg_desc_list(struct xe_gt *gt, u32 owner, u32 type,
enum guc_capture_list_class_type capture_class, bool is_ext);
struct __guc_capture_parsed_output *xe_guc_capture_get_matching_and_lock(struct xe_sched_job *job);
void xe_engine_manual_capture(struct xe_hw_engine *hwe, struct xe_hw_engine_snapshot *snapshot);
void xe_engine_snapshot_print(struct xe_hw_engine_snapshot *snapshot, struct drm_printer *p);
void xe_engine_snapshot_capture_for_job(struct xe_sched_job *job);
void xe_guc_capture_steered_list_init(struct xe_guc *guc);
void xe_guc_capture_put_matched_nodes(struct xe_guc *guc);
int xe_guc_capture_init(struct xe_guc *guc);
#endif

68
drivers/gpu/drm/xe/xe_guc_capture_types.h Normal file
View File

@ -0,0 +1,68 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2021-2024 Intel Corporation
*/
#ifndef _XE_GUC_CAPTURE_TYPES_H
#define _XE_GUC_CAPTURE_TYPES_H
#include <linux/types.h>
#include "regs/xe_reg_defs.h"
struct xe_guc;
/* data type of the register in register list */
enum capture_register_data_type {
REG_32BIT = 0,
REG_64BIT_LOW_DW,
REG_64BIT_HI_DW,
};
/**
* struct __guc_mmio_reg_descr - GuC mmio register descriptor
*
* xe_guc_capture module uses these structures to define a register
* (offsets, names, flags,...) that are used at the ADS regisration
* time as well as during runtime processing and reporting of error-
* capture states generated by GuC just prior to engine reset events.
*/
struct __guc_mmio_reg_descr {
/** @reg: the register */
struct xe_reg reg;
/**
* @data_type: data type of the register
* Could be 32 bit, low or hi dword of a 64 bit, see enum
* register_data_type
*/
enum capture_register_data_type data_type;
/** @flags: Flags for the register */
u32 flags;
/** @mask: The mask to apply */
u32 mask;
/** @regname: Name of the register */
const char *regname;
};
/**
* struct __guc_mmio_reg_descr_group - The group of register descriptor
*
* xe_guc_capture module uses these structures to maintain static
* tables (per unique platform) that consists of lists of registers
* (offsets, names, flags,...) that are used at the ADS regisration
* time as well as during runtime processing and reporting of error-
* capture states generated by GuC just prior to engine reset events.
*/
struct __guc_mmio_reg_descr_group {
/** @list: The register list */
const struct __guc_mmio_reg_descr *list;
/** @num_regs: Count of registers in the list */
u32 num_regs;
/** @owner: PF/VF owner, see enum guc_capture_list_index_type */
u32 owner;
/** @type: Capture register type, see enum guc_state_capture_type */
u32 type;
/** @engine: The engine class, see enum guc_capture_list_class_type */
u32 engine;
};
#endif

486
drivers/gpu/drm/xe/xe_guc_ct.c
View File

@ -8,6 +8,7 @@
#include <linux/bitfield.h>
#include <linux/circ_buf.h>
#include <linux/delay.h>
#include <linux/fault-inject.h>
#include <kunit/static_stub.h>
@ -17,6 +18,7 @@
#include "abi/guc_actions_sriov_abi.h"
#include "abi/guc_klvs_abi.h"
#include "xe_bo.h"
#include "xe_devcoredump.h"
#include "xe_device.h"
#include "xe_gt.h"
#include "xe_gt_pagefault.h"
@ -25,12 +27,48 @@
#include "xe_gt_sriov_pf_monitor.h"
#include "xe_gt_tlb_invalidation.h"
#include "xe_guc.h"
#include "xe_guc_log.h"
#include "xe_guc_relay.h"
#include "xe_guc_submit.h"
#include "xe_map.h"
#include "xe_pm.h"
#include "xe_trace_guc.h"
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
enum {
/* Internal states, not error conditions */
CT_DEAD_STATE_REARM, /* 0x0001 */
CT_DEAD_STATE_CAPTURE, /* 0x0002 */
/* Error conditions */
CT_DEAD_SETUP, /* 0x0004 */
CT_DEAD_H2G_WRITE, /* 0x0008 */
CT_DEAD_H2G_HAS_ROOM, /* 0x0010 */
CT_DEAD_G2H_READ, /* 0x0020 */
CT_DEAD_G2H_RECV, /* 0x0040 */
CT_DEAD_G2H_RELEASE, /* 0x0080 */
CT_DEAD_DEADLOCK, /* 0x0100 */
CT_DEAD_PROCESS_FAILED, /* 0x0200 */
CT_DEAD_FAST_G2H, /* 0x0400 */
CT_DEAD_PARSE_G2H_RESPONSE, /* 0x0800 */
CT_DEAD_PARSE_G2H_UNKNOWN, /* 0x1000 */
CT_DEAD_PARSE_G2H_ORIGIN, /* 0x2000 */
CT_DEAD_PARSE_G2H_TYPE, /* 0x4000 */
};
static void ct_dead_worker_func(struct work_struct *w);
static void ct_dead_capture(struct xe_guc_ct *ct, struct guc_ctb *ctb, u32 reason_code);
#define CT_DEAD(ct, ctb, reason_code) ct_dead_capture((ct), (ctb), CT_DEAD_##reason_code)
#else
#define CT_DEAD(ct, ctb, reason) \
do { \
struct guc_ctb *_ctb = (ctb); \
if (_ctb) \
_ctb->info.broken = true; \
} while (0)
#endif
/* Used when a CT send wants to block and / or receive data */
struct g2h_fence {
u32 *response_buffer;
@ -182,7 +220,11 @@ int xe_guc_ct_init(struct xe_guc_ct *ct)
spin_lock_init(&ct->fast_lock);
xa_init(&ct->fence_lookup);
INIT_WORK(&ct->g2h_worker, g2h_worker_func);
INIT_DELAYED_WORK(&ct->safe_mode_worker, safe_mode_worker_func);
INIT_DELAYED_WORK(&ct->safe_mode_worker, safe_mode_worker_func);
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
spin_lock_init(&ct->dead.lock);
INIT_WORK(&ct->dead.worker, ct_dead_worker_func);
#endif
init_waitqueue_head(&ct->wq);
init_waitqueue_head(&ct->g2h_fence_wq);
@ -209,6 +251,7 @@ int xe_guc_ct_init(struct xe_guc_ct *ct)
ct->state = XE_GUC_CT_STATE_DISABLED;
return 0;
}
ALLOW_ERROR_INJECTION(xe_guc_ct_init, ERRNO); /* See xe_pci_probe() */
#define desc_read(xe_, guc_ctb__, field_) \
xe_map_rd_field(xe_, &guc_ctb__->desc, 0, \
@ -395,6 +438,7 @@ int xe_guc_ct_enable(struct xe_guc_ct *ct)
xe_gt_assert(gt, !xe_guc_ct_enabled(ct));
xe_map_memset(xe, &ct->bo->vmap, 0, 0, ct->bo->size);
guc_ct_ctb_h2g_init(xe, &ct->ctbs.h2g, &ct->bo->vmap);
guc_ct_ctb_g2h_init(xe, &ct->ctbs.g2h, &ct->bo->vmap);
@ -419,10 +463,22 @@ int xe_guc_ct_enable(struct xe_guc_ct *ct)
if (ct_needs_safe_mode(ct))
ct_enter_safe_mode(ct);
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
/*
* The CT has now been reset so the dumper can be re-armed
* after any existing dead state has been dumped.
*/
spin_lock_irq(&ct->dead.lock);
if (ct->dead.reason)
ct->dead.reason |= (1 << CT_DEAD_STATE_REARM);
spin_unlock_irq(&ct->dead.lock);
#endif
return 0;
err_out:
xe_gt_err(gt, "Failed to enable GuC CT (%pe)\n", ERR_PTR(err));
CT_DEAD(ct, NULL, SETUP);
return err;
}
@ -466,6 +522,19 @@ static bool h2g_has_room(struct xe_guc_ct *ct, u32 cmd_len)
if (cmd_len > h2g->info.space) {
h2g->info.head = desc_read(ct_to_xe(ct), h2g, head);
if (h2g->info.head > h2g->info.size) {
struct xe_device *xe = ct_to_xe(ct);
u32 desc_status = desc_read(xe, h2g, status);
desc_write(xe, h2g, status, desc_status | GUC_CTB_STATUS_OVERFLOW);
xe_gt_err(ct_to_gt(ct), "CT: invalid head offset %u >= %u)\n",
h2g->info.head, h2g->info.size);
CT_DEAD(ct, h2g, H2G_HAS_ROOM);
return false;
}
h2g->info.space = CIRC_SPACE(h2g->info.tail, h2g->info.head,
h2g->info.size) -
h2g->info.resv_space;
@ -521,10 +590,24 @@ static void __g2h_reserve_space(struct xe_guc_ct *ct, u32 g2h_len, u32 num_g2h)
static void __g2h_release_space(struct xe_guc_ct *ct, u32 g2h_len)
{
bool bad = false;
lockdep_assert_held(&ct->fast_lock);
xe_gt_assert(ct_to_gt(ct), ct->ctbs.g2h.info.space + g2h_len <=
ct->ctbs.g2h.info.size - ct->ctbs.g2h.info.resv_space);
xe_gt_assert(ct_to_gt(ct), ct->g2h_outstanding);
bad = ct->ctbs.g2h.info.space + g2h_len >
ct->ctbs.g2h.info.size - ct->ctbs.g2h.info.resv_space;
bad |= !ct->g2h_outstanding;
if (bad) {
xe_gt_err(ct_to_gt(ct), "Invalid G2H release: %d + %d vs %d - %d -> %d vs %d, outstanding = %d!\n",
ct->ctbs.g2h.info.space, g2h_len,
ct->ctbs.g2h.info.size, ct->ctbs.g2h.info.resv_space,
ct->ctbs.g2h.info.space + g2h_len,
ct->ctbs.g2h.info.size - ct->ctbs.g2h.info.resv_space,
ct->g2h_outstanding);
CT_DEAD(ct, &ct->ctbs.g2h, G2H_RELEASE);
return;
}
ct->ctbs.g2h.info.space += g2h_len;
if (!--ct->g2h_outstanding)
@ -551,12 +634,43 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,
u32 full_len;
struct iosys_map map = IOSYS_MAP_INIT_OFFSET(&h2g->cmds,
tail * sizeof(u32));
u32 desc_status;
full_len = len + GUC_CTB_HDR_LEN;
lockdep_assert_held(&ct->lock);
xe_gt_assert(gt, full_len <= GUC_CTB_MSG_MAX_LEN);
xe_gt_assert(gt, tail <= h2g->info.size);
desc_status = desc_read(xe, h2g, status);
if (desc_status) {
xe_gt_err(gt, "CT write: non-zero status: %u\n", desc_status);
goto corrupted;
}
if (IS_ENABLED(CONFIG_DRM_XE_DEBUG)) {
u32 desc_tail = desc_read(xe, h2g, tail);
u32 desc_head = desc_read(xe, h2g, head);
if (tail != desc_tail) {
desc_write(xe, h2g, status, desc_status | GUC_CTB_STATUS_MISMATCH);
xe_gt_err(gt, "CT write: tail was modified %u != %u\n", desc_tail, tail);
goto corrupted;
}
if (tail > h2g->info.size) {
desc_write(xe, h2g, status, desc_status | GUC_CTB_STATUS_OVERFLOW);
xe_gt_err(gt, "CT write: tail out of range: %u vs %u\n",
tail, h2g->info.size);
goto corrupted;
}
if (desc_head >= h2g->info.size) {
desc_write(xe, h2g, status, desc_status | GUC_CTB_STATUS_OVERFLOW);
xe_gt_err(gt, "CT write: invalid head offset %u >= %u)\n",
desc_head, h2g->info.size);
goto corrupted;
}
}
/* Command will wrap, zero fill (NOPs), return and check credits again */
if (tail + full_len > h2g->info.size) {
@ -609,6 +723,10 @@ static int h2g_write(struct xe_guc_ct *ct, const u32 *action, u32 len,
desc_read(xe, h2g, head), h2g->info.tail);
return 0;
corrupted:
CT_DEAD(ct, &ct->ctbs.h2g, H2G_WRITE);
return -EPIPE;
}
/*
@ -667,16 +785,12 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action,
num_g2h = 1;
if (g2h_fence_needs_alloc(g2h_fence)) {
void *ptr;
g2h_fence->seqno = next_ct_seqno(ct, true);
ptr = xa_store(&ct->fence_lookup,
g2h_fence->seqno,
g2h_fence, GFP_ATOMIC);
if (IS_ERR(ptr)) {
ret = PTR_ERR(ptr);
ret = xa_err(xa_store(&ct->fence_lookup,
g2h_fence->seqno, g2h_fence,
GFP_ATOMIC));
if (ret)
goto out;
}
}
seqno = g2h_fence->seqno;
@ -720,7 +834,6 @@ static int guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, u32 len,
{
struct xe_device *xe = ct_to_xe(ct);
struct xe_gt *gt = ct_to_gt(ct);
struct drm_printer p = xe_gt_info_printer(gt);
unsigned int sleep_period_ms = 1;
int ret;
@ -773,8 +886,13 @@ static int guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, u32 len,
goto broken;
#undef g2h_avail
if (dequeue_one_g2h(ct) < 0)
ret = dequeue_one_g2h(ct);
if (ret < 0) {
if (ret != -ECANCELED)
xe_gt_err(ct_to_gt(ct), "CTB receive failed (%pe)",
ERR_PTR(ret));
goto broken;
}
goto try_again;
}
@ -783,8 +901,7 @@ static int guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, u32 len,
broken:
xe_gt_err(gt, "No forward process on H2G, reset required\n");
xe_guc_ct_print(ct, &p, true);
ct->ctbs.h2g.info.broken = true;
CT_DEAD(ct, &ct->ctbs.h2g, DEADLOCK);
return -EDEADLK;
}
@ -852,7 +969,7 @@ static bool retry_failure(struct xe_guc_ct *ct, int ret)
#define ct_alive(ct) \
(xe_guc_ct_enabled(ct) && !ct->ctbs.h2g.info.broken && \
!ct->ctbs.g2h.info.broken)
if (!wait_event_interruptible_timeout(ct->wq, ct_alive(ct), HZ * 5))
if (!wait_event_interruptible_timeout(ct->wq, ct_alive(ct), HZ * 5))
return false;
#undef ct_alive
@ -879,14 +996,11 @@ static int guc_ct_send_recv(struct xe_guc_ct *ct, const u32 *action, u32 len,
retry_same_fence:
ret = guc_ct_send(ct, action, len, 0, 0, &g2h_fence);
if (unlikely(ret == -ENOMEM)) {
void *ptr;
/* Retry allocation /w GFP_KERNEL */
ptr = xa_store(&ct->fence_lookup,
g2h_fence.seqno,
&g2h_fence, GFP_KERNEL);
if (IS_ERR(ptr))
return PTR_ERR(ptr);
ret = xa_err(xa_store(&ct->fence_lookup, g2h_fence.seqno,
&g2h_fence, GFP_KERNEL));
if (ret)
return ret;
goto retry_same_fence;
} else if (unlikely(ret)) {
@ -897,22 +1011,32 @@ static int guc_ct_send_recv(struct xe_guc_ct *ct, const u32 *action, u32 len,
goto retry_same_fence;
if (!g2h_fence_needs_alloc(&g2h_fence))
xa_erase_irq(&ct->fence_lookup, g2h_fence.seqno);
xa_erase(&ct->fence_lookup, g2h_fence.seqno);
return ret;
}
ret = wait_event_timeout(ct->g2h_fence_wq, g2h_fence.done, HZ);
/*
* Ensure we serialize with completion side to prevent UAF with fence going out of scope on
* the stack, since we have no clue if it will fire after the timeout before we can erase
* from the xa. Also we have some dependent loads and stores below for which we need the
* correct ordering, and we lack the needed barriers.
*/
mutex_lock(&ct->lock);
if (!ret) {
xe_gt_err(gt, "Timed out wait for G2H, fence %u, action %04x",
g2h_fence.seqno, action[0]);
xa_erase_irq(&ct->fence_lookup, g2h_fence.seqno);
xe_gt_err(gt, "Timed out wait for G2H, fence %u, action %04x, done %s",
g2h_fence.seqno, action[0], str_yes_no(g2h_fence.done));
xa_erase(&ct->fence_lookup, g2h_fence.seqno);
mutex_unlock(&ct->lock);
return -ETIME;
}
if (g2h_fence.retry) {
xe_gt_dbg(gt, "H2G action %#x retrying: reason %#x\n",
action[0], g2h_fence.reason);
mutex_unlock(&ct->lock);
goto retry;
}
if (g2h_fence.fail) {
@ -921,7 +1045,12 @@ static int guc_ct_send_recv(struct xe_guc_ct *ct, const u32 *action, u32 len,
ret = -EIO;
}
return ret > 0 ? response_buffer ? g2h_fence.response_len : g2h_fence.response_data : ret;
if (ret > 0)
ret = response_buffer ? g2h_fence.response_len : g2h_fence.response_data;
mutex_unlock(&ct->lock);
return ret;
}
/**
@ -1011,6 +1140,7 @@ static int parse_g2h_response(struct xe_guc_ct *ct, u32 *msg, u32 len)
else
xe_gt_err(gt, "unexpected response %u for FAST_REQ H2G fence 0x%x!\n",
type, fence);
CT_DEAD(ct, NULL, PARSE_G2H_RESPONSE);
return -EPROTO;
}
@ -1018,6 +1148,7 @@ static int parse_g2h_response(struct xe_guc_ct *ct, u32 *msg, u32 len)
g2h_fence = xa_erase(&ct->fence_lookup, fence);
if (unlikely(!g2h_fence)) {
/* Don't tear down channel, as send could've timed out */
/* CT_DEAD(ct, NULL, PARSE_G2H_UNKNOWN); */
xe_gt_warn(gt, "G2H fence (%u) not found!\n", fence);
g2h_release_space(ct, GUC_CTB_HXG_MSG_MAX_LEN);
return 0;
@ -1062,7 +1193,7 @@ static int parse_g2h_msg(struct xe_guc_ct *ct, u32 *msg, u32 len)
if (unlikely(origin != GUC_HXG_ORIGIN_GUC)) {
xe_gt_err(gt, "G2H channel broken on read, origin=%u, reset required\n",
origin);
ct->ctbs.g2h.info.broken = true;
CT_DEAD(ct, &ct->ctbs.g2h, PARSE_G2H_ORIGIN);
return -EPROTO;
}
@ -1080,7 +1211,7 @@ static int parse_g2h_msg(struct xe_guc_ct *ct, u32 *msg, u32 len)
default:
xe_gt_err(gt, "G2H channel broken on read, type=%u, reset required\n",
type);
ct->ctbs.g2h.info.broken = true;
CT_DEAD(ct, &ct->ctbs.g2h, PARSE_G2H_TYPE);
ret = -EOPNOTSUPP;
}
@ -1123,6 +1254,8 @@ static int process_g2h_msg(struct xe_guc_ct *ct, u32 *msg, u32 len)
/* Selftest only at the moment */
break;
case XE_GUC_ACTION_STATE_CAPTURE_NOTIFICATION:
ret = xe_guc_error_capture_handler(guc, payload, adj_len);
break;
case XE_GUC_ACTION_NOTIFY_FLUSH_LOG_BUFFER_TO_FILE:
/* FIXME: Handle this */
break;
@ -1157,9 +1290,11 @@ static int process_g2h_msg(struct xe_guc_ct *ct, u32 *msg, u32 len)
xe_gt_err(gt, "unexpected G2H action 0x%04x\n", action);
}
if (ret)
if (ret) {
xe_gt_err(gt, "G2H action 0x%04x failed (%pe)\n",
action, ERR_PTR(ret));
CT_DEAD(ct, NULL, PROCESS_FAILED);
}
return 0;
}
@ -1169,7 +1304,7 @@ static int g2h_read(struct xe_guc_ct *ct, u32 *msg, bool fast_path)
struct xe_device *xe = ct_to_xe(ct);
struct xe_gt *gt = ct_to_gt(ct);
struct guc_ctb *g2h = &ct->ctbs.g2h;
u32 tail, head, len;
u32 tail, head, len, desc_status;
s32 avail;
u32 action;
u32 *hxg;
@ -1188,6 +1323,63 @@ static int g2h_read(struct xe_guc_ct *ct, u32 *msg, bool fast_path)
xe_gt_assert(gt, xe_guc_ct_enabled(ct));
desc_status = desc_read(xe, g2h, status);
if (desc_status) {
if (desc_status & GUC_CTB_STATUS_DISABLED) {
/*
* Potentially valid if a CLIENT_RESET request resulted in
* contexts/engines being reset. But should never happen as
* no contexts should be active when CLIENT_RESET is sent.
*/
xe_gt_err(gt, "CT read: unexpected G2H after GuC has stopped!\n");
desc_status &= ~GUC_CTB_STATUS_DISABLED;
}
if (desc_status) {
xe_gt_err(gt, "CT read: non-zero status: %u\n", desc_status);
goto corrupted;
}
}
if (IS_ENABLED(CONFIG_DRM_XE_DEBUG)) {
u32 desc_tail = desc_read(xe, g2h, tail);
/*
u32 desc_head = desc_read(xe, g2h, head);
* info.head and desc_head are updated back-to-back at the end of
* this function and nowhere else. Hence, they cannot be different
* unless two g2h_read calls are running concurrently. Which is not
* possible because it is guarded by ct->fast_lock. And yet, some
* discrete platforms are reguarly hitting this error :(.
*
* desc_head rolling backwards shouldn't cause any noticeable
* problems - just a delay in GuC being allowed to proceed past that
* point in the queue. So for now, just disable the error until it
* can be root caused.
*
if (g2h->info.head != desc_head) {
desc_write(xe, g2h, status, desc_status | GUC_CTB_STATUS_MISMATCH);
xe_gt_err(gt, "CT read: head was modified %u != %u\n",
desc_head, g2h->info.head);
goto corrupted;
}
*/
if (g2h->info.head > g2h->info.size) {
desc_write(xe, g2h, status, desc_status | GUC_CTB_STATUS_OVERFLOW);
xe_gt_err(gt, "CT read: head out of range: %u vs %u\n",
g2h->info.head, g2h->info.size);
goto corrupted;
}
if (desc_tail >= g2h->info.size) {
desc_write(xe, g2h, status, desc_status | GUC_CTB_STATUS_OVERFLOW);
xe_gt_err(gt, "CT read: invalid tail offset %u >= %u)\n",
desc_tail, g2h->info.size);
goto corrupted;
}
}
/* Calculate DW available to read */
tail = desc_read(xe, g2h, tail);
avail = tail - g2h->info.head;
@ -1204,9 +1396,7 @@ static int g2h_read(struct xe_guc_ct *ct, u32 *msg, bool fast_path)
if (len > avail) {
xe_gt_err(gt, "G2H channel broken on read, avail=%d, len=%d, reset required\n",
avail, len);
g2h->info.broken = true;
return -EPROTO;
goto corrupted;
}
head = (g2h->info.head + 1) % g2h->info.size;
@ -1252,6 +1442,10 @@ static int g2h_read(struct xe_guc_ct *ct, u32 *msg, bool fast_path)
action, len, g2h->info.head, tail);
return len;
corrupted:
CT_DEAD(ct, &ct->ctbs.g2h, G2H_READ);
return -EPROTO;
}
static void g2h_fast_path(struct xe_guc_ct *ct, u32 *msg, u32 len)
@ -1278,9 +1472,11 @@ static void g2h_fast_path(struct xe_guc_ct *ct, u32 *msg, u32 len)
xe_gt_warn(gt, "NOT_POSSIBLE");
}
if (ret)
if (ret) {
xe_gt_err(gt, "G2H action 0x%04x failed (%pe)\n",
action, ERR_PTR(ret));
CT_DEAD(ct, NULL, FAST_G2H);
}
}
/**
@ -1340,7 +1536,6 @@ static int dequeue_one_g2h(struct xe_guc_ct *ct)
static void receive_g2h(struct xe_guc_ct *ct)
{
struct xe_gt *gt = ct_to_gt(ct);
bool ongoing;
int ret;
@ -1377,9 +1572,8 @@ static void receive_g2h(struct xe_guc_ct *ct)
mutex_unlock(&ct->lock);
if (unlikely(ret == -EPROTO || ret == -EOPNOTSUPP)) {
struct drm_printer p = xe_gt_info_printer(gt);
xe_guc_ct_print(ct, &p, false);
xe_gt_err(ct_to_gt(ct), "CT dequeue failed: %d", ret);
CT_DEAD(ct, NULL, G2H_RECV);
kick_reset(ct);
}
} while (ret == 1);
@ -1395,49 +1589,33 @@ static void g2h_worker_func(struct work_struct *w)
receive_g2h(ct);
}
static void guc_ctb_snapshot_capture(struct xe_device *xe, struct guc_ctb *ctb,
struct guc_ctb_snapshot *snapshot,
bool atomic)
struct xe_guc_ct_snapshot *xe_guc_ct_snapshot_alloc(struct xe_guc_ct *ct, bool atomic)
{
u32 head, tail;
struct xe_guc_ct_snapshot *snapshot;
snapshot = kzalloc(sizeof(*snapshot), atomic ? GFP_ATOMIC : GFP_KERNEL);
if (!snapshot)
return NULL;
if (ct->bo) {
snapshot->ctb_size = ct->bo->size;
snapshot->ctb = kmalloc(snapshot->ctb_size, atomic ? GFP_ATOMIC : GFP_KERNEL);
}
return snapshot;
}
static void guc_ctb_snapshot_capture(struct xe_device *xe, struct guc_ctb *ctb,
struct guc_ctb_snapshot *snapshot)
{
xe_map_memcpy_from(xe, &snapshot->desc, &ctb->desc, 0,
sizeof(struct guc_ct_buffer_desc));
memcpy(&snapshot->info, &ctb->info, sizeof(struct guc_ctb_info));
snapshot->cmds = kmalloc_array(ctb->info.size, sizeof(u32),
atomic ? GFP_ATOMIC : GFP_KERNEL);
if (!snapshot->cmds) {
drm_err(&xe->drm, "Skipping CTB commands snapshot. Only CTB info will be available.\n");
return;
}
head = snapshot->desc.head;
tail = snapshot->desc.tail;
if (head != tail) {
struct iosys_map map =
IOSYS_MAP_INIT_OFFSET(&ctb->cmds, head * sizeof(u32));
while (head != tail) {
snapshot->cmds[head] = xe_map_rd(xe, &map, 0, u32);
++head;
if (head == ctb->info.size) {
head = 0;
map = ctb->cmds;
} else {
iosys_map_incr(&map, sizeof(u32));
}
}
}
}
static void guc_ctb_snapshot_print(struct guc_ctb_snapshot *snapshot,
struct drm_printer *p)
{
u32 head, tail;
drm_printf(p, "\tsize: %d\n", snapshot->info.size);
drm_printf(p, "\tresv_space: %d\n", snapshot->info.resv_space);
drm_printf(p, "\thead: %d\n", snapshot->info.head);
@ -1447,25 +1625,6 @@ static void guc_ctb_snapshot_print(struct guc_ctb_snapshot *snapshot,
drm_printf(p, "\thead (memory): %d\n", snapshot->desc.head);
drm_printf(p, "\ttail (memory): %d\n", snapshot->desc.tail);
drm_printf(p, "\tstatus (memory): 0x%x\n", snapshot->desc.status);
if (!snapshot->cmds)
return;
head = snapshot->desc.head;
tail = snapshot->desc.tail;
while (head != tail) {
drm_printf(p, "\tcmd[%d]: 0x%08x\n", head,
snapshot->cmds[head]);
++head;
if (head == snapshot->info.size)
head = 0;
}
}
static void guc_ctb_snapshot_free(struct guc_ctb_snapshot *snapshot)
{
kfree(snapshot->cmds);
}
/**
@ -1486,23 +1645,22 @@ struct xe_guc_ct_snapshot *xe_guc_ct_snapshot_capture(struct xe_guc_ct *ct,
struct xe_device *xe = ct_to_xe(ct);
struct xe_guc_ct_snapshot *snapshot;
snapshot = kzalloc(sizeof(*snapshot),
atomic ? GFP_ATOMIC : GFP_KERNEL);
snapshot = xe_guc_ct_snapshot_alloc(ct, atomic);
if (!snapshot) {
drm_err(&xe->drm, "Skipping CTB snapshot entirely.\n");
xe_gt_err(ct_to_gt(ct), "Skipping CTB snapshot entirely.\n");
return NULL;
}
if (xe_guc_ct_enabled(ct) || ct->state == XE_GUC_CT_STATE_STOPPED) {
snapshot->ct_enabled = true;
snapshot->g2h_outstanding = READ_ONCE(ct->g2h_outstanding);
guc_ctb_snapshot_capture(xe, &ct->ctbs.h2g,
&snapshot->h2g, atomic);
guc_ctb_snapshot_capture(xe, &ct->ctbs.g2h,
&snapshot->g2h, atomic);
guc_ctb_snapshot_capture(xe, &ct->ctbs.h2g, &snapshot->h2g);
guc_ctb_snapshot_capture(xe, &ct->ctbs.g2h, &snapshot->g2h);
}
if (ct->bo && snapshot->ctb)
xe_map_memcpy_from(xe, snapshot->ctb, &ct->bo->vmap, 0, snapshot->ctb_size);
return snapshot;
}
@ -1523,11 +1681,17 @@ void xe_guc_ct_snapshot_print(struct xe_guc_ct_snapshot *snapshot,
drm_puts(p, "H2G CTB (all sizes in DW):\n");
guc_ctb_snapshot_print(&snapshot->h2g, p);
drm_puts(p, "\nG2H CTB (all sizes in DW):\n");
drm_puts(p, "G2H CTB (all sizes in DW):\n");
guc_ctb_snapshot_print(&snapshot->g2h, p);
drm_printf(p, "\tg2h outstanding: %d\n",
snapshot->g2h_outstanding);
if (snapshot->ctb) {
xe_print_blob_ascii85(p, "CTB data", snapshot->ctb, 0, snapshot->ctb_size);
} else {
drm_printf(p, "CTB snapshot missing!\n");
return;
}
} else {
drm_puts(p, "CT disabled\n");
}
@ -1545,8 +1709,7 @@ void xe_guc_ct_snapshot_free(struct xe_guc_ct_snapshot *snapshot)
if (!snapshot)
return;
guc_ctb_snapshot_free(&snapshot->h2g);
guc_ctb_snapshot_free(&snapshot->g2h);
kfree(snapshot->ctb);
kfree(snapshot);
}
@ -1554,16 +1717,119 @@ void xe_guc_ct_snapshot_free(struct xe_guc_ct_snapshot *snapshot)
* xe_guc_ct_print - GuC CT Print.
* @ct: GuC CT.
* @p: drm_printer where it will be printed out.
* @atomic: Boolean to indicate if this is called from atomic context like
* reset or CTB handler or from some regular path like debugfs.
*
* This function quickly capture a snapshot and immediately print it out.
*/
void xe_guc_ct_print(struct xe_guc_ct *ct, struct drm_printer *p, bool atomic)
void xe_guc_ct_print(struct xe_guc_ct *ct, struct drm_printer *p)
{
struct xe_guc_ct_snapshot *snapshot;
snapshot = xe_guc_ct_snapshot_capture(ct, atomic);
snapshot = xe_guc_ct_snapshot_capture(ct, false);
xe_guc_ct_snapshot_print(snapshot, p);
xe_guc_ct_snapshot_free(snapshot);
}
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
static void ct_dead_capture(struct xe_guc_ct *ct, struct guc_ctb *ctb, u32 reason_code)
{
struct xe_guc_log_snapshot *snapshot_log;
struct xe_guc_ct_snapshot *snapshot_ct;
struct xe_guc *guc = ct_to_guc(ct);
unsigned long flags;
bool have_capture;
if (ctb)
ctb->info.broken = true;
/* Ignore further errors after the first dump until a reset */
if (ct->dead.reported)
return;
spin_lock_irqsave(&ct->dead.lock, flags);
/* And only capture one dump at a time */
have_capture = ct->dead.reason & (1 << CT_DEAD_STATE_CAPTURE);
ct->dead.reason |= (1 << reason_code) |
(1 << CT_DEAD_STATE_CAPTURE);
spin_unlock_irqrestore(&ct->dead.lock, flags);
if (have_capture)
return;
snapshot_log = xe_guc_log_snapshot_capture(&guc->log, true);
snapshot_ct = xe_guc_ct_snapshot_capture((ct), true);
spin_lock_irqsave(&ct->dead.lock, flags);
if (ct->dead.snapshot_log || ct->dead.snapshot_ct) {
xe_gt_err(ct_to_gt(ct), "Got unexpected dead CT capture!\n");
xe_guc_log_snapshot_free(snapshot_log);
xe_guc_ct_snapshot_free(snapshot_ct);
} else {
ct->dead.snapshot_log = snapshot_log;
ct->dead.snapshot_ct = snapshot_ct;
}
spin_unlock_irqrestore(&ct->dead.lock, flags);
queue_work(system_unbound_wq, &(ct)->dead.worker);
}
static void ct_dead_print(struct xe_dead_ct *dead)
{
struct xe_guc_ct *ct = container_of(dead, struct xe_guc_ct, dead);
struct xe_device *xe = ct_to_xe(ct);
struct xe_gt *gt = ct_to_gt(ct);
static int g_count;
struct drm_printer ip = xe_gt_info_printer(gt);
struct drm_printer lp = drm_line_printer(&ip, "Capture", ++g_count);
if (!dead->reason) {
xe_gt_err(gt, "CTB is dead for no reason!?\n");
return;
}
drm_printf(&lp, "CTB is dead - reason=0x%X\n", dead->reason);
/* Can't generate a genuine core dump at this point, so just do the good bits */
drm_puts(&lp, "**** Xe Device Coredump ****\n");
xe_device_snapshot_print(xe, &lp);
drm_printf(&lp, "**** GT #%d ****\n", gt->info.id);
drm_printf(&lp, "\tTile: %d\n", gt->tile->id);
drm_puts(&lp, "**** GuC Log ****\n");
xe_guc_log_snapshot_print(dead->snapshot_log, &lp);
drm_puts(&lp, "**** GuC CT ****\n");
xe_guc_ct_snapshot_print(dead->snapshot_ct, &lp);
drm_puts(&lp, "Done.\n");
}
static void ct_dead_worker_func(struct work_struct *w)
{
struct xe_guc_ct *ct = container_of(w, struct xe_guc_ct, dead.worker);
if (!ct->dead.reported) {
ct->dead.reported = true;
ct_dead_print(&ct->dead);
}
spin_lock_irq(&ct->dead.lock);
xe_guc_log_snapshot_free(ct->dead.snapshot_log);
ct->dead.snapshot_log = NULL;
xe_guc_ct_snapshot_free(ct->dead.snapshot_ct);
ct->dead.snapshot_ct = NULL;
if (ct->dead.reason & (1 << CT_DEAD_STATE_REARM)) {
/* A reset has occurred so re-arm the error reporting */
ct->dead.reason = 0;
ct->dead.reported = false;
}
spin_unlock_irq(&ct->dead.lock);
}
#endif

10
drivers/gpu/drm/xe/xe_guc_ct.h
View File

@ -9,6 +9,7 @@
#include "xe_guc_ct_types.h"
struct drm_printer;
struct xe_device;
int xe_guc_ct_init(struct xe_guc_ct *ct);
int xe_guc_ct_enable(struct xe_guc_ct *ct);
@ -16,12 +17,11 @@ void xe_guc_ct_disable(struct xe_guc_ct *ct);
void xe_guc_ct_stop(struct xe_guc_ct *ct);
void xe_guc_ct_fast_path(struct xe_guc_ct *ct);
struct xe_guc_ct_snapshot *
xe_guc_ct_snapshot_capture(struct xe_guc_ct *ct, bool atomic);
void xe_guc_ct_snapshot_print(struct xe_guc_ct_snapshot *snapshot,
struct drm_printer *p);
struct xe_guc_ct_snapshot *xe_guc_ct_snapshot_alloc(struct xe_guc_ct *ct, bool atomic);
struct xe_guc_ct_snapshot *xe_guc_ct_snapshot_capture(struct xe_guc_ct *ct, bool atomic);
void xe_guc_ct_snapshot_print(struct xe_guc_ct_snapshot *snapshot, struct drm_printer *p);
void xe_guc_ct_snapshot_free(struct xe_guc_ct_snapshot *snapshot);
void xe_guc_ct_print(struct xe_guc_ct *ct, struct drm_printer *p, bool atomic);
void xe_guc_ct_print(struct xe_guc_ct *ct, struct drm_printer *p);
static inline bool xe_guc_ct_enabled(struct xe_guc_ct *ct)
{

29
drivers/gpu/drm/xe/xe_guc_ct_types.h
View File

@ -52,8 +52,6 @@ struct guc_ctb {
struct guc_ctb_snapshot {
/** @desc: snapshot of the CTB descriptor */
struct guc_ct_buffer_desc desc;
/** @cmds: snapshot of the CTB commands */
u32 *cmds;
/** @info: snapshot of the CTB info */
struct guc_ctb_info info;
};
@ -70,6 +68,10 @@ struct xe_guc_ct_snapshot {
struct guc_ctb_snapshot g2h;
/** @h2g: H2G CTB snapshot */
struct guc_ctb_snapshot h2g;
/** @ctb_size: size of the snapshot of the CTB */
size_t ctb_size;
/** @ctb: snapshot of the entire CTB */
u32 *ctb;
};
/**
@ -86,6 +88,24 @@ enum xe_guc_ct_state {
XE_GUC_CT_STATE_ENABLED,
};
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
/** struct xe_dead_ct - Information for debugging a dead CT */
struct xe_dead_ct {
/** @lock: protects memory allocation/free operations, and @reason updates */
spinlock_t lock;
/** @reason: bit mask of CT_DEAD_* reason codes */
unsigned int reason;
/** @reported: for preventing multiple dumps per error sequence */
bool reported;
/** @worker: worker thread to get out of interrupt context before dumping */
struct work_struct worker;
/** snapshot_ct: copy of CT state and CTB content at point of error */
struct xe_guc_ct_snapshot *snapshot_ct;
/** snapshot_log: copy of GuC log at point of error */
struct xe_guc_log_snapshot *snapshot_log;
};
#endif
/**
* struct xe_guc_ct - GuC command transport (CT) layer
*
@ -128,6 +148,11 @@ struct xe_guc_ct {
u32 msg[GUC_CTB_MSG_MAX_LEN];
/** @fast_msg: Message buffer */
u32 fast_msg[GUC_CTB_MSG_MAX_LEN];
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
/** @dead: information for debugging dead CTs */
struct xe_dead_ct dead;
#endif
};
#endif

26
drivers/gpu/drm/xe/xe_guc_fwif.h
View File

@ -8,7 +8,9 @@
#include <linux/bits.h>
#include "abi/guc_capture_abi.h"
#include "abi/guc_klvs_abi.h"
#include "xe_hw_engine_types.h"
#define G2H_LEN_DW_SCHED_CONTEXT_MODE_SET 4
#define G2H_LEN_DW_DEREGISTER_CONTEXT 3
@ -157,24 +159,6 @@ struct guc_policies {
u32 reserved[4];
} __packed;
/* GuC MMIO reg state struct */
struct guc_mmio_reg {
u32 offset;
u32 value;
u32 flags;
u32 mask;
#define GUC_REGSET_MASKED BIT(0)
#define GUC_REGSET_MASKED_WITH_VALUE BIT(2)
#define GUC_REGSET_RESTORE_ONLY BIT(3)
} __packed;
/* GuC register sets */
struct guc_mmio_reg_set {
u32 address;
u16 count;
u16 reserved;
} __packed;
/* Generic GT SysInfo data types */
#define GUC_GENERIC_GT_SYSINFO_SLICE_ENABLED 0
#define GUC_GENERIC_GT_SYSINFO_VDBOX_SFC_SUPPORT_MASK 1
@ -188,12 +172,6 @@ struct guc_gt_system_info {
u32 generic_gt_sysinfo[GUC_GENERIC_GT_SYSINFO_MAX];
} __packed;
enum {
GUC_CAPTURE_LIST_INDEX_PF = 0,
GUC_CAPTURE_LIST_INDEX_VF = 1,
GUC_CAPTURE_LIST_INDEX_MAX = 2,
};
/* GuC Additional Data Struct */
struct guc_ads {
struct guc_mmio_reg_set reg_state_list[GUC_MAX_ENGINE_CLASSES][GUC_MAX_INSTANCES_PER_CLASS];

7
drivers/gpu/drm/xe/xe_guc_klv_thresholds_set.h
View File

@ -17,6 +17,13 @@
#define MAKE_GUC_KLV_VF_CFG_THRESHOLD_KEY(TAG) \
MAKE_GUC_KLV_KEY(CONCATENATE(VF_CFG_THRESHOLD_, TAG))
/**
* MAKE_GUC_KLV_VF_CFG_THRESHOLD_LEN - Prepare the name of the KLV length constant.
* @TAG: unique tag of the GuC threshold KLV key.
*/
#define MAKE_GUC_KLV_VF_CFG_THRESHOLD_LEN(TAG) \
MAKE_GUC_KLV_LEN(CONCATENATE(VF_CFG_THRESHOLD_, TAG))
/**
* xe_guc_klv_threshold_key_to_index - Find index of the tracked GuC threshold.
* @key: GuC threshold KLV key.

320
drivers/gpu/drm/xe/xe_guc_log.c
View File

@ -5,13 +5,26 @@
#include "xe_guc_log.h"
#include <linux/fault-inject.h>
#include <drm/drm_managed.h>
#include "regs/xe_guc_regs.h"
#include "xe_bo.h"
#include "xe_devcoredump.h"
#include "xe_force_wake.h"
#include "xe_gt.h"
#include "xe_gt_printk.h"
#include "xe_map.h"
#include "xe_mmio.h"
#include "xe_module.h"
static struct xe_guc *
log_to_guc(struct xe_guc_log *log)
{
return container_of(log, struct xe_guc, log);
}
static struct xe_gt *
log_to_gt(struct xe_guc_log *log)
{
@ -49,32 +62,193 @@ static size_t guc_log_size(void)
CAPTURE_BUFFER_SIZE;
}
#define GUC_LOG_CHUNK_SIZE SZ_2M
static struct xe_guc_log_snapshot *xe_guc_log_snapshot_alloc(struct xe_guc_log *log, bool atomic)
{
struct xe_guc_log_snapshot *snapshot;
size_t remain;
int i;
snapshot = kzalloc(sizeof(*snapshot), atomic ? GFP_ATOMIC : GFP_KERNEL);
if (!snapshot)
return NULL;
/*
* NB: kmalloc has a hard limit well below the maximum GuC log buffer size.
* Also, can't use vmalloc as might be called from atomic context. So need
* to break the buffer up into smaller chunks that can be allocated.
*/
snapshot->size = log->bo->size;
snapshot->num_chunks = DIV_ROUND_UP(snapshot->size, GUC_LOG_CHUNK_SIZE);
snapshot->copy = kcalloc(snapshot->num_chunks, sizeof(*snapshot->copy),
atomic ? GFP_ATOMIC : GFP_KERNEL);
if (!snapshot->copy)
goto fail_snap;
remain = snapshot->size;
for (i = 0; i < snapshot->num_chunks; i++) {
size_t size = min(GUC_LOG_CHUNK_SIZE, remain);
snapshot->copy[i] = kmalloc(size, atomic ? GFP_ATOMIC : GFP_KERNEL);
if (!snapshot->copy[i])
goto fail_copy;
remain -= size;
}
return snapshot;
fail_copy:
for (i = 0; i < snapshot->num_chunks; i++)
kfree(snapshot->copy[i]);
kfree(snapshot->copy);
fail_snap:
kfree(snapshot);
return NULL;
}
/**
* xe_guc_log_snapshot_free - free a previously captured GuC log snapshot
* @snapshot: GuC log snapshot structure
*
* Return: pointer to a newly allocated snapshot object or null if out of memory. Caller is
* responsible for calling xe_guc_log_snapshot_free when done with the snapshot.
*/
void xe_guc_log_snapshot_free(struct xe_guc_log_snapshot *snapshot)
{
int i;
if (!snapshot)
return;
if (!snapshot->copy) {
for (i = 0; i < snapshot->num_chunks; i++)
kfree(snapshot->copy[i]);
kfree(snapshot->copy);
}
kfree(snapshot);
}
/**
* xe_guc_log_snapshot_capture - create a new snapshot copy the GuC log for later dumping
* @log: GuC log structure
* @atomic: is the call inside an atomic section of some kind?
*
* Return: pointer to a newly allocated snapshot object or null if out of memory. Caller is
* responsible for calling xe_guc_log_snapshot_free when done with the snapshot.
*/
struct xe_guc_log_snapshot *xe_guc_log_snapshot_capture(struct xe_guc_log *log, bool atomic)
{
struct xe_guc_log_snapshot *snapshot;
struct xe_device *xe = log_to_xe(log);
struct xe_guc *guc = log_to_guc(log);
struct xe_gt *gt = log_to_gt(log);
size_t remain;
int i, err;
if (!log->bo) {
xe_gt_err(gt, "GuC log buffer not allocated\n");
return NULL;
}
snapshot = xe_guc_log_snapshot_alloc(log, atomic);
if (!snapshot) {
xe_gt_err(gt, "GuC log snapshot not allocated\n");
return NULL;
}
remain = snapshot->size;
for (i = 0; i < snapshot->num_chunks; i++) {
size_t size = min(GUC_LOG_CHUNK_SIZE, remain);
xe_map_memcpy_from(xe, snapshot->copy[i], &log->bo->vmap,
i * GUC_LOG_CHUNK_SIZE, size);
remain -= size;
}
err = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
if (err) {
snapshot->stamp = ~0;
} else {
snapshot->stamp = xe_mmio_read32(&gt->mmio, GUC_PMTIMESTAMP);
xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
}
snapshot->ktime = ktime_get_boottime_ns();
snapshot->level = log->level;
snapshot->ver_found = guc->fw.versions.found[XE_UC_FW_VER_RELEASE];
snapshot->ver_want = guc->fw.versions.wanted;
snapshot->path = guc->fw.path;
return snapshot;
}
/**
* xe_guc_log_snapshot_print - dump a previously saved copy of the GuC log to some useful location
* @snapshot: a snapshot of the GuC log
* @p: the printer object to output to
*/
void xe_guc_log_snapshot_print(struct xe_guc_log_snapshot *snapshot, struct drm_printer *p)
{
size_t remain;
int i;
if (!snapshot) {
drm_printf(p, "GuC log snapshot not allocated!\n");
return;
}
drm_printf(p, "GuC firmware: %s\n", snapshot->path);
drm_printf(p, "GuC version: %u.%u.%u (wanted %u.%u.%u)\n",
snapshot->ver_found.major, snapshot->ver_found.minor, snapshot->ver_found.patch,
snapshot->ver_want.major, snapshot->ver_want.minor, snapshot->ver_want.patch);
drm_printf(p, "Kernel timestamp: 0x%08llX [%llu]\n", snapshot->ktime, snapshot->ktime);
drm_printf(p, "GuC timestamp: 0x%08X [%u]\n", snapshot->stamp, snapshot->stamp);
drm_printf(p, "Log level: %u\n", snapshot->level);
remain = snapshot->size;
for (i = 0; i < snapshot->num_chunks; i++) {
size_t size = min(GUC_LOG_CHUNK_SIZE, remain);
xe_print_blob_ascii85(p, i ? NULL : "Log data", snapshot->copy[i], 0, size);
remain -= size;
}
}
/**
* xe_guc_log_print_dmesg - dump a copy of the GuC log to dmesg
* @log: GuC log structure
*/
void xe_guc_log_print_dmesg(struct xe_guc_log *log)
{
struct xe_gt *gt = log_to_gt(log);
static int g_count;
struct drm_printer ip = xe_gt_info_printer(gt);
struct drm_printer lp = drm_line_printer(&ip, "Capture", ++g_count);
drm_printf(&lp, "Dumping GuC log for %ps...\n", __builtin_return_address(0));
xe_guc_log_print(log, &lp);
drm_printf(&lp, "Done.\n");
}
/**
* xe_guc_log_print - dump a copy of the GuC log to some useful location
* @log: GuC log structure
* @p: the printer object to output to
*/
void xe_guc_log_print(struct xe_guc_log *log, struct drm_printer *p)
{
struct xe_device *xe = log_to_xe(log);
size_t size;
int i, j;
struct xe_guc_log_snapshot *snapshot;
xe_assert(xe, log->bo);
drm_printf(p, "**** GuC Log ****\n");
size = log->bo->size;
#define DW_PER_READ 128
xe_assert(xe, !(size % (DW_PER_READ * sizeof(u32))));
for (i = 0; i < size / sizeof(u32); i += DW_PER_READ) {
u32 read[DW_PER_READ];
xe_map_memcpy_from(xe, read, &log->bo->vmap, i * sizeof(u32),
DW_PER_READ * sizeof(u32));
#define DW_PER_PRINT 4
for (j = 0; j < DW_PER_READ / DW_PER_PRINT; ++j) {
u32 *print = read + j * DW_PER_PRINT;
drm_printf(p, "0x%08x 0x%08x 0x%08x 0x%08x\n",
*(print + 0), *(print + 1),
*(print + 2), *(print + 3));
}
}
snapshot = xe_guc_log_snapshot_capture(log, false);
drm_printf(p, "CS reference clock: %u\n", log_to_gt(log)->info.reference_clock);
xe_guc_log_snapshot_print(snapshot, p);
xe_guc_log_snapshot_free(snapshot);
}
int xe_guc_log_init(struct xe_guc_log *log)
@ -96,3 +270,105 @@ int xe_guc_log_init(struct xe_guc_log *log)
return 0;
}
ALLOW_ERROR_INJECTION(xe_guc_log_init, ERRNO); /* See xe_pci_probe() */
static u32 xe_guc_log_section_size_crash(struct xe_guc_log *log)
{
return CRASH_BUFFER_SIZE;
}
static u32 xe_guc_log_section_size_debug(struct xe_guc_log *log)
{
return DEBUG_BUFFER_SIZE;
}
/**
* xe_guc_log_section_size_capture - Get capture buffer size within log sections.
* @log: The log object.
*
* This function will return the capture buffer size within log sections.
*
* Return: capture buffer size.
*/
u32 xe_guc_log_section_size_capture(struct xe_guc_log *log)
{
return CAPTURE_BUFFER_SIZE;
}
/**
* xe_guc_get_log_buffer_size - Get log buffer size for a type.
* @log: The log object.
* @type: The log buffer type
*
* Return: buffer size.
*/
u32 xe_guc_get_log_buffer_size(struct xe_guc_log *log, enum guc_log_buffer_type type)
{
switch (type) {
case GUC_LOG_BUFFER_CRASH_DUMP:
return xe_guc_log_section_size_crash(log);
case GUC_LOG_BUFFER_DEBUG:
return xe_guc_log_section_size_debug(log);
case GUC_LOG_BUFFER_CAPTURE:
return xe_guc_log_section_size_capture(log);
}
return 0;
}
/**
* xe_guc_get_log_buffer_offset - Get offset in log buffer for a type.
* @log: The log object.
* @type: The log buffer type
*
* This function will return the offset in the log buffer for a type.
* Return: buffer offset.
*/
u32 xe_guc_get_log_buffer_offset(struct xe_guc_log *log, enum guc_log_buffer_type type)
{
enum guc_log_buffer_type i;
u32 offset = PAGE_SIZE;/* for the log_buffer_states */
for (i = GUC_LOG_BUFFER_CRASH_DUMP; i < GUC_LOG_BUFFER_TYPE_MAX; ++i) {
if (i == type)
break;
offset += xe_guc_get_log_buffer_size(log, i);
}
return offset;
}
/**
* xe_guc_check_log_buf_overflow - Check if log buffer overflowed
* @log: The log object.
* @type: The log buffer type
* @full_cnt: The count of buffer full
*
* This function will check count of buffer full against previous, mismatch
* indicate overflowed.
* Update the sampled_overflow counter, if the 4 bit counter overflowed, add
* up 16 to correct the value.
*
* Return: True if overflowed.
*/
bool xe_guc_check_log_buf_overflow(struct xe_guc_log *log, enum guc_log_buffer_type type,
unsigned int full_cnt)
{
unsigned int prev_full_cnt = log->stats[type].sampled_overflow;
bool overflow = false;
if (full_cnt != prev_full_cnt) {
overflow = true;
log->stats[type].overflow = full_cnt;
log->stats[type].sampled_overflow += full_cnt - prev_full_cnt;
if (full_cnt < prev_full_cnt) {
/* buffer_full_cnt is a 4 bit counter */
log->stats[type].sampled_overflow += 16;
}
xe_gt_notice(log_to_gt(log), "log buffer overflow\n");
}
return overflow;
}

15
drivers/gpu/drm/xe/xe_guc_log.h
View File

@ -7,8 +7,10 @@
#define _XE_GUC_LOG_H_
#include "xe_guc_log_types.h"
#include "abi/guc_log_abi.h"
struct drm_printer;
struct xe_device;
#if IS_ENABLED(CONFIG_DRM_XE_LARGE_GUC_BUFFER)
#define CRASH_BUFFER_SIZE SZ_1M
@ -17,7 +19,7 @@ struct drm_printer;
#else
#define CRASH_BUFFER_SIZE SZ_8K
#define DEBUG_BUFFER_SIZE SZ_64K
#define CAPTURE_BUFFER_SIZE SZ_16K
#define CAPTURE_BUFFER_SIZE SZ_1M
#endif
/*
* While we're using plain log level in i915, GuC controls are much more...
@ -38,6 +40,10 @@ struct drm_printer;
int xe_guc_log_init(struct xe_guc_log *log);
void xe_guc_log_print(struct xe_guc_log *log, struct drm_printer *p);
void xe_guc_log_print_dmesg(struct xe_guc_log *log);
struct xe_guc_log_snapshot *xe_guc_log_snapshot_capture(struct xe_guc_log *log, bool atomic);
void xe_guc_log_snapshot_print(struct xe_guc_log_snapshot *snapshot, struct drm_printer *p);
void xe_guc_log_snapshot_free(struct xe_guc_log_snapshot *snapshot);
static inline u32
xe_guc_log_get_level(struct xe_guc_log *log)
@ -45,4 +51,11 @@ xe_guc_log_get_level(struct xe_guc_log *log)
return log->level;
}
u32 xe_guc_log_section_size_capture(struct xe_guc_log *log);
u32 xe_guc_get_log_buffer_size(struct xe_guc_log *log, enum guc_log_buffer_type type);
u32 xe_guc_get_log_buffer_offset(struct xe_guc_log *log, enum guc_log_buffer_type type);
bool xe_guc_check_log_buf_overflow(struct xe_guc_log *log,
enum guc_log_buffer_type type,
unsigned int full_cnt);
#endif

34
drivers/gpu/drm/xe/xe_guc_log_types.h
View File

@ -7,9 +7,37 @@
#define _XE_GUC_LOG_TYPES_H_
#include <linux/types.h>
#include "abi/guc_log_abi.h"
#include "xe_uc_fw_types.h"
struct xe_bo;
/**
* struct xe_guc_log_snapshot:
* Capture of the GuC log plus various state useful for decoding the log
*/
struct xe_guc_log_snapshot {
/** @size: Size in bytes of the @copy allocation */
size_t size;
/** @copy: Host memory copy of the log buffer for later dumping, split into chunks */
void **copy;
/** @num_chunks: Number of chunks within @copy */
int num_chunks;
/** @ktime: Kernel time the snapshot was taken */
u64 ktime;
/** @stamp: GuC timestamp at which the snapshot was taken */
u32 stamp;
/** @level: GuC log verbosity level */
u32 level;
/** @ver_found: GuC firmware version */
struct xe_uc_fw_version ver_found;
/** @ver_want: GuC firmware version that driver expected */
struct xe_uc_fw_version ver_want;
/** @path: Path of GuC firmware blob */
const char *path;
};
/**
* struct xe_guc_log - GuC log
*/
@ -18,6 +46,12 @@ struct xe_guc_log {
u32 level;
/** @bo: XE BO for GuC log */
struct xe_bo *bo;
/** @stats: logging related stats */
struct {
u32 sampled_overflow;
u32 overflow;
u32 flush;
} stats[GUC_LOG_BUFFER_TYPE_MAX];
};
#endif

34
drivers/gpu/drm/xe/xe_guc_pc.c
View File

@ -262,7 +262,7 @@ static void pc_set_manual_rp_ctrl(struct xe_guc_pc *pc, bool enable)
u32 state = enable ? RPSWCTL_ENABLE : RPSWCTL_DISABLE;
/* Allow/Disallow punit to process software freq requests */
xe_mmio_write32(gt, RP_CONTROL, state);
xe_mmio_write32(&gt->mmio, RP_CONTROL, state);
}
static void pc_set_cur_freq(struct xe_guc_pc *pc, u32 freq)
@ -274,7 +274,7 @@ static void pc_set_cur_freq(struct xe_guc_pc *pc, u32 freq)
/* Req freq is in units of 16.66 Mhz */
rpnswreq = REG_FIELD_PREP(REQ_RATIO_MASK, encode_freq(freq));
xe_mmio_write32(gt, RPNSWREQ, rpnswreq);
xe_mmio_write32(&gt->mmio, RPNSWREQ, rpnswreq);
/* Sleep for a small time to allow pcode to respond */
usleep_range(100, 300);
@ -334,9 +334,9 @@ static void mtl_update_rpe_value(struct xe_guc_pc *pc)
u32 reg;
if (xe_gt_is_media_type(gt))
reg = xe_mmio_read32(gt, MTL_MPE_FREQUENCY);
reg = xe_mmio_read32(&gt->mmio, MTL_MPE_FREQUENCY);
else
reg = xe_mmio_read32(gt, MTL_GT_RPE_FREQUENCY);
reg = xe_mmio_read32(&gt->mmio, MTL_GT_RPE_FREQUENCY);
pc->rpe_freq = decode_freq(REG_FIELD_GET(MTL_RPE_MASK, reg));
}
@ -353,9 +353,9 @@ static void tgl_update_rpe_value(struct xe_guc_pc *pc)
* PCODE at a different register
*/
if (xe->info.platform == XE_PVC)
reg = xe_mmio_read32(gt, PVC_RP_STATE_CAP);
reg = xe_mmio_read32(&gt->mmio, PVC_RP_STATE_CAP);
else
reg = xe_mmio_read32(gt, FREQ_INFO_REC);
reg = xe_mmio_read32(&gt->mmio, FREQ_INFO_REC);
pc->rpe_freq = REG_FIELD_GET(RPE_MASK, reg) * GT_FREQUENCY_MULTIPLIER;
}
@ -392,10 +392,10 @@ u32 xe_guc_pc_get_act_freq(struct xe_guc_pc *pc)
/* When in RC6, actual frequency reported will be 0. */
if (GRAPHICS_VERx100(xe) >= 1270) {
freq = xe_mmio_read32(gt, MTL_MIRROR_TARGET_WP1);
freq = xe_mmio_read32(&gt->mmio, MTL_MIRROR_TARGET_WP1);
freq = REG_FIELD_GET(MTL_CAGF_MASK, freq);
} else {
freq = xe_mmio_read32(gt, GT_PERF_STATUS);
freq = xe_mmio_read32(&gt->mmio, GT_PERF_STATUS);
freq = REG_FIELD_GET(CAGF_MASK, freq);
}
@ -425,7 +425,7 @@ int xe_guc_pc_get_cur_freq(struct xe_guc_pc *pc, u32 *freq)
if (ret)
return ret;
*freq = xe_mmio_read32(gt, RPNSWREQ);
*freq = xe_mmio_read32(&gt->mmio, RPNSWREQ);
*freq = REG_FIELD_GET(REQ_RATIO_MASK, *freq);
*freq = decode_freq(*freq);
@ -612,10 +612,10 @@ enum xe_gt_idle_state xe_guc_pc_c_status(struct xe_guc_pc *pc)
u32 reg, gt_c_state;
if (GRAPHICS_VERx100(gt_to_xe(gt)) >= 1270) {
reg = xe_mmio_read32(gt, MTL_MIRROR_TARGET_WP1);
reg = xe_mmio_read32(&gt->mmio, MTL_MIRROR_TARGET_WP1);
gt_c_state = REG_FIELD_GET(MTL_CC_MASK, reg);
} else {
reg = xe_mmio_read32(gt, GT_CORE_STATUS);
reg = xe_mmio_read32(&gt->mmio, GT_CORE_STATUS);
gt_c_state = REG_FIELD_GET(RCN_MASK, reg);
}
@ -638,7 +638,7 @@ u64 xe_guc_pc_rc6_residency(struct xe_guc_pc *pc)
struct xe_gt *gt = pc_to_gt(pc);
u32 reg;
reg = xe_mmio_read32(gt, GT_GFX_RC6);
reg = xe_mmio_read32(&gt->mmio, GT_GFX_RC6);
return reg;
}
@ -652,7 +652,7 @@ u64 xe_guc_pc_mc6_residency(struct xe_guc_pc *pc)
struct xe_gt *gt = pc_to_gt(pc);
u64 reg;
reg = xe_mmio_read32(gt, MTL_MEDIA_MC6);
reg = xe_mmio_read32(&gt->mmio, MTL_MEDIA_MC6);
return reg;
}
@ -665,9 +665,9 @@ static void mtl_init_fused_rp_values(struct xe_guc_pc *pc)
xe_device_assert_mem_access(pc_to_xe(pc));
if (xe_gt_is_media_type(gt))
reg = xe_mmio_read32(gt, MTL_MEDIAP_STATE_CAP);
reg = xe_mmio_read32(&gt->mmio, MTL_MEDIAP_STATE_CAP);
else
reg = xe_mmio_read32(gt, MTL_RP_STATE_CAP);
reg = xe_mmio_read32(&gt->mmio, MTL_RP_STATE_CAP);
pc->rp0_freq = decode_freq(REG_FIELD_GET(MTL_RP0_CAP_MASK, reg));
@ -683,9 +683,9 @@ static void tgl_init_fused_rp_values(struct xe_guc_pc *pc)
xe_device_assert_mem_access(pc_to_xe(pc));
if (xe->info.platform == XE_PVC)
reg = xe_mmio_read32(gt, PVC_RP_STATE_CAP);
reg = xe_mmio_read32(&gt->mmio, PVC_RP_STATE_CAP);
else
reg = xe_mmio_read32(gt, RP_STATE_CAP);
reg = xe_mmio_read32(&gt->mmio, RP_STATE_CAP);
pc->rp0_freq = REG_FIELD_GET(RP0_MASK, reg) * GT_FREQUENCY_MULTIPLIER;
pc->rpn_freq = REG_FIELD_GET(RPN_MASK, reg) * GT_FREQUENCY_MULTIPLIER;
}

2
drivers/gpu/drm/xe/xe_guc_relay.c
View File

@ -5,6 +5,7 @@
#include <linux/bitfield.h>
#include <linux/delay.h>
#include <linux/fault-inject.h>
#include <drm/drm_managed.h>
@ -355,6 +356,7 @@ int xe_guc_relay_init(struct xe_guc_relay *relay)
return drmm_add_action_or_reset(&xe->drm, __fini_relay, relay);
}
ALLOW_ERROR_INJECTION(xe_guc_relay_init, ERRNO); /* See xe_pci_probe() */
static u32 to_relay_error(int err)
{

94
drivers/gpu/drm/xe/xe_guc_submit.c
View File

@ -27,6 +27,7 @@
#include "xe_gt_clock.h"
#include "xe_gt_printk.h"
#include "xe_guc.h"
#include "xe_guc_capture.h"
#include "xe_guc_ct.h"
#include "xe_guc_exec_queue_types.h"
#include "xe_guc_id_mgr.h"
@ -393,7 +394,6 @@ static void __release_guc_id(struct xe_guc *guc, struct xe_exec_queue *q, u32 xa
static int alloc_guc_id(struct xe_guc *guc, struct xe_exec_queue *q)
{
int ret;
void *ptr;
int i;
/*
@ -413,12 +413,10 @@ static int alloc_guc_id(struct xe_guc *guc, struct xe_exec_queue *q)
q->guc->id = ret;
for (i = 0; i < q->width; ++i) {
ptr = xa_store(&guc->submission_state.exec_queue_lookup,
q->guc->id + i, q, GFP_NOWAIT);
if (IS_ERR(ptr)) {
ret = PTR_ERR(ptr);
ret = xa_err(xa_store(&guc->submission_state.exec_queue_lookup,
q->guc->id + i, q, GFP_NOWAIT));
if (ret)
goto err_release;
}
}
return 0;
@ -827,7 +825,7 @@ static void guc_exec_queue_free_job(struct drm_sched_job *drm_job)
xe_sched_job_put(job);
}
static int guc_read_stopped(struct xe_guc *guc)
int xe_guc_read_stopped(struct xe_guc *guc)
{
return atomic_read(&guc->submission_state.stopped);
}
@ -849,7 +847,7 @@ static void disable_scheduling_deregister(struct xe_guc *guc,
set_min_preemption_timeout(guc, q);
smp_rmb();
ret = wait_event_timeout(guc->ct.wq, !exec_queue_pending_enable(q) ||
guc_read_stopped(guc), HZ * 5);
xe_guc_read_stopped(guc), HZ * 5);
if (!ret) {
struct xe_gpu_scheduler *sched = &q->guc->sched;
@ -975,7 +973,7 @@ static void xe_guc_exec_queue_lr_cleanup(struct work_struct *w)
*/
ret = wait_event_timeout(guc->ct.wq,
!exec_queue_pending_disable(q) ||
guc_read_stopped(guc), HZ * 5);
xe_guc_read_stopped(guc), HZ * 5);
if (!ret) {
drm_warn(&xe->drm, "Schedule disable failed to respond");
xe_sched_submission_start(sched);
@ -1043,8 +1041,8 @@ static void enable_scheduling(struct xe_exec_queue *q)
ret = wait_event_timeout(guc->ct.wq,
!exec_queue_pending_enable(q) ||
guc_read_stopped(guc), HZ * 5);
if (!ret || guc_read_stopped(guc)) {
xe_guc_read_stopped(guc), HZ * 5);
if (!ret || xe_guc_read_stopped(guc)) {
xe_gt_warn(guc_to_gt(guc), "Schedule enable failed to respond");
set_exec_queue_banned(q);
xe_gt_reset_async(q->gt);
@ -1099,6 +1097,7 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
struct xe_gpu_scheduler *sched = &q->guc->sched;
struct xe_guc *guc = exec_queue_to_guc(q);
const char *process_name = "no process";
struct xe_device *xe = guc_to_xe(guc);
int err = -ETIME;
pid_t pid = -1;
int i = 0;
@ -1126,6 +1125,21 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
if (!skip_timeout_check && !xe_sched_job_started(job))
goto rearm;
/*
* If devcoredump not captured and GuC capture for the job is not ready
* do manual capture first and decide later if we need to use it
*/
if (!exec_queue_killed(q) && !xe->devcoredump.captured &&
!xe_guc_capture_get_matching_and_lock(job)) {
/* take force wake before engine register manual capture */
if (xe_force_wake_get(gt_to_fw(q->gt), XE_FORCEWAKE_ALL))
xe_gt_info(q->gt, "failed to get forcewake for coredump capture\n");
xe_engine_snapshot_capture_for_job(job);
xe_force_wake_put(gt_to_fw(q->gt), XE_FORCEWAKE_ALL);
}
/*
* XXX: Sampling timeout doesn't work in wedged mode as we have to
* modify scheduling state to read timestamp. We could read the
@ -1149,8 +1163,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
*/
ret = wait_event_timeout(guc->ct.wq,
!exec_queue_pending_enable(q) ||
guc_read_stopped(guc), HZ * 5);
if (!ret || guc_read_stopped(guc))
xe_guc_read_stopped(guc), HZ * 5);
if (!ret || xe_guc_read_stopped(guc))
goto trigger_reset;
/*
@ -1174,8 +1188,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
smp_rmb();
ret = wait_event_timeout(guc->ct.wq,
!exec_queue_pending_disable(q) ||
guc_read_stopped(guc), HZ * 5);
if (!ret || guc_read_stopped(guc)) {
xe_guc_read_stopped(guc), HZ * 5);
if (!ret || xe_guc_read_stopped(guc)) {
trigger_reset:
if (!ret)
xe_gt_warn(guc_to_gt(guc), "Schedule disable failed to respond");
@ -1364,7 +1378,7 @@ static void suspend_fence_signal(struct xe_exec_queue *q)
struct xe_device *xe = guc_to_xe(guc);
xe_assert(xe, exec_queue_suspended(q) || exec_queue_killed(q) ||
guc_read_stopped(guc));
xe_guc_read_stopped(guc));
xe_assert(xe, q->guc->suspend_pending);
__suspend_fence_signal(q);
@ -1378,9 +1392,9 @@ static void __guc_exec_queue_process_msg_suspend(struct xe_sched_msg *msg)
if (guc_exec_queue_allowed_to_change_state(q) && !exec_queue_suspended(q) &&
exec_queue_enabled(q)) {
wait_event(guc->ct.wq, q->guc->resume_time != RESUME_PENDING ||
guc_read_stopped(guc));
xe_guc_read_stopped(guc));
if (!guc_read_stopped(guc)) {
if (!xe_guc_read_stopped(guc)) {
s64 since_resume_ms =
ktime_ms_delta(ktime_get(),
q->guc->resume_time);
@ -1505,7 +1519,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
q->entity = &ge->entity;
if (guc_read_stopped(guc))
if (xe_guc_read_stopped(guc))
xe_sched_stop(sched);
mutex_unlock(&guc->submission_state.lock);
@ -1661,7 +1675,7 @@ static int guc_exec_queue_suspend_wait(struct xe_exec_queue *q)
ret = wait_event_interruptible_timeout(q->guc->suspend_wait,
!READ_ONCE(q->guc->suspend_pending) ||
exec_queue_killed(q) ||
guc_read_stopped(guc),
xe_guc_read_stopped(guc),
HZ * 5);
if (!ret) {
@ -1787,7 +1801,7 @@ int xe_guc_submit_reset_prepare(struct xe_guc *guc)
void xe_guc_submit_reset_wait(struct xe_guc *guc)
{
wait_event(guc->ct.wq, xe_device_wedged(guc_to_xe(guc)) ||
!guc_read_stopped(guc));
!xe_guc_read_stopped(guc));
}
void xe_guc_submit_stop(struct xe_guc *guc)
@ -1796,7 +1810,7 @@ void xe_guc_submit_stop(struct xe_guc *guc)
unsigned long index;
struct xe_device *xe = guc_to_xe(guc);
xe_assert(xe, guc_read_stopped(guc) == 1);
xe_assert(xe, xe_guc_read_stopped(guc) == 1);
mutex_lock(&guc->submission_state.lock);
@ -1835,7 +1849,7 @@ int xe_guc_submit_start(struct xe_guc *guc)
unsigned long index;
struct xe_device *xe = guc_to_xe(guc);
xe_assert(xe, guc_read_stopped(guc) == 1);
xe_assert(xe, xe_guc_read_stopped(guc) == 1);
mutex_lock(&guc->submission_state.lock);
atomic_dec(&guc->submission_state.stopped);
@ -2009,8 +2023,6 @@ int xe_guc_exec_queue_reset_handler(struct xe_guc *guc, u32 *msg, u32 len)
xe_gt_info(gt, "Engine reset: engine_class=%s, logical_mask: 0x%x, guc_id=%d",
xe_hw_engine_class_to_str(q->class), q->logical_mask, guc_id);
/* FIXME: Do error capture, most likely async */
trace_xe_exec_queue_reset(q);
/*
@ -2026,6 +2038,36 @@ int xe_guc_exec_queue_reset_handler(struct xe_guc *guc, u32 *msg, u32 len)
return 0;
}
/*
* xe_guc_error_capture_handler - Handler of GuC captured message
* @guc: The GuC object
* @msg: Point to the message
* @len: The message length
*
* When GuC captured data is ready, GuC will send message
* XE_GUC_ACTION_STATE_CAPTURE_NOTIFICATION to host, this function will be
* called 1st to check status before process the data comes with the message.
*
* Returns: error code. 0 if success
*/
int xe_guc_error_capture_handler(struct xe_guc *guc, u32 *msg, u32 len)
{
u32 status;
if (unlikely(len != XE_GUC_ACTION_STATE_CAPTURE_NOTIFICATION_DATA_LEN)) {
xe_gt_dbg(guc_to_gt(guc), "Invalid length %u", len);
return -EPROTO;
}
status = msg[0] & XE_GUC_STATE_CAPTURE_EVENT_STATUS_MASK;
if (status == XE_GUC_STATE_CAPTURE_EVENT_STATUS_NOSPACE)
xe_gt_warn(guc_to_gt(guc), "G2H-Error capture no space");
xe_guc_capture_process(guc);
return 0;
}
int xe_guc_exec_queue_memory_cat_error_handler(struct xe_guc *guc, u32 *msg,
u32 len)
{
@ -2240,7 +2282,7 @@ xe_guc_exec_queue_snapshot_print(struct xe_guc_submit_exec_queue_snapshot *snaps
if (!snapshot)
return;
drm_printf(p, "\nGuC ID: %d\n", snapshot->guc.id);
drm_printf(p, "GuC ID: %d\n", snapshot->guc.id);
drm_printf(p, "\tName: %s\n", snapshot->name);
drm_printf(p, "\tClass: %d\n", snapshot->class);
drm_printf(p, "\tLogical mask: 0x%x\n", snapshot->logical_mask);

2
drivers/gpu/drm/xe/xe_guc_submit.h
View File

@ -20,12 +20,14 @@ void xe_guc_submit_stop(struct xe_guc *guc);
int xe_guc_submit_start(struct xe_guc *guc);
void xe_guc_submit_wedge(struct xe_guc *guc);
int xe_guc_read_stopped(struct xe_guc *guc);
int xe_guc_sched_done_handler(struct xe_guc *guc, u32 *msg, u32 len);
int xe_guc_deregister_done_handler(struct xe_guc *guc, u32 *msg, u32 len);
int xe_guc_exec_queue_reset_handler(struct xe_guc *guc, u32 *msg, u32 len);
int xe_guc_exec_queue_memory_cat_error_handler(struct xe_guc *guc, u32 *msg,
u32 len);
int xe_guc_exec_queue_reset_failure_handler(struct xe_guc *guc, u32 *msg, u32 len);
int xe_guc_error_capture_handler(struct xe_guc *guc, u32 *msg, u32 len);
struct xe_guc_submit_exec_queue_snapshot *
xe_guc_exec_queue_snapshot_capture(struct xe_exec_queue *q);

2
drivers/gpu/drm/xe/xe_guc_types.h
View File

@ -58,6 +58,8 @@ struct xe_guc {
struct xe_guc_ads ads;
/** @ct: GuC ct */
struct xe_guc_ct ct;
/** @capture: the error-state-capture module's data and objects */
struct xe_guc_state_capture *capture;
/** @pc: GuC Power Conservation */
struct xe_guc_pc pc;
/** @dbm: GuC Doorbell Manager */

6
drivers/gpu/drm/xe/xe_huc.c
View File

@ -229,7 +229,7 @@ bool xe_huc_is_authenticated(struct xe_huc *huc, enum xe_huc_auth_types type)
{
struct xe_gt *gt = huc_to_gt(huc);
return xe_mmio_read32(gt, huc_auth_modes[type].reg) & huc_auth_modes[type].val;
return xe_mmio_read32(&gt->mmio, huc_auth_modes[type].reg) & huc_auth_modes[type].val;
}
int xe_huc_auth(struct xe_huc *huc, enum xe_huc_auth_types type)
@ -268,7 +268,7 @@ int xe_huc_auth(struct xe_huc *huc, enum xe_huc_auth_types type)
goto fail;
}
ret = xe_mmio_wait32(gt, huc_auth_modes[type].reg, huc_auth_modes[type].val,
ret = xe_mmio_wait32(&gt->mmio, huc_auth_modes[type].reg, huc_auth_modes[type].val,
huc_auth_modes[type].val, 100000, NULL, false);
if (ret) {
xe_gt_err(gt, "HuC: firmware not verified: %pe\n", ERR_PTR(ret));
@ -308,7 +308,7 @@ void xe_huc_print_info(struct xe_huc *huc, struct drm_printer *p)
return;
drm_printf(p, "\nHuC status: 0x%08x\n",
xe_mmio_read32(gt, HUC_KERNEL_LOAD_INFO));
xe_mmio_read32(&gt->mmio, HUC_KERNEL_LOAD_INFO));
xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
}

305
drivers/gpu/drm/xe/xe_hw_engine.c
View File

@ -12,6 +12,7 @@
#include "regs/xe_engine_regs.h"
#include "regs/xe_gt_regs.h"
#include "regs/xe_irq_regs.h"
#include "xe_assert.h"
#include "xe_bo.h"
#include "xe_device.h"
@ -23,6 +24,7 @@
#include "xe_gt_printk.h"
#include "xe_gt_mcr.h"
#include "xe_gt_topology.h"
#include "xe_guc_capture.h"
#include "xe_hw_engine_group.h"
#include "xe_hw_fence.h"
#include "xe_irq.h"
@ -295,7 +297,7 @@ void xe_hw_engine_mmio_write32(struct xe_hw_engine *hwe,
reg.addr += hwe->mmio_base;
xe_mmio_write32(hwe->gt, reg, val);
xe_mmio_write32(&hwe->gt->mmio, reg, val);
}
/**
@ -315,7 +317,7 @@ u32 xe_hw_engine_mmio_read32(struct xe_hw_engine *hwe, struct xe_reg reg)
reg.addr += hwe->mmio_base;
return xe_mmio_read32(hwe->gt, reg);
return xe_mmio_read32(&hwe->gt->mmio, reg);
}
void xe_hw_engine_enable_ring(struct xe_hw_engine *hwe)
@ -324,7 +326,7 @@ void xe_hw_engine_enable_ring(struct xe_hw_engine *hwe)
xe_hw_engine_mask_per_class(hwe->gt, XE_ENGINE_CLASS_COMPUTE);
if (hwe->class == XE_ENGINE_CLASS_COMPUTE && ccs_mask)
xe_mmio_write32(hwe->gt, RCU_MODE,
xe_mmio_write32(&hwe->gt->mmio, RCU_MODE,
_MASKED_BIT_ENABLE(RCU_MODE_CCS_ENABLE));
xe_hw_engine_mmio_write32(hwe, RING_HWSTAM(0), ~0x0);
@ -354,7 +356,7 @@ static bool xe_rtp_cfeg_wmtp_disabled(const struct xe_gt *gt,
hwe->class != XE_ENGINE_CLASS_RENDER)
return false;
return xe_mmio_read32(hwe->gt, XEHP_FUSE4) & CFEG_WMTP_DISABLE;
return xe_mmio_read32(&hwe->gt->mmio, XEHP_FUSE4) & CFEG_WMTP_DISABLE;
}
void
@ -460,6 +462,30 @@ hw_engine_setup_default_state(struct xe_hw_engine *hwe)
xe_rtp_process_to_sr(&ctx, engine_entries, &hwe->reg_sr);
}
static const struct engine_info *find_engine_info(enum xe_engine_class class, int instance)
{
const struct engine_info *info;
enum xe_hw_engine_id id;
for (id = 0; id < XE_NUM_HW_ENGINES; ++id) {
info = &engine_infos[id];
if (info->class == class && info->instance == instance)
return info;
}
return NULL;
}
static u16 get_msix_irq_offset(struct xe_gt *gt, enum xe_engine_class class)
{
/* For MSI-X, hw engines report to offset of engine instance zero */
const struct engine_info *info = find_engine_info(class, 0);
xe_gt_assert(gt, info);
return info ? info->irq_offset : 0;
}
static void hw_engine_init_early(struct xe_gt *gt, struct xe_hw_engine *hwe,
enum xe_hw_engine_id id)
{
@ -479,7 +505,9 @@ static void hw_engine_init_early(struct xe_gt *gt, struct xe_hw_engine *hwe,
hwe->class = info->class;
hwe->instance = info->instance;
hwe->mmio_base = info->mmio_base;
hwe->irq_offset = info->irq_offset;
hwe->irq_offset = xe_device_has_msix(gt_to_xe(gt)) ?
get_msix_irq_offset(gt, info->class) :
info->irq_offset;
hwe->domain = info->domain;
hwe->name = info->name;
hwe->fence_irq = &gt->fence_irq[info->class];
@ -612,7 +640,7 @@ static void read_media_fuses(struct xe_gt *gt)
xe_force_wake_assert_held(gt_to_fw(gt), XE_FW_GT);
media_fuse = xe_mmio_read32(gt, GT_VEBOX_VDBOX_DISABLE);
media_fuse = xe_mmio_read32(&gt->mmio, GT_VEBOX_VDBOX_DISABLE);
/*
* Pre-Xe_HP platforms had register bits representing absent engines,
@ -657,7 +685,7 @@ static void read_copy_fuses(struct xe_gt *gt)
xe_force_wake_assert_held(gt_to_fw(gt), XE_FW_GT);
bcs_mask = xe_mmio_read32(gt, MIRROR_FUSE3);
bcs_mask = xe_mmio_read32(&gt->mmio, MIRROR_FUSE3);
bcs_mask = REG_FIELD_GET(MEML3_EN_MASK, bcs_mask);
/* BCS0 is always present; only BCS1-BCS8 may be fused off */
@ -704,7 +732,7 @@ static void read_compute_fuses_from_reg(struct xe_gt *gt)
struct xe_device *xe = gt_to_xe(gt);
u32 ccs_mask;
ccs_mask = xe_mmio_read32(gt, XEHP_FUSE4);
ccs_mask = xe_mmio_read32(&gt->mmio, XEHP_FUSE4);
ccs_mask = REG_FIELD_GET(CCS_EN_MASK, ccs_mask);
for (int i = XE_HW_ENGINE_CCS0, j = 0; i <= XE_HW_ENGINE_CCS3; ++i, ++j) {
@ -742,8 +770,8 @@ static void check_gsc_availability(struct xe_gt *gt)
gt->info.engine_mask &= ~BIT(XE_HW_ENGINE_GSCCS0);
/* interrupts where previously enabled, so turn them off */
xe_mmio_write32(gt, GUNIT_GSC_INTR_ENABLE, 0);
xe_mmio_write32(gt, GUNIT_GSC_INTR_MASK, ~0);
xe_mmio_write32(&gt->mmio, GUNIT_GSC_INTR_ENABLE, 0);
xe_mmio_write32(&gt->mmio, GUNIT_GSC_INTR_MASK, ~0);
drm_info(&xe->drm, "gsccs disabled due to lack of FW\n");
}
@ -798,60 +826,10 @@ void xe_hw_engine_handle_irq(struct xe_hw_engine *hwe, u16 intr_vec)
xe_hw_fence_irq_run(hwe->fence_irq);
}
static bool
is_slice_common_per_gslice(struct xe_device *xe)
{
return GRAPHICS_VERx100(xe) >= 1255;
}
static void
xe_hw_engine_snapshot_instdone_capture(struct xe_hw_engine *hwe,
struct xe_hw_engine_snapshot *snapshot)
{
struct xe_gt *gt = hwe->gt;
struct xe_device *xe = gt_to_xe(gt);
unsigned int dss;
u16 group, instance;
snapshot->reg.instdone.ring = xe_hw_engine_mmio_read32(hwe, RING_INSTDONE(0));
if (snapshot->hwe->class != XE_ENGINE_CLASS_RENDER)
return;
if (is_slice_common_per_gslice(xe) == false) {
snapshot->reg.instdone.slice_common[0] =
xe_mmio_read32(gt, SC_INSTDONE);
snapshot->reg.instdone.slice_common_extra[0] =
xe_mmio_read32(gt, SC_INSTDONE_EXTRA);
snapshot->reg.instdone.slice_common_extra2[0] =
xe_mmio_read32(gt, SC_INSTDONE_EXTRA2);
} else {
for_each_geometry_dss(dss, gt, group, instance) {
snapshot->reg.instdone.slice_common[dss] =
xe_gt_mcr_unicast_read(gt, XEHPG_SC_INSTDONE, group, instance);
snapshot->reg.instdone.slice_common_extra[dss] =
xe_gt_mcr_unicast_read(gt, XEHPG_SC_INSTDONE_EXTRA, group, instance);
snapshot->reg.instdone.slice_common_extra2[dss] =
xe_gt_mcr_unicast_read(gt, XEHPG_SC_INSTDONE_EXTRA2, group, instance);
}
}
for_each_geometry_dss(dss, gt, group, instance) {
snapshot->reg.instdone.sampler[dss] =
xe_gt_mcr_unicast_read(gt, SAMPLER_INSTDONE, group, instance);
snapshot->reg.instdone.row[dss] =
xe_gt_mcr_unicast_read(gt, ROW_INSTDONE, group, instance);
if (GRAPHICS_VERx100(xe) >= 1255)
snapshot->reg.instdone.geom_svg[dss] =
xe_gt_mcr_unicast_read(gt, XEHPG_INSTDONE_GEOM_SVGUNIT,
group, instance);
}
}
/**
* xe_hw_engine_snapshot_capture - Take a quick snapshot of the HW Engine.
* @hwe: Xe HW Engine.
* @job: The job object.
*
* This can be printed out in a later stage like during dev_coredump
* analysis.
@ -860,11 +838,10 @@ xe_hw_engine_snapshot_instdone_capture(struct xe_hw_engine *hwe,
* caller, using `xe_hw_engine_snapshot_free`.
*/
struct xe_hw_engine_snapshot *
xe_hw_engine_snapshot_capture(struct xe_hw_engine *hwe)
xe_hw_engine_snapshot_capture(struct xe_hw_engine *hwe, struct xe_sched_job *job)
{
struct xe_hw_engine_snapshot *snapshot;
size_t len;
u64 val;
struct __guc_capture_parsed_output *node;
if (!xe_hw_engine_is_valid(hwe))
return NULL;
@ -874,28 +851,6 @@ xe_hw_engine_snapshot_capture(struct xe_hw_engine *hwe)
if (!snapshot)
return NULL;
/* Because XE_MAX_DSS_FUSE_BITS is defined in xe_gt_types.h and it
* includes xe_hw_engine_types.h the length of this 3 registers can't be
* set in struct xe_hw_engine_snapshot, so here doing additional
* allocations.
*/
len = (XE_MAX_DSS_FUSE_BITS * sizeof(u32));
snapshot->reg.instdone.slice_common = kzalloc(len, GFP_ATOMIC);
snapshot->reg.instdone.slice_common_extra = kzalloc(len, GFP_ATOMIC);
snapshot->reg.instdone.slice_common_extra2 = kzalloc(len, GFP_ATOMIC);
snapshot->reg.instdone.sampler = kzalloc(len, GFP_ATOMIC);
snapshot->reg.instdone.row = kzalloc(len, GFP_ATOMIC);
snapshot->reg.instdone.geom_svg = kzalloc(len, GFP_ATOMIC);
if (!snapshot->reg.instdone.slice_common ||
!snapshot->reg.instdone.slice_common_extra ||
!snapshot->reg.instdone.slice_common_extra2 ||
!snapshot->reg.instdone.sampler ||
!snapshot->reg.instdone.row ||
!snapshot->reg.instdone.geom_svg) {
xe_hw_engine_snapshot_free(snapshot);
return NULL;
}
snapshot->name = kstrdup(hwe->name, GFP_ATOMIC);
snapshot->hwe = hwe;
snapshot->logical_instance = hwe->logical_instance;
@ -903,157 +858,32 @@ xe_hw_engine_snapshot_capture(struct xe_hw_engine *hwe)
snapshot->forcewake.ref = xe_force_wake_ref(gt_to_fw(hwe->gt),
hwe->domain);
snapshot->mmio_base = hwe->mmio_base;
snapshot->kernel_reserved = xe_hw_engine_is_reserved(hwe);
/* no more VF accessible data below this point */
if (IS_SRIOV_VF(gt_to_xe(hwe->gt)))
return snapshot;
snapshot->reg.ring_execlist_status =
xe_hw_engine_mmio_read32(hwe, RING_EXECLIST_STATUS_LO(0));
val = xe_hw_engine_mmio_read32(hwe, RING_EXECLIST_STATUS_HI(0));
snapshot->reg.ring_execlist_status |= val << 32;
if (job) {
/* If got guc capture, set source to GuC */
node = xe_guc_capture_get_matching_and_lock(job);
if (node) {
struct xe_device *xe = gt_to_xe(hwe->gt);
struct xe_devcoredump *coredump = &xe->devcoredump;
snapshot->reg.ring_execlist_sq_contents =
xe_hw_engine_mmio_read32(hwe, RING_EXECLIST_SQ_CONTENTS_LO(0));
val = xe_hw_engine_mmio_read32(hwe, RING_EXECLIST_SQ_CONTENTS_HI(0));
snapshot->reg.ring_execlist_sq_contents |= val << 32;
snapshot->reg.ring_acthd = xe_hw_engine_mmio_read32(hwe, RING_ACTHD(0));
val = xe_hw_engine_mmio_read32(hwe, RING_ACTHD_UDW(0));
snapshot->reg.ring_acthd |= val << 32;
snapshot->reg.ring_bbaddr = xe_hw_engine_mmio_read32(hwe, RING_BBADDR(0));
val = xe_hw_engine_mmio_read32(hwe, RING_BBADDR_UDW(0));
snapshot->reg.ring_bbaddr |= val << 32;
snapshot->reg.ring_dma_fadd =
xe_hw_engine_mmio_read32(hwe, RING_DMA_FADD(0));
val = xe_hw_engine_mmio_read32(hwe, RING_DMA_FADD_UDW(0));
snapshot->reg.ring_dma_fadd |= val << 32;
snapshot->reg.ring_hwstam = xe_hw_engine_mmio_read32(hwe, RING_HWSTAM(0));
snapshot->reg.ring_hws_pga = xe_hw_engine_mmio_read32(hwe, RING_HWS_PGA(0));
snapshot->reg.ring_start = xe_hw_engine_mmio_read32(hwe, RING_START(0));
if (GRAPHICS_VERx100(hwe->gt->tile->xe) >= 2000) {
val = xe_hw_engine_mmio_read32(hwe, RING_START_UDW(0));
snapshot->reg.ring_start |= val << 32;
}
if (xe_gt_has_indirect_ring_state(hwe->gt)) {
snapshot->reg.indirect_ring_state =
xe_hw_engine_mmio_read32(hwe, INDIRECT_RING_STATE(0));
}
snapshot->reg.ring_head =
xe_hw_engine_mmio_read32(hwe, RING_HEAD(0)) & HEAD_ADDR;
snapshot->reg.ring_tail =
xe_hw_engine_mmio_read32(hwe, RING_TAIL(0)) & TAIL_ADDR;
snapshot->reg.ring_ctl = xe_hw_engine_mmio_read32(hwe, RING_CTL(0));
snapshot->reg.ring_mi_mode =
xe_hw_engine_mmio_read32(hwe, RING_MI_MODE(0));
snapshot->reg.ring_mode = xe_hw_engine_mmio_read32(hwe, RING_MODE(0));
snapshot->reg.ring_imr = xe_hw_engine_mmio_read32(hwe, RING_IMR(0));
snapshot->reg.ring_esr = xe_hw_engine_mmio_read32(hwe, RING_ESR(0));
snapshot->reg.ring_emr = xe_hw_engine_mmio_read32(hwe, RING_EMR(0));
snapshot->reg.ring_eir = xe_hw_engine_mmio_read32(hwe, RING_EIR(0));
snapshot->reg.ipehr = xe_hw_engine_mmio_read32(hwe, RING_IPEHR(0));
xe_hw_engine_snapshot_instdone_capture(hwe, snapshot);
if (snapshot->hwe->class == XE_ENGINE_CLASS_COMPUTE)
snapshot->reg.rcu_mode = xe_mmio_read32(hwe->gt, RCU_MODE);
return snapshot;
}
static void
xe_hw_engine_snapshot_instdone_print(struct xe_hw_engine_snapshot *snapshot, struct drm_printer *p)
{
struct xe_gt *gt = snapshot->hwe->gt;
struct xe_device *xe = gt_to_xe(gt);
u16 group, instance;
unsigned int dss;
drm_printf(p, "\tRING_INSTDONE: 0x%08x\n", snapshot->reg.instdone.ring);
if (snapshot->hwe->class != XE_ENGINE_CLASS_RENDER)
return;
if (is_slice_common_per_gslice(xe) == false) {
drm_printf(p, "\tSC_INSTDONE[0]: 0x%08x\n",
snapshot->reg.instdone.slice_common[0]);
drm_printf(p, "\tSC_INSTDONE_EXTRA[0]: 0x%08x\n",
snapshot->reg.instdone.slice_common_extra[0]);
drm_printf(p, "\tSC_INSTDONE_EXTRA2[0]: 0x%08x\n",
snapshot->reg.instdone.slice_common_extra2[0]);
} else {
for_each_geometry_dss(dss, gt, group, instance) {
drm_printf(p, "\tSC_INSTDONE[%u]: 0x%08x\n", dss,
snapshot->reg.instdone.slice_common[dss]);
drm_printf(p, "\tSC_INSTDONE_EXTRA[%u]: 0x%08x\n", dss,
snapshot->reg.instdone.slice_common_extra[dss]);
drm_printf(p, "\tSC_INSTDONE_EXTRA2[%u]: 0x%08x\n", dss,
snapshot->reg.instdone.slice_common_extra2[dss]);
coredump->snapshot.matched_node = node;
snapshot->source = XE_ENGINE_CAPTURE_SOURCE_GUC;
xe_gt_dbg(hwe->gt, "Found and locked GuC-err-capture node");
return snapshot;
}
}
for_each_geometry_dss(dss, gt, group, instance) {
drm_printf(p, "\tSAMPLER_INSTDONE[%u]: 0x%08x\n", dss,
snapshot->reg.instdone.sampler[dss]);
drm_printf(p, "\tROW_INSTDONE[%u]: 0x%08x\n", dss,
snapshot->reg.instdone.row[dss]);
/* otherwise, do manual capture */
xe_engine_manual_capture(hwe, snapshot);
snapshot->source = XE_ENGINE_CAPTURE_SOURCE_MANUAL;
xe_gt_dbg(hwe->gt, "Proceeding with manual engine snapshot");
if (GRAPHICS_VERx100(xe) >= 1255)
drm_printf(p, "\tINSTDONE_GEOM_SVGUNIT[%u]: 0x%08x\n",
dss, snapshot->reg.instdone.geom_svg[dss]);
}
}
/**
* xe_hw_engine_snapshot_print - Print out a given Xe HW Engine snapshot.
* @snapshot: Xe HW Engine snapshot object.
* @p: drm_printer where it will be printed out.
*
* This function prints out a given Xe HW Engine snapshot object.
*/
void xe_hw_engine_snapshot_print(struct xe_hw_engine_snapshot *snapshot,
struct drm_printer *p)
{
if (!snapshot)
return;
drm_printf(p, "%s (physical), logical instance=%d\n",
snapshot->name ? snapshot->name : "",
snapshot->logical_instance);
drm_printf(p, "\tForcewake: domain 0x%x, ref %d\n",
snapshot->forcewake.domain, snapshot->forcewake.ref);
drm_printf(p, "\tHWSTAM: 0x%08x\n", snapshot->reg.ring_hwstam);
drm_printf(p, "\tRING_HWS_PGA: 0x%08x\n", snapshot->reg.ring_hws_pga);
drm_printf(p, "\tRING_EXECLIST_STATUS: 0x%016llx\n",
snapshot->reg.ring_execlist_status);
drm_printf(p, "\tRING_EXECLIST_SQ_CONTENTS: 0x%016llx\n",
snapshot->reg.ring_execlist_sq_contents);
drm_printf(p, "\tRING_START: 0x%016llx\n", snapshot->reg.ring_start);
drm_printf(p, "\tRING_HEAD: 0x%08x\n", snapshot->reg.ring_head);
drm_printf(p, "\tRING_TAIL: 0x%08x\n", snapshot->reg.ring_tail);
drm_printf(p, "\tRING_CTL: 0x%08x\n", snapshot->reg.ring_ctl);
drm_printf(p, "\tRING_MI_MODE: 0x%08x\n", snapshot->reg.ring_mi_mode);
drm_printf(p, "\tRING_MODE: 0x%08x\n",
snapshot->reg.ring_mode);
drm_printf(p, "\tRING_IMR: 0x%08x\n", snapshot->reg.ring_imr);
drm_printf(p, "\tRING_ESR: 0x%08x\n", snapshot->reg.ring_esr);
drm_printf(p, "\tRING_EMR: 0x%08x\n", snapshot->reg.ring_emr);
drm_printf(p, "\tRING_EIR: 0x%08x\n", snapshot->reg.ring_eir);
drm_printf(p, "\tACTHD: 0x%016llx\n", snapshot->reg.ring_acthd);
drm_printf(p, "\tBBADDR: 0x%016llx\n", snapshot->reg.ring_bbaddr);
drm_printf(p, "\tDMA_FADDR: 0x%016llx\n", snapshot->reg.ring_dma_fadd);
drm_printf(p, "\tINDIRECT_RING_STATE: 0x%08x\n",
snapshot->reg.indirect_ring_state);
drm_printf(p, "\tIPEHR: 0x%08x\n", snapshot->reg.ipehr);
xe_hw_engine_snapshot_instdone_print(snapshot, p);
if (snapshot->hwe->class == XE_ENGINE_CLASS_COMPUTE)
drm_printf(p, "\tRCU_MODE: 0x%08x\n",
snapshot->reg.rcu_mode);
drm_puts(p, "\n");
return snapshot;
}
/**
@ -1065,15 +895,18 @@ void xe_hw_engine_snapshot_print(struct xe_hw_engine_snapshot *snapshot,
*/
void xe_hw_engine_snapshot_free(struct xe_hw_engine_snapshot *snapshot)
{
struct xe_gt *gt;
if (!snapshot)
return;
kfree(snapshot->reg.instdone.slice_common);
kfree(snapshot->reg.instdone.slice_common_extra);
kfree(snapshot->reg.instdone.slice_common_extra2);
kfree(snapshot->reg.instdone.sampler);
kfree(snapshot->reg.instdone.row);
kfree(snapshot->reg.instdone.geom_svg);
gt = snapshot->hwe->gt;
/*
* xe_guc_capture_put_matched_nodes is called here and from
* xe_devcoredump_snapshot_free, to cover the 2 calling paths
* of hw_engines - debugfs and devcoredump free.
*/
xe_guc_capture_put_matched_nodes(&gt->uc.guc);
kfree(snapshot->name);
kfree(snapshot);
}
@ -1089,8 +922,8 @@ void xe_hw_engine_print(struct xe_hw_engine *hwe, struct drm_printer *p)
{
struct xe_hw_engine_snapshot *snapshot;
snapshot = xe_hw_engine_snapshot_capture(hwe);
xe_hw_engine_snapshot_print(snapshot, p);
snapshot = xe_hw_engine_snapshot_capture(hwe, NULL);
xe_engine_snapshot_print(snapshot, p);
xe_hw_engine_snapshot_free(snapshot);
}
@ -1150,7 +983,7 @@ const char *xe_hw_engine_class_to_str(enum xe_engine_class class)
u64 xe_hw_engine_read_timestamp(struct xe_hw_engine *hwe)
{
return xe_mmio_read64_2x32(hwe->gt, RING_TIMESTAMP(hwe->mmio_base));
return xe_mmio_read64_2x32(&hwe->gt->mmio, RING_TIMESTAMP(hwe->mmio_base));
}
enum xe_force_wake_domains xe_hw_engine_to_fw_domain(struct xe_hw_engine *hwe)

6
drivers/gpu/drm/xe/xe_hw_engine.h
View File

@ -11,6 +11,7 @@
struct drm_printer;
struct drm_xe_engine_class_instance;
struct xe_device;
struct xe_sched_job;
#ifdef CONFIG_DRM_XE_JOB_TIMEOUT_MIN
#define XE_HW_ENGINE_JOB_TIMEOUT_MIN CONFIG_DRM_XE_JOB_TIMEOUT_MIN
@ -54,12 +55,9 @@ void xe_hw_engine_handle_irq(struct xe_hw_engine *hwe, u16 intr_vec);
void xe_hw_engine_enable_ring(struct xe_hw_engine *hwe);
u32 xe_hw_engine_mask_per_class(struct xe_gt *gt,
enum xe_engine_class engine_class);
struct xe_hw_engine_snapshot *
xe_hw_engine_snapshot_capture(struct xe_hw_engine *hwe);
xe_hw_engine_snapshot_capture(struct xe_hw_engine *hwe, struct xe_sched_job *job);
void xe_hw_engine_snapshot_free(struct xe_hw_engine_snapshot *snapshot);
void xe_hw_engine_snapshot_print(struct xe_hw_engine_snapshot *snapshot,
struct drm_printer *p);
void xe_hw_engine_print(struct xe_hw_engine *hwe, struct drm_printer *p);
void xe_hw_engine_setup_default_lrc_state(struct xe_hw_engine *hwe);

68
drivers/gpu/drm/xe/xe_hw_engine_types.h
View File

@ -152,6 +152,11 @@ struct xe_hw_engine {
struct xe_hw_engine_group *hw_engine_group;
};
enum xe_hw_engine_snapshot_source_id {
XE_ENGINE_CAPTURE_SOURCE_MANUAL,
XE_ENGINE_CAPTURE_SOURCE_GUC
};
/**
* struct xe_hw_engine_snapshot - Hardware engine snapshot
*
@ -160,6 +165,8 @@ struct xe_hw_engine {
struct xe_hw_engine_snapshot {
/** @name: name of the hw engine */
char *name;
/** @source: Data source, either manual or GuC */
enum xe_hw_engine_snapshot_source_id source;
/** @hwe: hw engine */
struct xe_hw_engine *hwe;
/** @logical_instance: logical instance of this hw engine */
@ -173,65 +180,8 @@ struct xe_hw_engine_snapshot {
} forcewake;
/** @mmio_base: MMIO base address of this hw engine*/
u32 mmio_base;
/** @reg: Useful MMIO register snapshot */
struct {
/** @reg.ring_execlist_status: RING_EXECLIST_STATUS */
u64 ring_execlist_status;
/** @reg.ring_execlist_sq_contents: RING_EXECLIST_SQ_CONTENTS */
u64 ring_execlist_sq_contents;
/** @reg.ring_acthd: RING_ACTHD */
u64 ring_acthd;
/** @reg.ring_bbaddr: RING_BBADDR */
u64 ring_bbaddr;
/** @reg.ring_dma_fadd: RING_DMA_FADD */
u64 ring_dma_fadd;
/** @reg.ring_hwstam: RING_HWSTAM */
u32 ring_hwstam;
/** @reg.ring_hws_pga: RING_HWS_PGA */
u32 ring_hws_pga;
/** @reg.ring_start: RING_START */
u64 ring_start;
/** @reg.ring_head: RING_HEAD */
u32 ring_head;
/** @reg.ring_tail: RING_TAIL */
u32 ring_tail;
/** @reg.ring_ctl: RING_CTL */
u32 ring_ctl;
/** @reg.ring_mi_mode: RING_MI_MODE */
u32 ring_mi_mode;
/** @reg.ring_mode: RING_MODE */
u32 ring_mode;
/** @reg.ring_imr: RING_IMR */
u32 ring_imr;
/** @reg.ring_esr: RING_ESR */
u32 ring_esr;
/** @reg.ring_emr: RING_EMR */
u32 ring_emr;
/** @reg.ring_eir: RING_EIR */
u32 ring_eir;
/** @reg.indirect_ring_state: INDIRECT_RING_STATE */
u32 indirect_ring_state;
/** @reg.ipehr: IPEHR */
u32 ipehr;
/** @reg.rcu_mode: RCU_MODE */
u32 rcu_mode;
struct {
/** @reg.instdone.ring: RING_INSTDONE */
u32 ring;
/** @reg.instdone.slice_common: SC_INSTDONE */
u32 *slice_common;
/** @reg.instdone.slice_common_extra: SC_INSTDONE_EXTRA */
u32 *slice_common_extra;
/** @reg.instdone.slice_common_extra2: SC_INSTDONE_EXTRA2 */
u32 *slice_common_extra2;
/** @reg.instdone.sampler: SAMPLER_INSTDONE */
u32 *sampler;
/** @reg.instdone.row: ROW_INSTDONE */
u32 *row;
/** @reg.instdone.geom_svg: INSTDONE_GEOM_SVGUNIT */
u32 *geom_svg;
} instdone;
} reg;
/** @kernel_reserved: Engine reserved, can't be used by userspace */
bool kernel_reserved;
};
#endif

16
drivers/gpu/drm/xe/xe_hwmon.c
View File

@ -149,7 +149,7 @@ static void xe_hwmon_power_max_read(struct xe_hwmon *hwmon, int channel, long *v
u64 reg_val, min, max;
struct xe_device *xe = hwmon->xe;
struct xe_reg rapl_limit, pkg_power_sku;
struct xe_gt *mmio = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
rapl_limit = xe_hwmon_get_reg(hwmon, REG_PKG_RAPL_LIMIT, channel);
pkg_power_sku = xe_hwmon_get_reg(hwmon, REG_PKG_POWER_SKU, channel);
@ -190,7 +190,7 @@ static void xe_hwmon_power_max_read(struct xe_hwmon *hwmon, int channel, long *v
static int xe_hwmon_power_max_write(struct xe_hwmon *hwmon, int channel, long value)
{
struct xe_gt *mmio = xe_root_mmio_gt(hwmon->xe);
struct xe_mmio *mmio = xe_root_tile_mmio(hwmon->xe);
int ret = 0;
u64 reg_val;
struct xe_reg rapl_limit;
@ -222,7 +222,7 @@ static int xe_hwmon_power_max_write(struct xe_hwmon *hwmon, int channel, long va
static void xe_hwmon_power_rated_max_read(struct xe_hwmon *hwmon, int channel, long *value)
{
struct xe_gt *mmio = xe_root_mmio_gt(hwmon->xe);
struct xe_mmio *mmio = xe_root_tile_mmio(hwmon->xe);
struct xe_reg reg = xe_hwmon_get_reg(hwmon, REG_PKG_POWER_SKU, channel);
u64 reg_val;
@ -259,7 +259,7 @@ static void xe_hwmon_power_rated_max_read(struct xe_hwmon *hwmon, int channel, l
static void
xe_hwmon_energy_get(struct xe_hwmon *hwmon, int channel, long *energy)
{
struct xe_gt *mmio = xe_root_mmio_gt(hwmon->xe);
struct xe_mmio *mmio = xe_root_tile_mmio(hwmon->xe);
struct xe_hwmon_energy_info *ei = &hwmon->ei[channel];
u64 reg_val;
@ -282,7 +282,7 @@ xe_hwmon_power_max_interval_show(struct device *dev, struct device_attribute *at
char *buf)
{
struct xe_hwmon *hwmon = dev_get_drvdata(dev);
struct xe_gt *mmio = xe_root_mmio_gt(hwmon->xe);
struct xe_mmio *mmio = xe_root_tile_mmio(hwmon->xe);
u32 x, y, x_w = 2; /* 2 bits */
u64 r, tau4, out;
int sensor_index = to_sensor_dev_attr(attr)->index;
@ -323,7 +323,7 @@ xe_hwmon_power_max_interval_store(struct device *dev, struct device_attribute *a
const char *buf, size_t count)
{
struct xe_hwmon *hwmon = dev_get_drvdata(dev);
struct xe_gt *mmio = xe_root_mmio_gt(hwmon->xe);
struct xe_mmio *mmio = xe_root_tile_mmio(hwmon->xe);
u32 x, y, rxy, x_w = 2; /* 2 bits */
u64 tau4, r, max_win;
unsigned long val;
@ -498,7 +498,7 @@ static int xe_hwmon_power_curr_crit_write(struct xe_hwmon *hwmon, int channel,
static void xe_hwmon_get_voltage(struct xe_hwmon *hwmon, int channel, long *value)
{
struct xe_gt *mmio = xe_root_mmio_gt(hwmon->xe);
struct xe_mmio *mmio = xe_root_tile_mmio(hwmon->xe);
u64 reg_val;
reg_val = xe_mmio_read32(mmio, xe_hwmon_get_reg(hwmon, REG_GT_PERF_STATUS, channel));
@ -781,7 +781,7 @@ static const struct hwmon_chip_info hwmon_chip_info = {
static void
xe_hwmon_get_preregistration_info(struct xe_device *xe)
{
struct xe_gt *mmio = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
struct xe_hwmon *hwmon = xe->hwmon;
long energy;
u64 val_sku_unit = 0;

78
drivers/gpu/drm/xe/xe_irq.c
View File

@ -10,8 +10,7 @@
#include <drm/drm_managed.h>
#include "display/xe_display.h"
#include "regs/xe_gt_regs.h"
#include "regs/xe_regs.h"
#include "regs/xe_irq_regs.h"
#include "xe_device.h"
#include "xe_drv.h"
#include "xe_gsc_proxy.h"
@ -30,14 +29,14 @@
#define IIR(offset) XE_REG(offset + 0x8)
#define IER(offset) XE_REG(offset + 0xc)
static void assert_iir_is_zero(struct xe_gt *mmio, struct xe_reg reg)
static void assert_iir_is_zero(struct xe_mmio *mmio, struct xe_reg reg)
{
u32 val = xe_mmio_read32(mmio, reg);
if (val == 0)
return;
drm_WARN(&gt_to_xe(mmio)->drm, 1,
drm_WARN(&mmio->tile->xe->drm, 1,
"Interrupt register 0x%x is not zero: 0x%08x\n",
reg.addr, val);
xe_mmio_write32(mmio, reg, 0xffffffff);
@ -52,7 +51,7 @@ static void assert_iir_is_zero(struct xe_gt *mmio, struct xe_reg reg)
*/
static void unmask_and_enable(struct xe_tile *tile, u32 irqregs, u32 bits)
{
struct xe_gt *mmio = tile->primary_gt;
struct xe_mmio *mmio = &tile->mmio;
/*
* If we're just enabling an interrupt now, it shouldn't already
@ -70,7 +69,7 @@ static void unmask_and_enable(struct xe_tile *tile, u32 irqregs, u32 bits)
/* Mask and disable all interrupts. */
static void mask_and_disable(struct xe_tile *tile, u32 irqregs)
{
struct xe_gt *mmio = tile->primary_gt;
struct xe_mmio *mmio = &tile->mmio;
xe_mmio_write32(mmio, IMR(irqregs), ~0);
/* Posting read */
@ -87,7 +86,7 @@ static void mask_and_disable(struct xe_tile *tile, u32 irqregs)
static u32 xelp_intr_disable(struct xe_device *xe)
{
struct xe_gt *mmio = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
xe_mmio_write32(mmio, GFX_MSTR_IRQ, 0);
@ -103,7 +102,7 @@ static u32 xelp_intr_disable(struct xe_device *xe)
static u32
gu_misc_irq_ack(struct xe_device *xe, const u32 master_ctl)
{
struct xe_gt *mmio = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
u32 iir;
if (!(master_ctl & GU_MISC_IRQ))
@ -118,7 +117,7 @@ gu_misc_irq_ack(struct xe_device *xe, const u32 master_ctl)
static inline void xelp_intr_enable(struct xe_device *xe, bool stall)
{
struct xe_gt *mmio = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
xe_mmio_write32(mmio, GFX_MSTR_IRQ, MASTER_IRQ);
if (stall)
@ -129,12 +128,13 @@ static inline void xelp_intr_enable(struct xe_device *xe, bool stall)
void xe_irq_enable_hwe(struct xe_gt *gt)
{
struct xe_device *xe = gt_to_xe(gt);
struct xe_mmio *mmio = &gt->mmio;
u32 ccs_mask, bcs_mask;
u32 irqs, dmask, smask;
u32 gsc_mask = 0;
u32 heci_mask = 0;
if (IS_SRIOV_VF(xe) && xe_device_has_memirq(xe))
if (xe_device_uses_memirq(xe))
return;
if (xe_device_uc_enabled(xe)) {
@ -155,35 +155,35 @@ void xe_irq_enable_hwe(struct xe_gt *gt)
if (!xe_gt_is_media_type(gt)) {
/* Enable interrupts for each engine class */
xe_mmio_write32(gt, RENDER_COPY_INTR_ENABLE, dmask);
xe_mmio_write32(mmio, RENDER_COPY_INTR_ENABLE, dmask);
if (ccs_mask)
xe_mmio_write32(gt, CCS_RSVD_INTR_ENABLE, smask);
xe_mmio_write32(mmio, CCS_RSVD_INTR_ENABLE, smask);
/* Unmask interrupts for each engine instance */
xe_mmio_write32(gt, RCS0_RSVD_INTR_MASK, ~smask);
xe_mmio_write32(gt, BCS_RSVD_INTR_MASK, ~smask);
xe_mmio_write32(mmio, RCS0_RSVD_INTR_MASK, ~smask);
xe_mmio_write32(mmio, BCS_RSVD_INTR_MASK, ~smask);
if (bcs_mask & (BIT(1)|BIT(2)))
xe_mmio_write32(gt, XEHPC_BCS1_BCS2_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, XEHPC_BCS1_BCS2_INTR_MASK, ~dmask);
if (bcs_mask & (BIT(3)|BIT(4)))
xe_mmio_write32(gt, XEHPC_BCS3_BCS4_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, XEHPC_BCS3_BCS4_INTR_MASK, ~dmask);
if (bcs_mask & (BIT(5)|BIT(6)))
xe_mmio_write32(gt, XEHPC_BCS5_BCS6_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, XEHPC_BCS5_BCS6_INTR_MASK, ~dmask);
if (bcs_mask & (BIT(7)|BIT(8)))
xe_mmio_write32(gt, XEHPC_BCS7_BCS8_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, XEHPC_BCS7_BCS8_INTR_MASK, ~dmask);
if (ccs_mask & (BIT(0)|BIT(1)))
xe_mmio_write32(gt, CCS0_CCS1_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, CCS0_CCS1_INTR_MASK, ~dmask);
if (ccs_mask & (BIT(2)|BIT(3)))
xe_mmio_write32(gt, CCS2_CCS3_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, CCS2_CCS3_INTR_MASK, ~dmask);
}
if (xe_gt_is_media_type(gt) || MEDIA_VER(xe) < 13) {
/* Enable interrupts for each engine class */
xe_mmio_write32(gt, VCS_VECS_INTR_ENABLE, dmask);
xe_mmio_write32(mmio, VCS_VECS_INTR_ENABLE, dmask);
/* Unmask interrupts for each engine instance */
xe_mmio_write32(gt, VCS0_VCS1_INTR_MASK, ~dmask);
xe_mmio_write32(gt, VCS2_VCS3_INTR_MASK, ~dmask);
xe_mmio_write32(gt, VECS0_VECS1_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, VCS0_VCS1_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, VCS2_VCS3_INTR_MASK, ~dmask);
xe_mmio_write32(mmio, VECS0_VECS1_INTR_MASK, ~dmask);
/*
* the heci2 interrupt is enabled via the same register as the
@ -197,17 +197,17 @@ void xe_irq_enable_hwe(struct xe_gt *gt)
}
if (gsc_mask) {
xe_mmio_write32(gt, GUNIT_GSC_INTR_ENABLE, gsc_mask | heci_mask);
xe_mmio_write32(gt, GUNIT_GSC_INTR_MASK, ~gsc_mask);
xe_mmio_write32(mmio, GUNIT_GSC_INTR_ENABLE, gsc_mask | heci_mask);
xe_mmio_write32(mmio, GUNIT_GSC_INTR_MASK, ~gsc_mask);
}
if (heci_mask)
xe_mmio_write32(gt, HECI2_RSVD_INTR_MASK, ~(heci_mask << 16));
xe_mmio_write32(mmio, HECI2_RSVD_INTR_MASK, ~(heci_mask << 16));
}
}
static u32
gt_engine_identity(struct xe_device *xe,
struct xe_gt *mmio,
struct xe_mmio *mmio,
const unsigned int bank,
const unsigned int bit)
{
@ -279,7 +279,7 @@ static struct xe_gt *pick_engine_gt(struct xe_tile *tile,
return tile->media_gt;
default:
break;
};
}
fallthrough;
default:
return tile->primary_gt;
@ -291,7 +291,7 @@ static void gt_irq_handler(struct xe_tile *tile,
u32 *identity)
{
struct xe_device *xe = tile_to_xe(tile);
struct xe_gt *mmio = tile->primary_gt;
struct xe_mmio *mmio = &tile->mmio;
unsigned int bank, bit;
u16 instance, intr_vec;
enum xe_engine_class class;
@ -376,7 +376,7 @@ static irqreturn_t xelp_irq_handler(int irq, void *arg)
static u32 dg1_intr_disable(struct xe_device *xe)
{
struct xe_gt *mmio = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
u32 val;
/* First disable interrupts */
@ -394,7 +394,7 @@ static u32 dg1_intr_disable(struct xe_device *xe)
static void dg1_intr_enable(struct xe_device *xe, bool stall)
{
struct xe_gt *mmio = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
xe_mmio_write32(mmio, DG1_MSTR_TILE_INTR, DG1_MSTR_IRQ);
if (stall)
@ -431,7 +431,7 @@ static irqreturn_t dg1_irq_handler(int irq, void *arg)
}
for_each_tile(tile, xe, id) {
struct xe_gt *mmio = tile->primary_gt;
struct xe_mmio *mmio = &tile->mmio;
if ((master_tile_ctl & DG1_MSTR_TILE(tile->id)) == 0)
continue;
@ -474,7 +474,7 @@ static irqreturn_t dg1_irq_handler(int irq, void *arg)
static void gt_irq_reset(struct xe_tile *tile)
{
struct xe_gt *mmio = tile->primary_gt;
struct xe_mmio *mmio = &tile->mmio;
u32 ccs_mask = xe_hw_engine_mask_per_class(tile->primary_gt,
XE_ENGINE_CLASS_COMPUTE);
@ -504,7 +504,7 @@ static void gt_irq_reset(struct xe_tile *tile)
if (ccs_mask & (BIT(0)|BIT(1)))
xe_mmio_write32(mmio, CCS0_CCS1_INTR_MASK, ~0);
if (ccs_mask & (BIT(2)|BIT(3)))
xe_mmio_write32(mmio, CCS2_CCS3_INTR_MASK, ~0);
xe_mmio_write32(mmio, CCS2_CCS3_INTR_MASK, ~0);
if ((tile->media_gt &&
xe_hw_engine_mask_per_class(tile->media_gt, XE_ENGINE_CLASS_OTHER)) ||
@ -547,7 +547,7 @@ static void dg1_irq_reset(struct xe_tile *tile)
static void dg1_irq_reset_mstr(struct xe_tile *tile)
{
struct xe_gt *mmio = tile->primary_gt;
struct xe_mmio *mmio = &tile->mmio;
xe_mmio_write32(mmio, GFX_MSTR_IRQ, ~0);
}
@ -566,7 +566,7 @@ static void vf_irq_reset(struct xe_device *xe)
for_each_tile(tile, xe, id) {
if (xe_device_has_memirq(xe))
xe_memirq_reset(&tile->sriov.vf.memirq);
xe_memirq_reset(&tile->memirq);
else
gt_irq_reset(tile);
}
@ -609,7 +609,7 @@ static void vf_irq_postinstall(struct xe_device *xe)
for_each_tile(tile, xe, id)
if (xe_device_has_memirq(xe))
xe_memirq_postinstall(&tile->sriov.vf.memirq);
xe_memirq_postinstall(&tile->memirq);
if (GRAPHICS_VERx100(xe) < 1210)
xelp_intr_enable(xe, true);
@ -652,7 +652,7 @@ static irqreturn_t vf_mem_irq_handler(int irq, void *arg)
spin_unlock(&xe->irq.lock);
for_each_tile(tile, xe, id)
xe_memirq_handler(&tile->sriov.vf.memirq);
xe_memirq_handler(&tile->memirq);
return IRQ_HANDLED;
}

2
drivers/gpu/drm/xe/xe_lmtt.c
View File

@ -193,7 +193,7 @@ static void lmtt_setup_dir_ptr(struct xe_lmtt *lmtt)
lmtt_assert(lmtt, xe_bo_is_vram(lmtt->pd->bo));
lmtt_assert(lmtt, IS_ALIGNED(offset, SZ_64K));
xe_mmio_write32(tile->primary_gt,
xe_mmio_write32(&tile->mmio,
GRAPHICS_VER(xe) >= 20 ? XE2_LMEM_CFG : LMEM_CFG,
LMEM_EN | REG_FIELD_PREP(LMTT_DIR_PTR, offset / SZ_64K));
}

26
drivers/gpu/drm/xe/xe_lrc.c
View File

@ -38,24 +38,6 @@
#define LRC_INDIRECT_RING_STATE_SIZE SZ_4K
struct xe_lrc_snapshot {
struct xe_bo *lrc_bo;
void *lrc_snapshot;
unsigned long lrc_size, lrc_offset;
u32 context_desc;
u32 indirect_context_desc;
u32 head;
struct {
u32 internal;
u32 memory;
} tail;
u32 start_seqno;
u32 seqno;
u32 ctx_timestamp;
u32 ctx_job_timestamp;
};
static struct xe_device *
lrc_to_xe(struct xe_lrc *lrc)
{
@ -599,10 +581,10 @@ static void set_context_control(u32 *regs, struct xe_hw_engine *hwe)
static void set_memory_based_intr(u32 *regs, struct xe_hw_engine *hwe)
{
struct xe_memirq *memirq = &gt_to_tile(hwe->gt)->sriov.vf.memirq;
struct xe_memirq *memirq = &gt_to_tile(hwe->gt)->memirq;
struct xe_device *xe = gt_to_xe(hwe->gt);
if (!IS_SRIOV_VF(xe) || !xe_device_has_memirq(xe))
if (!xe_device_uses_memirq(xe))
return;
regs[CTX_LRM_INT_MASK_ENABLE] = MI_LOAD_REGISTER_MEM |
@ -613,9 +595,9 @@ static void set_memory_based_intr(u32 *regs, struct xe_hw_engine *hwe)
regs[CTX_LRI_INT_REPORT_PTR] = MI_LOAD_REGISTER_IMM | MI_LRI_NUM_REGS(2) |
MI_LRI_LRM_CS_MMIO | MI_LRI_FORCE_POSTED;
regs[CTX_INT_STATUS_REPORT_REG] = RING_INT_STATUS_RPT_PTR(0).addr;
regs[CTX_INT_STATUS_REPORT_PTR] = xe_memirq_status_ptr(memirq);
regs[CTX_INT_STATUS_REPORT_PTR] = xe_memirq_status_ptr(memirq, hwe);
regs[CTX_INT_SRC_REPORT_REG] = RING_INT_SRC_RPT_PTR(0).addr;
regs[CTX_INT_SRC_REPORT_PTR] = xe_memirq_source_ptr(memirq);
regs[CTX_INT_SRC_REPORT_PTR] = xe_memirq_source_ptr(memirq, hwe);
}
static int lrc_ring_mi_mode(struct xe_hw_engine *hwe)

19
drivers/gpu/drm/xe/xe_lrc.h
View File

@ -17,9 +17,26 @@ enum xe_engine_class;
struct xe_gt;
struct xe_hw_engine;
struct xe_lrc;
struct xe_lrc_snapshot;
struct xe_vm;
struct xe_lrc_snapshot {
struct xe_bo *lrc_bo;
void *lrc_snapshot;
unsigned long lrc_size, lrc_offset;
u32 context_desc;
u32 indirect_context_desc;
u32 head;
struct {
u32 internal;
u32 memory;
} tail;
u32 start_seqno;
u32 seqno;
u32 ctx_timestamp;
u32 ctx_job_timestamp;
};
#define LRC_PPHWSP_SCRATCH_ADDR (0x34 * 4)
struct xe_lrc *xe_lrc_create(struct xe_hw_engine *hwe, struct xe_vm *vm,

203
drivers/gpu/drm/xe/xe_memirq.c
View File

@ -5,8 +5,8 @@
#include <drm/drm_managed.h>
#include "regs/xe_gt_regs.h"
#include "regs/xe_guc_regs.h"
#include "regs/xe_irq_regs.h"
#include "regs/xe_regs.h"
#include "xe_assert.h"
@ -19,15 +19,25 @@
#include "xe_hw_engine.h"
#include "xe_map.h"
#include "xe_memirq.h"
#include "xe_sriov.h"
#include "xe_sriov_printk.h"
#define memirq_assert(m, condition) xe_tile_assert(memirq_to_tile(m), condition)
#define memirq_debug(m, msg...) xe_sriov_dbg_verbose(memirq_to_xe(m), "MEMIRQ: " msg)
#define memirq_printk(m, _level, _fmt, ...) \
drm_##_level(&memirq_to_xe(m)->drm, "MEMIRQ%u: " _fmt, \
memirq_to_tile(m)->id, ##__VA_ARGS__)
#ifdef CONFIG_DRM_XE_DEBUG_MEMIRQ
#define memirq_debug(m, _fmt, ...) memirq_printk(m, dbg, _fmt, ##__VA_ARGS__)
#else
#define memirq_debug(...)
#endif
#define memirq_err(m, _fmt, ...) memirq_printk(m, err, _fmt, ##__VA_ARGS__)
#define memirq_err_ratelimited(m, _fmt, ...) \
memirq_printk(m, err_ratelimited, _fmt, ##__VA_ARGS__)
static struct xe_tile *memirq_to_tile(struct xe_memirq *memirq)
{
return container_of(memirq, struct xe_tile, sriov.vf.memirq);
return container_of(memirq, struct xe_tile, memirq);
}
static struct xe_device *memirq_to_xe(struct xe_memirq *memirq)
@ -105,6 +115,44 @@ static const char *guc_name(struct xe_guc *guc)
* | |
* | |
* +-----------+
*
*
* MSI-X use case
*
* When using MSI-X, hw engines report interrupt status and source to engine
* instance 0. For this scenario, in order to differentiate between the
* engines, we need to pass different status/source pointers in the LRC.
*
* The requirements on those pointers are:
* - Interrupt status should be 4KiB aligned
* - Interrupt source should be 64 bytes aligned
*
* To accommodate this, we duplicate the memirq page layout above -
* allocating a page for each engine instance and pass this page in the LRC.
* Note that the same page can be reused for different engine types.
* For example, an LRC executing on CCS #x will have pointers to page #x,
* and an LRC executing on BCS #x will have the same pointers.
*
* ::
*
* 0x0000 +==============================+ <== page for instance 0 (BCS0, CCS0, etc.)
* | Interrupt Status Report Page |
* 0x0400 +==============================+
* | Interrupt Source Report Page |
* 0x0440 +==============================+
* | Interrupt Enable Mask |
* +==============================+
* | Not used |
* 0x1000 +==============================+ <== page for instance 1 (BCS1, CCS1, etc.)
* | Interrupt Status Report Page |
* 0x1400 +==============================+
* | Interrupt Source Report Page |
* 0x1440 +==============================+
* | Not used |
* 0x2000 +==============================+ <== page for instance 2 (BCS2, CCS2, etc.)
* | ... |
* +==============================+
*
*/
static void __release_xe_bo(struct drm_device *drm, void *arg)
@ -114,18 +162,30 @@ static void __release_xe_bo(struct drm_device *drm, void *arg)
xe_bo_unpin_map_no_vm(bo);
}
static inline bool hw_reports_to_instance_zero(struct xe_memirq *memirq)
{
/*
* When the HW engines are configured to use MSI-X,
* they report interrupt status and source to the offset of
* engine instance 0.
*/
return xe_device_has_msix(memirq_to_xe(memirq));
}
static int memirq_alloc_pages(struct xe_memirq *memirq)
{
struct xe_device *xe = memirq_to_xe(memirq);
struct xe_tile *tile = memirq_to_tile(memirq);
size_t bo_size = hw_reports_to_instance_zero(memirq) ?
XE_HW_ENGINE_MAX_INSTANCE * SZ_4K : SZ_4K;
struct xe_bo *bo;
int err;
BUILD_BUG_ON(!IS_ALIGNED(XE_MEMIRQ_SOURCE_OFFSET, SZ_64));
BUILD_BUG_ON(!IS_ALIGNED(XE_MEMIRQ_STATUS_OFFSET, SZ_4K));
BUILD_BUG_ON(!IS_ALIGNED(XE_MEMIRQ_SOURCE_OFFSET(0), SZ_64));
BUILD_BUG_ON(!IS_ALIGNED(XE_MEMIRQ_STATUS_OFFSET(0), SZ_4K));
/* XXX: convert to managed bo */
bo = xe_bo_create_pin_map(xe, tile, NULL, SZ_4K,
bo = xe_bo_create_pin_map(xe, tile, NULL, bo_size,
ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM |
XE_BO_FLAG_GGTT |
@ -140,25 +200,25 @@ static int memirq_alloc_pages(struct xe_memirq *memirq)
memirq_assert(memirq, !xe_bo_is_vram(bo));
memirq_assert(memirq, !memirq->bo);
iosys_map_memset(&bo->vmap, 0, 0, SZ_4K);
iosys_map_memset(&bo->vmap, 0, 0, bo_size);
memirq->bo = bo;
memirq->source = IOSYS_MAP_INIT_OFFSET(&bo->vmap, XE_MEMIRQ_SOURCE_OFFSET);
memirq->status = IOSYS_MAP_INIT_OFFSET(&bo->vmap, XE_MEMIRQ_STATUS_OFFSET);
memirq->source = IOSYS_MAP_INIT_OFFSET(&bo->vmap, XE_MEMIRQ_SOURCE_OFFSET(0));
memirq->status = IOSYS_MAP_INIT_OFFSET(&bo->vmap, XE_MEMIRQ_STATUS_OFFSET(0));
memirq->mask = IOSYS_MAP_INIT_OFFSET(&bo->vmap, XE_MEMIRQ_ENABLE_OFFSET);
memirq_assert(memirq, !memirq->source.is_iomem);
memirq_assert(memirq, !memirq->status.is_iomem);
memirq_assert(memirq, !memirq->mask.is_iomem);
memirq_debug(memirq, "page offsets: source %#x status %#x\n",
xe_memirq_source_ptr(memirq), xe_memirq_status_ptr(memirq));
memirq_debug(memirq, "page offsets: bo %#x bo_size %zu source %#x status %#x\n",
xe_bo_ggtt_addr(bo), bo_size, XE_MEMIRQ_SOURCE_OFFSET(0),
XE_MEMIRQ_STATUS_OFFSET(0));
return drmm_add_action_or_reset(&xe->drm, __release_xe_bo, memirq->bo);
out:
xe_sriov_err(memirq_to_xe(memirq),
"Failed to allocate memirq page (%pe)\n", ERR_PTR(err));
memirq_err(memirq, "Failed to allocate memirq page (%pe)\n", ERR_PTR(err));
return err;
}
@ -178,9 +238,7 @@ static void memirq_set_enable(struct xe_memirq *memirq, bool enable)
*
* These allocations are managed and will be implicitly released on unload.
*
* Note: This function shall be called only by the VF driver.
*
* If this function fails then VF driver won't be able to operate correctly.
* If this function fails then the driver won't be able to operate correctly.
* If `Memory Based Interrupts`_ are not used this function will return 0.
*
* Return: 0 on success or a negative error code on failure.
@ -190,9 +248,7 @@ int xe_memirq_init(struct xe_memirq *memirq)
struct xe_device *xe = memirq_to_xe(memirq);
int err;
memirq_assert(memirq, IS_SRIOV_VF(xe));
if (!xe_device_has_memirq(xe))
if (!xe_device_uses_memirq(xe))
return 0;
err = memirq_alloc_pages(memirq);
@ -205,55 +261,70 @@ int xe_memirq_init(struct xe_memirq *memirq)
return 0;
}
static u32 __memirq_source_page(struct xe_memirq *memirq, u16 instance)
{
memirq_assert(memirq, instance <= XE_HW_ENGINE_MAX_INSTANCE);
memirq_assert(memirq, memirq->bo);
instance = hw_reports_to_instance_zero(memirq) ? instance : 0;
return xe_bo_ggtt_addr(memirq->bo) + XE_MEMIRQ_SOURCE_OFFSET(instance);
}
/**
* xe_memirq_source_ptr - Get GGTT's offset of the `Interrupt Source Report Page`_.
* @memirq: the &xe_memirq to query
* @hwe: the hw engine for which we want the report page
*
* Shall be called only on VF driver when `Memory Based Interrupts`_ are used
* Shall be called when `Memory Based Interrupts`_ are used
* and xe_memirq_init() didn't fail.
*
* Return: GGTT's offset of the `Interrupt Source Report Page`_.
*/
u32 xe_memirq_source_ptr(struct xe_memirq *memirq)
u32 xe_memirq_source_ptr(struct xe_memirq *memirq, struct xe_hw_engine *hwe)
{
memirq_assert(memirq, IS_SRIOV_VF(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_has_memirq(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_uses_memirq(memirq_to_xe(memirq)));
return __memirq_source_page(memirq, hwe->instance);
}
static u32 __memirq_status_page(struct xe_memirq *memirq, u16 instance)
{
memirq_assert(memirq, instance <= XE_HW_ENGINE_MAX_INSTANCE);
memirq_assert(memirq, memirq->bo);
return xe_bo_ggtt_addr(memirq->bo) + XE_MEMIRQ_SOURCE_OFFSET;
instance = hw_reports_to_instance_zero(memirq) ? instance : 0;
return xe_bo_ggtt_addr(memirq->bo) + XE_MEMIRQ_STATUS_OFFSET(instance);
}
/**
* xe_memirq_status_ptr - Get GGTT's offset of the `Interrupt Status Report Page`_.
* @memirq: the &xe_memirq to query
* @hwe: the hw engine for which we want the report page
*
* Shall be called only on VF driver when `Memory Based Interrupts`_ are used
* Shall be called when `Memory Based Interrupts`_ are used
* and xe_memirq_init() didn't fail.
*
* Return: GGTT's offset of the `Interrupt Status Report Page`_.
*/
u32 xe_memirq_status_ptr(struct xe_memirq *memirq)
u32 xe_memirq_status_ptr(struct xe_memirq *memirq, struct xe_hw_engine *hwe)
{
memirq_assert(memirq, IS_SRIOV_VF(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_has_memirq(memirq_to_xe(memirq)));
memirq_assert(memirq, memirq->bo);
memirq_assert(memirq, xe_device_uses_memirq(memirq_to_xe(memirq)));
return xe_bo_ggtt_addr(memirq->bo) + XE_MEMIRQ_STATUS_OFFSET;
return __memirq_status_page(memirq, hwe->instance);
}
/**
* xe_memirq_enable_ptr - Get GGTT's offset of the Interrupt Enable Mask.
* @memirq: the &xe_memirq to query
*
* Shall be called only on VF driver when `Memory Based Interrupts`_ are used
* Shall be called when `Memory Based Interrupts`_ are used
* and xe_memirq_init() didn't fail.
*
* Return: GGTT's offset of the Interrupt Enable Mask.
*/
u32 xe_memirq_enable_ptr(struct xe_memirq *memirq)
{
memirq_assert(memirq, IS_SRIOV_VF(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_has_memirq(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_uses_memirq(memirq_to_xe(memirq)));
memirq_assert(memirq, memirq->bo);
return xe_bo_ggtt_addr(memirq->bo) + XE_MEMIRQ_ENABLE_OFFSET;
@ -267,7 +338,7 @@ u32 xe_memirq_enable_ptr(struct xe_memirq *memirq)
* Register `Interrupt Source Report Page`_ and `Interrupt Status Report Page`_
* to be used by the GuC when `Memory Based Interrupts`_ are required.
*
* Shall be called only on VF driver when `Memory Based Interrupts`_ are used
* Shall be called when `Memory Based Interrupts`_ are used
* and xe_memirq_init() didn't fail.
*
* Return: 0 on success or a negative error code on failure.
@ -279,12 +350,10 @@ int xe_memirq_init_guc(struct xe_memirq *memirq, struct xe_guc *guc)
u32 source, status;
int err;
memirq_assert(memirq, IS_SRIOV_VF(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_has_memirq(memirq_to_xe(memirq)));
memirq_assert(memirq, memirq->bo);
memirq_assert(memirq, xe_device_uses_memirq(memirq_to_xe(memirq)));
source = xe_memirq_source_ptr(memirq) + offset;
status = xe_memirq_status_ptr(memirq) + offset * SZ_16;
source = __memirq_source_page(memirq, 0) + offset;
status = __memirq_status_page(memirq, 0) + offset * SZ_16;
err = xe_guc_self_cfg64(guc, GUC_KLV_SELF_CFG_MEMIRQ_SOURCE_ADDR_KEY,
source);
@ -299,9 +368,8 @@ int xe_memirq_init_guc(struct xe_memirq *memirq, struct xe_guc *guc)
return 0;
failed:
xe_sriov_err(memirq_to_xe(memirq),
"Failed to setup report pages in %s (%pe)\n",
guc_name(guc), ERR_PTR(err));
memirq_err(memirq, "Failed to setup report pages in %s (%pe)\n",
guc_name(guc), ERR_PTR(err));
return err;
}
@ -311,13 +379,12 @@ int xe_memirq_init_guc(struct xe_memirq *memirq, struct xe_guc *guc)
*
* This is part of the driver IRQ setup flow.
*
* This function shall only be used by the VF driver on platforms that use
* This function shall only be used on platforms that use
* `Memory Based Interrupts`_.
*/
void xe_memirq_reset(struct xe_memirq *memirq)
{
memirq_assert(memirq, IS_SRIOV_VF(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_has_memirq(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_uses_memirq(memirq_to_xe(memirq)));
if (memirq->bo)
memirq_set_enable(memirq, false);
@ -329,13 +396,12 @@ void xe_memirq_reset(struct xe_memirq *memirq)
*
* This is part of the driver IRQ setup flow.
*
* This function shall only be used by the VF driver on platforms that use
* This function shall only be used on platforms that use
* `Memory Based Interrupts`_.
*/
void xe_memirq_postinstall(struct xe_memirq *memirq)
{
memirq_assert(memirq, IS_SRIOV_VF(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_has_memirq(memirq_to_xe(memirq)));
memirq_assert(memirq, xe_device_uses_memirq(memirq_to_xe(memirq)));
if (memirq->bo)
memirq_set_enable(memirq, true);
@ -349,9 +415,9 @@ static bool memirq_received(struct xe_memirq *memirq, struct iosys_map *vector,
value = iosys_map_rd(vector, offset, u8);
if (value) {
if (value != 0xff)
xe_sriov_err_ratelimited(memirq_to_xe(memirq),
"Unexpected memirq value %#x from %s at %u\n",
value, name, offset);
memirq_err_ratelimited(memirq,
"Unexpected memirq value %#x from %s at %u\n",
value, name, offset);
iosys_map_wr(vector, offset, u8, 0x00);
}
@ -378,6 +444,28 @@ static void memirq_dispatch_guc(struct xe_memirq *memirq, struct iosys_map *stat
xe_guc_irq_handler(guc, GUC_INTR_GUC2HOST);
}
/**
* xe_memirq_hwe_handler - Check and process interrupts for a specific HW engine.
* @memirq: the &xe_memirq
* @hwe: the hw engine to process
*
* This function reads and dispatches `Memory Based Interrupts` for the provided HW engine.
*/
void xe_memirq_hwe_handler(struct xe_memirq *memirq, struct xe_hw_engine *hwe)
{
u16 offset = hwe->irq_offset;
u16 instance = hw_reports_to_instance_zero(memirq) ? hwe->instance : 0;
struct iosys_map src_offset = IOSYS_MAP_INIT_OFFSET(&memirq->bo->vmap,
XE_MEMIRQ_SOURCE_OFFSET(instance));
if (memirq_received(memirq, &src_offset, offset, "SRC")) {
struct iosys_map status_offset =
IOSYS_MAP_INIT_OFFSET(&memirq->bo->vmap,
XE_MEMIRQ_STATUS_OFFSET(instance) + offset * SZ_16);
memirq_dispatch_engine(memirq, &status_offset, hwe);
}
}
/**
* xe_memirq_handler - The `Memory Based Interrupts`_ Handler.
* @memirq: the &xe_memirq
@ -405,13 +493,8 @@ void xe_memirq_handler(struct xe_memirq *memirq)
if (gt->tile != tile)
continue;
for_each_hw_engine(hwe, gt, id) {
if (memirq_received(memirq, &memirq->source, hwe->irq_offset, "SRC")) {
map = IOSYS_MAP_INIT_OFFSET(&memirq->status,
hwe->irq_offset * SZ_16);
memirq_dispatch_engine(memirq, &map, hwe);
}
}
for_each_hw_engine(hwe, gt, id)
xe_memirq_hwe_handler(memirq, hwe);
}
/* GuC and media GuC (if present) must be checked separately */

6
drivers/gpu/drm/xe/xe_memirq.h
View File

@ -9,16 +9,18 @@
#include <linux/types.h>
struct xe_guc;
struct xe_hw_engine;
struct xe_memirq;
int xe_memirq_init(struct xe_memirq *memirq);
u32 xe_memirq_source_ptr(struct xe_memirq *memirq);
u32 xe_memirq_status_ptr(struct xe_memirq *memirq);
u32 xe_memirq_source_ptr(struct xe_memirq *memirq, struct xe_hw_engine *hwe);
u32 xe_memirq_status_ptr(struct xe_memirq *memirq, struct xe_hw_engine *hwe);
u32 xe_memirq_enable_ptr(struct xe_memirq *memirq);
void xe_memirq_reset(struct xe_memirq *memirq);
void xe_memirq_postinstall(struct xe_memirq *memirq);
void xe_memirq_hwe_handler(struct xe_memirq *memirq, struct xe_hw_engine *hwe);
void xe_memirq_handler(struct xe_memirq *memirq);
int xe_memirq_init_guc(struct xe_memirq *memirq, struct xe_guc *guc);

4
drivers/gpu/drm/xe/xe_memirq_types.h
View File

@ -11,9 +11,9 @@
struct xe_bo;
/* ISR */
#define XE_MEMIRQ_STATUS_OFFSET 0x0
#define XE_MEMIRQ_STATUS_OFFSET(inst) ((inst) * SZ_4K + 0x0)
/* IIR */
#define XE_MEMIRQ_SOURCE_OFFSET 0x400
#define XE_MEMIRQ_SOURCE_OFFSET(inst) ((inst) * SZ_4K + 0x400)
/* IMR */
#define XE_MEMIRQ_ENABLE_OFFSET 0x440

139
drivers/gpu/drm/xe/xe_mmio.c
View File

@ -36,13 +36,19 @@ static void tiles_fini(void *arg)
/*
* On multi-tile devices, partition the BAR space for MMIO on each tile,
* possibly accounting for register override on the number of tiles available.
* tile_mmio_size contains both the tile's 4MB register space, as well as
* additional space for the GTT and other (possibly unused) regions).
* Resulting memory layout is like below:
*
* .----------------------. <- tile_count * tile_mmio_size
* | .... |
* |----------------------| <- 2 * tile_mmio_size
* | tile1 GTT + other |
* |----------------------| <- 1 * tile_mmio_size + 4MB
* | tile1->mmio.regs |
* |----------------------| <- 1 * tile_mmio_size
* | tile0 GTT + other |
* |----------------------| <- 4MB
* | tile0->mmio.regs |
* '----------------------' <- 0MB
*/
@ -61,16 +67,16 @@ static void mmio_multi_tile_setup(struct xe_device *xe, size_t tile_mmio_size)
/* Possibly override number of tile based on configuration register */
if (!xe->info.skip_mtcfg) {
struct xe_gt *gt = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
u8 tile_count;
u32 mtcfg;
/*
* Although the per-tile mmio regs are not yet initialized, this
* is fine as it's going to the root gt, that's guaranteed to be
* initialized earlier in xe_mmio_init()
* is fine as it's going to the root tile's mmio, that's
* guaranteed to be initialized earlier in xe_mmio_init()
*/
mtcfg = xe_mmio_read64_2x32(gt, XEHP_MTCFG_ADDR);
mtcfg = xe_mmio_read64_2x32(mmio, XEHP_MTCFG_ADDR);
tile_count = REG_FIELD_GET(TILE_COUNT, mtcfg) + 1;
if (tile_count < xe->info.tile_count) {
@ -90,8 +96,9 @@ static void mmio_multi_tile_setup(struct xe_device *xe, size_t tile_mmio_size)
regs = xe->mmio.regs;
for_each_tile(tile, xe, id) {
tile->mmio.size = tile_mmio_size;
tile->mmio.regs_size = SZ_4M;
tile->mmio.regs = regs;
tile->mmio.tile = tile;
regs += tile_mmio_size;
}
}
@ -126,8 +133,9 @@ static void mmio_extension_setup(struct xe_device *xe, size_t tile_mmio_size,
regs = xe->mmio.regs + tile_mmio_size * xe->info.tile_count;
for_each_tile(tile, xe, id) {
tile->mmio_ext.size = tile_mmio_ext_size;
tile->mmio_ext.regs_size = tile_mmio_ext_size;
tile->mmio_ext.regs = regs;
tile->mmio_ext.tile = tile;
regs += tile_mmio_ext_size;
}
}
@ -157,137 +165,132 @@ int xe_mmio_init(struct xe_device *xe)
{
struct xe_tile *root_tile = xe_device_get_root_tile(xe);
struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
const int mmio_bar = 0;
/*
* Map the entire BAR.
* The first 16MB of the BAR, belong to the root tile, and include:
* registers (0-4MB), reserved space (4MB-8MB) and GGTT (8MB-16MB).
*/
xe->mmio.size = pci_resource_len(pdev, mmio_bar);
xe->mmio.regs = pci_iomap(pdev, mmio_bar, GTTMMADR_BAR);
xe->mmio.size = pci_resource_len(pdev, GTTMMADR_BAR);
xe->mmio.regs = pci_iomap(pdev, GTTMMADR_BAR, 0);
if (xe->mmio.regs == NULL) {
drm_err(&xe->drm, "failed to map registers\n");
return -EIO;
}
/* Setup first tile; other tiles (if present) will be setup later. */
root_tile->mmio.size = SZ_16M;
root_tile->mmio.regs_size = SZ_4M;
root_tile->mmio.regs = xe->mmio.regs;
root_tile->mmio.tile = root_tile;
return devm_add_action_or_reset(xe->drm.dev, mmio_fini, xe);
}
static void mmio_flush_pending_writes(struct xe_gt *gt)
static void mmio_flush_pending_writes(struct xe_mmio *mmio)
{
#define DUMMY_REG_OFFSET 0x130030
struct xe_tile *tile = gt_to_tile(gt);
int i;
if (tile->xe->info.platform != XE_LUNARLAKE)
if (mmio->tile->xe->info.platform != XE_LUNARLAKE)
return;
/* 4 dummy writes */
for (i = 0; i < 4; i++)
writel(0, tile->mmio.regs + DUMMY_REG_OFFSET);
writel(0, mmio->regs + DUMMY_REG_OFFSET);
}
u8 xe_mmio_read8(struct xe_gt *gt, struct xe_reg reg)
u8 xe_mmio_read8(struct xe_mmio *mmio, struct xe_reg reg)
{
struct xe_tile *tile = gt_to_tile(gt);
u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
u32 addr = xe_mmio_adjusted_addr(mmio, reg.addr);
u8 val;
/* Wa_15015404425 */
mmio_flush_pending_writes(gt);
mmio_flush_pending_writes(mmio);
val = readb((reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
trace_xe_reg_rw(gt, false, addr, val, sizeof(val));
val = readb(mmio->regs + addr);
trace_xe_reg_rw(mmio, false, addr, val, sizeof(val));
return val;
}
u16 xe_mmio_read16(struct xe_gt *gt, struct xe_reg reg)
u16 xe_mmio_read16(struct xe_mmio *mmio, struct xe_reg reg)
{
struct xe_tile *tile = gt_to_tile(gt);
u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
u32 addr = xe_mmio_adjusted_addr(mmio, reg.addr);
u16 val;
/* Wa_15015404425 */
mmio_flush_pending_writes(gt);
mmio_flush_pending_writes(mmio);
val = readw((reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
trace_xe_reg_rw(gt, false, addr, val, sizeof(val));
val = readw(mmio->regs + addr);
trace_xe_reg_rw(mmio, false, addr, val, sizeof(val));
return val;
}
void xe_mmio_write32(struct xe_gt *gt, struct xe_reg reg, u32 val)
void xe_mmio_write32(struct xe_mmio *mmio, struct xe_reg reg, u32 val)
{
struct xe_tile *tile = gt_to_tile(gt);
u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
u32 addr = xe_mmio_adjusted_addr(mmio, reg.addr);
trace_xe_reg_rw(gt, true, addr, val, sizeof(val));
trace_xe_reg_rw(mmio, true, addr, val, sizeof(val));
if (!reg.vf && IS_SRIOV_VF(gt_to_xe(gt)))
xe_gt_sriov_vf_write32(gt, reg, val);
if (!reg.vf && mmio->sriov_vf_gt)
xe_gt_sriov_vf_write32(mmio->sriov_vf_gt, reg, val);
else
writel(val, (reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
writel(val, mmio->regs + addr);
}
u32 xe_mmio_read32(struct xe_gt *gt, struct xe_reg reg)
u32 xe_mmio_read32(struct xe_mmio *mmio, struct xe_reg reg)
{
struct xe_tile *tile = gt_to_tile(gt);
u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
u32 addr = xe_mmio_adjusted_addr(mmio, reg.addr);
u32 val;
/* Wa_15015404425 */
mmio_flush_pending_writes(gt);
mmio_flush_pending_writes(mmio);
if (!reg.vf && IS_SRIOV_VF(gt_to_xe(gt)))
val = xe_gt_sriov_vf_read32(gt, reg);
if (!reg.vf && mmio->sriov_vf_gt)
val = xe_gt_sriov_vf_read32(mmio->sriov_vf_gt, reg);
else
val = readl((reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
val = readl(mmio->regs + addr);
trace_xe_reg_rw(gt, false, addr, val, sizeof(val));
trace_xe_reg_rw(mmio, false, addr, val, sizeof(val));
return val;
}
u32 xe_mmio_rmw32(struct xe_gt *gt, struct xe_reg reg, u32 clr, u32 set)
u32 xe_mmio_rmw32(struct xe_mmio *mmio, struct xe_reg reg, u32 clr, u32 set)
{
u32 old, reg_val;
old = xe_mmio_read32(gt, reg);
old = xe_mmio_read32(mmio, reg);
reg_val = (old & ~clr) | set;
xe_mmio_write32(gt, reg, reg_val);
xe_mmio_write32(mmio, reg, reg_val);
return old;
}
int xe_mmio_write32_and_verify(struct xe_gt *gt,
int xe_mmio_write32_and_verify(struct xe_mmio *mmio,
struct xe_reg reg, u32 val, u32 mask, u32 eval)
{
u32 reg_val;
xe_mmio_write32(gt, reg, val);
reg_val = xe_mmio_read32(gt, reg);
xe_mmio_write32(mmio, reg, val);
reg_val = xe_mmio_read32(mmio, reg);
return (reg_val & mask) != eval ? -EINVAL : 0;
}
bool xe_mmio_in_range(const struct xe_gt *gt,
bool xe_mmio_in_range(const struct xe_mmio *mmio,
const struct xe_mmio_range *range,
struct xe_reg reg)
{
u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
u32 addr = xe_mmio_adjusted_addr(mmio, reg.addr);
return range && addr >= range->start && addr <= range->end;
}
/**
* xe_mmio_read64_2x32() - Read a 64-bit register as two 32-bit reads
* @gt: MMIO target GT
* @mmio: MMIO target
* @reg: register to read value from
*
* Although Intel GPUs have some 64-bit registers, the hardware officially
@ -307,21 +310,21 @@ bool xe_mmio_in_range(const struct xe_gt *gt,
*
* Returns the value of the 64-bit register.
*/
u64 xe_mmio_read64_2x32(struct xe_gt *gt, struct xe_reg reg)
u64 xe_mmio_read64_2x32(struct xe_mmio *mmio, struct xe_reg reg)
{
struct xe_reg reg_udw = { .addr = reg.addr + 0x4 };
u32 ldw, udw, oldudw, retries;
reg.addr = xe_mmio_adjusted_addr(gt, reg.addr);
reg_udw.addr = xe_mmio_adjusted_addr(gt, reg_udw.addr);
reg.addr = xe_mmio_adjusted_addr(mmio, reg.addr);
reg_udw.addr = xe_mmio_adjusted_addr(mmio, reg_udw.addr);
/* we shouldn't adjust just one register address */
xe_gt_assert(gt, reg_udw.addr == reg.addr + 0x4);
xe_tile_assert(mmio->tile, reg_udw.addr == reg.addr + 0x4);
oldudw = xe_mmio_read32(gt, reg_udw);
oldudw = xe_mmio_read32(mmio, reg_udw);
for (retries = 5; retries; --retries) {
ldw = xe_mmio_read32(gt, reg);
udw = xe_mmio_read32(gt, reg_udw);
ldw = xe_mmio_read32(mmio, reg);
udw = xe_mmio_read32(mmio, reg_udw);
if (udw == oldudw)
break;
@ -329,13 +332,13 @@ u64 xe_mmio_read64_2x32(struct xe_gt *gt, struct xe_reg reg)
oldudw = udw;
}
xe_gt_WARN(gt, retries == 0,
"64-bit read of %#x did not stabilize\n", reg.addr);
drm_WARN(&mmio->tile->xe->drm, retries == 0,
"64-bit read of %#x did not stabilize\n", reg.addr);
return (u64)udw << 32 | ldw;
}
static int __xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
static int __xe_mmio_wait32(struct xe_mmio *mmio, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
u32 *out_val, bool atomic, bool expect_match)
{
ktime_t cur = ktime_get_raw();
@ -346,7 +349,7 @@ static int __xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 v
bool check;
for (;;) {
read = xe_mmio_read32(gt, reg);
read = xe_mmio_read32(mmio, reg);
check = (read & mask) == val;
if (!expect_match)
@ -372,7 +375,7 @@ static int __xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 v
}
if (ret != 0) {
read = xe_mmio_read32(gt, reg);
read = xe_mmio_read32(mmio, reg);
check = (read & mask) == val;
if (!expect_match)
@ -390,7 +393,7 @@ static int __xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 v
/**
* xe_mmio_wait32() - Wait for a register to match the desired masked value
* @gt: MMIO target GT
* @mmio: MMIO target
* @reg: register to read value from
* @mask: mask to be applied to the value read from the register
* @val: desired value after applying the mask
@ -407,15 +410,15 @@ static int __xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 v
* @timeout_us for different reasons, specially in non-atomic contexts. Thus,
* it is possible that this function succeeds even after @timeout_us has passed.
*/
int xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
int xe_mmio_wait32(struct xe_mmio *mmio, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
u32 *out_val, bool atomic)
{
return __xe_mmio_wait32(gt, reg, mask, val, timeout_us, out_val, atomic, true);
return __xe_mmio_wait32(mmio, reg, mask, val, timeout_us, out_val, atomic, true);
}
/**
* xe_mmio_wait32_not() - Wait for a register to return anything other than the given masked value
* @gt: MMIO target GT
* @mmio: MMIO target
* @reg: register to read value from
* @mask: mask to be applied to the value read from the register
* @val: value not to be matched after applying the mask
@ -426,8 +429,8 @@ int xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 t
* This function works exactly like xe_mmio_wait32() with the exception that
* @val is expected not to be matched.
*/
int xe_mmio_wait32_not(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
int xe_mmio_wait32_not(struct xe_mmio *mmio, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
u32 *out_val, bool atomic)
{
return __xe_mmio_wait32(gt, reg, mask, val, timeout_us, out_val, atomic, false);
return __xe_mmio_wait32(mmio, reg, mask, val, timeout_us, out_val, atomic, false);
}

Some files were not shown because too many files have changed in this diff Show More