Can Guo <can.guo@oss.qualcomm.com> says:
Hi,
The UFS v5.0 and UFSHCI v5.0 standards have published, introducing support
for HS-G6 (46.6 Gbps per lane) through the new UniPro V3.0 interconnect
layer and M-PHY V6.0 physical layer specifications. To achieve reliable
operation at these higher speeds, UniPro V3.0 introduces TX Equalization
and Pre-Coding mechanisms that are essential for signal integrity.
This patch series implements TX Equalization support in the UFS core
driver as specified in UFSHCI v5.0, along with the necessary vendor
operations and a reference implementation for Qualcomm UFS host
controllers.
Background
==========
TX Equalization is a signal conditioning technique that compensates for
channel impairments at high data rates (HS-G4 through HS-G6). It works
by adjusting two key parameters:
- PreShoot: Pre-emphasis applied before the main signal transition
- DeEmphasis: De-emphasis applied after the main signal transition
UniPro V3.0 defines TX Equalization Training (EQTR) procedure to
automatically discover optimal TX Equalization settings. The EQTR
procedure:
1. Starts from the most reliable link state (HS-G1)
2. Iterates through all possible PreShoot and DeEmphasis combinations
3. Evaluates signal quality using Figure of Merit (FOM) measurements
4. Selects the best settings for both host and device TX lanes
For HS-G6, Pre-Coding is also introduced to further improve signal
quality. Pre-Coding must be enabled on both transmitter and receiver
when the RX_FOM indicates it is required.
Implementation Overview
=======================
The implementation follows the UFSHCI v5.0 specification and consists of:
Core Infrastructure (Patches 1-6):
- New vops callback negotiate_pwr_mode() to allow vendors to negotiate
power mode parameters before applying TX Equalization settings
- Support for HS-G6 gear enumeration
- Complete TX EQTR procedure implementation in ufs-txeq.c
- Debugfs interface for TX Equalization parameter inspection and manual
retraining
- Module parameters for adaptive TX Equalization control
Qualcomm Implementation (Patches 7-11):
- PHY-specific configurations for TX EQTR procedure
- Vendor-specific FOM measurement support
- TX Equalization settings application
- Enable TX Equalization for HW version 0x7 and onwards
The implementation is designed to be vendor-agnostic, with platform-
specific details handled through the vops callbacks. Other vendors can
add support by implementing the three new vops:
- tx_eqtr_notify(): Called before/after TX EQTR for vendor setup
- apply_tx_eqtr_settings(): Apply vendor-specific PHY configurations
- get_rx_fom(): Retrieve vendor-specific FOM measurements if needed
Module Parameters
=================
The implementation provides several module parameters for flexibility:
- use_adaptive_txeq: Enable/disable adaptive TX Equalization (default: false)
- adaptive_txeq_gear: Minimum gear for adaptive TX EQ (default: HS-G6)
- use_txeq_presets: Use only the 8 standaird presets (default: false)
- txeq_presets_selected[]: Select specific presets for EQTR
Testing
=======
This patch series has been tested on Qualcomm platforms with UFS 5.0
devices, validating:
- Successful TX EQTR completion for HS-G6
- Proper FOM evaluation and optimal settings selection
- Pre-Coding enablement for HS-G6
- Power mode changes with TX Equalization settings applied
- Report of TX Equalization settings via debugfs entries
- Report of TX EQTR histories via debug entries (see next section)
- Re-training TX Equalization via debugfs entry
Example of TX EQTR history
==========================
Device TX EQTR record summary -
Target Power Mode: HS-G6, Rate-B
Most recent record index: 2
Most recent record timestamp: 219573378 us
TX Lane 0 FOM - PreShoot\DeEmphasis
\ 0 1 2 3 4 5 6 7
0 50 70 65 - - - - x
1 x x x x x x x x
2 100 90 70 - - - - x
3 x x x x x x x x
4 95 90 - - - - - x
5 - - - - - - - x
6 x x x x x x x x
7 x x x x x x x x
TX Lane 1 FOM - PreShoot\DeEmphasis
\ 0 1 2 3 4 5 6 7
0 50 70 60 - - - - x
1 x x x x x x x x
2 100 80 65 - - - - x
3 x x x x x x x x
4 95 85 - - - - - x
5 - - - - - - - x
6 x x x x x x x x
7 x x x x x x x x
Patch Structure
===============
Patches 1-3: Preparatory changes for power mode negotiation and HS-G6
Patch 4: Core TX Equalization and EQTR implementation
Patches 5-7: Debugfs support for TX Equalization
Patches 8-12: Qualcomm vendor implementation
Next
====
One more series has been developed to enhance TX Equalization support,
which will be submitted for review after this series is accepted:
- Provide board specific (static) TX Equalization settings from DTS
- Parse static TX Equalization settings from DTS if provided
- Apply static TX Equalization settings if use_adaptive_txeq is disabled
- Add support for UFS v5.0 attributes qTxEQGnSettings & wTxEQGnSettingsExt
- Enable persistent storage and retrieval of optimal TX Equalization settings
Link: https://patch.msgid.link/20260325152154.1604082-1-can.guo@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
On some platforms, host's M-PHY RX_FOM Attribute always reads 0, meaning SW
cannot rely on Figure of Merit (FOM) to identify the optimal TX
Equalization settings for device's TX Lanes. Implement the vops
ufs_qcom_get_rx_fom() such that SW can utilize the UFS Eye Opening Monitor
(EOM) to evaluate the TX Equalization settings for device's TX Lanes.
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Can Guo <can.guo@oss.qualcomm.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/20260325152154.1604082-11-can.guo@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Drastic environmental changes, such as significant temperature shifts, can
impact link signal integrity. In such cases, retraining TX Equalization is
necessary to compensate for these environmental changes.
Add a debugfs entry, 'tx_eq_ctrl', to allow userspace to manually trigger
the TX Equalization training (EQTR) procedure and apply the identified
optimal settings on the fly. These entries are created on a per-gear basis
for High Speed Gear 4 (HS-G4) and above, as TX EQTR is not supported for
lower gears.
The 'tx_eq_ctrl' entry currently accepts the 'retrain' command to initiate
the procedure. The interface is designed to be scalable to support
additional commands in the future.
Reading the 'tx_eq_ctrl' entry provides a usage hint to the user, ensuring
the interface is self-documenting.
The ufshcd's debugfs folder structure will look like below:
/sys/kernel/debug/ufshcd/*ufs*/
|--tx_eq_hs_gear1/
| |--device_tx_eq_params
| |--host_tx_eq_params
|--tx_eq_hs_gear2/
|--tx_eq_hs_gear3/
|--tx_eq_hs_gear4/
|--tx_eq_hs_gear5/
|--tx_eq_hs_gear6/
|--device_tx_eq_params
|--device_tx_eqtr_record
|--host_tx_eq_params
|--host_tx_eqtr_record
|--tx_eq_ctrl
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Can Guo <can.guo@oss.qualcomm.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/20260325152154.1604082-8-can.guo@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
MIPI Unipro3.0 introduced PA_TxEQGnSetting and PA_PreCodeEn attributes for
TX Equalization and Pre-Coding. It is Host Software's responsibility to
configure these attributes for both host and device before initiating Power
Mode Change to High-Speed Gears.
MIPI Unipro3.0 also introduced TX Equalization Training (EQTR) to identify
optimal TX Equalization settings for use by both Host's and Device's
UniPro. TX EQTR shall be initiated from the most reliable High-Speed Gear
(HS-G1) targeting High-Speed Gears (HS-G4 to HS-G6).
Implement TX Equalization configuration and TX EQTR procedure as defined
in UFSHCI v5.0 specification. The TX EQTR procedure determines the optimal
TX Equalization settings by iterating through all possible PreShoot and
DeEmphasis combinations and selecting the best combinations for both Host
and Device based on Figure of Merit (FOM) evaluation.
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Can Guo <can.guo@oss.qualcomm.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/20260325152154.1604082-5-can.guo@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Currently, callers must manually toggle hba->force_pmc before and after
calling ufshcd_config_pwr_mode() to force a Power Mode change. Introduce
enum ufshcd_pmc_policy and refactor ufshcd_config_pwr_mode() to accept
pmc_policy as a parameter to force a Power Mode change.
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Bean Huo <beanhuo@micron.com>
Signed-off-by: Can Guo <can.guo@oss.qualcomm.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/20260325152154.1604082-3-can.guo@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Most vendor specific implemenations of vops pwr_change_notify(PRE_CHANGE)
are fulfilling two things at once:
- Vendor specific target power mode negotiation
- Vendor specific power mode change preparation
When TX Equalization is added into consideration, before power mode change
to a target power mode, TX Equalization Training (EQTR) needs be done for
that target power mode. In addition, UFSHCI spec requires to start TX EQTR
from HS-G1 (the most reliable High Speed Gear).
Adding TX EQTR before pwr_change_notify(PRE_CHANGE) is not applicable
because we don't know the negotiated power mode yet.
Adding TX EQTR post pwr_change_notify(PRE_CHANGE) is inappropriate because
pwr_change_notify(PRE_CHANGE) has finished preparation for a power mode
change to negotiated power mode, yet we are changing power mode to HS-G1
for TX EQTR.
Add a new vops negotiate_pwr_mode() so that vendor specific power mode
negotiation can be fulfilled in its vendor specific implementations. Later
on, TX EQTR can be added post vops negotiate_pwr_mode() and before vops
pwr_change_notify(PRE_CHANGE).
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Can Guo <can.guo@oss.qualcomm.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/20260325152154.1604082-2-can.guo@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
On some platforms, the VCC regulator has a slow ramp-up time. Add a delay
after enabling VCC to ensure voltage has fully stabilized before we enable
the clocks.
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Ed Tsai <ed.tsai@mediatek.com>
Link: https://patch.msgid.link/20260310005230.4001904-4-ed.tsai@mediatek.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The UniPro stack manages to repair many potential Link problems without the
need to notify the Application Layer. Repair mechanisms of the stack
include L2 re-transmission and successful handling of PA_INIT.req.
Nevertheless, any successful repair sequence requires Link bandwidth that
is no longer vailable for the Application. Therefore, it may be useful for
an Application to understand how often such repair attempts are made.
The DME implements Quality of Service monitoring using a simple counting
scheme, counting error events and comparing them against the number of
correctly received or transmitted bytes. When the error counter exceeds a
programmed threshold before the byte counter overflows, a DME_QoS.ind is
issued to the Application and both counters are reset. When the byte
counter overflows before the error counter has reached the programmed
threshold, both counters are reset without triggering a DME_QoS.ind.
The DME provides Link quality monitoring for the following purposes:
1. Detection of re-occurring repaired fatal error conditions on the Link
(PA_INIT loop). This kind of detection is useful if capabilities
exchanged between local and peer permit a potential operation at a
higher M-PHY Gear, but the physical interconnect between local and peer
Device does not, or, after Line quality degradation, no longer satisfies
channel characteristics.
2. Detection of degraded inbound or outbound Link quality, to allow an
Application to issue an ADAPT sequence for a Link running in HS-G4 or
higher HS Gears. This kind of detection is used to monitor a slowly
degrading Link quality, e.g., one being affected by temperature and
voltage variations, against the expected M-PHY bit error rate.
Userspace can configure and enable UniPro QoS via UniPro QoS Attributes
(via UFS BSG) and get notified by dme_qos_notification without polling
UniPro QoS Status attribute. The dme_qos_notification attribute is a
bitfield with the following bit assignments:
Bit Description
=== ======================================
0 DME QoS Monitor has been reset by host
1 QoS from TX is detected
2 QoS from RX is detected
3 QoS from PA_INIT is detected
Signed-off-by: Can Guo <can.guo@oss.qualcomm.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/20260305110856.959211-2-can.guo@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The added print statement from a recent fix causes the driver to fail
building when CONFIG_PM is disabled:
drivers/ufs/host/ufs-mediatek.c: In function 'ufs_mtk_resume':
drivers/ufs/host/ufs-mediatek.c:1890:40: error: 'struct dev_pm_info' has no member named 'request'
1890 | hba->dev->power.request,
It seems unlikely that the driver can work at all without CONFIG_PM, so
just add a dependency and remove the existing ifdef checks, rather than
adding another ifdef.
Fixes: 15ef3f5aa8 ("scsi: ufs: host: mediatek: Enhance recovery on resume failure")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
Link: https://patch.msgid.link/20260202095052.1232703-1-arnd@kernel.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Some UFS platforms only support a limited subset of power levels.
Currently, the sysfs interface allows users to set any PM level without
validating the minimum supported value. If an unsupported level is
selected, suspend may fail.
Introduce an pm_lvl_min field in the ufs_hba structure and use it to
clamp the PM level requested via sysfs so that only supported levels are
accepted. Platforms that require a minimum PM level can set this field
during probe.
Signed-off-by: Ram Kumar Dwivedi <ram.dwivedi@oss.qualcomm.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20260113080046.284089-4-ram.dwivedi@oss.qualcomm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Bean Huo <beanhuo@iokpp.de> says:
This patch series introduces OP-TEE based RPMB (Replay Protected
Memory Block) support for UFS devices, extending the kernel-level
secure storage capabilities that are currently available for eMMC
devices.
Previously, OP-TEE required a userspace supplicant to access RPMB
partitions, which created complex dependencies and reliability issues,
especially during early boot scenarios. Recent work by Linaro has
moved core supplicant functionality directly into the Linux kernel for
eMMC devices, eliminating userspace dependencies and enabling
immediate secure storage access. This series extends the same approach
to UFS devices, which are used in enterprise and mobile applications
that require secure storage capabilities.
Benefits:
- Eliminates dependency on userspace supplicant for UFS RPMB access
- Enables early boot secure storage access (e.g., fTPM, secure UEFI
variables)
- Provides kernel-level RPMB access as soon as UFS driver is
initialized
- Removes complex initramfs dependencies and boot ordering
requirements
- Ensures reliable and deterministic secure storage operations
- Supports both built-in and modular fTPM configurations.
Prerequisites:
--------------
This patch series depends on commit 7e8242405b ("rpmb: move struct
rpmb_frame to common header") which has been merged into mainline
v6.18-rc2.
Link: https://patch.msgid.link/20251107230518.4060231-1-beanhuo@iokpp.de
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Instead of storing the tag of the reserved command in hba->reserved_slot,
use scsi_get_internal_cmd() and scsi_put_internal_cmd() to allocate the
tag for the reserved command dynamically. Add
ufshcd_queue_reserved_command() for submitting reserved commands. Add
support in ufshcd_abort() for device management commands. Use
blk_execute_rq() for submitting reserved commands. Remove the code and
data structures that became superfluous. This includes
ufshcd_wait_for_dev_cmd(), hba->reserved_slot and ufs_dev_cmd.complete.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20251031204029.2883185-29-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Remove the ufshcd_lrb task_tag member and use scsi_cmd_to_rq(cmd)->tag
instead. Use rq->tag instead of lrbp->task_tag. This patch reduces the
size of struct ufshcd_lrb.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20251031204029.2883185-26-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Set .cmd_size in the SCSI host template such that the SCSI core makes
struct scsi_cmnd and struct ufshcd_lrb adjacent. Convert the cmd->lrbp
and lrbp->cmd memory loads into pointer offset calculations. Remove the
data structure members that became superfluous, namely ufshcd_lrb.cmd
and ufs_hba.lrb. Since ufshcd_lrb.cmd is removed, this pointer cannot be
used anymore to test whether or not a command is a SCSI command.
Introduce a new function for this purpose, namely ufshcd_is_scsi_cmd().
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20251031204029.2883185-24-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Add OP-TEE based RPMB support for UFS devices. This enables secure RPMB
operations on UFS devices through OP-TEE, providing the same
functionality available for eMMC devices and extending kernel-based
secure storage support to UFS-based systems.
Benefits of OP-TEE based RPMB implementation:
- Eliminates dependency on userspace supplicant for RPMB access
- Enables early boot secure storage access (e.g., fTPM, secure UEFI
variables)
- Provides kernel-level RPMB access as soon as UFS driver is
initialized
- Removes complex initramfs dependencies and boot ordering requirements
- Ensures reliable and deterministic secure storage operations
- Supports both built-in and modular fTPM configurations
[mkp: make this build as a module]
Co-developed-by: Can Guo <can.guo@oss.qualcomm.com>
Signed-off-by: Can Guo <can.guo@oss.qualcomm.com>
Reviewed-by: Avri Altman <avri.altman@sandisk.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Bean Huo <beanhuo@micron.com>
Link: https://patch.msgid.link/20251107230518.4060231-4-beanhuo@iokpp.de
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Convert SD_ASCII_STD and SD_RAW from boolean macros to enum values for
improved code readability. This makes ufshcd_read_string_desc() calls
self-documenting by using explicit enum values instead of true/false.
Move the ufshcd_read_string_desc() declaration from include/ufs/ufshcd.h
to drivers/ufs/core/ufshcd-priv.h since this function is not exported.
Co-developed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Suggested-by: Avri Altman <Avri.Altman@sandisk.com>
Signed-off-by: Bean Huo <beanhuo@micron.com>
Link: https://patch.msgid.link/20251107230518.4060231-2-beanhuo@iokpp.de
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
ufshcd_link_startup() has a facility (link_startup_again) to issue
DME_LINKSTARTUP a 2nd time even though the 1st time was successful.
Some older hardware benefits from that, however the behaviour is
non-standard, and has been found to cause link startup to be unreliable
for some Intel Alder Lake based host controllers.
Add UFSHCD_QUIRK_PERFORM_LINK_STARTUP_ONCE to suppress
link_startup_again, in preparation for setting the quirk for affected
controllers.
Fixes: 7dc9fb47bc ("scsi: ufs: ufs-pci: Add support for Intel ADL")
Cc: stable@vger.kernel.org
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20251024085918.31825-3-adrian.hunter@intel.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Bart Van Assche <bvanassche@acm.org> says:
Hi Martin,
This patch series includes two bug fixes for this development cycle
and six small patches that are intended for the next merge window. If
applying the first two patches only during the current development
cycle would be inconvenient, postponing all patches until the next
merge window is fine with me.
Please consider including these patches in the upstream kernel.
Thanks,
Bart.
[mkp: Applied patches #1 and #2 to 6.18/scsi-fixes]
Link: https://patch.msgid.link/20251014200118.3390839-1-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
ufshcd_enable_intr() is not exported and hence should not be declared in
include/ufs/ufshcd.h.
Fixes: 2537577979 ("scsi: ufs: core: Change MCQ interrupt enable flow")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/20251014200118.3390839-7-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Since uic_command::cmd_active is used as a boolean variable, change its
type from 'int' into 'bool'. No functionality has been changed.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20251014200118.3390839-5-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Peter Wang <peter.wang@mediatek.com says>:
Improves the UFS Mediatek driver by correcting clock scaling with PM
QoS, and adjusting power management flows. It addresses
shutdown/suspend race conditions, and removes redundant
functions. Support for new platforms is added with the MMIO_OTSD_CTRL
register, and MT6991 performance is optimized with MRTT and random
improvements. These changes collectively enhance driver performance,
stability, and compatibility.
Changes since v1:
1. Remove two patches that will be fixed in UFS core.
- ufs: host: mediatek: Fix runtime suspend error deadlock
- ufs: host: mediatek: Enable interrupts for MCQ mode
2. Use hba->shutting_down instead of ufshcd_is_user_access_allowed
v1:
https://patch.msgid.link/20250918104000.208856-1-peter.wang@mediatek.com
Link: https://patch.msgid.link/20250924094527.2992256-1-peter.wang@mediatek.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Remove the duplicate ufs_mtk_us_to_ahit() function in the UFS Mediatek
driver and export the existing ufshcd_us_to_ahit() function for shared
use. This change reduces redundancy and maintains consistency across the
codebase.
Signed-off-by: Peter Wang <peter.wang@mediatek.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Acked-by: Chun-Hung Wu <chun-hung.wu@mediatek.com>
Link: https://patch.msgid.link/20250924094527.2992256-7-peter.wang@mediatek.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Correct clock scaling with PM QoS during suspend and resume. Ensure PM
QoS is released during suspend if scaling up and re-applied after
resume. This prevents performance issues and maintains proper power
management.
Signed-off-by: Peter Wang <peter.wang@mediatek.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Acked-by: Chun-Hung Wu <chun-hung.wu@mediatek.com>
Link: https://patch.msgid.link/20250924094527.2992256-2-peter.wang@mediatek.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
After the UFS device VCC is powered off, all the UFS device
manufacturers require a minimum of 1ms of power-off time before VCC can
be powered on again. This requirement has been verified with all the UFS
device manufacturer's datasheets.
Replace the hard coded 5ms delay with a variable with a default setting
of 2ms to improve the system resume latency. The platform drivers can
override this setting as needed.
Signed-off-by: Bao D. Nguyen <quic_nguyenb@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Link: https://patch.msgid.link/72fa649406a0bf02271575b7d58f22c968aa5d7e.1760383740.git.quic_nguyenb@quicinc.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The cpu_latency_qos_add/remove/update_request interfaces lack internal
synchronization by design, requiring the caller to ensure thread safety.
The current implementation relies on the 'pm_qos_enabled' flag, which is
insufficient to prevent concurrent access and cannot serve as a proper
synchronization mechanism. This has led to data races and list
corruption issues.
A typical race condition call trace is:
[Thread A]
ufshcd_pm_qos_exit()
--> cpu_latency_qos_remove_request()
--> cpu_latency_qos_apply();
--> pm_qos_update_target()
--> plist_del <--(1) delete plist node
--> memset(req, 0, sizeof(*req));
--> hba->pm_qos_enabled = false;
[Thread B]
ufshcd_devfreq_target
--> ufshcd_devfreq_scale
--> ufshcd_scale_clks
--> ufshcd_pm_qos_update <--(2) pm_qos_enabled is true
--> cpu_latency_qos_update_request
--> pm_qos_update_target
--> plist_del <--(3) plist node use-after-free
Introduces a dedicated mutex to serialize PM QoS operations, preventing
data races and ensuring safe access to PM QoS resources, including sysfs
interface reads.
Fixes: 2777e73fc1 ("scsi: ufs: core: Add CPU latency QoS support for UFS driver")
Signed-off-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Tested-by: Huan Tang <tanghuan@vivo.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Move the MCQ interrupt enable process to
ufshcd_mcq_make_queues_operational() to ensure that interrupts are set
correctly when making queues operational, similar to
ufshcd_make_hba_operational(). This change addresses the issue where
ufshcd_mcq_make_queues_operational() was not fully operational due to
missing interrupt enablement.
This change only affects host drivers that call
ufshcd_mcq_make_queues_operational(), i.e. ufs-mediatek.
Signed-off-by: Peter Wang <peter.wang@mediatek.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Nitin Rawat <nitin.rawat@oss.qualcomm.com> says:
The patch series simplifies the UFS MCQ (Multi Circular Queue)
resource mapping in the Qualcomm UFS host controller driver by
replacing the complex multi-resource approach with a streamlined
single-resource implementation.
The current MCQ implementation uses multiple separate resource
mappings (RES_UFS, RES_MCQ, RES_MCQ_SQD, RES_MCQ_VS) with dynamic
resource allocation, which increases code complexity and potential for
resource mapping errors. This approach also doesn't align with the
device tree binding specification that defines a single 'mcq' memory
region.
Replace the multi-resource mapping with a single "mcq" resource that
encompasses the entire MCQ configuration space. The doorbell registers
(SQD, CQD, SQIS, CQIS) are accessed using predefined offsets relative
to the MCQ base address, providing clearer memory layout organization.
Tested on Qualcomm platforms SM8650 and SM8750 with UFS MCQ enabled.
Changes from v3:
1. Addressed Krzysztof comment to separate device tree and driver
patch independently in different patch series. This series caters
driver changes.
2. Addressed Manivannan's change to update commit text and remove
redundant null check in mcq code.
3. Addressed Manivannan's to Update few offsets as fixed definition
instead of enum.
Changes from v2:
1. Removed dt-bindings patch as existing binding supports required
reg-names format.
2. Added patch to refactor MCQ register dump logic for new resource
mapping.
3. Added patch to remove unused ufshcd_res_info structure from UFS core.
4. Changed reg-names from "ufs_mem" to "std" in device tree patches.
5. Reordered patches with driver changes first, then device tree changes.
6. Updated SM8750 MCQ region size from 0x2000 to 0x15000
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Refactor MCQ register dump to align with the new resource mapping. As
part of refactor, below changes are done:
- Update ufs_qcom_dump_regs() function signature to accept direct
base address instead of resource ID enum
- Modify ufs_qcom_dump_mcq_hci_regs() to use hba->mcq_base and
calculated addresses from MCQ operation info
- Replace enum ufshcd_res with direct memory-mapped I/O addresses
Additionally remove the ufshcd_res_info structure and associated enum
ufshcd_res definitions from the UFS host controller header. These were
previously used for MCQ resource mapping but are no longer needed
following recent refactoring to use direct base addresses instead of
multiple separate resource regions.
Signed-off-by: Nitin Rawat <quic_nitirawa@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Peter Wang <peter.wang@mediatek.com> says:
These patches collectively enhance the UFS host driver's reliability,
power management efficiency, and error recovery mechanisms on MediaTek
platforms. They address critical issues and introduce optimizations
that improve system stability and performance.
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Improve the recovery process for hibernation exit failures. Trigger the
error handler and break the suspend operation to ensure effective
recovery from hibernation errors. Activate the error handling mechanism
by ufshcd_force_error_recovery and scheduling the error handler work.
Signed-off-by: Peter Wang <peter.wang@mediatek.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The size of the data structures that are used in the hot path matters
for performance (IOPS). Hence this patch that reduces the size of struct
ufshcd_lrb on 64-bit systems by 16 bytes. The size of this data
structure is reduced from 152 to 136 bytes.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20250819154356.2256952-1-bvanassche@acm.org
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Introduce ufshcd_dme_rmw() API to read, modify, and write DME attributes
in UFS host controllers using a mask and value.
Signed-off-by: Nitin Rawat <quic_nitirawa@quicinc.com>
Link: https://lore.kernel.org/r/20250714075336.2133-3-quic_nitirawa@quicinc.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Pull in fixes from 6.15 and resolve a few conflicts so we can have a
clean base for UFS patches.
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Follow JESD220G, support a WB buffer resize function through sysfs. The
host can obtain resize hint and resize status, and enable the resize
operation. Add three sysfs nodes:
1. wb_resize_enable
2. wb_resize_hint
3. wb_resize_status
The detailed definition of the three nodes can be found in the sysfs
documentation.
Signed-off-by: Huan Tang <tanghuan@vivo.com>
Signed-off-by: Lu Hongfei <luhongfei@vivo.com>
Link: https://lore.kernel.org/r/20250411092924.1116-1-tanghuan@vivo.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
In preparation of adding a threaded interrupt handler, track when the
MCQ ESI interrupt handlers were installed so we can optimize the MCQ
interrupt handling to avoid walking the threaded handler in the case ESI
handlers are enabled.
Signed-off-by: Neil Armstrong <neil.armstrong@linaro.org>
Link: https://lore.kernel.org/r/20250407-topic-ufs-use-threaded-irq-v3-2-08bee980f71e@linaro.org
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
In order to prepare introduction of a threaded interrupt handler, drop
last_intr_status & last_intr_ts drop the ufs_stats struct, and the
associated debug code.
Suggested-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Neil Armstrong <neil.armstrong@linaro.org>
Link: https://lore.kernel.org/r/20250407-topic-ufs-use-threaded-irq-v3-1-08bee980f71e@linaro.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The ufs device JEDEC specification version 4.1 adds support for the
device level exception events. To support this new device level
exception feature, expose two new sysfs nodes below to provide the user
space access to the device level exception information.
/sys/bus/platform/drivers/ufshcd/*/device_lvl_exception_count
/sys/bus/platform/drivers/ufshcd/*/device_lvl_exception_id
The device_lvl_exception_count sysfs node reports the number of device
level exceptions that have occurred since the last time this variable is
reset. Writing a value of 0 will reset it. The device_lvl_exception_id
reports the exception ID which is the qDeviceLevelExceptionID attribute
of the device JEDEC specifications version 4.1 and later. The user space
application can query these sysfs nodes to get more information about
the device level exception.
Signed-off-by: Bao D. Nguyen <quic_nguyenb@quicinc.com>
Link: https://lore.kernel.org/r/6278d7c125b2f0cf5056f4a647a4b9c1fdd24fc7.1743198325.git.quic_nguyenb@quicinc.com
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Arthur Simchaev <arthur.simchaev@sandisk.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
There is a TOCTOU race in ufshcd_compl_one_cqe(): hba->dev_cmd.complete may
be cleared from another thread after it has been checked and before it is
used. Fix this race by moving the device command completion from the stack
of the device command submitter into struct ufs_hba. This patch fixes the
following kernel crash:
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
Call trace:
_raw_spin_lock_irqsave+0x34/0x80
complete+0x24/0xb8
ufshcd_compl_one_cqe+0x13c/0x4f0
ufshcd_mcq_poll_cqe_lock+0xb4/0x108
ufshcd_intr+0x2f4/0x444
__handle_irq_event_percpu+0xbc/0x250
handle_irq_event+0x48/0xb0
Fixes: 5a0b0cb9be ("[SCSI] ufs: Add support for sending NOP OUT UPIU")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20250314225206.1487838-1-bvanassche@acm.org
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Shawn Lin <shawn.lin@rock-chips.com> says:
This patchset adds initial UFS controller supprt for RK3576 SoC.
Patch 1 is the dt-bindings. Patch 2-4 deal with rpm and spm support
in advanced suggested by Ulf. Patch 5 exports two new APIs for host
driver. Patch 6 and 7 are the host driver and dtsi support.
Link: https://lore.kernel.org/r/1738736156-119203-1-git-send-email-shawn.lin@rock-chips.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
These two APIs will be used by glue driver if they need a different HCE
process.
Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
Link: https://lore.kernel.org/r/1738736156-119203-6-git-send-email-shawn.lin@rock-chips.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Ziqi Chen <quic_ziqichen@quicinc.com> says:
With OPP V2 enabled, devfreq can scale clocks amongst multiple frequency
plans. However, the gear speed is only toggled between min and max during
clock scaling. Enable multi-level gear scaling by mapping clock frequencies
to gear speeds, so that when devfreq scales clock frequencies we can put
the UFS link at the appropraite gear speeds accordingly.
This series has been tested on below platforms -
sm8550 mtp + UFS3.1
SM8650 MTP + UFS3.1
SM8750 MTP + UFS4.0
Tested-by: Neil Armstrong <neil.armstrong@linaro.org> # on SM8550-QRD
Tested-by: Neil Armstrong <neil.armstrong@linaro.org> # on SM8550-HDK
Tested-by: Neil Armstrong <neil.armstrong@linaro.org> # on SM8650-HDK
Link: https://lore.kernel.org/r/20250213080008.2984807-1-quic_ziqichen@quicinc.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
During clock scaling, Write Booster is toggled on or off based on whether
the clock is scaled up or down. However, with OPP V2 powered multi-level
gear scaling, the gear can be scaled amongst multiple gear speeds, e.g., it
may scale down from G5 to G4, or from G4 to G2. To provide flexibilities,
add a new field for clock scaling such that during clock scaling Write
Booster can be enabled or disabled based on gear speeds but not based on
scaling up or down.
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Co-developed-by: Ziqi Chen <quic_ziqichen@quicinc.com>
Signed-off-by: Ziqi Chen <quic_ziqichen@quicinc.com>
Link: https://lore.kernel.org/r/20250213080008.2984807-8-quic_ziqichen@quicinc.com
Reviewed-by: Bean Huo <beanhuo@micron.com>
Tested-by: Neil Armstrong <neil.armstrong@linaro.org>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Add a vop to map UFS host controller clock frequencies to the corresponding
maximum supported UFS high speed gear speeds. During clock scaling, we can
map the target clock frequency, demanded by devfreq, to the maximum
supported gear speed, so that devfreq can scale the gear to the highest
gear speed supported at the target clock frequency, instead of just scaling
up/down the gear between the min and max gear speeds.
Co-developed-by: Ziqi Chen <quic_ziqichen@quicinc.com>
Signed-off-by: Ziqi Chen <quic_ziqichen@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Link: https://lore.kernel.org/r/20250213080008.2984807-4-quic_ziqichen@quicinc.com
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Tested-by: Neil Armstrong <neil.armstrong@linaro.org>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Instead of only two frequencies, if OPP V2 is used, the UFS devfreq clock
scaling may scale the clock among multiple frequencies, so just passing
up/down to vop clk_scale_notify() is not enough to cover the intermediate
clock freqs between the min and max freqs. Hence pass the target_freq,
which will be used in successive commits, to clk_scale_notify() to allow
the vop to perform corresponding configurations with regard to the clock
freqs.
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Co-developed-by: Ziqi Chen <quic_ziqichen@quicinc.com>
Signed-off-by: Ziqi Chen <quic_ziqichen@quicinc.com>
Link: https://lore.kernel.org/r/20250213080008.2984807-2-quic_ziqichen@quicinc.com
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Tested-by: Neil Armstrong <neil.armstrong@linaro.org>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The third pwr_change_notify() argument is an input parameter. Make this
explicit by declaring it 'const'.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20250212213838.1044917-1-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Martin hi,
The UFS4.1 standard, released on January 8, 2025, added a new exception
event: HEALTH_CRITICAL, which notifies the host of a device's critical
health condition. This notification implies that the device is approaching
the end of its lifetime based on the amount of performed program/erase
cycles.
Once an EOL (End-of-Life) exception event is received, we increment a
designated member, which is exposed via a sysfs entry. This new entry, will
report the number of times a critical health event has been reported by a
UFS device.
To handle this new sysfs entry, userspace applications can use select(),
poll(), or epoll() to monitor changes in the critical_health attribute. The
kernel will call sysfs_notify() to signal changes, allowing the userspace
application to detect and respond to these changes efficiently.
The host can gain further insight into the specific issue by reading one of
the following attributes: bPreEOLInfo, bDeviceLifeTimeEstA,
bDeviceLifeTimeEstB, bWriteBoosterBufferLifeTimeEst, and
bRPMBLifeTimeEst. All those are available for reading via the driver's
sysfs entries or through an applicable utility. It is up to userspace to
read these attributes if needed.
Signed-off-by: Avri Altman <avri.altman@wdc.com>
Link: https://lore.kernel.org/r/20250211065813.58091-1-avri.altman@wdc.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
devm_blk_crypto_profile_init() registers a cleanup handler to run when
the associated (platform-) device is being released. For UFS, the
crypto private data and pointers are stored as part of the ufs_hba's
data structure 'struct ufs_hba::crypto_profile'. This structure is
allocated as part of the underlying ufshcd and therefore Scsi_host
allocation.
During driver release or during error handling in ufshcd_pltfrm_init(),
this structure is released as part of ufshcd_dealloc_host() before the
(platform-) device associated with the crypto call above is released.
Once this device is released, the crypto cleanup code will run, using
the just-released 'struct ufs_hba::crypto_profile'. This causes a
use-after-free situation:
Call trace:
kfree+0x60/0x2d8 (P)
kvfree+0x44/0x60
blk_crypto_profile_destroy_callback+0x28/0x70
devm_action_release+0x1c/0x30
release_nodes+0x6c/0x108
devres_release_all+0x98/0x100
device_unbind_cleanup+0x20/0x70
really_probe+0x218/0x2d0
In other words, the initialisation code flow is:
platform-device probe
ufshcd_pltfrm_init()
ufshcd_alloc_host()
scsi_host_alloc()
allocation of struct ufs_hba
creation of scsi-host devices
devm_blk_crypto_profile_init()
devm registration of cleanup handler using platform-device
and during error handling of ufshcd_pltfrm_init() or during driver
removal:
ufshcd_dealloc_host()
scsi_host_put()
put_device(scsi-host)
release of struct ufs_hba
put_device(platform-device)
crypto cleanup handler
To fix this use-after free, change ufshcd_alloc_host() to register a
devres action to automatically cleanup the underlying SCSI device on
ufshcd destruction, without requiring explicit calls to
ufshcd_dealloc_host(). This way:
* the crypto profile and all other ufs_hba-owned resources are
destroyed before SCSI (as they've been registered after)
* a memleak is plugged in tc-dwc-g210-pci.c remove() as a
side-effect
* EXPORT_SYMBOL_GPL(ufshcd_dealloc_host) can be removed fully as
it's not needed anymore
* no future drivers using ufshcd_alloc_host() could ever forget
adding the cleanup
Fixes: cb77cb5abe ("blk-crypto: rename blk_keyslot_manager to blk_crypto_profile")
Fixes: d76d9d7d10 ("scsi: ufs: use devm_blk_ksm_init()")
Cc: stable@vger.kernel.org
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Link: https://lore.kernel.org/r/20250124-ufshcd-fix-v4-1-c5d0144aae59@linaro.org
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Acked-by: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This reverts commit 6e1d850acf.
Although added a while ago, to date no one make use of ext_iid,
specifically incorporates it in the upiu header. Therefore, remove it as
it is currently unused and not serving any purpose.
Signed-off-by: Avri Altman <avri.altman@wdc.com>
Link: https://lore.kernel.org/r/20250103080204.63951-1-avri.altman@wdc.com
Cc: Can Guo <quic_cang@quicinc.com>
Cc: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
