github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

DRM Rust changes for v7.4-rc1

Browse Source 
- Cross-subsystem:
 
   - I/O (shared from driver-core tree via signed tag rust-io-7.4-rc1):
 
     - Reimplement register!() as a procedural macro, require matching
       base types for register access, and replace relative registers
       with typed I/O projections. Convert Nova, Tyr and the PCI sample
 
     - Add AsRepr/AsReprMut conversions for I/O on compatible wrapper
       types and transmute helpers with deferred size checks
 
   - DMA (shared from driver-core tree via signed tag rust-dma-7.4-rc1):
 
     - Tie Coherent, CoherentBox and CoherentHandle to the device's bound
       lifetime, and allow scoped debugfs files backed by non-'static
       data
 
   - Allow auxiliary registration-data closures to return covariant
     sub-fields and add UserSliceWriter::write_truncated()
 
   - Implement Send and Sync for MapleTree
 
   - Add PCI resource_flags() and IORESOURCE_MEM_64, and convert PCI IRQ
     flags to impl_flags!()
 
   - Add I/O view cast() with compile-time size and alignment checks
 
 - DRM:
 
   - Add Rust dma_fence abstractions with separate producer (DriverFence)
     and consumer (Fence) types, per-context sequencing, signal callbacks
     and preallocation. Additional backend operations remain follow-up
     work.
 
   - Initialize DRM registrations and their data in place, removing a
     separate allocation
 
   - Drop the unused flags argument from drm::Registration::new()
 
 - Nova:
 
   - UAPI and Cross Module API:
     - Add NovaCoreApi and a lifetime-bound handle for auxiliary child
       drivers, carried in nova-drm registration data
     - Expose a render node and extensible INFO ioctl reporting GPU
       architecture, opaque chip ID, usable VRAM and VRAM BAR sizes,
       full and short GPU names, and GID, with size discovery and
       truncation to the userspace buffer size
     - Move BAR-size queries into GPU info, removing GETPARAM and
       changing the UAPI
 
   - PRAMIN access:
     - Add typed CPU access to VRAM through the movable 1 MiB BAR0
       PRAMIN window, with per-architecture support from Turing through
       Blackwell
     - Add GpuMm centralized memory manager, VramAddress type, aperture
       documentation and optional probe-time self-tests under
       CONFIG_NOVA_CORE_SELFTESTS
 
   - GPU virtual memory and BAR1:
     - Add VRAM buddy allocation, TLB flushing, MMU v2/v3 page-table
       types with trait-based generics (PteOps, PdeOps, DualPdeOps,
       MmuConfig) and a generic walker
     - Add maple-tree virtual address tracking and multi-page mapping,
       separating allocation from mapping execution
     - Add CPU access to mapped VRAM through BAR1 and memory-management
       self-tests; handle 64-bit BAR0 when locating BAR1
 
   - GPU interrupts and GSP events:
     - Add GIN (GPU Interrupt and Notification) interrupt-tree support
       with MSI-X, MSI fallback and per-architecture rearming
     - Drain GSP events between commands via a threaded SWGEN0 handler
     - Report OS errors, robust-channel notifications and unknown
       messages without aborting command waits; hold the queue lock and
       use one deadline per wait
     - Disable GSP event interrupts on persistent falcon faults or
       drain errors
     - Add a delivery/rearm self-test, interrupt-tree KUnit coverage
       and interrupt-controller documentation
 
   - GSP messaging fixes:
     - Queue setup RPCs before GSP-RM boot
     - Correct DMA barriers in both queue directions
 
   - Firmware and transport:
     - Extract a reusable radix3 buffer mapper; zero-pad page-table
       levels to whole pages to prevent booter reads of stale padding
     - Correct GSP-FMC naming and boot-flow documentation
     - Set MCTP header version 1 for FSP messages and retain the
       command queue's BAR0 reference
 
   - Resource handling:
     - Replace unnecessary BarUser reference counting with in-place
       ownership and borrowed access; consolidate probe initialization
     - Declare an unlimited DMA maximum segment size
 
 - Tyr:
 
   - Use bitfield!() for firmware section flags and memory attributes
     that do not describe registers
 
   - Zero unsupported and newly added GPU-info fields, and limit the
     doorbell array to 17 entries to fit the mapped register region
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQS2q/xV6QjXAdC7k+1FlHeO1qrKLgUCastrnAAKCRBFlHeO1qrK
 LqE4AP9SjAxwHqRYDR2MKlULdBOzRfDKk+Z5OTxYLoOH/6AXZAEArb9tXybEvja+
 gjTesUT+hZFI6O5yfsqfTwxjypvv6AY=
 =X0Mt
 -----END PGP SIGNATURE-----

Merge tag 'drm-rust-next-2026-10-11' of https://gitlab.freedesktop.org/drm/rust/kernel into drm-next

DRM Rust changes for v7.4-rc1

- Cross-subsystem:

  - I/O (shared from driver-core tree via signed tag rust-io-7.4-rc1):

    - Reimplement register!() as a procedural macro, require matching
      base types for register access, and replace relative registers
      with typed I/O projections. Convert Nova, Tyr and the PCI sample

    - Add AsRepr/AsReprMut conversions for I/O on compatible wrapper
      types and transmute helpers with deferred size checks

  - DMA (shared from driver-core tree via signed tag rust-dma-7.4-rc1):

    - Tie Coherent, CoherentBox and CoherentHandle to the device's bound
      lifetime, and allow scoped debugfs files backed by non-'static
      data

  - Allow auxiliary registration-data closures to return covariant
    sub-fields and add UserSliceWriter::write_truncated()

  - Implement Send and Sync for MapleTree

  - Add PCI resource_flags() and IORESOURCE_MEM_64, and convert PCI IRQ
    flags to impl_flags!()

  - Add I/O view cast() with compile-time size and alignment checks

- DRM:

  - Add Rust dma_fence abstractions with separate producer (DriverFence)
    and consumer (Fence) types, per-context sequencing, signal callbacks
    and preallocation. Additional backend operations remain follow-up
    work.

  - Initialize DRM registrations and their data in place, removing a
    separate allocation

  - Drop the unused flags argument from drm::Registration::new()

- Nova:

  - UAPI and Cross Module API:
    - Add NovaCoreApi and a lifetime-bound handle for auxiliary child
      drivers, carried in nova-drm registration data
    - Expose a render node and extensible INFO ioctl reporting GPU
      architecture, opaque chip ID, usable VRAM and VRAM BAR sizes,
      full and short GPU names, and GID, with size discovery and
      truncation to the userspace buffer size
    - Move BAR-size queries into GPU info, removing GETPARAM and
      changing the UAPI

  - PRAMIN access:
    - Add typed CPU access to VRAM through the movable 1 MiB BAR0
      PRAMIN window, with per-architecture support from Turing through
      Blackwell
    - Add GpuMm centralized memory manager, VramAddress type, aperture
      documentation and optional probe-time self-tests under
      CONFIG_NOVA_CORE_SELFTESTS

  - GPU virtual memory and BAR1:
    - Add VRAM buddy allocation, TLB flushing, MMU v2/v3 page-table
      types with trait-based generics (PteOps, PdeOps, DualPdeOps,
      MmuConfig) and a generic walker
    - Add maple-tree virtual address tracking and multi-page mapping,
      separating allocation from mapping execution
    - Add CPU access to mapped VRAM through BAR1 and memory-management
      self-tests; handle 64-bit BAR0 when locating BAR1

  - GPU interrupts and GSP events:
    - Add GIN (GPU Interrupt and Notification) interrupt-tree support
      with MSI-X, MSI fallback and per-architecture rearming
    - Drain GSP events between commands via a threaded SWGEN0 handler
    - Report OS errors, robust-channel notifications and unknown
      messages without aborting command waits; hold the queue lock and
      use one deadline per wait
    - Disable GSP event interrupts on persistent falcon faults or
      drain errors
    - Add a delivery/rearm self-test, interrupt-tree KUnit coverage
      and interrupt-controller documentation

  - GSP messaging fixes:
    - Queue setup RPCs before GSP-RM boot
    - Correct DMA barriers in both queue directions

  - Firmware and transport:
    - Extract a reusable radix3 buffer mapper; zero-pad page-table
      levels to whole pages to prevent booter reads of stale padding
    - Correct GSP-FMC naming and boot-flow documentation
    - Set MCTP header version 1 for FSP messages and retain the
      command queue's BAR0 reference

  - Resource handling:
    - Replace unnecessary BarUser reference counting with in-place
      ownership and borrowed access; consolidate probe initialization
    - Declare an unlimited DMA maximum segment size

- Tyr:

  - Use bitfield!() for firmware section flags and memory attributes
    that do not describe registers

  - Zero unsupported and newly added GPU-info fields, and limit the
    doorbell array to 17 entries to fit the mapped register region

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: "Danilo Krummrich" <dakr@kernel.org>
Link: https://patch.msgid.link/DM371PN4TBJT.3LY1B0EYWSI4@kernel.org
This commit is contained in:
Dave Airlie 2026-10-13 17:59:46 +10:00
commit a56b45995b
106 changed files with 10057 additions and 1840 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Documentation/gpu/nova/core/fsp.rst
View File

@ -106,6 +106,8 @@ All FSP messages share a common header format consisting of two 32-bit words:
- Bit 30: EOM (End of Message)
- Bits 29:28: Packet sequence number
- Bits 23:16: Source Endpoint ID
- Bits 15:8: Destination Endpoint ID
- Bits 3:0: MCTP header version (1)
**NVDM header** (NVIDIA Vendor Defined Message):

603
Documentation/gpu/nova/core/interrupts.rst Normal file
View File

@ -0,0 +1,603 @@
.. SPDX-License-Identifier: GPL-2.0
.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
=============================================
GPU interrupt handling: GIN and the GSP event
=============================================
This document describes how nova-core receives interrupts from the GPU on Turing
and later chipsets. It covers the GPU Interrupt and Notification unit (GIN),
which is the GPU's interrupt controller, and the GSP event, the interrupt that
nova-core services in normal operation.
Throughout, *CPU* means the CPU and the nova-core driver running on it. The GPU
also has on-chip processors that run their own firmware and receive their own
interrupts. The GSP (GPU System Processor) is one of them.
Register names are the names from the GPU hardware reference headers. The
pre-Hopper headers call the controller ``NV_CTRL`` and the Hopper-plus headers
call it ``NV_GIN``. This document calls it GIN throughout, because the tree that
nova-core services is the same on every supported chipset. OpenRM, NVIDIA's
open-source GPU kernel driver, is cited wherever nova-core follows it.
Terminology
===========
The three levels of the controller, innermost first:
leaf
One ``LEAF`` register. Each of its 32 bits is the pending bit of one
interrupt source. A Turing, Ampere, or Ada tree has 8 leaves. A Hopper or
Blackwell tree has 16.
subtree
Two consecutive leaves, summarized by one bit of ``TOP``. A subtree is the
unit of enabling at ``TOP``, and under MSI-X it is the unit of delivery:
every interrupt from one subtree arrives on one MSI-X entry.
tree
One ``TOP`` register and the leaves under it. Every PCIe function has its
own tree, and nova-core services the CPU tree of one function.
The hardware headers, OpenRM, and the Linux PCI API all use the word "vector",
each for a different number. This document gives each one its own name, and a
bare "vector" always means a GIN vector.
GIN vector
The GPU-internal interrupt source number. It addresses one bit of one leaf.
A 16-leaf tree holds vectors 0 through 511, and an 8-leaf tree holds 0
through 255. The CPU doorbell is vector 129 and the GSP event is vector
155.
MSI-X entry
An index into the device's MSI-X table. One entry serves one subtree.
PCI vector
One of the interrupts that ``pci_alloc_irq_vectors()`` allocates: an MSI-X
entry, or the single MSI message.
Linux IRQ number
What ``request_irq()`` takes, obtained from ``pci_irq_vector()`` for a PCI
vector. Linux's ``struct msix_entry`` calls this number ``.vector`` as
well.
The remaining terms, each named for the register or the specification that
defines it:
enable, disable a vector
Writes to ``LEAF_EN_SET`` and ``LEAF_EN_CLEAR``.
enable, disable a subtree
Writes to ``TOP_EN_SET`` and ``TOP_EN_CLEAR``.
serviced subtree
A subtree that nova-core enables and has a handler for.
rearm
Restoring PCI interrupt delivery after servicing an interrupt. See
"Rearming PCI interrupt delivery".
mask
Reserved for the two places where hardware and the PCI specification use
the word: the MSI-X per-entry Vector Control mask bit, which Linux
controls, and the falcon interrupt masks. It never names a GIN enable.
latched, pending
Two names for one state, a set ``LEAF`` bit. The vector's source sets the
bit whether or not the vector is enabled.
clear a vector
Write a 1 to the vector's bit in ``LEAF``. OpenRM calls the same operation
``intrClearLeafVector_HAL``.
pending bits
The plain 32-bit value read from a ``LEAF`` register.
notification
An interrupt whose only content is that something happened, such as a
posted message. Servicing a notification means reading what it announces.
The unit that raised it needs no attention. The GSP event is one.
unit
Any block that raises an interrupt. "Engine" is reserved for the blocks
that do user work: GR, CE, NVDEC, and the like.
falcon
One of the GPU's microcontrollers (see
Documentation/gpu/nova/core/falcon.rst). The GSP runs on the RISC-V core
inside its falcon. A falcon latches each of its interrupt causes and routes
it either to the host, meaning the CPU, or to its own core.
The GIN controller
==================
A GPU has many interrupt sources: the GSP, the copy engines, the graphics
engine, video decode and encode, the MMU fault path, timers, and others. GIN
records which of them are pending and raises the PCI interrupt to the CPU.
Trees
-----
GIN keeps one tree for each destination it can deliver an interrupt to. The CPU
has one tree per PCIe function, so the physical function and each virtual
function have their own. The GSP has a tree, and so do the other on-chip
processors that receive interrupts. Every tree has the same two-level layout,
and a function reaches its own tree through the per-function register aperture.
nova-core services the CPU tree of one function. A virtual function's tree
belongs to that function's driver, and a processor's tree belongs to the
firmware running on that processor.
The two-level tree
------------------
A tree is a set of ``LEAF`` registers and one ``TOP`` register.
* ``LEAF(i)`` is a 32-bit register that holds the pending bits of vectors
``32i`` through ``32i + 31``. A set bit is a pending vector.
* ``TOP`` is a 32-bit read-only register. Bit ``N`` summarizes subtree ``N``,
which is ``LEAF(2N)`` and ``LEAF(2N + 1)``. The bit is set when an enabled
vector is pending in either leaf.
A tree with L leaves has L / 2 subtrees and uses TOP bits 0 through L / 2 - 1.
The other TOP bits read 0. The leaves and subtrees that a chipset has are its
implemented leaves and subtrees, and "Per-architecture differences" gives the
counts. For an 8-leaf tree::
TOP bit 0 -> subtree 0 -> LEAF(0), LEAF(1) vectors 0..63
TOP bit 1 -> subtree 1 -> LEAF(2), LEAF(3) vectors 64..127
TOP bit 2 -> subtree 2 -> LEAF(4), LEAF(5) vectors 128..191
TOP bit 3 -> subtree 3 -> LEAF(6), LEAF(7) vectors 192..255
LEAF(4), one bit per vector, holds vectors 128..159:
bit 1 = vector 129 (CPU doorbell)
bit 27 = vector 155 (GSP event)
A vector's number fixes its place in the tree::
leaf = vector / 32
bit = vector % 32
subtree = leaf / 2
Registers
---------
nova-core defines the tree's registers in the ``irq`` module's ``regs.rs``. The
leaf registers are arrays indexed by leaf number.
* ``LEAF(i)`` reads as the pending bits of leaf ``i``. Writing a 1 to a bit
clears that vector, and a 0 leaves the bit as it was.
* ``LEAF_EN_SET(i)`` and ``LEAF_EN_CLEAR(i)`` enable and disable the vectors
of leaf ``i``, one bit per vector.
* ``TOP_EN_SET`` and ``TOP_EN_CLEAR`` enable and disable subtrees, one bit per
subtree.
* ``LEAF_TRIGGER`` takes a vector number and latches that vector, exactly as
the vector's own source would. It is write-only. The self-test uses it.
nova-core does not read ``TOP``. "Servicing the tree" says why.
Every set and clear register acts per bit: a 1 performs the action for that
bit, and a 0 leaves the bit alone. No register needs a read-modify-write.
GIN delivers a vector to the CPU only when its leaf enable bit and its
subtree's TOP enable bit are both set. The enables do not affect the latch. The
source of a disabled vector still sets its ``LEAF`` bit. ``TOP`` does not show
that bit, so reading the leaf is the only way to see it.
How a unit interrupt reaches the CPU
------------------------------------
A unit does not write a ``LEAF`` register. Each unit has an interrupt control
register that GSP firmware programs. The control register holds the unit's
vector, the GFID that identifies the PCIe function whose tree receives the
interrupt, and one enable bit per destination: the CPU, the GSP, and the other
on-chip processors. When the unit has an event::
1. The unit sends GIN an interrupt message carrying the vector, the GFID,
and the destination enables from its control register.
2. In the tree of each destination that the message selects, GIN sets bit
(vector % 32) of LEAF(vector / 32).
3. If the vector and its subtree are enabled in the CPU tree, GIN raises
the PCI interrupt.
Because firmware assigns the vectors, nova-core does not hardcode which vector
belongs to which unit, with two exceptions. The hardware headers of every
supported chipset define the GSP event as vector 155, and GSP firmware's own
interrupt table uses that definition. The CPU doorbell is vector 129. Pre-Hopper
hardware fixes that number, and GSP firmware keeps it on Hopper and later.
nova-core names both by number. A driver can fetch the full unit-to-vector
table from the GSP by RPC, and nouveau does. nova-core does not, because a
fixed vector needs no lookup.
Edge-triggered delivery
-----------------------
A ``LEAF`` bit is a latch. Its source sets it on a rising edge, and it stays set
until the CPU clears it. A source that stays high does not set the bit again.
GIN raises the PCI interrupt for a subtree when the subtree's enabled pending
state goes from low to high::
Per vector, in leaf i at bit b:
LEAF(i)[b] AND LEAF_EN(i)[b]
Per subtree N, across leaves 2N and 2N + 1:
OR of every enabled pending bit -> TOP[N]
Delivery for subtree N:
TOP[N] AND TOP_EN[N] -> rising edge -> PCI interrupt
``TOP_EN`` applies after the summary, so disabling a subtree stops delivery
without changing what ``TOP`` reports.
Three consequences:
* Code that must find every pending vector reads the leaves. A vector that
latched while disabled is not in ``TOP``.
* Writing ``TOP_EN_SET`` for a subtree with an enabled pending bit produces a
new edge. GIN delivers an interrupt for a pending bit left uncleared as soon
as its subtree is enabled again.
* A source that holds its signal high produces no new edge after the CPU
clears the leaf bit. Such a source has to re-emit its interrupt. The falcons
do that through ``INTR_RETRIGGER`` (see "Retriggering a falcon").
Delivery over PCI
=================
GIN delivers the tree's interrupts to the CPU as MSI or MSI-X, whichever Linux
grants. nova-core requests MSI-X first and falls back to MSI, and never uses
INTx.
MSI has a single message, and every subtree raises that one message, so one
Linux IRQ serves the whole tree.
MSI-X gives each subtree its own table entry, at the index equal to the subtree
number. Linux masks every entry until a driver requests its Linux IRQ number,
and a masked entry sends no message: the GPU records the interrupt in the MSI-X
pending bit array, where it stays until Linux unmasks the entry. An entry that
the driver never requests is never unmasked. A driver that enables a subtree
without requesting that subtree's entry loses every interrupt from that
subtree, with nothing reported: the leaf and TOP registers show the vector
pending and enabled while no handler runs.
The serviced-subtree invariant
------------------------------
Every subtree enabled at ``TOP`` has an allocated PCI vector with a registered
handler.
MSI satisfies this with its single message. MSI-X needs one allocated entry per
serviced subtree, and a PCI allocation cannot be sparse, so nova-core requests
entries 0 through the highest serviced subtree::
MSI-X, with subtree 2 serviced:
subtree 0 -> entry 0 allocated, no handler, stays masked
subtree 1 -> entry 1 allocated, no handler, stays masked
subtree 2 -> entry 2 handler here, and its rearm covers subtree 2
MSI, with any serviced set:
every serviced subtree -> the one allocated PCI vector, whose
handler's rearm covers the whole serviced set
An allocated entry whose subtree nova-core does not service costs nothing. The
entry stays masked, and a disabled subtree raises no interrupt.
nova-core services one subtree. The GSP event, vector 155, is in leaf 4, which
is in subtree 2. OpenRM's headers place its UVM_SHARED interrupt category in
subtree 2 on every chipset that nova-core supports. The self-test doorbell,
vector 129, is in the same leaf, and the test allocates its own vectors for it.
Rearming PCI interrupt delivery
-------------------------------
A message-signaled interrupt is delivered once per edge, and the PCI side
delivers no further interrupt until the CPU rearms it. The rearm operation
depends on the GPU family and on the interrupt type that Linux granted:
================== ===== ===========================================
Architecture Type Rearm operation
================== ===== ===========================================
Turing through Ada MSI write the configuration-mirror EOI register
Hopper and later MSI clear then set the serviced TOP enables
Any MSI-X clear then set the handler's own TOP enable
================== ===== ===========================================
The end-of-interrupt register is ``NV_XVE_CYA_2`` in the BAR0 mirror of PCI
configuration space, and the value written does not matter. The ``TOP_EN``
cycle produces a new delivery edge. The MSI forms cover every serviced subtree,
because one message serves all of them. The MSI-X form covers one subtree,
because each serviced subtree has its own entry and its own handler.
A handler rearms once per delivered interrupt, on every path, including the
path where it finds its vector not pending. A handler that skips the rearm
receives no further interrupts.
OpenRM makes the same split. It writes the configuration-space EOI for MSI on
pre-Hopper chipsets, and cycles the TOP enables of the subtrees that it
services for Hopper-plus MSI and for MSI-X.
Servicing the tree
==================
Servicing a leaf has a required order: read its pending bits, then clear them.
Clearing a leaf before reading it discards every vector latched in it, and no
register reports the loss. In nova-core, reading a leaf produces the handle that
clears it, so that the wrong order does not compile. The handle clears exactly
the bits that it read, so a vector that latched after the read stays pending.
A handler clears its bit before it services the vector. Clearing afterwards
would discard an interrupt that the source raised while the handler ran.
nova-core services the tree in two ways.
The notification path services one vector. It reads the vector's leaf, clears
only the vector's bit, and rearms. The subtree stays enabled, and a vector
pending beside it in the same leaf keeps its bit set for the code that services
that vector. The GSP event handler takes this path, and so does the self-test
handler.
The startup drain walks the whole tree, because it must clear whatever is
pending across every subtree rather than one known vector. It disables the
serviced subtrees at ``TOP``, reads and clears every implemented leaf, and
leaves the subtrees disabled for its caller to enable once the caller is ready
for deliveries. The drain reads every leaf rather than descending from ``TOP``,
because sources latch vectors during boot while those vectors are disabled, and
``TOP`` does not show them. OpenRM's stall-interrupt path reads every leaf for
the same reason.
The two paths as register operations::
Startup drain, run once during probe:
write TOP_EN_CLEAR = serviced stop new deliveries
for each implemented leaf i:
pending = read LEAF(i)
write LEAF(i) = pending clear what was read
(returns with TOP_EN still clear)
Notification, the subtree stays enabled:
pending = read LEAF(leaf) is the handler's bit set?
write LEAF(leaf) = bit clear that one bit
rearm PCI interrupt delivery
The drain clears every pending bit, including bits that nova-core never
services. An uncleared bit holds its subtree in the pending state, and enabling
that subtree again would deliver an interrupt for a vector that no handler
services.
The drain's ``TOP_EN_CLEAR`` is not a rearm, and pre-Hopper MSI rearms through
the configuration mirror, which the drain never writes. An interrupt delivered
before probe had no handler to rearm it, so the tree reset rearms explicitly
after the drain. A TOP-enable rearm leaves the serviced subtrees enabled, so
the reset then disables them again.
nova-core does not serialize access to the tree. The GSP event handler
touches only its own leaf, and the drain runs during probe, before that handler
is registered.
Per-architecture differences
============================
The tree is the same on every supported GPU except for its size, which changes
at Hopper:
=================== ====== ======== ====================
GPUs Leaves Subtrees Implemented subtrees
=================== ====== ======== ====================
Turing, Ampere, Ada 8 4 ``0x0f``
Hopper, Blackwell 16 8 ``0xff``
=================== ====== ======== ====================
The interrupt HAL provides the leaf count, and the subtree count and the
implemented-subtree set derive from it. A subtree that the chipset does not
implement has no TOP bit, so building a tree that services one fails with
``EINVAL``. Vectors 129 and 155 are in the 8-leaf tree, so every supported
chipset has them.
OpenRM's headers assign every interrupt category of a 16-leaf tree to leaves 0
through 11. The drain reads all 16, because a vector can be latched in any
implemented leaf.
The HAL performs the rearm as well (see "Rearming PCI interrupt delivery"). Two
falcon properties also differ by family, and the falcon HAL carries them:
Turing falcons have no ``INTR_RETRIGGER``, and the RISC-V routing registers
moved at GA102 (see "Retriggering a falcon").
The GSP event
=============
When the GSP has output for the CPU, it writes messages into the GSP-to-CPU
queue in shared memory and raises SWGEN0, one of the software-generated
interrupt causes of the GSP falcon. SWGEN0 is routed to the host, at vector
155, leaf 4 bit 27, in subtree 2.
The queue carries notifications (log records, error records, lifecycle events)
and command replies. A thread waiting for a reply reads the queue itself, so
the interrupt is only the trigger to drain the queue (see "Draining the
GSP-to-CPU queue").
The falcon latches every cause that it raises, SWGEN0 among them, in its
``IRQSTAT`` register. The handler services the host-routed causes and clears
their latches, and then it writes ``INTR_RETRIGGER`` so that the falcon re-emits
any cause that latched in the meantime. "Retriggering a falcon" has the details.
Draining the queue takes the command-queue mutex and walks shared memory, so it
cannot run in hard interrupt context. nova-core registers a threaded handler,
under the name ``nova-core`` in ``/proc/interrupts``. The top half runs in hard
interrupt context and reads and writes only registers, and it wakes the IRQ
thread to drain the queue::
GSP writes messages into the GSP-to-CPU queue
GSP raises SWGEN0
GIN sets bit 27 of LEAF(4), and subtree 2 becomes pending
PCI interrupt -> Linux IRQ -> top half, in hard interrupt context:
read LEAF(4), and if bit 27 is clear, rearm and return
clear bit 27 (the subtree stays enabled)
read the falcon causes routed to the host, clearing SWGEN0 if set
for every other host cause: log it, clear its latch, and read the
host causes back
if the clear ended all of them: retrigger the falcon
otherwise: disable vector 155 at its leaf and skip the retrigger
rearm PCI interrupt delivery
wake the IRQ thread if SWGEN0 was set
IRQ thread, which may sleep:
take the command-queue mutex and drain the GSP-to-CPU queue
A halt and a posted message can be pending together, so the top half services
every cause that the status reports.
A drain fails when a message's framing or checksum is bad (see "Draining the
GSP-to-CPU queue"). The message stays at the queue head, so every later drain
would fail the same way, and the IRQ thread disables vector 155 and logs the
failure, which leaves the queue unserviced until the device is reset.
The handler, the self-test, and the rest of the driver read BAR0 through one
shared mapping. nova-core unregisters an interrupt handler when the device
unbinds, so a handler runs only while the mapping exists.
Retriggering a falcon
---------------------
A falcon signals the tree when its set of host-routed causes goes from empty to
non-empty. A cause left latched keeps the set non-empty, so no later cause
signals the tree, and the vector is lost. For a cause that stays latched, the
handler can clear the tree leaf first or the falcon latch first, and the loss
is the same.
``IRQSTAT`` latches every cause in the falcon, including the causes routed to
the falcon's own RISC-V core and owned by the firmware running on it. A host
handler owns only the causes that ``PRISCV_RISCV_IRQMASK`` and
``PRISCV_RISCV_IRQDEST`` both select, so it intersects ``IRQSTAT`` with both
before it reads or clears a cause. OpenRM computes the same intersection in
``kflcnRiscvReadIntrStatus``. GA100 keeps the Turing offsets of the two routing
registers and GA102 moves them, so the offsets change at GA102 rather than at
the Ampere boundary. The handler masks no cause: ``PRISCV_RISCV_IRQMASK`` is
read-only to the host, and ``FALCON_IRQMASK`` has no effect on host routing on
a RISC-V falcon.
``INTR_RETRIGGER`` makes the falcon re-emit its host-routed causes into the
tree, which supplies the transition that clearing the leaf lost. The handler
writes ``INTR_RETRIGGER`` only on a path where it ended every cause that it
read, because a re-emitted cause that stays set arrives again at once and
on every pass after that.
``IRQSCLR`` ends a latch and does not end the source behind it, so a cause
driven from outside the falcon stays set after the write. On Blackwell the
fault-containment and ECC causes are driven that way: they appear in
``IRQSTAT`` but come from ``PRISCV_RISCV_FAULT_CONTAINMENT_SRCSTAT`` and
``PGSP_ECC_INTR_STATUS``, and only a device reset ends them. So the handler
clears the latch of every host cause other than SWGEN0, reads the host causes
back, and retriggers only when the read-back is empty. When a cause is still
set, the handler disables vector 155 instead and reports that the device needs
a reset. Disabling loses no notification: the cause that is still set holds the
host-routed set non-empty, so the falcon would signal nothing further either
way. OpenRM makes the same choice, and ``kgspService_TU102`` skips
``kflcnIntrRetrigger`` once it has recorded a fatal error.
The handler cannot distinguish a fault cause that arrives after the clear from
one that the clear failed to end, so it disables the vector in that case too.
Both mean that the GSP has faulted.
Turing falcons have no ``INTR_RETRIGGER``, so a Turing handler cannot re-create
a transition that it has lost. It must leave no host cause latched: it reads the
host-routed status once and takes every cause that the status reports, rather
than stopping at the first one that it recognizes. One window stays open. A
cause that arrives after the handler has read the status is not in the value
that the handler clears, so it stays latched after the leaf has been cleared,
and no later cause from that falcon signals the tree. OpenRM has the same window
on Turing, where ``kflcnIntrRetrigger`` does nothing.
Enabling the GSP event
----------------------
SWGEN0 is a latch, and the GSP drives no new edge into the tree while it stays
set. nova-core's GSP boot code consumes the GSP's notifications by polling the
queue, which leaves the latch set and leaves pending bits in the tree. The
handoff from polling to interrupts has a required order::
disable every implemented vector drop enables left by boot, or by a
driver that ran before this one
drain the tree clear stale pending bits
rearm PCI interrupt delivery required under pre-Hopper MSI, which
no later step rearms
disable subtree 2 at TOP a TOP-enable rearm enabled it again
clear the SWGEN0 latch so the next message makes an edge
register the threaded handler no delivery can reach it yet
enable vector 155 at LEAF(4) the subtree is still disabled
enable subtree 2 at TOP deliveries become possible here
drain the GSP-to-CPU queue messages posted before the clear
The first four steps are the tree reset. nova-core runs them, and clears the
latch, before it registers the handler. Registering unmasks the PCI interrupt,
and GIN would then deliver a vector that boot left enabled to a handler that
services one vector and has no way to service any other. OpenRM clears every
leaf enable at the same point for the same reason.
nova-core clears the latch after the drain. If nova-core cleared the latch
first, a message posted before the drain could set it again, along with bit 27
of LEAF(4). The drain would then clear the leaf bit while the latch stays set,
and no later message would signal the tree. Clearing after the drain can
instead leave the leaf bit pending with the latch already clear. Enabling the
subtree then delivers one interrupt whose ``IRQSTAT`` reads zero. The top half
clears the leaf bit, rearms, and does not wake the IRQ thread, and the queue
drain that follows reads the message.
Clearing the latch makes the first interrupt possible. A message that the GSP
posted before that clear produces no interrupt, so the sequence ends by
draining the queue.
The subtree is enabled at ``TOP`` only once the handler is registered. A
TOP-enable rearm sets the enables that it cycles, so the reset disables the
serviced subtrees again after its rearm, and the tree reaches the registration
with subtree 2 disabled under every rearm method. On teardown nova-core disables
the vector at its leaf, so that no vector in the subtree can be delivered, then
calls ``free_irq()``, and disables the subtree last. Disabling the subtree
earlier would let a handler still in flight enable it again through the
``TOP_EN`` cycle of its rearm, which would leave the subtree enabled with no
handler registered. The driver's ``Gpu`` object, which owns the tree, holds the
subtree enable, and a handler's registration holds only the enable of its own
vector, so that tearing down one handler does not disable a subtree that another
handler shares. nova-core tears down the registration before it frees the queue
that the handler drains and the falcon that the handler reads, and before it
unloads the GSP.
Draining the GSP-to-CPU queue
-----------------------------
The queue carries command replies and unsolicited events, and a message's
function code says which it is.
* A function code that matches the awaited reply: the message is decoded and
returned to the caller that sent the command.
* Anything else is an event. An OS error record and a robust-channel record
are logged at error level, and an unrecognized function code at warning
level. The other known events (GSP logs, libos prints, assertion records,
lifecycle notices) need no action and get no line of their own, because the
receive trace at debug level already records every message's arrival with
its sequence number, function code, and length.
The sequence number takes no part in the match, because the GSP does not echo
the command's sequence number on every reply. On r570 the reply to
``UnloadingGuestDriver`` carries sequence 0.
The read pointer advances past every message that passes framing and checksum
validation, whether it matched or was an event. A matched message that is too
short to decode is the exception, and it stays at the queue head.
A message's length is inside the region that the checksum covers, so once the
framing or the checksum fails there is no trustworthy length with which to skip
the message. Such a message also stays at the queue head, and every later
receive fails on it with ``EIO`` until the device is reset.
The polling path and the IRQ thread both read the queue under the command-queue
mutex. Replies and events share one queue and one read pointer, so one lock is
held across the whole drain. A thread waiting for a reply logs each event that
arrives before the reply and keeps waiting. One deadline of 5 seconds applies to
the whole wait, rather than a fresh timeout after each message, and the thread
holds the mutex for the whole wait, so that no other caller consumes the message
that it waits for.
With one lock, a drain waits for an in-flight command's receive to finish or
time out. For log and error records that delay does not matter.

128
Documentation/gpu/nova/core/pramin.rst Normal file
View File

@ -0,0 +1,128 @@
.. SPDX-License-Identifier: GPL-2.0
=========================
PRAMIN aperture mechanism
=========================
.. note::
The following description is approximate and current as of the Ampere
family. It may change for future generations and is intended to assist in
understanding the driver code.
Introduction
============
PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video
RAM (VRAM) before the GPU's Memory Management Unit (MMU) and page tables are
initialized. This 1 MiB sliding window, located at a fixed offset within BAR0,
is essential for setting up page tables and other critical GPU data structures
without relying on the GPU's MMU.
Architecture Overview
=====================
The PRAMIN aperture mechanism is logically implemented by the GPU's PBUS (PCIe
Bus Controller Unit) and provides a CPU-accessible window into VRAM through the
PCIe interface::
+-----------------+ PCIe +------------------------------+
| CPU |<----------->| GPU |
+-----------------+ | |
| +----------------------+ |
| | PBUS | |
| | (Bus Controller) | |
| | | |
| | +--------------+ <------------ [1]
| | | PRAMIN | | |
| | | Window | | |
| | | (1 MiB) | | |
| | +--------------+ | |
| | | | |
| +---------|------------+ |
| | |
| v |
| +----------------------+ <------- [2]
| | VRAM | |
| | (Several GiB) | |
| | | |
| | FB[0x0000000000] | |
| | ... | |
| | FB[0xFFFFFFFFFF] | |
| +----------------------+ |
+------------------------------+
[1] Window starts at BAR0 + 0x700000.
[2] Program PRAMIN to any 64 KiB-aligned VRAM boundary.
PBUS is responsible for, among other things, handling MMIO
accesses to the BAR registers.
PRAMIN Window Operation
=======================
The PRAMIN window provides a 1 MiB sliding aperture that can be repositioned
over the entire VRAM address space using the ``NV_PBUS_BAR0_WINDOW`` register.
Window Control Mechanism
-------------------------
::
NV_PBUS_BAR0_WINDOW Register (0x1700):
+-------+--------+--------------------------------------+
| 31:26 | 25:24 | 23:0 |
| RSVD | TARGET | BASE_ADDR |
| | | (bits 39:16 of VRAM address) |
+-------+--------+--------------------------------------+
The 24-bit BASE_ADDR field encodes bits [39:16] of the target VRAM address,
providing 40-bit (1 TiB) address space coverage with 64 KiB alignment.
TARGET field (bits 25:24):
- 0x0: VRAM (Video Memory)
- 0x1: Reserved (unused)
- 0x2: SYS_MEM_COH (Coherent System Memory)
- 0x3: SYS_MEM_NONCOH (Non-coherent System Memory)
.. note::
Nova only uses TARGET=VRAM (0x0) for video memory access. The SYS_MEM
target values are documented here for hardware completeness but are
not used by the driver.
64 KiB Alignment Requirement
----------------------------
The PRAMIN window must be aligned to 64 KiB boundaries in VRAM. This is enforced
by the ``BASE_ADDR`` field representing bits [39:16] of the target address::
VRAM Address Calculation:
actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
Where:
- BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
- pramin_offset: 20-bit offset within the PRAMIN window [0x00000-0xFFFFF]
Example Window Positioning:
+---------------------------------------------------------+
| VRAM Space |
| |
| 0x0000000000 +-----------------+ <-- 64 KiB aligned |
| | PRAMIN Window | |
| | (1 MiB) | |
| 0x00000FFFFF +-----------------+ |
| |
| | ^ |
| | | Window can slide |
| v | to any 64 KiB-aligned boundary |
| |
| 0x0123400000 +-----------------+ <-- 64 KiB aligned |
| | PRAMIN Window | |
| | (1 MiB) | |
| 0x01234FFFFF +-----------------+ |
| |
| ... |
| |
| 0xFFFFF00000 +-----------------+ <-- 64 KiB aligned |
| | PRAMIN Window | |
| | (1 MiB) | |
| 0xFFFFFFFFFF +-----------------+ |
+---------------------------------------------------------+

2
Documentation/gpu/nova/core/todo.rst
View File

@ -33,7 +33,7 @@ A good example from nova-core would be the ``Chipset`` enum type, which defines
the value ``AD102``. When probing the GPU the value ``0x192`` can be read from a
certain register indication the chipset AD102. Hence, the enum value ``AD102``
should be derived from the number ``0x192``. Currently, nova-core uses a custom
implementation (``Chipset::from_u32`` for this.
implementation (``Chipset::from_u32``) for this.
Instead, it would be desirable to have something like the ``FromPrimitive``
trait [1] from the num crate.

2
Documentation/gpu/nova/index.rst
View File

@ -34,3 +34,5 @@ vGPU manager VFIO driver and the nova-drm driver.
core/fwsec
core/falcon
core/tlv
core/pramin
core/interrupts

7
MAINTAINERS
View File

@ -2333,6 +2333,7 @@ ARM MALI TYR DRM DRIVER
M: Daniel Almeida <daniel.almeida@collabora.com>
M: Alice Ryhl <aliceryhl@google.com>
L: dri-devel@lists.freedesktop.org
L: rust-for-linux@vger.kernel.org
S: Supported
W: https://rust-for-linux.com/tyr-gpu-driver
W: https://drm.pages.freedesktop.org/maintainer-tools/drm-rust.html
@ -7497,6 +7498,7 @@ F: rust/kernel/io.rs
F: rust/kernel/io/
F: rust/kernel/irq.rs
F: rust/kernel/irq/
F: rust/macros/io/
DEVICE RESOURCE MANAGEMENT HELPERS
M: Hans de Goede <hansg@kernel.org>
@ -7683,6 +7685,7 @@ F: fs/dlm/
DMA BUFFER SHARING FRAMEWORK
M: Sumit Semwal <sumit.semwal@linaro.org>
M: Christian König <christian.koenig@amd.com>
M: Philipp Stanner <phasta@kernel.org>
L: linux-media@vger.kernel.org
L: dri-devel@lists.freedesktop.org
L: linaro-mm-sig@lists.linaro.org (moderated for non-subscribers)
@ -7696,6 +7699,8 @@ F: include/linux/dma-buf.h
F: include/linux/dma-buf/
F: include/linux/dma-resv.h
F: rust/helpers/dma-resv.c
F: rust/helpers/dma_fence.c
F: rust/kernel/dma_buf/
K: \bdma_(?:buf|fence|resv)\b
DMA GENERIC OFFLOAD ENGINE SUBSYSTEM
@ -8707,7 +8712,9 @@ T: git https://gitlab.freedesktop.org/drm/rust/kernel.git
F: drivers/gpu/drm/nova/
F: drivers/gpu/drm/tyr/
F: drivers/gpu/nova-core/
F: rust/helpers/dma_fence.c
F: rust/helpers/gpu.c
F: rust/kernel/dma_buf/
F: rust/kernel/drm/
F: rust/kernel/gpu.rs
F: rust/kernel/gpu/

37
drivers/gpu/drm/nova/driver.rs
View File

@ -18,12 +18,23 @@
use crate::file::File;
use crate::gem::NovaObject;
use nova_core::api::{
NovaCoreApi,
NovaCoreApiHandle, //
};
pub(crate) struct NovaDriver;
#[pin_data]
pub(crate) struct Nova<'bound> {
#[expect(unused)]
drm: ARef<drm::Device<NovaDriver>>,
_reg: drm::Registration<'bound, NovaDriver>,
#[pin]
reg: drm::Registration<'bound, NovaDriver>,
}
/// DRM registration data, accessible from ioctl handlers via the registration guard.
pub(crate) struct DrmRegData<'bound> {
pub(crate) api: NovaCoreApiHandle<'bound>,
}
/// Convienence type alias for the DRM device type for this driver
@ -58,14 +69,19 @@ fn probe<'bound>(
adev: &'bound auxiliary::Device<Core<'_>>,
_info: &'bound Self::IdInfo,
) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
let drm = drm::UnregisteredDevice::<Self>::new(adev, Ok(()))?;
// SAFETY: `reg` is stored in `Nova` and dropped when the driver is unbound; it is
// never forgotten.
let reg = unsafe { drm::Registration::new(adev.as_ref(), drm, (), 0)? };
try_pin_init!(Self::Data {
reg <- {
let drm = drm::UnregisteredDevice::<Self>::new(adev, Ok(()))?;
Ok(Nova {
let reg_data = DrmRegData {
api: NovaCoreApi::of(adev)?,
};
// SAFETY: `reg` is stored in `Self::Data` and dropped when the driver is unbound;
// it is never forgotten.
unsafe { drm::Registration::new(adev.as_ref(), drm, reg_data) }
},
drm: reg.device().into(),
_reg: reg,
})
}
}
@ -73,15 +89,16 @@ fn probe<'bound>(
#[vtable]
impl drm::Driver for NovaDriver {
type Data = ();
type RegistrationData<'a> = ();
type RegistrationData<'a> = DrmRegData<'a>;
type File = File;
type Object = gem::Object<NovaObject>;
type ParentDevice<Ctx: DeviceContext> = auxiliary::Device<Ctx>;
const INFO: drm::DriverInfo = INFO;
const FEAT_RENDER: bool = true;
kernel::declare_drm_ioctls! {
(NOVA_GETPARAM, drm_nova_getparam, ioctl::RENDER_ALLOW, File::get_param),
(NOVA_INFO, drm_nova_info, ioctl::RENDER_ALLOW, File::info),
(NOVA_GEM_CREATE, drm_nova_gem_create, ioctl::AUTH | ioctl::RENDER_ALLOW, File::gem_create),
(NOVA_GEM_INFO, drm_nova_gem_info, ioctl::AUTH | ioctl::RENDER_ALLOW, File::gem_info),
}

129
drivers/gpu/drm/nova/file.rs
View File

@ -1,23 +1,105 @@
// SPDX-License-Identifier: GPL-2.0
use crate::driver::{NovaDevice, NovaDriver};
use crate::driver::{
DrmRegData,
NovaDevice,
NovaDriver, //
};
use crate::gem::NovaObject;
use kernel::{
alloc::flags::*,
auxiliary,
device::Bound,
drm::{
self,
gem::BaseObject,
Registered, //
},
pci,
num::casts::{
arch::IntoSafeCastArch,
IntoSafeCast, //
},
prelude::*,
transmute::AsBytes,
uaccess::UserSlice,
uapi,
};
pub(crate) struct File;
/// GPU information returned to userspace.
#[repr(transparent)]
struct GpuInfo(uapi::drm_nova_info_gpu);
/// Copies `name` into the zero initialised, fixed size uAPI buffer `dst`, keeping it
/// NUL-terminated.
///
/// Fails with [`ENAMETOOLONG`] if `name` does not fit in `dst` with room for the terminator.
fn copy_name(dst: &mut [u8], name: &str) -> Result {
let bytes = name.as_bytes();
if bytes.len() >= dst.len() {
return Err(ENAMETOOLONG);
}
dst[..bytes.len()].copy_from_slice(bytes);
Ok(())
}
impl GpuInfo {
/// Collects the GPU information reported to userspace.
///
/// This is fallible so that unexpected GSP behaviour, such as a malformed name string, is
/// reported to userspace instead of being silently replaced with a made up value. When adding
/// a field, take care that a value the GSP simply does not provide, for example because the
/// firmware predates it, does not cause a failure. Such fields must fall back to zero or
/// another documented default instead.
fn new(reg_data: &DrmRegData<'_>) -> Result<Self> {
let spec = reg_data.api.with(|api| api.get_ref().spec());
let gsp_static_info = reg_data.api.with(|api| api.get_ref().gsp_static_info());
let mut info = uapi::drm_nova_info_gpu {
architecture: spec.chipset.arch().into(),
chipid: spec.chipset.into(),
vram_size: gsp_static_info.vram_size(),
vram_bar_size: reg_data.api.with(|api| api.bar1_size())?,
gpu_gid: gsp_static_info.gpu_gid,
..pin_init::zeroed()
};
copy_name(
&mut info.gpu_name,
gsp_static_info.gpu_name().map_err(|_| EINVAL)?,
)?;
copy_name(
&mut info.gpu_short_name,
gsp_static_info.gpu_short_name().map_err(|_| EINVAL)?,
)?;
Ok(Self(info))
}
}
// SAFETY: `GpuInfo` has no interior mutability, and there are no uninitialized bytes.
unsafe impl AsBytes for GpuInfo {}
fn write_info<T: AsBytes>(info: &mut uapi::drm_nova_info, value: T) -> Result {
// A NULL data pointer requests the size of the information structure.
if info.data == 0 {
info.size = size_of::<T>().try_into()?;
return Ok(());
}
let mut writer = UserSlice::new(
UserPtr::from_addr(info.data.into_safe_cast_arch()),
info.size.into_safe_cast(),
)
.writer();
info.size = writer.write_truncated(&value)?.try_into()?;
Ok(())
}
impl drm::file::DriverFile for File {
type Driver = NovaDriver;
@ -27,30 +109,10 @@ fn open(_dev: &NovaDevice) -> Result<Pin<KBox<Self>>> {
}
impl File {
/// IOCTL: get_param: Query GPU / driver metadata.
pub(crate) fn get_param(
dev: &NovaDevice<Registered>,
_reg_data: &(),
getparam: &mut uapi::drm_nova_getparam,
_file: &drm::File<File>,
) -> Result<u32> {
let adev: &auxiliary::Device<Bound> = dev.as_ref();
let pdev: &pci::Device<Bound> = adev.parent().try_into()?;
let value = match getparam.param as u32 {
uapi::NOVA_GETPARAM_VRAM_BAR_SIZE => pdev.resource_len(1)?,
_ => return Err(EINVAL),
};
getparam.value = Into::<u64>::into(value);
Ok(0)
}
/// IOCTL: gem_create: Create a new DRM GEM object.
pub(crate) fn gem_create(
dev: &NovaDevice<Registered>,
_reg_data: &(),
_reg_data: &DrmRegData<'_>,
req: &mut uapi::drm_nova_gem_create,
file: &drm::File<File>,
) -> Result<u32> {
@ -64,7 +126,7 @@ pub(crate) fn gem_create(
/// IOCTL: gem_info: Query GEM metadata.
pub(crate) fn gem_info(
_dev: &NovaDevice<Registered>,
_reg_data: &(),
_reg_data: &DrmRegData<'_>,
req: &mut uapi::drm_nova_gem_info,
file: &drm::File<File>,
) -> Result<u32> {
@ -74,4 +136,19 @@ pub(crate) fn gem_info(
Ok(0)
}
/// IOCTL: info: Query device information.
pub(crate) fn info(
_dev: &NovaDevice<Registered>,
reg_data: &DrmRegData<'_>,
info: &mut uapi::drm_nova_info,
_file: &drm::File<File>,
) -> Result<u32> {
match info.id {
uapi::DRM_NOVA_INFO_GPU => write_info(info, GpuInfo::new(reg_data)?)?,
_ => return Err(EINVAL),
}
Ok(0)
}
}

114
drivers/gpu/drm/tyr/driver.rs
View File

@ -46,6 +46,7 @@
};
pub(crate) type IoMem<'a> = kernel::io::mem::IoMem<'a, SZ_2M>;
pub(crate) type TyrRegisters = kernel::io::Region<SZ_2M>;
pub(crate) struct TyrDrmDriver;
@ -56,6 +57,7 @@
#[pin_data(PinnedDrop)]
pub(crate) struct TyrPlatformDriverData<'bound> {
#[pin]
_reg: drm::Registration<'bound, TyrDrmDriver>,
}
@ -116,73 +118,75 @@ fn probe<'bound>(
pdev: &'bound platform::Device<Core<'_>>,
_info: Option<&'bound Self::IdInfo>,
) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
let core_clk = Clk::get(pdev.as_ref(), Some(c"core"))?;
let stacks_clk = OptionalClk::get(pdev.as_ref(), Some(c"stacks"))?;
let coregroup_clk = OptionalClk::get(pdev.as_ref(), Some(c"coregroup"))?;
pin_init::pin_init_scope(move || {
let core_clk = Clk::get(pdev.as_ref(), Some(c"core"))?;
let stacks_clk = OptionalClk::get(pdev.as_ref(), Some(c"stacks"))?;
let coregroup_clk = OptionalClk::get(pdev.as_ref(), Some(c"coregroup"))?;
core_clk.prepare_enable()?;
stacks_clk.prepare_enable()?;
coregroup_clk.prepare_enable()?;
core_clk.prepare_enable()?;
stacks_clk.prepare_enable()?;
coregroup_clk.prepare_enable()?;
let mali_regulator = Regulator::<regulator::Enabled>::get(pdev.as_ref(), c"mali")?;
let sram_regulator = Regulator::<regulator::Enabled>::get(pdev.as_ref(), c"sram")?;
let mali_regulator = Regulator::<regulator::Enabled>::get(pdev.as_ref(), c"mali")?;
let sram_regulator = Regulator::<regulator::Enabled>::get(pdev.as_ref(), c"sram")?;
let request = pdev.io_request_by_index(0).ok_or(ENODEV)?;
let request = pdev.io_request_by_index(0).ok_or(ENODEV)?;
let iomem = Arc::new(request.iomap_sized::<SZ_2M>()?, GFP_KERNEL)?;
let iomem = Arc::new(request.iomap_sized::<SZ_2M>()?, GFP_KERNEL)?;
issue_soft_reset(pdev.as_ref(), &iomem)?;
gpu::l2_power_on(pdev.as_ref(), &iomem)?;
issue_soft_reset(pdev.as_ref(), &iomem)?;
gpu::l2_power_on(pdev.as_ref(), &iomem)?;
let gpu_info = GpuInfo::new(&iomem);
gpu_info.log(pdev.as_ref());
let gpu_info = GpuInfo::new(&iomem);
gpu_info.log(pdev.as_ref());
let pa_bits = MMU_FEATURES::from_raw(gpu_info.mmu_features)
.pa_bits()
.get();
// SAFETY: No concurrent DMA allocations or mappings can be made because
// the device is still being probed and therefore isn't being used by
// other threads of execution.
unsafe { pdev.dma_set_mask_and_coherent(DmaMask::try_new(pa_bits)?)? };
let pa_bits = MMU_FEATURES::from_raw(gpu_info.mmu_features)
.pa_bits()
.get();
// SAFETY: No concurrent DMA allocations or mappings can be made because
// the device is still being probed and therefore isn't being used by
// other threads of execution.
unsafe { pdev.dma_set_mask_and_coherent(DmaMask::try_new(pa_bits)?)? };
let unreg_dev = drm::UnregisteredDevice::<TyrDrmDriver>::new(pdev, Ok(()))?;
let unreg_dev = drm::UnregisteredDevice::<TyrDrmDriver>::new(pdev, Ok(()))?;
let mmu = Mmu::new(pdev.as_ref(), iomem.as_arc_borrow(), &gpu_info)?;
let mmu = Mmu::new(pdev.as_ref(), iomem.as_arc_borrow(), &gpu_info)?;
let firmware = Firmware::new(
pdev.as_ref(),
iomem.clone(),
&unreg_dev,
mmu.as_arc_borrow(),
&gpu_info,
)?;
let firmware = Firmware::new(
pdev.as_ref(),
iomem.clone(),
&unreg_dev,
mmu.as_arc_borrow(),
&gpu_info,
)?;
firmware.boot()?;
firmware.boot()?;
let reg_data = pin_init!(TyrDrmRegistrationData {
pdev,
fw: firmware,
clks <- new_mutex!(Clocks {
core: core_clk,
stacks: stacks_clk,
coregroup: coregroup_clk,
}),
regulators <- new_mutex!(Regulators {
_mali: mali_regulator,
_sram: sram_regulator,
}),
iomem,
gpu_info,
});
// SAFETY: `reg` is stored in `TyrPlatformDriverData` and dropped when the driver is
// unbound; it is never forgotten.
let reg = unsafe { drm::Registration::new(pdev.as_ref(), unreg_dev, reg_data, 0)? };
let driver = TyrPlatformDriverData { _reg: reg };
dev_dbg!(pdev, "Tyr initialized correctly.");
Ok(driver)
Ok(try_pin_init!(TyrPlatformDriverData {
// SAFETY: `_reg` is stored in `TyrPlatformDriverData` and dropped when the
// driver is unbound; it is never forgotten.
_reg <- unsafe { drm::Registration::new(
pdev.as_ref(),
unreg_dev,
pin_init!(TyrDrmRegistrationData {
pdev,
fw: firmware,
clks <- new_mutex!(Clocks {
core: core_clk,
stacks: stacks_clk,
coregroup: coregroup_clk,
}),
regulators <- new_mutex!(Regulators {
_mali: mali_regulator,
_sram: sram_regulator,
}),
iomem,
gpu_info,
}),
)},
_: { dev_dbg!(pdev, "Tyr initialized correctly.") },
}))
})
}
}

7
drivers/gpu/drm/tyr/fw.rs
View File

@ -14,6 +14,7 @@
//! [`Section`]: crate::fw::Section
use kernel::{
bitfield,
device::{
Bound,
Device, //
@ -27,7 +28,6 @@
},
num::Bounded,
prelude::*,
register,
str::CString,
sync::{
Arc,
@ -100,9 +100,8 @@ fn from(value: CacheMode) -> Self {
}
}
register! {
#[allow(non_upper_case_globals)]
pub(super) SectionFlags(u32) @ 0x0 {
bitfield! {
pub(super) struct SectionFlags(u32) {
0:0 read => bool;
1:1 write => bool;
2:2 exec => bool;

6
drivers/gpu/drm/tyr/gpu.rs
View File

@ -74,10 +74,8 @@ pub(crate) fn new(io: &IoMem<'_>) -> Self {
io.read(TILER_PRESENT_HI).into_raw(),
),
core_features: io.read(CORE_FEATURES).into_raw(),
// Padding must be zero.
pad: 0,
//GPU_FEATURES register is not available; it was introduced in arch 11.x.
gpu_features: 0,
// Zero unsupported and newly added UAPI fields.
..pin_init::zeroed()
})
}

57
drivers/gpu/drm/tyr/regs.rs
View File

@ -57,7 +57,11 @@ pub(crate) mod gpu_control {
uapi, //
};
use crate::driver::TyrRegisters;
register! {
base: TyrRegisters;
/// GPU identification register.
pub(crate) GPU_ID(u32) @ 0x0 {
/// Status of the GPU release.
@ -315,6 +319,8 @@ fn from(mode: FlushMode) -> Self {
}
register! {
base: TyrRegisters;
/// GPU command register.
///
/// Use the constructor methods to create commands:
@ -380,6 +386,8 @@ pub(crate) fn clear_fault() -> Self {
}
register! {
base: TyrRegisters;
/// GPU status register. Read only.
pub(crate) GPU_STATUS(u32) @ 0x34 {
/// GPU active, a 1-bit boolean flag.
@ -463,6 +471,8 @@ fn from(access: AccessType) -> Self {
}
register! {
base: TyrRegisters;
/// GPU fault status register. Read only.
pub(crate) GPU_FAULTSTATUS(u32) @ 0x3c {
/// Exception type.
@ -768,6 +778,8 @@ fn from(mode: CoherencyMode) -> Self {
}
register! {
base: TyrRegisters;
/// Coherency enable. An index of which coherency protocols should be used.
/// This register only selects the protocol for coherency messages on the
/// interconnect. This is not to enable or disable coherency controlled by MMU.
@ -808,6 +820,8 @@ fn from(mode: McuControlMode) -> Self {
}
register! {
base: TyrRegisters;
/// MCU control.
pub(crate) MCU_CONTROL(u32) @ 0x700 {
/// Request MCU state change.
@ -849,6 +863,8 @@ fn from(status: McuStatus) -> Self {
}
register! {
base: TyrRegisters;
/// MCU status. Read only.
pub(crate) MCU_STATUS(u32) @ 0x704 {
/// Read current state of MCU.
@ -862,7 +878,11 @@ fn from(status: McuStatus) -> Self {
pub(crate) mod job_control {
use kernel::register;
use crate::driver::TyrRegisters;
register! {
base: TyrRegisters;
/// Raw status of job interrupts.
///
/// Write to this register to trigger these interrupts.
@ -912,7 +932,11 @@ pub(crate) mod job_control {
pub(crate) mod mmu_control {
use kernel::register;
use crate::driver::TyrRegisters;
register! {
base: TyrRegisters;
/// IRQ sources raw status.
///
/// This register contains the raw unmasked interrupt sources for MMU status and exception
@ -962,13 +986,15 @@ pub(crate) mod mmu_control {
/// This array contains 16 instances of the MMU_AS_CONTROL register page.
pub(crate) mod mmu_as_control {
use kernel::{
bitfield,
num::Bounded,
prelude::*,
register, //
};
use pin_init::Zeroable;
use crate::driver::TyrRegisters;
/// Maximum number of hardware address space slots.
/// The actual number of slots available is usually lower.
pub(crate) const MAX_AS: usize = 16;
@ -977,6 +1003,8 @@ pub(crate) mod mmu_as_control {
const STRIDE: usize = 0x40;
register! {
base: TyrRegisters;
/// Translation table base address. A 64-bit pointer.
///
/// This field contains the address of the top level of a translation table structure.
@ -1103,12 +1131,12 @@ fn from(val: MemoryType) -> Self {
}
}
register! {
bitfield! {
/// Stage 1 memory attributes (8-bit bitfield).
///
/// This is not an actual register, but a bitfield definition used by the MEMATTR
/// register. Each of the 8 bytes in MEMATTR follows this layout.
MMU_MEMATTR_STAGE1(u8) @ 0x0 {
/// Each of the 8 bytes in MEMATTR follows this layout.
#[allow(non_camel_case_types)]
struct MMU_MEMATTR_STAGE1(u8) {
/// Inner cache write allocation policy.
0:0 alloc_w => bool;
/// Inner cache read allocation policy.
@ -1137,6 +1165,8 @@ fn from(val: MMU_MEMATTR_STAGE1) -> Self {
}
register! {
base: TyrRegisters;
/// Memory attributes.
///
/// Each address space can configure up to 8 different memory attribute profiles.
@ -1292,6 +1322,8 @@ pub(crate) fn from_mair(mair: u64) -> Self {
}
register! {
base: TyrRegisters;
/// Lock region address for each address space.
pub(crate) LOCKADDR(u64)[MAX_AS, stride = STRIDE] @ 0x2410 {
/// Lock region size.
@ -1353,6 +1385,8 @@ fn from(cmd: MmuCommand) -> Self {
}
register! {
base: TyrRegisters;
/// MMU command register for each address space. Write only.
pub(crate) COMMAND(u32)[MAX_AS, stride = STRIDE] @ 0x2418 {
7:0 command ?=> MmuCommand;
@ -1480,6 +1514,8 @@ fn from(access: MmuAccessType) -> Self {
}
register! {
base: TyrRegisters;
/// Fault status register for each address space. Read only.
pub(crate) FAULTSTATUS(u32)[MAX_AS, stride = STRIDE] @ 0x241c {
/// Exception type.
@ -1705,6 +1741,8 @@ fn from(sh: PtwShareability) -> Self {
}
register! {
base: TyrRegisters;
/// Translation configuration and control.
pub(crate) TRANSCFG(u64)[MAX_AS, stride = STRIDE] @ 0x2430 {
/// Address space mode.
@ -1760,8 +1798,13 @@ fn from(sh: PtwShareability) -> Self {
pub(crate) mod doorbell_block {
use kernel::register;
use crate::driver::TyrRegisters;
/// Number of doorbells available.
pub(crate) const NUM_DOORBELLS: usize = 64;
/// The architectural maximum is 64 but this full array does not
/// fit in the 2 MiB iomem region. Panthor currently uses a maximum of
/// 17 doorbells so do the same here.
pub(crate) const NUM_DOORBELLS: usize = 17;
/// Doorbell block stride (64KiB).
///
@ -1770,6 +1813,8 @@ pub(crate) mod doorbell_block {
const STRIDE: usize = 0x10000;
register! {
base: TyrRegisters;
/// Doorbell request register. Write-only.
pub(crate) DOORBELL(u32)[NUM_DOORBELLS, stride = STRIDE] @ 0x80000 {
/// Doorbell set. Writing 1 triggers the doorbell.

15
drivers/gpu/nova-core/Kconfig
View File

@ -5,6 +5,7 @@ config NOVA_CORE
depends on RUST
depends on !CPU_BIG_ENDIAN
select AUXILIARY_BUS
select GPU_BUDDY
select RUST_FW_LOADER_ABSTRACTIONS
default n
help
@ -15,3 +16,17 @@ config NOVA_CORE
This driver is work in progress and may not be functional.
If M is selected, the module will be called nova-core.
config NOVA_CORE_SELFTESTS
bool "Nova Core driver self-tests"
depends on NOVA_CORE
default n
help
Build the driver self-tests and run them when the GPU is probed.
If the interrupt delivery test fails, the probe fails and the driver
does not bind to the GPU. A broken interrupt path would otherwise
show up later as a hang, far from its cause. Every other self-test
logs its failure and lets the probe continue.
If unsure, say N.

73
drivers/gpu/nova-core/api.rs Normal file
View File

@ -0,0 +1,73 @@
// SPDX-License-Identifier: GPL-2.0
//! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
//! to interact with nova-core.
use core::pin::Pin;
use kernel::{
auxiliary,
device::Bound,
pci,
prelude::*,
types::ForLt, //
};
pub use crate::gpu::Spec;
use crate::{
driver,
gpu::Gpu,
gsp::commands::GetGspStaticInfoReply, //
};
/// API handle for the auxiliary bus child drivers to interact with nova-core.
pub struct NovaCoreApi<'bound> {
pub(crate) gpu: Pin<&'bound Gpu<'bound>>,
pub(crate) pdev: &'bound pci::Device<Bound>,
}
impl NovaCoreApi<'_> {
/// Obtain a [`NovaCoreApi`] handle from an auxiliary device registered
/// by nova-core.
pub fn of(adev: &auxiliary::Device<Bound>) -> Result<NovaCoreApiHandle<'_>> {
NovaCoreApiHandle::of(adev)
}
/// Returns the GPU [`GetGspStaticInfoReply`].
pub fn gsp_static_info(&self) -> &GetGspStaticInfoReply {
&self.gpu.gsp_static_info
}
/// Returns the GPU [`Spec`].
pub fn spec(&self) -> &Spec {
&self.gpu.spec
}
/// Returns the size of the PCIe BAR used for accessing VRAM, typically
/// BAR1.
pub fn bar1_size(&self) -> Result<u64> {
let bar1 = driver::bar1_resource_index(self.pdev)?;
self.pdev.resource_len(bar1)
}
}
/// Expose a handle to nova-core API
pub struct NovaCoreApiHandle<'a> {
adev: &'a auxiliary::Device<Bound>,
}
impl<'a> NovaCoreApiHandle<'a> {
fn of(adev: &'a auxiliary::Device<Bound>) -> Result<Self> {
adev.registration_data_with::<ForLt!(NovaCoreApi<'_>), ()>(|_| ())?;
Ok(Self { adev })
}
/// Access the [`NovaCoreApi`] through a closure.
pub fn with<R>(&self, f: impl for<'b> FnOnce(Pin<&'a NovaCoreApi<'b>>) -> R) -> R {
self.adev
.registration_data_with::<ForLt!(NovaCoreApi<'_>), R>(f)
.expect("TypeId was validated in NovaCoreApiHandle::of()")
}
}

125
drivers/gpu/nova-core/driver.rs
View File

@ -2,7 +2,11 @@
use kernel::{
auxiliary,
device::Core,
device::{
Bound,
Core, //
},
io::resource,
pci,
pci::{
Class,
@ -15,21 +19,25 @@
Atomic,
Relaxed, //
},
types::CovariantForLt,
types::ForLt,
};
use crate::gpu::Gpu;
use crate::{
api::NovaCoreApi,
gpu::Gpu, //
};
/// Counter for generating unique auxiliary device IDs.
static AUXILIARY_ID_COUNTER: Atomic<u32> = Atomic::new(0);
#[pin_data]
pub(crate) struct NovaCore<'bound> {
#[allow(clippy::type_complexity)]
_reg: auxiliary::Registration<'bound, ForLt!(NovaCoreApi<'_>)>,
#[pin]
pub(crate) gpu: Gpu<'bound>,
bar: pci::Bar<'bound, BAR0_SIZE>,
#[allow(clippy::type_complexity)]
_reg: auxiliary::Registration<'bound, CovariantForLt!(())>,
bar1: pci::Bar<'bound>,
}
pub(crate) struct NovaCoreDriver;
@ -37,6 +45,27 @@ pub(crate) struct NovaCore<'bound> {
const BAR0_SIZE: usize = SZ_16M;
pub(crate) type Bar0<'a> = &'a pci::Bar<'a, BAR0_SIZE>;
pub(crate) type NovaRegisters = kernel::io::Region<BAR0_SIZE>;
pub(crate) type Bar1<'a> = &'a pci::Bar<'a>;
/// Returns the Linux PCI resource index that holds BAR1 for an NVIDIA GPU.
///
/// On Maxwell through Ada, BAR0 is a 32-bit memory BAR occupying a single
/// Linux PCI resource slot, so BAR1 lives at index 1. Starting with Blackwell
/// (and on some Ampere GA100 / Hopper SKUs) BAR0 is a 64-bit memory BAR that
/// consumes two consecutive resource slots: index 0 holds the low 32 bits and
/// index 1 holds the high 32 bits (with no `flags` / or size of its own),
/// shifting BAR1 to index 2.
pub(crate) fn bar1_resource_index(pdev: &pci::Device<Bound>) -> Result<u32> {
// Probe the `IORESOURCE_MEM_64` flag of BAR0 as a robust way of exposing
// if BAR0 and hence BAR1 is 64-bit.
let flags0 = pdev.resource_flags(0)?;
if flags0.contains(resource::Flags::IORESOURCE_MEM_64) {
Ok(2)
} else {
Ok(1)
}
}
kernel::pci_device_table!(
PCI_TABLE,
@ -71,30 +100,74 @@ fn probe<'bound>(
pdev: &'bound pci::Device<Core<'_>>,
_info: Option<&'bound Self::IdInfo>,
) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
pin_init::pin_init_scope(move || {
dev_dbg!(pdev, "Probe Nova Core GPU driver.\n");
dev_dbg!(pdev, "Probe Nova Core GPU driver.\n");
pdev.enable_device_mem()?;
pdev.set_master();
try_pin_init!(NovaCore {
_: {
pdev.enable_device_mem()?;
pdev.set_master();
},
Ok(try_pin_init!(NovaCore {
bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
// TODO: Use `&bar` self-referential pin-init syntax once available.
bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
bar1: {
let bar1_idx = bar1_resource_index(pdev)?;
pdev.iomap_region(bar1_idx, c"nova-core/bar1")?
},
// Run self-tests that do not depend on the `Gpu` instance.
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
_: {
let spec = crate::gpu::Spec::new(pdev.as_ref(), bar)?;
// We must wait for GFW_BOOT completion before doing any significant setup on
// the GPU.
Gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
// The self-test disables and drains the whole tree, so it has to run before
// `Gpu::new` boots the GSP.
crate::irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
},
// TODO: Use self-referential pin-init syntax once available.
gpu <- Gpu::new(
pdev,
// SAFETY: `bar` is initialized above, pinned, and outlives `gpu`.
unsafe { &*core::ptr::from_ref(bar) },
// SAFETY: `bar1` is initialized above, pinned, and outlives `gpu`.
unsafe { &*core::ptr::from_ref(bar1) },
).pin_chain(|_gpu| {
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
_gpu.run_selftests(pdev);
Ok(())
}),
_reg: {
// TODO: Use `&gpu` self-referential pin-init syntax once available.
//
// SAFETY: `bar` is initialized before this expression is evaluated
// (`try_pin_init!()` initializes fields in declaration order), lives at a pinned
// stable address, and is dropped after `gpu` (struct field drop order).
gpu <- Gpu::new(pdev, unsafe { &*core::ptr::from_ref(bar) }),
_reg: auxiliary::Registration::new(
pdev.as_ref(),
c"nova-drm",
// TODO[XARR]: Use XArray or perhaps IDA for proper ID allocation/recycling. For
// now, use a simple atomic counter that never recycles IDs.
AUXILIARY_ID_COUNTER.fetch_add(1, Relaxed),
crate::MODULE_NAME,
(),
)?,
}))
// SAFETY: `gpu` is initialized before this expression is evaluated
// (`try_pin_init!()` initializes fields in initializer order), lives at
// a pinned stable address, and is dropped after `_reg` (struct field
// drop order).
let gpu = unsafe {
Pin::new_unchecked(&*core::ptr::from_ref(gpu.as_ref().get_ref()))
};
// SAFETY: `NovaCore` is dropped when the device is unbound;
// i.e. `mem::forget()` is never called on it.
unsafe {
auxiliary::Registration::new_with_lt(
pdev.as_ref(),
c"nova-drm",
// TODO[XARR]: Use XArray or perhaps IDA for proper ID
// allocation/recycling. For now, use a simple atomic counter that
// never recycles IDs.
AUXILIARY_ID_COUNTER.fetch_add(1, Relaxed),
crate::MODULE_NAME,
NovaCoreApi { gpu, pdev },
)?
}
},
})
}
}

164
drivers/gpu/nova-core/falcon.rs
View File

@ -14,13 +14,12 @@
io::{
io_project,
poll::read_poll_timeout,
register::{
RegisterBase,
WithBase, //
},
register::Array,
Io,
Mmio, //
},
prelude::*,
sizes::SZ_4K,
time::Delta,
};
@ -165,18 +164,25 @@ pub(crate) enum FalconFbifMemType with From<Bounded<u32, 1>> {
}
}
/// Type used to represent the `PFALCON` registers address base for a given falcon engine.
pub(crate) struct PFalconBase(());
const PFALCON_REGION_SIZE: usize = SZ_4K;
const PFALCON2_REGION_SIZE: usize = SZ_4K;
/// Type used to represent the `PFALCON2` registers address base for a given falcon engine.
pub(crate) struct PFalcon2Base(());
/// Type used to represent the `PFALCON` registers.
#[repr(align(4))]
#[derive(FromBytes, IntoBytes)]
pub(crate) struct PFalconRegisters([u8; PFALCON_REGION_SIZE]);
/// Type used to represent the `PFALCON2` registers.
#[repr(align(4))]
#[derive(FromBytes, IntoBytes)]
pub(crate) struct PFalcon2Registers([u8; PFALCON2_REGION_SIZE]);
/// Trait defining the parameters of a given Falcon engine.
///
/// Each engine provides one base for `PFALCON` and `PFALCON2` registers.
pub(crate) trait FalconEngine:
Send + Sync + RegisterBase<PFalconBase> + RegisterBase<PFalcon2Base> + Sized
{
pub(crate) trait FalconEngine: Send + Sync + Sized {
fn pfalcon(io: Bar0<'_>) -> Mmio<'_, PFalconRegisters>;
fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, PFalcon2Registers>;
}
/// Represents a portion of the firmware to be loaded into a particular memory (e.g. IMEM or DMEM)
@ -358,6 +364,9 @@ pub(crate) struct Falcon<'a, E: FalconEngine> {
hal: KBox<dyn FalconHal<E>>,
dev: &'a device::Device<device::Bound>,
bar: Bar0<'a>,
// TODO: make private
pub(crate) pfalcon: Mmio<'a, PFalconRegisters>,
pfalcon2: Mmio<'a, PFalcon2Registers>,
}
impl<'a, E: FalconEngine + 'static> Falcon<'a, E> {
@ -371,19 +380,19 @@ pub(crate) fn new(
hal: hal::falcon_hal(chipset)?,
dev,
bar,
pfalcon: E::pfalcon(bar),
pfalcon2: E::pfalcon2(bar),
})
}
/// Resets DMA-related registers.
pub(crate) fn dma_reset(&self) {
self.bar.update(regs::NV_PFALCON_FBIF_CTL::of::<E>(), |v| {
self.pfalcon.update(regs::NV_PFALCON_FBIF_CTL, |v| {
v.with_allow_phys_no_ctx(true)
});
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_DMACTL::zeroed(),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_DMACTL::zeroed());
}
/// Reset the controller, select the falcon core, and wait for memory scrubbing to complete.
@ -392,10 +401,10 @@ pub(crate) fn reset(&self) -> Result {
self.hal.select_core(self)?;
self.hal.reset_wait_mem_scrubbing(self)?;
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_RM::from(self.bar.read(regs::NV_PMC_BOOT_0).into_raw()),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_RM::from(crate::gpu::boot_0_raw(
self.bar,
)));
Ok(())
}
@ -413,8 +422,8 @@ fn pio_wr_imem_slice(&self, load_offsets: FalconPioImemLoadTarget<'_>) -> Result
return Err(EINVAL);
}
self.bar.write(
WithBase::of::<E>().at(Self::PIO_PORT),
self.pfalcon.write(
Array::at(Self::PIO_PORT),
regs::NV_PFALCON_FALCON_IMEMC::zeroed()
.with_secure(load_offsets.secure)
.with_aincw(true)
@ -424,14 +433,14 @@ fn pio_wr_imem_slice(&self, load_offsets: FalconPioImemLoadTarget<'_>) -> Result
for (n, block) in load_offsets.data.chunks(MEM_BLOCK_ALIGNMENT).enumerate() {
let n = u16::try_from(n)?;
let tag: u16 = load_offsets.start_tag.checked_add(n).ok_or(ERANGE)?;
self.bar.write(
WithBase::of::<E>().at(Self::PIO_PORT),
self.pfalcon.write(
Array::at(Self::PIO_PORT),
regs::NV_PFALCON_FALCON_IMEMT::zeroed().with_tag(tag),
);
for word in block.chunks_exact(4) {
let w = [word[0], word[1], word[2], word[3]];
self.bar.write(
WithBase::of::<E>().at(Self::PIO_PORT),
self.pfalcon.write(
Array::at(Self::PIO_PORT),
regs::NV_PFALCON_FALCON_IMEMD::zeroed().with_data(u32::from_le_bytes(w)),
);
}
@ -450,8 +459,8 @@ fn pio_wr_dmem_slice(&self, load_offsets: FalconPioDmemLoadTarget<'_>) -> Result
return Err(EINVAL);
}
self.bar.write(
WithBase::of::<E>().at(Self::PIO_PORT),
self.pfalcon.write(
Array::at(Self::PIO_PORT),
regs::NV_PFALCON_FALCON_DMEMC::zeroed()
.with_aincw(true)
.with_offs(load_offsets.dst_start),
@ -459,8 +468,8 @@ fn pio_wr_dmem_slice(&self, load_offsets: FalconPioDmemLoadTarget<'_>) -> Result
for word in load_offsets.data.chunks_exact(4) {
let w = [word[0], word[1], word[2], word[3]];
self.bar.write(
WithBase::of::<E>().at(Self::PIO_PORT),
self.pfalcon.write(
Array::at(Self::PIO_PORT),
regs::NV_PFALCON_FALCON_DMEMD::zeroed().with_data(u32::from_le_bytes(w)),
);
}
@ -473,14 +482,12 @@ pub(crate) fn pio_load<F: FalconFirmware<Target = E> + FalconPioLoadable>(
&self,
fw: &F,
) -> Result {
self.bar.update(regs::NV_PFALCON_FBIF_CTL::of::<E>(), |v| {
self.pfalcon.update(regs::NV_PFALCON_FBIF_CTL, |v| {
v.with_allow_phys_no_ctx(true)
});
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_DMACTL::zeroed(),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_DMACTL::zeroed());
if let Some(imem_ns) = fw.imem_ns_load_params() {
self.pio_wr_imem_slice(imem_ns)?;
@ -492,10 +499,8 @@ pub(crate) fn pio_load<F: FalconFirmware<Target = E> + FalconPioLoadable>(
self.hal.program_brom(self, &fw.brom_params());
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_BOOTVEC::zeroed().with_value(fw.boot_addr()),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_BOOTVEC::zeroed().with_value(fw.boot_addr()));
Ok(())
}
@ -506,7 +511,7 @@ pub(crate) fn pio_load<F: FalconFirmware<Target = E> + FalconPioLoadable>(
/// `sec` is set if the loaded firmware is expected to run in secure mode.
fn dma_wr(
&self,
dma_obj: &Coherent<[u8]>,
dma_obj: &Coherent<'_, [u8]>,
target_mem: FalconMem,
load_offsets: FalconDmaLoadTarget,
) -> Result {
@ -547,16 +552,13 @@ fn dma_wr(
// Set up the base source DMA address.
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_DMATRFBASE::zeroed().with_base(
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_DMATRFBASE::zeroed().with_base(
// CAST: `as u32` is used on purpose since we do want to strip the upper bits,
// which will be written to `NV_PFALCON_FALCON_DMATRFBASE1`.
(dma_address >> 8) as u32,
),
);
self.bar.write(
WithBase::of::<E>(),
));
self.pfalcon.write_reg(
regs::NV_PFALCON_FALCON_DMATRFBASE1::zeroed().try_with_base(dma_address >> 40)?,
);
@ -566,23 +568,21 @@ fn dma_wr(
for pos in (0..num_transfers).map(|i| i * DMA_LEN) {
// Perform a transfer of size `DMA_LEN`.
self.bar.write(
WithBase::of::<E>(),
self.pfalcon.write_reg(
regs::NV_PFALCON_FALCON_DMATRFMOFFS::zeroed()
.try_with_offs(load_offsets.dst_start + pos)?,
);
self.bar.write(
WithBase::of::<E>(),
self.pfalcon.write_reg(
regs::NV_PFALCON_FALCON_DMATRFFBOFFS::zeroed().with_offs(src_start + pos),
);
self.bar.write(WithBase::of::<E>(), cmd);
self.pfalcon.write_reg(cmd);
// Wait for the transfer to complete.
// TIMEOUT: arbitrarily large value, no DMA transfer to the falcon's small memories
// should ever take that long.
read_poll_timeout(
|| Ok(self.bar.read(regs::NV_PFALCON_FALCON_DMATRFCMD::of::<E>())),
|| Ok(self.pfalcon.read(regs::NV_PFALCON_FALCON_DMATRFCMD)),
|r| r.idle(),
Delta::ZERO,
Delta::from_secs(2),
@ -614,8 +614,8 @@ fn dma_load<F: FalconFirmware<Target = E> + FalconDmaLoadable>(&self, fw: &F) ->
};
self.dma_reset();
self.bar
.update(regs::NV_PFALCON_FBIF_TRANSCFG::of::<E>().at(0), |v| {
self.pfalcon
.update(regs::NV_PFALCON_FBIF_TRANSCFG::at(0), |v| {
v.with_target(FalconFbifTarget::CoherentSysmem)
.with_mem_type(FalconFbifMemType::Physical)
});
@ -626,10 +626,8 @@ fn dma_load<F: FalconFirmware<Target = E> + FalconDmaLoadable>(&self, fw: &F) ->
self.hal.program_brom(self, &fw.brom_params());
// Set `BootVec` to start of non-secure code.
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_BOOTVEC::zeroed().with_value(fw.boot_addr()),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_BOOTVEC::zeroed().with_value(fw.boot_addr()));
Ok(())
}
@ -638,7 +636,7 @@ fn dma_load<F: FalconFirmware<Target = E> + FalconDmaLoadable>(&self, fw: &F) ->
pub(crate) fn wait_till_halted(&self) -> Result<()> {
// TIMEOUT: arbitrarily large value, firmwares should complete in less than 2 seconds.
read_poll_timeout(
|| Ok(self.bar.read(regs::NV_PFALCON_FALCON_CPUCTL::of::<E>())),
|| Ok(self.pfalcon.read(regs::NV_PFALCON_FALCON_CPUCTL)),
|r| r.halted(),
Delta::ZERO,
Delta::from_secs(2),
@ -649,19 +647,13 @@ pub(crate) fn wait_till_halted(&self) -> Result<()> {
/// Start the falcon CPU.
pub(crate) fn start(&self) -> Result<()> {
match self
.bar
.read(regs::NV_PFALCON_FALCON_CPUCTL::of::<E>())
.alias_en()
{
true => self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_CPUCTL_ALIAS::zeroed().with_startcpu(true),
),
false => self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_CPUCTL::zeroed().with_startcpu(true),
),
match self.pfalcon.read(regs::NV_PFALCON_FALCON_CPUCTL).alias_en() {
true => self
.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_CPUCTL_ALIAS::zeroed().with_startcpu(true)),
false => self
.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_CPUCTL::zeroed().with_startcpu(true)),
}
Ok(())
@ -670,32 +662,24 @@ pub(crate) fn start(&self) -> Result<()> {
/// Writes values to the mailbox registers if provided.
pub(crate) fn write_mailboxes(&self, mbox0: Option<u32>, mbox1: Option<u32>) {
if let Some(mbox0) = mbox0 {
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_MAILBOX0::zeroed().with_value(mbox0),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_MAILBOX0::zeroed().with_value(mbox0));
}
if let Some(mbox1) = mbox1 {
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_MAILBOX1::zeroed().with_value(mbox1),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_MAILBOX1::zeroed().with_value(mbox1));
}
}
/// Reads the value from `mbox0` register.
pub(crate) fn read_mailbox0(&self) -> u32 {
self.bar
.read(regs::NV_PFALCON_FALCON_MAILBOX0::of::<E>())
.value()
self.pfalcon.read(regs::NV_PFALCON_FALCON_MAILBOX0).value()
}
/// Reads the value from `mbox1` register.
pub(crate) fn read_mailbox1(&self) -> u32 {
self.bar
.read(regs::NV_PFALCON_FALCON_MAILBOX1::of::<E>())
.value()
self.pfalcon.read(regs::NV_PFALCON_FALCON_MAILBOX1).value()
}
/// Reads values from both mailbox registers.
@ -760,9 +744,7 @@ pub(crate) fn load<F: FalconFirmware<Target = E> + FalconDmaLoadable>(&self, fw:
/// Write the application version to the OS register.
pub(crate) fn write_os_version(&self, app_version: u32) {
self.bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON_FALCON_OS::zeroed().with_value(app_version),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_OS::zeroed().with_value(app_version));
}
}

67
drivers/gpu/nova-core/falcon/fsp.rs
View File

@ -8,13 +8,12 @@
use kernel::{
io::{
io_project,
poll::read_poll_timeout,
register::{
Array,
RegisterBase,
WithBase, //
},
Io, //
register,
register::Array,
Io,
Mmio, //
},
prelude::*,
sizes::SZ_1K,
@ -22,11 +21,13 @@
};
use crate::{
driver::{
Bar0,
NovaRegisters, //
},
falcon::{
Falcon,
FalconEngine,
PFalcon2Base,
PFalconBase, //
FalconEngine, //
},
num,
regs, //
@ -41,15 +42,24 @@
/// Type specifying the `Fsp` falcon engine. Cannot be instantiated.
pub(crate) struct Fsp(());
impl RegisterBase<PFalconBase> for Fsp {
const BASE: usize = 0x8f2000;
register! {
base: NovaRegisters;
PFALCON: super::PFalconRegisters @ 0x8f2000;
PFALCON2: super::PFalcon2Registers @ 0x8f3000;
}
impl RegisterBase<PFalcon2Base> for Fsp {
const BASE: usize = 0x8f3000;
}
impl FalconEngine for Fsp {
#[inline]
fn pfalcon(io: Bar0<'_>) -> Mmio<'_, super::PFalconRegisters> {
io_project!(io, build: PFALCON)
}
impl FalconEngine for Fsp {}
#[inline]
fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, super::PFalcon2Registers> {
io_project!(io, build: PFALCON2)
}
}
impl<'a> Falcon<'a, Fsp> {
/// Writes `data` to FSP external memory at offset `0`.
@ -62,19 +72,15 @@ fn write_emem(&mut self, data: &[u8]) -> Result {
}
// Begin a write burst at offset `0`, auto-incrementing on each write.
self.bar.write(
WithBase::of::<Fsp>(),
regs::NV_PFALCON_FALCON_EMEMC::zeroed().with_aincw(true),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_EMEMC::zeroed().with_aincw(true));
for chunk in data.chunks_exact(4) {
let value = u32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]);
// Write the next 32-bit `value`; hardware advances the offset.
self.bar.write(
WithBase::of::<Fsp>(),
regs::NV_PFALCON_FALCON_EMEMD::zeroed().with_data(value),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_EMEMD::zeroed().with_data(value));
}
Ok(())
@ -90,17 +96,12 @@ fn read_emem(&mut self, data: &mut [u8]) -> Result {
}
// Begin a read burst at offset `0`, auto-incrementing on each read.
self.bar.write(
WithBase::of::<Fsp>(),
regs::NV_PFALCON_FALCON_EMEMC::zeroed().with_aincr(true),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_EMEMC::zeroed().with_aincr(true));
for chunk in data.chunks_exact_mut(4) {
// Read the next 32-bit word; hardware advances the offset.
let value = self
.bar
.read(regs::NV_PFALCON_FALCON_EMEMD::of::<Fsp>())
.data();
let value = self.pfalcon.read(regs::NV_PFALCON_FALCON_EMEMD).data();
chunk.copy_from_slice(&value.to_le_bytes());
}
@ -172,9 +173,7 @@ pub(crate) fn recv_msg(&mut self) -> Result<KVec<u8>> {
return Err(EMSGSIZE);
}
let mut buffer = KVec::<u8>::new();
buffer.resize(msg_size, 0, GFP_KERNEL)?;
let mut buffer = KVec::zeroed(msg_size, GFP_KERNEL)?;
self.read_emem(&mut buffer)?;
// Reset message queue pointers after reading.

98
drivers/gpu/nova-core/falcon/gsp.rs
View File

@ -2,23 +2,24 @@
use kernel::{
io::{
io_project,
poll::read_poll_timeout,
register::{
RegisterBase,
WithBase, //
},
register,
Io,
Mmio, //
},
prelude::*,
time::Delta, //
};
use crate::{
driver::{
Bar0,
NovaRegisters, //
},
falcon::{
Falcon,
FalconEngine,
PFalcon2Base,
PFalconBase, //
FalconEngine, //
},
regs,
};
@ -26,24 +27,74 @@
/// Type specifying the `Gsp` falcon engine. Cannot be instantiated.
pub(crate) struct Gsp(());
impl RegisterBase<PFalconBase> for Gsp {
const BASE: usize = 0x00110000;
register! {
base: NovaRegisters;
PFALCON: super::PFalconRegisters @ 0x00110000;
PFALCON2: super::PFalcon2Registers @ 0x00111000;
}
impl RegisterBase<PFalcon2Base> for Gsp {
const BASE: usize = 0x00111000;
}
impl FalconEngine for Gsp {
#[inline]
fn pfalcon(io: Bar0<'_>) -> Mmio<'_, super::PFalconRegisters> {
io_project!(io, build: PFALCON)
}
impl FalconEngine for Gsp {}
#[inline]
fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, super::PFalcon2Registers> {
io_project!(io, build: PFALCON2)
}
}
impl<'a> Falcon<'a, Gsp> {
/// Clears the SWGEN0 bit in the Falcon's IRQ status clear register to
/// allow GSP to signal CPU for processing new messages in message queue.
/// Clears the SWGEN0 latch in the GSP falcon.
///
/// While the latch is set, no later message signals the tree, so a caller that consumed a
/// notification by polling must clear it.
pub(crate) fn clear_swgen0_intr(&self) {
self.bar.write(
WithBase::of::<Gsp>(),
regs::NV_PFALCON_FALCON_IRQSCLR::zeroed().with_swgen0(true),
);
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::zeroed().with_swgen0(true));
}
/// Reads the GSP falcon causes that are routed to the host, without clearing any latch.
///
/// Every one of them other than SWGEN0 reports a GSP fault.
pub(crate) fn read_host_intr(&self) -> regs::NV_PFALCON_FALCON_IRQSTAT {
let latched = self.pfalcon.read(regs::NV_PFALCON_FALCON_IRQSTAT);
self.hal.host_routed_causes(self, latched)
}
/// Reads the host-routed causes and clears the SWGEN0 latch if it was set.
///
/// Returns the causes as read, before the clear. No other latch changes.
pub(crate) fn take_host_intr(&self) -> regs::NV_PFALCON_FALCON_IRQSTAT {
let status = self.read_host_intr();
if status.swgen0() {
self.clear_swgen0_intr();
}
status
}
/// Clears the latch of every interrupt cause set in `status`.
///
/// A cause driven from outside the falcon is still set on return, and
/// [`Self::read_host_intr`] reports the causes that remain.
pub(crate) fn clear_intr(&self, status: regs::NV_PFALCON_FALCON_IRQSTAT) {
self.pfalcon
.write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::from(status.into_raw()));
}
/// Retriggers the GSP falcon, which then re-emits its host-routed causes into the tree.
///
/// Call this only once every host cause is clear. A cause still set is re-emitted at once, and
/// its vector arrives again as soon as delivery is rearmed.
///
/// Does nothing on Turing, whose falcons have no retrigger register.
pub(crate) fn retrigger_intr(&self) {
self.hal.retrigger(self);
}
/// Checks if GSP reload/resume has completed during the boot process.
@ -59,8 +110,8 @@ pub(crate) fn check_reload_completed(&self, timeout: Delta) -> Result<bool> {
/// Returns whether the RISC-V branch privilege lockdown bit is set.
pub(crate) fn riscv_branch_privilege_lockdown(&self) -> bool {
self.bar
.read(regs::NV_PFALCON_FALCON_HWCFG2::of::<Gsp>())
self.pfalcon
.read(regs::NV_PFALCON_FALCON_HWCFG2)
.riscv_br_priv_lockdown()
}
@ -71,10 +122,7 @@ pub(crate) fn priv_target_mask_released(&self) -> bool {
const LOCKED_PATTERN: u32 = 0xbadf_4100;
const LOCKED_MASK: u32 = 0xffff_ff00;
let hwcfg2 = self
.bar
.read(regs::NV_PFALCON_FALCON_HWCFG2::of::<Gsp>())
.into_raw();
let hwcfg2 = self.pfalcon.read(regs::NV_PFALCON_FALCON_HWCFG2).into_raw();
hwcfg2 != 0 && (hwcfg2 & LOCKED_MASK) != LOCKED_PATTERN
}

22
drivers/gpu/nova-core/falcon/hal.rs
View File

@ -1,4 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
use kernel::prelude::*;
@ -12,6 +13,7 @@
Architecture,
Chipset, //
},
regs,
};
mod ga102;
@ -70,6 +72,20 @@ fn signature_reg_fuse_version(
/// these. For anything above, the PIO registers appear to be masked to the CPU, so DMA is the
/// only usable method.
fn load_method(&self) -> LoadMethod;
/// Returns the causes in `latched` that are routed to the host, meaning the CPU, rather than
/// to the falcon's own RISC-V core.
///
/// The causes routed to the core belong to the firmware running on it, and the host does not
/// service them.
fn host_routed_causes(
&self,
falcon: &Falcon<'_, E>,
latched: regs::NV_PFALCON_FALCON_IRQSTAT,
) -> regs::NV_PFALCON_FALCON_IRQSTAT;
/// Retriggers the falcon, which then re-emits its host-routed causes into the interrupt tree.
fn retrigger(&self, falcon: &Falcon<'_, E>);
}
/// Returns a boxed falcon HAL adequate for `chipset`.
@ -82,11 +98,11 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(
) -> Result<KBox<dyn FalconHal<E>>> {
let hal = match chipset.arch() {
Architecture::Turing => {
KBox::new(tu102::Tu102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
KBox::new(tu102::Tu102::<E>::new(false), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
}
// GA100 boots like Turing so use Turing HAL
// GA100 boots like Turing, but features the retrigger register.
Architecture::Ampere if chipset == Chipset::GA100 => {
KBox::new(tu102::Tu102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
KBox::new(tu102::Tu102::<E>::new(true), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
}
Architecture::Ampere
| Architecture::Ada

76
drivers/gpu/nova-core/falcon/hal/ga102.rs
View File

@ -1,4 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
use core::marker::PhantomData;
@ -6,11 +7,9 @@
device,
io::{
poll::read_poll_timeout,
register::{
Array,
WithBase, //
},
Io, //
register::Array,
Io,
Mmio, //
},
prelude::*,
time::Delta, //
@ -24,6 +23,7 @@
FalconBromParams,
FalconEngine,
FalconModSelAlgo,
PFalcon2Registers,
PeregrineCoreSelect, //
},
regs,
@ -31,17 +31,16 @@
use super::FalconHal;
fn select_core_ga102<E: FalconEngine>(bar: Bar0<'_>) -> Result {
let bcr_ctrl = bar.read(regs::NV_PRISCV_RISCV_BCR_CTRL::of::<E>());
fn select_core_ga102(pfalcon2: Mmio<'_, PFalcon2Registers>) -> Result {
let bcr_ctrl = pfalcon2.read(regs::NV_PRISCV_RISCV_BCR_CTRL);
if bcr_ctrl.core_select() != PeregrineCoreSelect::Falcon {
bar.write(
WithBase::of::<E>(),
pfalcon2.write_reg(
regs::NV_PRISCV_RISCV_BCR_CTRL::zeroed().with_core_select(PeregrineCoreSelect::Falcon),
);
// TIMEOUT: falcon core should take less than 10ms to report being enabled.
read_poll_timeout(
|| Ok(bar.read(regs::NV_PRISCV_RISCV_BCR_CTRL::of::<E>())),
|| Ok(pfalcon2.read(regs::NV_PRISCV_RISCV_BCR_CTRL)),
|r| r.valid(),
Delta::ZERO,
Delta::from_millis(10),
@ -86,24 +85,20 @@ fn signature_reg_fuse_version_ga102(
Ok(u16::BITS - reg_fuse_version.leading_zeros())
}
fn program_brom_ga102<E: FalconEngine>(bar: Bar0<'_>, params: &FalconBromParams) {
bar.write(
WithBase::of::<E>().at(0),
fn program_brom_ga102(pfalcon2: Mmio<'_, PFalcon2Registers>, params: &FalconBromParams) {
pfalcon2.write(
Array::at(0),
regs::NV_PFALCON2_FALCON_BROM_PARAADDR::zeroed().with_value(params.pkc_data_offset),
);
bar.write(
WithBase::of::<E>(),
pfalcon2.write_reg(
regs::NV_PFALCON2_FALCON_BROM_ENGIDMASK::zeroed()
.with_value(u32::from(params.engine_id_mask)),
);
bar.write(
WithBase::of::<E>(),
pfalcon2.write_reg(
regs::NV_PFALCON2_FALCON_BROM_CURR_UCODE_ID::zeroed().with_ucode_id(params.ucode_id),
);
bar.write(
WithBase::of::<E>(),
regs::NV_PFALCON2_FALCON_MOD_SEL::zeroed().with_algo(FalconModSelAlgo::Rsa3k),
);
pfalcon2
.write_reg(regs::NV_PFALCON2_FALCON_MOD_SEL::zeroed().with_algo(FalconModSelAlgo::Rsa3k));
}
pub(super) struct Ga102<E: FalconEngine>(PhantomData<E>);
@ -116,7 +111,7 @@ pub(super) fn new() -> Self {
impl<E: FalconEngine> FalconHal<E> for Ga102<E> {
fn select_core(&self, falcon: &Falcon<'_, E>) -> Result {
select_core_ga102::<E>(falcon.bar)
select_core_ga102(falcon.pfalcon2)
}
fn signature_reg_fuse_version(
@ -129,27 +124,24 @@ fn signature_reg_fuse_version(
}
fn program_brom(&self, falcon: &Falcon<'_, E>, params: &FalconBromParams) {
program_brom_ga102::<E>(falcon.bar, params);
program_brom_ga102(falcon.pfalcon2, params);
}
fn is_riscv_active(&self, falcon: &Falcon<'_, E>) -> bool {
falcon
.bar
.read(regs::NV_PRISCV_RISCV_CPUCTL::of::<E>())
.pfalcon2
.read(regs::NV_PRISCV_RISCV_CPUCTL)
.active_stat()
}
fn is_riscv_halted(&self, falcon: &Falcon<'_, E>) -> Result<bool> {
Ok(falcon
.bar
.read(regs::NV_PRISCV_RISCV_CPUCTL::of::<E>())
.halted())
Ok(falcon.pfalcon2.read(regs::NV_PRISCV_RISCV_CPUCTL).halted())
}
fn reset_wait_mem_scrubbing(&self, falcon: &Falcon<'_, E>) -> Result {
// TIMEOUT: memory scrubbing should complete in less than 20ms.
read_poll_timeout(
|| Ok(falcon.bar.read(regs::NV_PFALCON_FALCON_HWCFG2::of::<E>())),
|| Ok(falcon.pfalcon.read(regs::NV_PFALCON_FALCON_HWCFG2)),
|r| r.mem_scrubbing_done(),
Delta::ZERO,
Delta::from_millis(20),
@ -158,20 +150,18 @@ fn reset_wait_mem_scrubbing(&self, falcon: &Falcon<'_, E>) -> Result {
}
fn reset_eng(&self, falcon: &Falcon<'_, E>) -> Result {
let bar = falcon.bar;
let _ = bar.read(regs::NV_PFALCON_FALCON_HWCFG2::of::<E>());
let _ = falcon.pfalcon.read(regs::NV_PFALCON_FALCON_HWCFG2);
// According to OpenRM's `kflcnPreResetWait_GA102` documentation, HW sometimes does not set
// RESET_READY so a non-failing timeout is used.
let _ = read_poll_timeout(
|| Ok(bar.read(regs::NV_PFALCON_FALCON_HWCFG2::of::<E>())),
|| Ok(falcon.pfalcon.read(regs::NV_PFALCON_FALCON_HWCFG2)),
|r| r.reset_ready(),
Delta::ZERO,
Delta::from_micros(150),
);
regs::NV_PFALCON_FALCON_ENGINE::reset_engine::<E>(bar);
regs::NV_PFALCON_FALCON_ENGINE::reset_engine(falcon.pfalcon);
self.reset_wait_mem_scrubbing(falcon)?;
Ok(())
@ -180,4 +170,20 @@ fn reset_eng(&self, falcon: &Falcon<'_, E>) -> Result {
fn load_method(&self) -> LoadMethod {
LoadMethod::Dma
}
fn host_routed_causes(
&self,
falcon: &Falcon<'_, E>,
latched: regs::NV_PFALCON_FALCON_IRQSTAT,
) -> regs::NV_PFALCON_FALCON_IRQSTAT {
let pfalcon2 = falcon.pfalcon2;
let mask = pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQMASK).value();
let dest = pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQDEST).value();
regs::NV_PFALCON_FALCON_IRQSTAT::from(latched.into_raw() & mask & dest)
}
fn retrigger(&self, falcon: &Falcon<'_, E>) {
super::tu102::retrigger_ga100(falcon);
}
}

55
drivers/gpu/nova-core/falcon/hal/tu102.rs
View File

@ -1,11 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
use core::marker::PhantomData;
use kernel::{
io::{
poll::read_poll_timeout,
register::WithBase,
register::Array,
Io, //
},
prelude::*,
@ -24,11 +25,31 @@
use super::FalconHal;
pub(super) struct Tu102<E: FalconEngine>(PhantomData<E>);
/// Writes the trigger register available on GA100 and more recent.
pub(super) fn retrigger_ga100<E: FalconEngine>(falcon: &Falcon<'_, E>) {
falcon.pfalcon.write(
Array::at(0),
regs::NV_PFALCON_FALCON_INTR_RETRIGGER::zeroed().with_trigger(true),
);
}
/// The falcon HAL for Turing and GA100. GA100 boots like a Turing, but unlike Turing, it has a
/// retrigger register.
pub(super) struct Tu102<E: FalconEngine> {
/// If `true`, the falcons have `NV_PFALCON_FALCON_INTR_RETRIGGER`.
has_intr_retrigger: bool,
_engine: PhantomData<E>,
}
impl<E: FalconEngine> Tu102<E> {
pub(super) fn new() -> Self {
Self(PhantomData)
/// Creates a new HAL.
///
/// `has_intr_retrigger` indicates whether the retrigger register can be used.
pub(super) fn new(has_intr_retrigger: bool) -> Self {
Self {
has_intr_retrigger,
_engine: PhantomData,
}
}
}
@ -50,8 +71,8 @@ fn program_brom(&self, _falcon: &Falcon<'_, E>, _params: &FalconBromParams) {}
fn is_riscv_active(&self, falcon: &Falcon<'_, E>) -> bool {
falcon
.bar
.read(regs::NV_PRISCV_RISCV_CORE_SWITCH_RISCV_STATUS::of::<E>())
.pfalcon2
.read(regs::NV_PRISCV_RISCV_CORE_SWITCH_RISCV_STATUS)
.active_stat()
}
@ -62,7 +83,7 @@ fn is_riscv_halted(&self, _falcon: &Falcon<'_, E>) -> Result<bool> {
fn reset_wait_mem_scrubbing(&self, falcon: &Falcon<'_, E>) -> Result {
// TIMEOUT: memory scrubbing should complete in less than 10ms.
read_poll_timeout(
|| Ok(falcon.bar.read(regs::NV_PFALCON_FALCON_DMACTL::of::<E>())),
|| Ok(falcon.pfalcon.read(regs::NV_PFALCON_FALCON_DMACTL)),
|r| r.mem_scrubbing_done(),
Delta::ZERO,
Delta::from_millis(10),
@ -71,7 +92,7 @@ fn reset_wait_mem_scrubbing(&self, falcon: &Falcon<'_, E>) -> Result {
}
fn reset_eng(&self, falcon: &Falcon<'_, E>) -> Result {
regs::NV_PFALCON_FALCON_ENGINE::reset_engine::<E>(falcon.bar);
regs::NV_PFALCON_FALCON_ENGINE::reset_engine(falcon.pfalcon);
self.reset_wait_mem_scrubbing(falcon)?;
Ok(())
@ -80,4 +101,22 @@ fn reset_eng(&self, falcon: &Falcon<'_, E>) -> Result {
fn load_method(&self) -> LoadMethod {
LoadMethod::Pio
}
fn host_routed_causes(
&self,
falcon: &Falcon<'_, E>,
latched: regs::NV_PFALCON_FALCON_IRQSTAT,
) -> regs::NV_PFALCON_FALCON_IRQSTAT {
let pfalcon2 = falcon.pfalcon2;
let mask = pfalcon2.read(regs::tu102::NV_PRISCV_RISCV_IRQMASK).value();
let dest = pfalcon2.read(regs::tu102::NV_PRISCV_RISCV_IRQDEST).value();
regs::NV_PFALCON_FALCON_IRQSTAT::from(latched.into_raw() & mask & dest)
}
fn retrigger(&self, falcon: &Falcon<'_, E>) {
if self.has_intr_retrigger {
retrigger_ga100(falcon);
}
}
}

37
drivers/gpu/nova-core/falcon/sec2.rs
View File

@ -1,22 +1,37 @@
// SPDX-License-Identifier: GPL-2.0
use kernel::io::register::RegisterBase;
use kernel::io::{
io_project,
register,
Mmio, //
};
use crate::falcon::{
FalconEngine,
PFalcon2Base,
PFalconBase, //
use crate::{
driver::{
Bar0,
NovaRegisters, //
},
falcon::FalconEngine, //
};
/// Type specifying the `Sec2` falcon engine. Cannot be instantiated.
pub(crate) struct Sec2(());
impl RegisterBase<PFalconBase> for Sec2 {
const BASE: usize = 0x00840000;
register! {
base: NovaRegisters;
PFALCON: super::PFalconRegisters @ 0x00840000;
PFALCON2: super::PFalcon2Registers @ 0x00841000;
}
impl RegisterBase<PFalcon2Base> for Sec2 {
const BASE: usize = 0x00841000;
}
impl FalconEngine for Sec2 {
#[inline]
fn pfalcon(io: Bar0<'_>) -> Mmio<'_, super::PFalconRegisters> {
io_project!(io, build: PFALCON)
}
impl FalconEngine for Sec2 {}
#[inline]
fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, super::PFalcon2Registers> {
io_project!(io, build: PFALCON2)
}
}

22
drivers/gpu/nova-core/fb.rs
View File

@ -49,7 +49,7 @@ pub(crate) struct SysmemFlush<'sys> {
device: &'sys device::Device,
bar: Bar0<'sys>,
/// Keep the page alive as long as we need it.
page: CoherentHandle,
page: CoherentHandle<'sys>,
}
impl<'sys> SysmemFlush<'sys> {
@ -159,7 +159,7 @@ pub(crate) struct FbRanges {
/// Memory area containing the GSP bootloader image.
pub(crate) boot: FbRange,
/// Memory area containing the GSP firmware image.
pub(crate) elf: FbRange,
pub(crate) fw_image: FbRange,
/// WPR2 heap.
pub(crate) wpr2_heap: FbRange,
/// WPR2 region range, starting with an instance of `GspFwWprMeta`.
@ -177,7 +177,7 @@ impl FbRanges {
pub(crate) fn new(
chipset: Chipset,
bar: Bar0<'_>,
gsp_fw: &GspFirmware,
gsp_fw: &GspFirmware<'_>,
vgpu_state: VgpuState,
) -> Result<Self> {
let hal = hal::fb_hal(chipset);
@ -234,25 +234,25 @@ pub(crate) fn new(
FbRange(bootloader_base..bootloader_base + bootloader_size)
};
let elf = {
const ELF_DOWN_ALIGN: Alignment = Alignment::new::<SZ_64K>();
let elf_size = u64::from_safe_cast(gsp_fw.size);
let elf_addr = (boot.start - elf_size).align_down(ELF_DOWN_ALIGN);
let fw_image = {
const FW_IMAGE_DOWN_ALIGN: Alignment = Alignment::new::<SZ_64K>();
let fw_image_size = u64::from_safe_cast(gsp_fw.size());
let fw_image_addr = (boot.start - fw_image_size).align_down(FW_IMAGE_DOWN_ALIGN);
FbRange(elf_addr..elf_addr + elf_size)
FbRange(fw_image_addr..fw_image_addr + fw_image_size)
};
let (vf_partition_count, wpr2_heap_size) = wpr2_heap_params(chipset, vgpu_state, fb.end)?;
let wpr2_heap = {
const WPR2_HEAP_DOWN_ALIGN: Alignment = Alignment::new::<SZ_1M>();
let wpr2_heap_addr = elf
let wpr2_heap_addr = fw_image
.start
.checked_sub(wpr2_heap_size)
.ok_or(EOVERFLOW)?
.align_down(WPR2_HEAP_DOWN_ALIGN);
FbRange(wpr2_heap_addr..(elf.start).align_down(WPR2_HEAP_DOWN_ALIGN))
FbRange(wpr2_heap_addr..(fw_image.start).align_down(WPR2_HEAP_DOWN_ALIGN))
};
let wpr2 = {
@ -273,7 +273,7 @@ pub(crate) fn new(
vga_workspace,
frts,
boot,
elf,
fw_image,
wpr2_heap,
wpr2,
non_wpr_heap,

59
drivers/gpu/nova-core/fb/hal/gb100.rs
View File

@ -5,11 +5,10 @@
use kernel::{
io::{
register::{
RegisterBase,
WithBase, //
},
Io, //
io_project,
register,
Io,
Mmio, //
},
num::Bounded,
prelude::*,
@ -21,7 +20,10 @@
};
use crate::{
driver::Bar0,
driver::{
Bar0,
NovaRegisters, //
},
fb::{
hal::FbHal,
regs, //
@ -31,17 +33,26 @@
struct Gb100;
impl RegisterBase<regs::Hshub0Base> for Gb100 {
const BASE: usize = 0x0087_0000;
register! {
base: NovaRegisters;
HSHUB0: regs::Hshub0Registers @ 0x0087_0000;
}
fn read_sysmem_flush_page_gb100(bar: Bar0<'_>) -> u64 {
#[inline]
fn hshub0(bar: Bar0<'_>) -> Mmio<'_, regs::Hshub0Registers> {
io_project!(bar, build: HSHUB0)
}
fn read_sysmem_flush_page_gb100(hshub0: Mmio<'_, regs::Hshub0Registers>) -> u64 {
let lo = u64::from(
bar.read(regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_LO::of::<Gb100>())
hshub0
.read(regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_LO)
.adr(),
);
let hi = u64::from(
bar.read(regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_HI::of::<Gb100>())
hshub0
.read(regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_HI)
.adr(),
);
@ -52,7 +63,7 @@ fn read_sysmem_flush_page_gb100(bar: Bar0<'_>) -> u64 {
///
/// Both the primary and EG (egress) register pairs must be programmed to the same address,
/// as required by hardware.
fn write_sysmem_flush_page_gb100(bar: Bar0<'_>, addr: Bounded<u64, 52>) {
fn write_sysmem_flush_page_gb100(hshub0: Mmio<'_, regs::Hshub0Registers>, addr: Bounded<u64, 52>) {
// CAST: lower 32 bits. Hardware ignores bits 7:0.
let addr_lo = *addr as u32;
let addr_hi = addr.shr::<32, 20>().cast::<u32>();
@ -60,24 +71,12 @@ fn write_sysmem_flush_page_gb100(bar: Bar0<'_>, addr: Bounded<u64, 52>) {
// Write HI first. The hardware will trigger the flush on the LO write.
// Primary HSHUB pair.
bar.write(
regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_HI::of::<Gb100>(),
regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_HI::zeroed().with_adr(addr_hi),
);
bar.write(
regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_LO::of::<Gb100>(),
regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_LO::zeroed().with_adr(addr_lo),
);
hshub0.write_reg(regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_HI::zeroed().with_adr(addr_hi));
hshub0.write_reg(regs::NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_LO::zeroed().with_adr(addr_lo));
// EG (egress) pair -- must match the primary pair.
bar.write(
regs::NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_HI::of::<Gb100>(),
regs::NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_HI::zeroed().with_adr(addr_hi),
);
bar.write(
regs::NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_LO::of::<Gb100>(),
regs::NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_LO::zeroed().with_adr(addr_lo),
);
hshub0.write_reg(regs::NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_HI::zeroed().with_adr(addr_hi));
hshub0.write_reg(regs::NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_LO::zeroed().with_adr(addr_lo));
}
// This PMU reservation size is r570-specific.
@ -88,13 +87,13 @@ pub(super) const fn pmu_reserved_size_gb100() -> u32 {
impl FbHal for Gb100 {
fn read_sysmem_flush_page(&self, bar: Bar0<'_>) -> u64 {
read_sysmem_flush_page_gb100(bar)
read_sysmem_flush_page_gb100(hshub0(bar))
}
fn write_sysmem_flush_page(&self, bar: Bar0<'_>, addr: u64) -> Result {
let addr = Bounded::<u64, 52>::try_new(addr).ok_or(EINVAL)?;
write_sysmem_flush_page_gb100(bar, addr);
write_sysmem_flush_page_gb100(hshub0(bar), addr);
Ok(())
}

34
drivers/gpu/nova-core/fb/regs.rs
View File

@ -2,12 +2,20 @@
use kernel::{
io::register,
sizes::SizeConstants, //
prelude::*,
sizes::{
SizeConstants,
SZ_4K, //
}, //
};
use crate::driver::NovaRegisters;
// PDISP
register! {
base: NovaRegisters;
pub(super) NV_PDISP_VGA_WORKSPACE_BASE(u32) @ 0x00625f04 {
/// VGA workspace base address divided by 0x10000.
31:8 addr;
@ -30,6 +38,8 @@ pub(super) fn vga_workspace_addr(self) -> Option<u64> {
// PFB
register! {
base: NovaRegisters;
/// Low bits of the physical system memory address used by the GPU to perform sysmembar
/// operations (see [`crate::fb::SysmemFlush`]).
pub(super) NV_PFB_NISO_FLUSH_SYSMEM_ADDR(u32) @ 0x00100c10 {
@ -59,34 +69,42 @@ pub(super) fn vga_workspace_addr(self) -> Option<u64> {
}
}
/// Base of the GB10x HSHUB0 register window (`NV_HSHUB0_PRIV_BASE` in Open RM).
const HSHUB0_REGION_SIZE: usize = SZ_4K;
/// The GB10x HSHUB0 register window (Base defined as `NV_HSHUB0_PRIV_BASE` in Open RM).
///
/// The base is provided by the GB10x framebuffer HAL.
pub(super) struct Hshub0Base(());
#[repr(align(4))]
#[derive(FromBytes, IntoBytes)]
pub(super) struct Hshub0Registers([u8; HSHUB0_REGION_SIZE]);
register! {
base: Hshub0Registers;
// GB10x sysmem flush registers, relative to the HSHUB0 base. GB10x routes sysmembar
// through a primary and an EG (egress) pair that must both be programmed to the same
// address. Hardware ignores bits 7:0 of each LO register. The boot path uses a fixed
// HSHUB0 base, so the multiple runtime-discovered HSHUB bases are not needed here.
pub(super) NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_LO(u32) @ Hshub0Base + 0x00000e50 {
pub(super) NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_LO(u32) @ 0x00000e50 {
31:0 adr => u32;
}
pub(super) NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_HI(u32) @ Hshub0Base + 0x00000e54 {
pub(super) NV_PFB_HSHUB_PCIE_FLUSH_SYSMEM_ADDR_HI(u32) @ 0x00000e54 {
19:0 adr;
}
pub(super) NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_LO(u32) @ Hshub0Base + 0x000006c0 {
pub(super) NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_LO(u32) @ 0x000006c0 {
31:0 adr => u32;
}
pub(super) NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_HI(u32) @ Hshub0Base + 0x000006c4 {
pub(super) NV_PFB_HSHUB_EG_PCIE_FLUSH_SYSMEM_ADDR_HI(u32) @ 0x000006c4 {
19:0 adr;
}
}
register! {
base: NovaRegisters;
// GB20x FBHUB0 sysmem flush registers. Unlike the older
// NV_PFB_NISO_FLUSH_SYSMEM_ADDR registers, which encode the address with an
// 8-bit right-shift, these take the raw address split into lower and upper
@ -101,6 +119,8 @@ pub(super) fn vga_workspace_addr(self) -> Option<u64> {
}
register! {
base: NovaRegisters;
/// Low bits of the physical system memory address used by the GPU to perform
/// sysmembar operations on Hopper.
///

3
drivers/gpu/nova-core/firmware.rs
View File

@ -23,9 +23,10 @@
};
pub(crate) mod booter;
pub(crate) mod fsp;
pub(crate) mod fwsec;
pub(crate) mod gsp;
pub(crate) mod gsp_fmc;
pub(crate) mod radix3;
pub(crate) mod riscv;
pub(crate) mod tlv;

2
drivers/gpu/nova-core/firmware/booter.rs
View File

@ -186,7 +186,7 @@ pub(crate) fn run<T>(
&self,
dev: &device::Device<device::Bound>,
sec2_falcon: &Falcon<'_, Sec2>,
wpr_meta: &Coherent<T>,
wpr_meta: &Coherent<'_, T>,
) -> Result {
sec2_falcon.reset()?;
sec2_falcon.load(self)?;

30
drivers/gpu/nova-core/firmware/fwsec/bootloader.rs
View File

@ -12,7 +12,10 @@
Device, //
},
dma::Coherent,
io::{register::WithBase, Io},
io::{
register::Array,
Io, //
},
prelude::*,
ptr::{
Alignable,
@ -23,7 +26,6 @@
};
use crate::{
driver::Bar0,
falcon::{
self,
gsp::Gsp,
@ -98,9 +100,9 @@ unsafe impl AsBytes for BootloaderDmemDescV2 {}
/// Wrapper for [`FwsecFirmware`] that includes the bootloader performing the actual load
/// operation.
pub(crate) struct FwsecFirmwareWithBl {
pub(crate) struct FwsecFirmwareWithBl<'a> {
/// DMA object the bootloader will copy the firmware from.
_firmware_dma: Coherent<[u8]>,
_firmware_dma: Coherent<'a, [u8]>,
/// Code of the bootloader to be loaded into non-secure IMEM.
ucode: KVec<u8>,
/// Descriptor to be loaded into DMEM for the bootloader to read.
@ -113,12 +115,12 @@ pub(crate) struct FwsecFirmwareWithBl {
start_tag: u16,
}
impl FwsecFirmwareWithBl {
impl<'a> FwsecFirmwareWithBl<'a> {
/// Loads the bootloader firmware for `dev` and `chipset`, and wrap `firmware` so it can be
/// loaded using it.
pub(crate) fn new(
firmware: FwsecFirmware,
dev: &Device<device::Bound>,
dev: &'a Device<device::Bound>,
chipset: Chipset,
) -> Result<Self> {
let fw = request_tlv(dev, chipset, "gen_bootloader")?;
@ -235,12 +237,7 @@ pub(crate) fn new(
///
/// The bootloader will load the FWSEC firmware and then execute it. This function returns
/// after FWSEC has reached completion.
pub(crate) fn run(
&self,
dev: &Device<device::Bound>,
falcon: &Falcon<'_, Gsp>,
bar: Bar0<'_>,
) -> Result<()> {
pub(crate) fn run(&self, dev: &Device<device::Bound>, falcon: &Falcon<'_, Gsp>) -> Result<()> {
// Reset falcon, load the firmware, and run it.
falcon
.reset()
@ -250,9 +247,8 @@ pub(crate) fn run(
.inspect_err(|e| dev_err!(dev, "Failed to load FWSEC firmware: {:?}\n", e))?;
// Configure DMA index for the bootloader to fetch the FWSEC firmware from system memory.
bar.update(
regs::NV_PFALCON_FBIF_TRANSCFG::of::<Gsp>()
.try_at(usize::from_safe_cast(self.dmem_desc.ctx_dma))
falcon.pfalcon.update(
regs::NV_PFALCON_FBIF_TRANSCFG::try_at(usize::from_safe_cast(self.dmem_desc.ctx_dma))
.ok_or(EINVAL)?,
|v| {
v.with_target(FalconFbifTarget::CoherentSysmem)
@ -272,7 +268,7 @@ pub(crate) fn run(
}
}
impl FalconFirmware for FwsecFirmwareWithBl {
impl FalconFirmware for FwsecFirmwareWithBl<'_> {
type Target = Gsp;
fn brom_params(&self) -> FalconBromParams {
@ -286,7 +282,7 @@ fn boot_addr(&self) -> u32 {
}
}
impl FalconPioLoadable for FwsecFirmwareWithBl {
impl FalconPioLoadable for FwsecFirmwareWithBl<'_> {
fn imem_sec_load_params(&self) -> Option<FalconPioImemLoadTarget<'_>> {
None
}

119
drivers/gpu/nova-core/firmware/gsp.rs
View File

@ -4,21 +4,16 @@
device,
dma::{
Coherent,
CoherentBox,
DataDirection,
DmaAddress, //
},
firmware,
prelude::*,
scatterlist::{
Owned,
SGTable, //
},
str::CString,
};
use crate::{
firmware::{
radix3::Radix3,
riscv::RiscvFirmware, //
tlv::{
request_tlv, //
@ -26,48 +21,25 @@
},
},
gpu::Chipset,
gsp::GSP_PAGE_SIZE,
num::FromSafeCast,
};
/// GSP firmware with 3-level radix page tables for the GSP bootloader.
///
/// The bootloader expects firmware to be mapped starting at address 0 in GSP's virtual address
/// space:
///
/// ```text
/// Level 0: 1 page, 1 entry -> points to first level 1 page
/// Level 1: Multiple pages/entries -> each entry points to a level 2 page
/// Level 2: Multiple pages/entries -> each entry points to a firmware page
/// ```
///
/// Each page is 4KB, each entry is 8 bytes (64-bit DMA address).
/// Also known as "Radix3" firmware.
/// The GSP firmware image, its signatures, and the GSP bootloader.
#[pin_data]
pub(crate) struct GspFirmware {
/// The GSP firmware inside a [`VVec`], device-mapped via a SG table.
pub(crate) struct GspFirmware<'a> {
/// The firmware image and the radix3 table that maps it.
#[pin]
fw: SGTable<Owned<VVec<u8>>>,
/// Level 2 page table whose entries contain DMA addresses of firmware pages.
#[pin]
level2: SGTable<Owned<VVec<u8>>>,
/// Level 1 page table whose entries contain DMA addresses of level 2 pages.
#[pin]
level1: SGTable<Owned<VVec<u8>>>,
/// Level 0 page table (single 4KB page) with one entry: DMA address of first level 1 page.
level0: Coherent<[u64]>,
/// Size in bytes of the firmware contained in [`Self::fw`].
pub(crate) size: usize,
radix3: Radix3<'a>,
/// Device-mapped GSP signatures matching the GPU's [`Chipset`].
pub(crate) signatures: Coherent<[u8]>,
pub(crate) signatures: Coherent<'a, [u8]>,
/// GSP bootloader, verifies the GSP firmware before loading and running it.
pub(crate) bootloader: RiscvFirmware,
pub(crate) bootloader: RiscvFirmware<'a>,
}
impl GspFirmware {
impl<'a> GspFirmware<'a> {
/// Loads the GSP firmware binaries, map them into `dev`'s address-space, and creates the page
/// tables expected by the GSP bootloader to load it.
pub(crate) fn new<'a>(
pub(crate) fn new(
dev: &'a device::Device<device::Bound>,
chipset: Chipset,
) -> impl PinInit<Self, Error> + 'a {
@ -87,49 +59,7 @@ pub(crate) fn new<'a>(
let signatures = Coherent::from_slice(dev, tlv.get_bytes(b"SIGN")?, GFP_KERNEL)?;
Ok(try_pin_init!(Self {
fw <- SGTable::new(dev, fw_vvec, DataDirection::ToDevice, GFP_KERNEL),
level2 <- {
// Allocate the level 2 page table, map the firmware onto it, and map it into
// the device address space.
VVec::<u8>::with_capacity(
fw.iter().count() * core::mem::size_of::<u64>(),
GFP_KERNEL,
)
.map_err(|_| ENOMEM)
.and_then(|level2| map_into_lvl(&fw, level2))
.map(|level2| SGTable::new(dev, level2, DataDirection::ToDevice, GFP_KERNEL))?
},
level1 <- {
// Allocate the level 1 page table, map the level 2 page table onto it, and map
// it into the device address space.
VVec::<u8>::with_capacity(
level2.iter().count() * core::mem::size_of::<u64>(),
GFP_KERNEL,
)
.map_err(|_| ENOMEM)
.and_then(|level1| map_into_lvl(&level2, level1))
.map(|level1| SGTable::new(dev, level1, DataDirection::ToDevice, GFP_KERNEL))?
},
level0: {
// Allocate the level 0 page table as a device-visible DMA object, and map the
// level 1 page table onto it.
// Fill level 1 page entry.
let level1_entry = level1.iter().next().ok_or(EINVAL)?;
let level1_entry_addr = level1_entry.dma_address();
// Create level 0 page table data and fill its first entry with the level 1
// table.
let mut level0 = CoherentBox::<[u64]>::zeroed_slice(
dev,
GSP_PAGE_SIZE / size_of::<u64>(),
GFP_KERNEL
)?;
level0[0] = level1_entry_addr.to_le();
level0.into()
},
size,
radix3 <- Radix3::new(dev, fw_vvec),
signatures,
bootloader: {
let bl = request_tlv(dev, chipset, "gsp_bootloader")?;
@ -140,28 +70,13 @@ pub(crate) fn new<'a>(
})
}
/// Returns the DMA address of the radix3 level 0 page table.
/// Returns the size of the firmware image, in bytes.
pub(crate) fn size(&self) -> usize {
self.radix3.size()
}
/// Returns the DMA address of the radix3 table that maps the firmware image.
pub(crate) fn radix3_dma_address(&self) -> DmaAddress {
self.level0.dma_address()
self.radix3.dma_address()
}
}
/// Build a page table from a scatter-gather list.
///
/// Takes each DMA-mapped region from `sg_table` and writes page table entries
/// for all 4KB pages within that region. For example, a 16KB SG entry becomes
/// 4 consecutive page table entries.
fn map_into_lvl(sg_table: &SGTable<Owned<VVec<u8>>>, mut dst: VVec<u8>) -> Result<VVec<u8>> {
for sg_entry in sg_table.iter() {
// Number of pages we need to map.
let num_pages = usize::from_safe_cast(sg_entry.dma_len()).div_ceil(GSP_PAGE_SIZE);
for i in 0..num_pages {
let entry = sg_entry.dma_address()
+ (u64::from_safe_cast(i) * u64::from_safe_cast(GSP_PAGE_SIZE));
dst.extend_from_slice(&entry.to_le_bytes(), GFP_KERNEL)?;
}
}
Ok(dst)
}

59
drivers/gpu/nova-core/firmware/fsp.rs → drivers/gpu/nova-core/firmware/gsp_fmc.rs
View File

@ -1,7 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! FSP is a hardware unit that runs FMC firmware.
//! GSP-FMC (First Mutable Code) is loaded by FSP into GSP to serve as the loader and verifier of
//! GSP-RM.
use kernel::{
device,
@ -17,16 +18,16 @@
gpu::Chipset, //
};
/// Size of the FSP SHA-384 hash, in bytes.
const FSP_HASH_SIZE: usize = 48;
/// Maximum size of the FSP public key (RSA-3072), in bytes.
/// Size of the GSP-FMC SHA-384 hash, in bytes.
const FMC_HASH_SIZE: usize = 48;
/// Maximum size of the GSP-FMC public key (RSA-3072), in bytes.
///
/// The FMC `PKEY` tag may be shorter, so the remaining bytes are zero-padded.
const FSP_PKEY_SIZE: usize = 384;
/// Maximum size of the FSP signature (RSA-3072), in bytes.
/// The `PKEY` tag may be shorter, so the remaining bytes are zero-padded.
const FMC_PKEY_SIZE: usize = 384;
/// Maximum size of the GSP-FMC signature (RSA-3072), in bytes.
///
/// The FMC `SIGN` tag may be shorter, so the remaining bytes are zero-padded.
const FSP_SIG_SIZE: usize = 384;
/// The `SIGN` tag may be shorter, so the remaining bytes are zero-padded.
const FMC_SIG_SIZE: usize = 384;
/// Structure to hold FMC signatures.
///
@ -34,23 +35,27 @@
#[derive(Debug, Clone, Copy, Zeroable)]
#[repr(C)]
pub(crate) struct FmcSignatures {
pub(crate) hash384: [u8; FSP_HASH_SIZE],
pub(crate) public_key: [u8; FSP_PKEY_SIZE],
pub(crate) signature: [u8; FSP_SIG_SIZE],
pub(crate) hash384: [u8; FMC_HASH_SIZE],
pub(crate) public_key: [u8; FMC_PKEY_SIZE],
pub(crate) signature: [u8; FMC_SIG_SIZE],
}
pub(crate) struct FspFirmware {
pub(crate) struct GspFmcFirmware<'a> {
/// FMC firmware image data
pub(crate) fmc_image: Coherent<[u8]>,
pub(crate) fmc_image: Coherent<'a, [u8]>,
/// FMC firmware signatures.
pub(crate) fmc_sigs: KBox<FmcSignatures>,
}
impl FspFirmware {
pub(crate) fn new(dev: &device::Device<device::Bound>, chipset: Chipset) -> Result<Self> {
impl<'a> GspFmcFirmware<'a> {
pub(crate) fn new(dev: &'a device::Device<device::Bound>, chipset: Chipset) -> Result<Self> {
let fw = request_tlv(dev, chipset, "fmc")?;
let tlv = Tlv::new(fw.data())?;
dev_dbg!(dev, "loaded fsp firmware v{}\n", tlv.get_string(b"VERS")?);
dev_dbg!(
dev,
"loaded GSP-FMC firmware v{}\n",
tlv.get_string(b"VERS")?
);
let fmc_image_data = tlv.get_bytes(b"BLOB")?;
let fmc_image = Coherent::from_slice(dev, fmc_image_data, GFP_KERNEL)?;
@ -70,34 +75,34 @@ fn extract_fmc_signatures(tlv: &Tlv<'_>, dev: &device::Device) -> Result<KBox<Fm
let pkey_section = tlv.get_bytes(b"PKEY")?;
let sig_section = tlv.get_bytes(b"SIGN")?;
// The hash section is a SHA-384 output: it must be exactly FSP_HASH_SIZE bytes.
if hash_section.len() != FSP_HASH_SIZE {
// The hash section is a SHA-384 output: it must be exactly `FMC_HASH_SIZE` bytes.
if hash_section.len() != FMC_HASH_SIZE {
dev_err!(
dev,
"FMC hash section size {} != expected {}\n",
hash_section.len(),
FSP_HASH_SIZE
FMC_HASH_SIZE
);
return Err(EINVAL);
}
// The key and signature sections are zero-padded to a fixed maximum, so they may be
// shorter, but must not exceed the destination buffers.
if pkey_section.len() > FSP_PKEY_SIZE {
if pkey_section.len() > FMC_PKEY_SIZE {
dev_err!(
dev,
"FMC public key section size {} > maximum {}\n",
pkey_section.len(),
FSP_PKEY_SIZE
FMC_PKEY_SIZE
);
return Err(EINVAL);
}
if sig_section.len() > FSP_SIG_SIZE {
if sig_section.len() > FMC_SIG_SIZE {
dev_err!(
dev,
"FMC signature section size {} > maximum {}\n",
sig_section.len(),
FSP_SIG_SIZE
FMC_SIG_SIZE
);
return Err(EINVAL);
}
@ -106,11 +111,11 @@ fn extract_fmc_signatures(tlv: &Tlv<'_>, dev: &device::Device) -> Result<KBox<Fm
// stack, then fill each section from the firmware.
let signatures = KBox::init(
pin_init::init_zeroed::<FmcSignatures>().chain(|sigs| {
// PANIC: src and dst lengths are both FSP_HASH_SIZE (verified above).
// PANIC: src and dst lengths are both `FMC_HASH_SIZE` (verified above).
sigs.hash384.copy_from_slice(hash_section);
// PANIC: dst is sliced to src.len(); src.len() <= FSP_PKEY_SIZE (verified above).
// PANIC: dst is sliced to src.len(); src.len() <= `FMC_PKEY_SIZE` (verified above).
sigs.public_key[..pkey_section.len()].copy_from_slice(pkey_section);
// PANIC: dst is sliced to src.len(); src.len() <= FSP_SIG_SIZE (verified above).
// PANIC: dst is sliced to src.len(); src.len() <= `FMC_SIG_SIZE` (verified above).
sigs.signature[..sig_section.len()].copy_from_slice(sig_section);
Ok(())
}),

144
drivers/gpu/nova-core/firmware/radix3.rs Normal file
View File

@ -0,0 +1,144 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! The radix3 page table, through which firmware running on the GPU reads a buffer in system
//! memory. LibOS, the operating system of the GSP, defines the format: three levels of
//! [`GSP_PAGE_SIZE`] pages, each entry the little-endian DMA address of one page.
//!
//! ```text
//! Level 0: one page, one entry -> the first level 1 page
//! Level 1: pages of entries -> each entry a level 2 page
//! Level 2: pages of entries -> each entry a page of the buffer
//! ```
use kernel::{
device,
dma::{
Coherent,
CoherentBox,
DataDirection,
DmaAddress, //
},
prelude::*,
scatterlist::{
Owned,
SGTable, //
},
};
use crate::{
gsp::GSP_PAGE_SIZE,
num::FromSafeCast, //
};
/// A radix3 page table and the data it maps.
#[pin_data]
pub(crate) struct Radix3<'a> {
/// The mapped data.
#[pin]
data: SGTable<Owned<VVec<u8>>>,
/// Level 2: one entry per page of `data`.
#[pin]
level2: SGTable<Owned<VVec<u8>>>,
/// Level 1: one entry per page of `level2`.
#[pin]
level1: SGTable<Owned<VVec<u8>>>,
/// Level 0: one page, whose single entry is the DMA address of the first `level1` page.
level0: Coherent<'a, [u64]>,
/// Length of `data`, in bytes.
size: usize,
}
impl<'a> Radix3<'a> {
/// Builds a radix3 page table over `data`, mapped for `dev` to read. May sleep.
pub(crate) fn new(
dev: &'a device::Device<device::Bound>,
data: VVec<u8>,
) -> impl PinInit<Self, Error> + 'a {
let size = data.len();
pin_init::pin_init_scope(move || {
Ok(try_pin_init!(Self {
data <- SGTable::new(dev, data, DataDirection::ToDevice, GFP_KERNEL),
level2 <- {
build_lvl(&data)
.map(|l2| SGTable::new(dev, l2, DataDirection::ToDevice, GFP_KERNEL))?
},
level1 <- {
build_lvl(&level2)
.map(|l1| SGTable::new(dev, l1, DataDirection::ToDevice, GFP_KERNEL))?
},
level0: {
// Allocate the level 0 page table as a device-visible DMA object, and map the
// level 1 page table onto it.
// Fill level 1 page entry.
let level1_entry = level1.iter().next().ok_or(EINVAL)?;
let level1_entry_addr = level1_entry.dma_address();
// Create level 0 page table data and fill its first entry with the level 1
// table.
let mut level0 = CoherentBox::<'_, [u64]>::zeroed_slice(
dev,
GSP_PAGE_SIZE / size_of::<u64>(),
GFP_KERNEL
)?;
level0[0] = level1_entry_addr.to_le();
level0.into()
},
size,
}))
})
}
/// Returns the DMA address of the level 0 page, which is the address of the table.
pub(crate) fn dma_address(&self) -> DmaAddress {
self.level0.dma_address()
}
/// Returns the length of the mapped data, in bytes.
pub(crate) fn size(&self) -> usize {
self.size
}
}
/// Returns the size of the level that maps `sg_table`: one `u64` entry per [`GSP_PAGE_SIZE`]
/// page of each DMA-mapped region, rounded up to whole pages.
fn lvl_size(sg_table: &SGTable<Owned<VVec<u8>>>) -> usize {
let entries: usize = sg_table
.iter()
.map(|sg_entry| usize::from_safe_cast(sg_entry.dma_len()).div_ceil(GSP_PAGE_SIZE))
.sum();
(entries * size_of::<u64>()).next_multiple_of(GSP_PAGE_SIZE)
}
/// Builds one level of the table over `sg_table`: one entry per [`GSP_PAGE_SIZE`] page of each
/// DMA-mapped region, in region order. The level is a whole number of pages and every byte past
/// the last entry is zero, because the booter reads a level a whole page at a time.
///
/// # Errors
///
/// - `ENOMEM` if the level cannot be allocated.
/// - `EINVAL` if `sg_table` yields more entries than [`lvl_size`] sized the level for.
fn build_lvl(sg_table: &SGTable<Owned<VVec<u8>>>) -> Result<VVec<u8>> {
let mut dst = VVec::<u8>::zeroed(lvl_size(sg_table), GFP_KERNEL).map_err(|_| ENOMEM)?;
let mut entries = dst.chunks_exact_mut(size_of::<u64>());
for sg_entry in sg_table.iter() {
let num_pages = usize::from_safe_cast(sg_entry.dma_len()).div_ceil(GSP_PAGE_SIZE);
for i in 0..num_pages {
let entry = sg_entry.dma_address()
+ (u64::from_safe_cast(i) * u64::from_safe_cast(GSP_PAGE_SIZE));
entries
.next()
.ok_or(EINVAL)?
.copy_from_slice(&entry.to_le_bytes());
}
}
Ok(dst)
}

8
drivers/gpu/nova-core/firmware/riscv.rs
View File

@ -13,7 +13,7 @@
use crate::firmware::tlv::Tlv;
/// A parsed firmware for a RISC-V core, ready to be loaded and run.
pub(crate) struct RiscvFirmware {
pub(crate) struct RiscvFirmware<'a> {
/// Offset at which the code starts in the firmware image.
pub(crate) code_offset: u32,
/// Offset at which the data starts in the firmware image.
@ -23,12 +23,12 @@ pub(crate) struct RiscvFirmware {
/// Application version.
pub(crate) app_version: u32,
/// Device-mapped firmware image.
pub(crate) ucode: Coherent<[u8]>,
pub(crate) ucode: Coherent<'a, [u8]>,
}
impl RiscvFirmware {
impl<'a> RiscvFirmware<'a> {
/// Parses the RISC-V firmware image contained in `fw`.
pub(crate) fn new(dev: &device::Device<device::Bound>, fw: &Firmware) -> Result<Self> {
pub(crate) fn new(dev: &'a device::Device<device::Bound>, fw: &Firmware) -> Result<Self> {
let tlv = Tlv::new(fw.data())?;
dev_dbg!(
dev,

45
drivers/gpu/nova-core/fsp.rs
View File

@ -3,9 +3,12 @@
//! FSP (Foundation Security Processor) interface for Hopper/Blackwell GPUs.
//!
//! Hopper/Blackwell use a simplified firmware boot sequence: FMC, then FSP, then GSP.
//! Hopper/Blackwell use a simplified firmware boot sequence: FSP secure-boots independently before
//! the driver starts. The driver then sends FSP a Chain-of-Trust request containing the GSP-FMC
//! image. FSP authenticates the image and launches GSP-FMC on the GSP RISC-V core; GSP-FMC
//! subsequently authenticates and boots GSP-RM.
//!
//! Unlike Turing/Ampere/Ada, there is no SEC2 (Security Engine 2) usage.
//! FSP handles secure boot directly using FMC firmware and Chain of Trust.
use kernel::{
device,
@ -32,9 +35,9 @@
Falcon, //
},
fb::FbSizes,
firmware::fsp::{
firmware::gsp_fmc::{
FmcSignatures,
FspFirmware, //
GspFmcFirmware, //
},
gpu::Chipset,
gsp::{
@ -267,7 +270,7 @@ fn frts_vidmem_offset(hal: &dyn hal::FspHal, fb_info: &FbSizes) -> Result<u64> {
/// Returns an in-place initializer for [`FspCotMessage`].
fn new<'a>(
fb_info: &FbSizes,
fsp_fw: &'a FspFirmware,
fmc_fw: &'a GspFmcFirmware<'_>,
args: &'a FmcBootArgs<'_>,
) -> Result<impl Init<Self> + 'a> {
let hal = hal::fsp_hal(args.chipset).ok_or(ENOTSUPP)?;
@ -296,13 +299,13 @@ fn new<'a>(
.chain(move |msg| {
msg.cot.version = version;
msg.cot.size = size;
msg.cot.gsp_fmc_sysmem_offset = fsp_fw.fmc_image.dma_address();
msg.cot.gsp_fmc_sysmem_offset = fmc_fw.fmc_image.dma_address();
msg.cot.frts_vidmem_offset = frts_vidmem_offset;
msg.cot.frts_vidmem_size = frts_size;
// frts_sysmem_* are left at zero because this path places FRTS in vidmem. The sysmem
// fields point to an FRTS buffer in sysmem instead, for systems without VRAM.
msg.cot.gsp_boot_args_sysmem_offset = args.fmc_boot_params.dma_address();
msg.cot.sigs = *fsp_fw.fmc_sigs;
msg.cot.sigs = *fmc_fw.fmc_sigs;
Ok(())
}))
@ -345,28 +348,28 @@ impl MessageToFsp for FspPrcMessage {
/// Bundled arguments for FMC boot via FSP Chain of Trust.
pub(crate) struct FmcBootArgs<'a> {
chipset: Chipset,
fmc_boot_params: Coherent<GspFmcBootParams>,
fmc_boot_params: Coherent<'a, GspFmcBootParams>,
resume: bool,
// Additional dependencies required to be kept alive for FMC boot.
_wpr_meta: Coherent<GspFwWprMeta>,
_libos: &'a Coherent<[LibosMemoryRegionInitArgument]>,
_wpr_meta: Coherent<'a, GspFwWprMeta>,
_libos: &'a Coherent<'a, [LibosMemoryRegionInitArgument]>,
}
impl<'a> FmcBootArgs<'a> {
/// Builds FMC boot arguments, allocating the DMA-coherent boot parameter
/// structure that FSP will read.
pub(crate) fn new(
dev: &device::Device<device::Bound>,
dev: &'a device::Device<device::Bound>,
chipset: Chipset,
wpr_meta: Coherent<GspFwWprMeta>,
libos: &'a Coherent<[LibosMemoryRegionInitArgument]>,
wpr_meta: Coherent<'a, GspFwWprMeta>,
libos: &'a Coherent<'a, [LibosMemoryRegionInitArgument]>,
resume: bool,
) -> Result<Self> {
let init = GspFmcBootParams::new(wpr_meta.dma_address(), libos.dma_address());
Ok(Self {
chipset,
fmc_boot_params: Coherent::<GspFmcBootParams>::init(dev, GFP_KERNEL, init)?,
fmc_boot_params: Coherent::init(dev, GFP_KERNEL, init)?,
resume,
_wpr_meta: wpr_meta,
_libos: libos,
@ -374,19 +377,19 @@ pub(crate) fn new(
}
/// Returns the FMC boot parameters allocation.
pub(crate) fn boot_params(&self) -> &Coherent<GspFmcBootParams> {
pub(crate) fn boot_params(&self) -> &Coherent<'_, GspFmcBootParams> {
&self.fmc_boot_params
}
}
/// FSP interface for Hopper/Blackwell GPUs.
///
/// An `Fsp` is produced by [`Fsp::wait_secure_boot`], which only returns once FSP secure boot
/// has completed. It owns the FSP falcon and the FMC firmware, which are used for the subsequent
/// An `Fsp` is produced by [`Fsp::wait_secure_boot`], which only returns once FSP secure boot has
/// completed. It owns the FSP falcon and the GSP-FMC firmware, which are used for the subsequent
/// Chain of Trust boot.
pub(crate) struct Fsp<'a> {
falcon: Falcon<'a, FspEngine>,
fsp_fw: FspFirmware,
fmc_fw: GspFmcFirmware<'a>,
}
impl<'a> Fsp<'a> {
@ -422,7 +425,7 @@ fn wait_secure_boot(
const FSP_SECURE_BOOT_TIMEOUT_MS: i64 = 5000;
let falcon = Falcon::<FspEngine>::new(dev, chipset, bar)?;
let fsp_fw = FspFirmware::new(dev, chipset)?;
let fmc_fw = GspFmcFirmware::new(dev, chipset)?;
read_poll_timeout(
|| Ok(hal.fsp_boot_status(bar)),
@ -434,7 +437,7 @@ fn wait_secure_boot(
dev_err!(dev, "FSP secure boot completion error: {:?}\n", e);
})?;
Ok(Fsp { falcon, fsp_fw })
Ok(Fsp { falcon, fmc_fw })
}
/// Sends a message to FSP and waits for the response.
@ -540,7 +543,7 @@ pub(crate) fn boot_fmc(
) -> Result {
dev_dbg!(dev, "Starting FSP boot sequence for {}\n", args.chipset);
let msg = KBox::init(FspCotMessage::new(fb_info, &self.fsp_fw, args)?, GFP_KERNEL)?;
let msg = KBox::init(FspCotMessage::new(fb_info, &self.fmc_fw, args)?, GFP_KERNEL)?;
let _response_buf = self.send_sync_fsp(dev, &*msg)?;

354
drivers/gpu/nova-core/gpu.rs
View File

@ -6,16 +6,25 @@
device,
dma::Device,
fmt,
gpu::buddy::GpuBuddyParams,
io::Io,
num::Bounded,
pci,
prelude::*,
sizes::SizeConstants, //
ptr::Alignment,
sizes::{
SizeConstants,
SZ_4K, //
},
uapi, //
};
use crate::{
bounded_enum,
driver::Bar0,
driver::{
Bar0,
Bar1, //
},
falcon::{
gsp::Gsp as GspFalcon,
sec2::Sec2 as Sec2Falcon,
@ -25,31 +34,49 @@
fsp::Fsp,
gsp::{
self,
cmdq::Cmdq,
commands::GetGspStaticInfoReply,
Gsp,
GspBootContext, //
},
regs,
irq::{
self,
gsp::GspIrq,
interrupt_tree::{
TopEnableGuard,
Tree, //
}, //
},
mm::{
bar_user::BarUser,
pagetable::MmuVersion,
GpuMm,
VramAddress, //
},
vgpu::VgpuManager, //
};
mod hal;
mod regs;
macro_rules! define_chipset {
({ $($variant:ident = $value:expr),* $(,)* }) =>
({ $($variant:ident),* $(,)* }) =>
{
::kernel::macros::paste!(
/// Enum representation of the GPU chipset.
#[derive(fmt::Debug, Copy, Clone, PartialOrd, Ord, PartialEq, Eq)]
pub(crate) enum Chipset {
$($variant = $value),*,
#[repr(u32)]
#[allow(missing_docs)]
pub enum Chipset {
$($variant = uapi::[<drm_nova_chipid_NOVA_DRM_CHIPID_ $variant:upper>]),*,
}
impl Chipset {
pub(crate) const ALL: &'static [Chipset] = &[
/// All chipsets known to the driver.
pub const ALL: &'static [Chipset] = &[
$( Chipset::$variant, )*
];
::kernel::macros::paste!(
/// Returns the name of this chipset, in lowercase.
///
/// # Examples
@ -65,7 +92,6 @@ pub(crate) const fn name(&self) -> &'static str {
)*
}
}
);
}
// TODO[FPRI]: replace with something like derive(FromPrimitive)
@ -74,49 +100,55 @@ impl TryFrom<u32> for Chipset {
fn try_from(value: u32) -> Result<Self, Self::Error> {
match value {
$( $value => Ok(Chipset::$variant), )*
$(
uapi::[<drm_nova_chipid_NOVA_DRM_CHIPID_ $variant:upper>] => {
Ok(Chipset::$variant)
}
)*
_ => Err(ENODEV),
}
}
}
);
}
}
define_chipset!({
// Turing
TU102 = 0x162,
TU104 = 0x164,
TU106 = 0x166,
TU117 = 0x167,
TU116 = 0x168,
TU102,
TU104,
TU106,
TU117,
TU116,
// Ampere
GA100 = 0x170,
GA102 = 0x172,
GA103 = 0x173,
GA104 = 0x174,
GA106 = 0x176,
GA107 = 0x177,
GA100,
GA102,
GA103,
GA104,
GA106,
GA107,
// Hopper
GH100 = 0x180,
GH100,
// Ada
AD102 = 0x192,
AD103 = 0x193,
AD104 = 0x194,
AD106 = 0x196,
AD107 = 0x197,
AD102,
AD103,
AD104,
AD106,
AD107,
// Blackwell GB10x
GB100 = 0x1a0,
GB102 = 0x1a2,
GB100,
GB102,
// Blackwell GB20x
GB202 = 0x1b2,
GB203 = 0x1b3,
GB205 = 0x1b5,
GB206 = 0x1b6,
GB207 = 0x1b7,
GB202,
GB203,
GB205,
GB206,
GB207,
});
impl Chipset {
pub(crate) const fn arch(self) -> Architecture {
/// Returns the [`Architecture`] generation of this chipset.
pub const fn arch(self) -> Architecture {
match self {
Self::TU102 | Self::TU104 | Self::TU106 | Self::TU117 | Self::TU116 => {
Architecture::Turing
@ -139,6 +171,19 @@ pub(crate) const fn arch(self) -> Architecture {
pub(crate) fn pci_config_mirror_range(self) -> Range<u32> {
hal::gpu_hal(self).pci_config_mirror_range()
}
/// Returns the MMU version for this chipset.
pub(crate) fn mmu_version(self) -> MmuVersion {
MmuVersion::from(self.arch())
}
}
impl From<Chipset> for u32 {
#[inline]
fn from(value: Chipset) -> Self {
// CAST: `Chipset` is `repr(u32)` and can thus be cast losslessly.
value as u32
}
}
// TODO
@ -158,13 +203,30 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
bounded_enum! {
/// Enum representation of the GPU generation.
#[derive(fmt::Debug, Copy, Clone)]
pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> {
Turing = 0x16,
Ampere = 0x17,
Hopper = 0x18,
Ada = 0x19,
BlackwellGB10x = 0x1a,
BlackwellGB20x = 0x1b,
#[repr(u32)]
pub enum Architecture with TryFrom<Bounded<u32, 6>> {
/// Turing (TU1xx).
Turing = uapi::drm_nova_architecture_NOVA_DRM_ARCHITECTURE_TURING,
/// Ampere (GA10x).
Ampere = uapi::drm_nova_architecture_NOVA_DRM_ARCHITECTURE_AMPERE,
/// Hopper (GH100).
Hopper = uapi::drm_nova_architecture_NOVA_DRM_ARCHITECTURE_HOPPER,
/// Ada Lovelace (AD10x).
Ada = uapi::drm_nova_architecture_NOVA_DRM_ARCHITECTURE_ADA,
/// Blackwell (GB10x).
BlackwellGB10x =
uapi::drm_nova_architecture_NOVA_DRM_ARCHITECTURE_BLACKWELL_GB10X,
/// Blackwell (GB20x).
BlackwellGB20x =
uapi::drm_nova_architecture_NOVA_DRM_ARCHITECTURE_BLACKWELL_GB20X,
}
}
impl From<Architecture> for u32 {
#[inline]
fn from(value: Architecture) -> Self {
// CAST: `Architecture` is `repr(u32)` and can thus be cast losslessly.
value as u32
}
}
@ -191,13 +253,14 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
/// Structure holding a basic description of the GPU: `Chipset` and `Revision`.
#[derive(Clone, Copy)]
pub(crate) struct Spec {
chipset: Chipset,
pub struct Spec {
/// The GPU chipset.
pub chipset: Chipset,
revision: Revision,
}
impl Spec {
fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
pub(crate) fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
// Some brief notes about boot0 and boot42, in chronological order:
//
// NV04 through NV50:
@ -272,17 +335,68 @@ struct GspResources<'gpu> {
vgpu: VgpuManager,
/// GSP runtime data.
#[pin]
gsp: Gsp,
gsp: Gsp<'gpu>,
/// GSP unload firmware bundle, if any.
unload_bundle: Option<gsp::UnloadBundle>,
unload_bundle: Option<gsp::UnloadBundle<'gpu>>,
}
/// The GSP event handler's registration and the enable of its subtree at `TOP`.
///
/// The two drop as a unit, the handler first, on the drop of [`Gpu`] and on the error path of
/// its constructor alike, so that the subtree is disabled only after the handler is freed.
#[pin_data]
struct GspSubtree<'a> {
#[pin]
irq: GspIrq<'a>,
/// Must be kept declared *after* `irq`. A handler still in flight enables the subtree again
/// through its rearm.
_top: TopEnableGuard<'a>,
}
impl<'a> GspSubtree<'a> {
/// Returns an initializer that registers the GSP event handler and then enables its subtree at
/// `TOP`.
///
/// # Safety
///
/// Callers must not `mem::forget()` the initialized `GspSubtree` or otherwise prevent its
/// [`Drop`] implementation, which runs `free_irq`, from running.
unsafe fn new(
pdev: &'a pci::Device<device::Bound>,
tree: &'a Tree<'a>,
falcon: &'a Falcon<'a, GspFalcon>,
cmdq: &'a Cmdq<'a>,
) -> impl PinInit<Self, Error> + 'a {
try_pin_init!(Self {
// SAFETY: this function's caller must not leak the `GspSubtree` that owns this
// registration, so the registration's `Drop` runs.
irq <- unsafe { GspIrq::new(pdev, tree, falcon, cmdq) },
_top: tree.enable_top_guarded(),
})
}
}
/// Structure holding the resources required to operate the GPU.
#[pin_data]
pub(crate) struct Gpu<'gpu> {
spec: Spec,
pub(crate) spec: Spec,
/// GSP event interrupt registration, and the enable of its subtree.
///
/// Must be kept declared *before* `gsp_resources`, so that the handler is unregistered, and
/// any in-flight run of it has finished, before the command queue that it drains and the
/// falcon that it reads are freed, and before the GSP is unloaded.
#[pin]
_gsp_subtree: GspSubtree<'gpu>,
/// Static GPU information as provided by the GSP.
gsp_static_info: GetGspStaticInfoReply,
pub(crate) gsp_static_info: GetGspStaticInfoReply,
/// GPU memory manager owning memory management resources.
///
/// Must be kept declared *before* `gsp_resources`, so that its components are dropped while
/// the GSP is still operational.
mm: GpuMm<'gpu>,
/// BAR1 user interface for CPU access to GPU virtual memory.
#[pin]
bar_user: BarUser<'gpu>,
/// GSP and its resources.
#[pin]
gsp_resources: GspResources<'gpu>,
@ -292,6 +406,14 @@ pub(crate) struct Gpu<'gpu> {
/// Must be kept declared *after* `gsp_resources`, as the latter's `PinnedDrop` implementation
/// requires the sysmem flush page to be in place.
sysmem_flush: SysmemFlush<'gpu>,
/// Borrow of `tree` that `_gsp_subtree` holds. A field that borrows a sibling field is
/// self-referential, which `pin_init` cannot express, so the borrow is taken by hand.
tree_ref: &'gpu Tree<'gpu>,
/// The GIN CPU interrupt tree and the PCI vectors that deliver it.
///
/// Must be kept declared *after* `_gsp_subtree`, which holds a borrow of it.
#[pin]
tree: Tree<'gpu>,
}
#[pinned_drop]
@ -326,6 +448,7 @@ impl<'gpu> Gpu<'gpu> {
pub(crate) fn new<'a>(
pdev: &'gpu pci::Device<device::Core<'a>>,
bar: Bar0<'gpu>,
bar1: Bar1<'gpu>,
) -> impl PinInit<Self, Error> + use<'gpu, 'a> {
let dev = pdev.as_ref();
@ -334,17 +457,28 @@ pub(crate) fn new<'a>(
dev_info!(dev,"NVIDIA ({})\n", spec);
})?,
// We must wait for GFW_BOOT completion before doing any significant setup on the GPU.
tree: Tree::new(pdev, bar, spec.chipset, irq::gsp::GSP_SUBTREE.into())?,
// SAFETY: `tree` is initialized above, is pinned at a stable address, and is dropped
// after every field that uses `tree_ref` (struct field drop order).
tree_ref: unsafe { &*core::ptr::from_ref(tree.as_ref().get_ref()) },
_: {
let hal = hal::gpu_hal(spec.chipset);
let dma_mask = hal.dma_mask();
let dma_mask = hal::gpu_hal(spec.chipset).dma_mask();
// SAFETY: `Gpu` owns all DMA allocations for this device, and we are
// still constructing it, so no concurrent DMA allocations can exist.
unsafe { pdev.dma_set_mask_and_coherent(dma_mask)? };
hal.wait_gfw_boot_completion(bar)
.inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))?;
// Nova walks SG segments to build page tables, so their length is
// irrelevant to the device.
//
// SAFETY: `Gpu` owns all DMA allocations for this device, and we are
// still constructing it, so no concurrent DMA allocations can exist.
unsafe { pdev.dma_set_max_seg_size(u32::MAX) };
// We must wait for GFW_BOOT completion before doing any significant setup on the GPU.
Self::wait_gfw_boot_completion(dev, bar, spec.chipset)?;
},
// Initialize this early because `gsp_resources` depends on it.
@ -357,12 +491,7 @@ pub(crate) fn new<'a>(
bar,
gsp_falcon: Falcon::new(
dev,
spec.chipset,
bar
)
.inspect(|falcon| falcon.clear_swgen0_intr())?,
gsp_falcon: Falcon::new(dev, spec.chipset, bar)?,
sec2_falcon: Falcon::new(dev, spec.chipset, bar)?,
@ -370,7 +499,7 @@ pub(crate) fn new<'a>(
vgpu: VgpuManager::new(pdev, spec.chipset, fsp.as_mut()),
gsp <- Gsp::new(pdev),
gsp <- Gsp::new(pdev, bar),
// This member must be initialized last, so the `UnloadBundle` can never be dropped
// from outside of the constructed `GspResources`, ensuring that the unload sequence
@ -386,9 +515,32 @@ pub(crate) fn new<'a>(
})?,
}),
_: {
irq::gsp::quiesce(tree_ref, &gsp_resources.gsp_falcon);
},
// SAFETY: the GSP falcon and the command queue are fields of `gsp_resources`, which
// is initialized above and pinned, so both references outlive the registration. The
// registration is a field of `Gpu` and is never leaked, so its `Drop` runs, and field
// drop order runs it before either is freed.
_gsp_subtree <- unsafe {
GspSubtree::new(
pdev,
tree_ref,
&*core::ptr::from_ref(&gsp_resources.gsp_falcon),
&*core::ptr::from_ref(&gsp_resources.gsp.cmdq),
)
},
// No interrupt announces the messages that the GSP posted during boot, before the
// SWGEN0 latch was cleared.
_: {
gsp_resources.gsp.cmdq.drain()?;
},
gsp_static_info: {
// Obtain and display basic GPU information.
let info = gsp_resources.gsp.get_static_info(bar)?;
let info = gsp_resources.gsp.get_static_info()?;
match info.gpu_name() {
Ok(name) => dev_info!(dev, "GPU name: {}\n", name),
Err(e) => dev_warn!(dev, "GPU name unavailable: {:?}\n", e),
@ -403,14 +555,84 @@ pub(crate) fn new<'a>(
dev_dbg!(
dev,
"Total usable VRAM: {} MiB\n",
info.usable_fb_regions.iter().fold(0u64, |res, region| res
.saturating_add(region.end - region.start))
/ u64::SZ_1M
info.vram_size() / u64::SZ_1M
);
}
info
}
},
// Create GPU memory manager owning memory management resources.
mm: {
let usable_vram = gsp_static_info.usable_fb_regions.first().ok_or(ENODEV)?;
let buddy_params = GpuBuddyParams {
base_offset: usable_vram.start,
size: usable_vram.end - usable_vram.start,
chunk_size: Alignment::new::<SZ_4K>(),
};
GpuMm::new(
bar,
gsp_resources.spec.chipset,
buddy_params,
VramAddress::from_raw(gsp_static_info.total_fb_end),
)?
},
// Create BAR1 user interface for CPU access to GPU virtual memory.
bar_user <- {
let pdb_addr = VramAddress::from_raw(gsp_static_info.bar1_pde_base);
let bar1_idx = crate::driver::bar1_resource_index(pdev)?;
let bar1_size = pdev.resource_len(bar1_idx)?;
BarUser::new(
pdb_addr,
gsp_resources.spec.chipset,
bar1_size,
bar1,
)
},
})
}
/// Waits for GFW, the GPU's boot firmware, to report completion.
///
/// The driver must not program the GPU before then.
///
/// # Errors
///
/// `ETIMEDOUT` if GFW does not report completion in time.
pub(crate) fn wait_gfw_boot_completion(
dev: &device::Device<device::Bound>,
bar: Bar0<'_>,
chipset: Chipset,
) -> Result {
hal::gpu_hal(chipset)
.wait_gfw_boot_completion(bar)
.inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))
}
/// Runs self-tests on the constructed [`Gpu`], logging failures without failing probe.
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
pub(crate) fn run_selftests(self: Pin<&mut Self>, pdev: &pci::Device<device::Bound>) {
let this = self.project();
let dev = pdev.as_ref();
let regions = &this.gsp_static_info.usable_fb_regions;
if let Err(err) = crate::mm::selftest::run(
dev,
this.mm,
regions,
&this.bar_user,
this.gsp_static_info.bar1_pde_base,
this.spec.chipset,
) {
dev_err!(dev, "self-tests failed: {:?}\n", err);
}
}
}
/// Reads the boot0 register and returns its raw value.
pub(crate) fn boot_0_raw(bar: Bar0<'_>) -> u32 {
bar.read(regs::NV_PMC_BOOT_0).into_raw()
}

86
drivers/gpu/nova-core/gpu/regs.rs Normal file
View File

@ -0,0 +1,86 @@
// SPDX-License-Identifier: GPL-2.0
use kernel::{
io::register,
prelude::*, //
};
use super::{
Architecture,
Chipset, //
};
use crate::driver::NovaRegisters;
// PMC
register! {
base: NovaRegisters;
/// Basic revision information about the GPU.
pub(super) NV_PMC_BOOT_0(u32) @ 0x00000000 {
/// Lower bits of the architecture.
28:24 architecture_0;
/// Implementation version of the architecture.
23:20 implementation;
/// MSB of the architecture.
8:8 architecture_1;
/// Major revision of the chip.
7:4 major_revision;
/// Minor revision of the chip.
3:0 minor_revision;
}
/// Extended architecture information.
pub(super) NV_PMC_BOOT_42(u32) @ 0x00000a00 {
/// Architecture value.
29:24 architecture ?=> Architecture;
/// Implementation version of the architecture.
23:20 implementation;
/// Major revision of the chip.
19:16 major_revision;
/// Minor revision of the chip.
15:12 minor_revision;
}
}
impl NV_PMC_BOOT_0 {
pub(super) fn is_older_than_fermi(self) -> bool {
// From https://github.com/NVIDIA/open-gpu-doc/tree/master/manuals :
const NV_PMC_BOOT_0_ARCHITECTURE_GF100: u32 = 0xc;
// Older chips left arch1 zeroed out. That, combined with an arch0 value that is less than
// GF100, means "older than Fermi".
self.architecture_1() == 0 && self.architecture_0() < NV_PMC_BOOT_0_ARCHITECTURE_GF100
}
}
impl NV_PMC_BOOT_42 {
/// Combines `architecture` and `implementation` to obtain a code unique to the chipset.
pub(super) fn chipset(self) -> Result<Chipset> {
self.architecture()
.map(|arch| {
((arch as u32) << Self::IMPLEMENTATION_RANGE.len())
| u32::from(self.implementation())
})
.and_then(Chipset::try_from)
}
/// Returns the raw architecture value from the register.
fn architecture_raw(self) -> u8 {
((self.into_raw() >> Self::ARCHITECTURE_RANGE.start())
& ((1 << Self::ARCHITECTURE_RANGE.len()) - 1)) as u8
}
}
impl kernel::fmt::Display for NV_PMC_BOOT_42 {
fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result {
write!(
f,
"boot42 = 0x{:08x} (architecture 0x{:x}, implementation 0x{:x})",
self.inner,
self.architecture_raw(),
self.implementation()
)
}
}

39
drivers/gpu/nova-core/gsp.rs
View File

@ -115,11 +115,11 @@ fn init(view: CoherentView<'_, Self>, start: DmaAddress) -> Result<()> {
/// then pp points to index into the buffer where the next logging entry will
/// be written. Therefore, the logging data is valid if:
/// 1 <= pp < sizeof(buffer)/sizeof(u64)
struct LogBuffer(Coherent<[u8; LOG_BUFFER_SIZE]>);
struct LogBuffer<'a>(Coherent<'a, [u8; LOG_BUFFER_SIZE]>);
impl LogBuffer {
impl<'a> LogBuffer<'a> {
/// Creates a new `LogBuffer` mapped on `dev`.
fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
fn new(dev: &'a device::Device<device::Bound>) -> Result<Self> {
let obj = Self(Coherent::zeroed(dev, GFP_KERNEL)?);
let start_addr = obj.0.dma_address();
@ -135,33 +135,36 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
}
}
struct LogBuffers {
struct LogBuffers<'a> {
/// Init log buffer.
loginit: LogBuffer,
loginit: LogBuffer<'a>,
/// Interrupts log buffer.
logintr: LogBuffer,
logintr: LogBuffer<'a>,
/// RM log buffer.
logrm: LogBuffer,
logrm: LogBuffer<'a>,
}
/// GSP runtime data.
#[pin_data]
pub(crate) struct Gsp {
pub(crate) struct Gsp<'gsp> {
/// Libos arguments.
pub(crate) libos: Coherent<[LibosMemoryRegionInitArgument]>,
pub(crate) libos: Coherent<'gsp, [LibosMemoryRegionInitArgument]>,
/// Log buffers, optionally exposed via debugfs.
#[pin]
logs: debugfs::Scope<LogBuffers>,
logs: debugfs::Scope<LogBuffers<'gsp>>,
/// Command queue.
#[pin]
pub(crate) cmdq: Cmdq,
pub(crate) cmdq: Cmdq<'gsp>,
/// RM arguments.
rmargs: Coherent<GspArgumentsPadded>,
rmargs: Coherent<'gsp, GspArgumentsPadded>,
}
impl Gsp {
impl<'gsp> Gsp<'gsp> {
// Creates an in-place initializer for a `Gsp` manager for `pdev`.
pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error> + '_ {
pub(crate) fn new(
pdev: &'gsp pci::Device<device::Bound>,
bar: Bar0<'gsp>,
) -> impl PinInit<Self, Error> + 'gsp {
pin_init::pin_init_scope(move || {
let dev = pdev.as_ref();
@ -173,7 +176,7 @@ pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error
// _kgspInitLibosLoggingStructures (allocates memory for buffers)
// kgspSetupLibosInitArgs_IMPL (creates pLibosInitArgs[] array)
Ok(try_pin_init!(Self {
cmdq <- Cmdq::new(dev),
cmdq <- Cmdq::new(dev, bar),
rmargs: Coherent::init(dev, GFP_KERNEL, GspArgumentsPadded::new(&cmdq))?,
libos: {
let mut libos = CoherentBox::zeroed_slice(
@ -217,10 +220,10 @@ pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error
}
/// Query the GSP for the static GPU information.
pub(crate) fn get_static_info(&self, bar: Bar0<'_>) -> Result<commands::GetGspStaticInfoReply> {
self.cmdq.send_command(bar, commands::GetGspStaticInfo)
pub(crate) fn get_static_info(&self) -> Result<commands::GetGspStaticInfoReply> {
self.cmdq.send_command(commands::GetGspStaticInfo)
}
}
/// Opaque bundle required to unload the GSP. Created by [`Gsp::boot`], consumed by [`Gsp::unload`].
pub(crate) struct UnloadBundle(KBox<dyn hal::UnloadBundle>);
pub(crate) struct UnloadBundle<'a>(KBox<dyn hal::UnloadBundle + 'a>);

26
drivers/gpu/nova-core/gsp/boot.rs
View File

@ -10,7 +10,6 @@
};
use crate::{
driver::Bar0,
falcon::{
gsp::Gsp,
Falcon, //
@ -22,7 +21,7 @@
},
};
impl super::Gsp {
impl<'gsp> super::Gsp<'gsp> {
/// Attempt to boot the GSP.
///
/// This is a GPU-dependent and complex procedure that involves loading firmware files from
@ -33,10 +32,9 @@ impl super::Gsp {
/// [`Self::unload`]) returned.
pub(crate) fn boot(
self: Pin<&mut Self>,
mut ctx: super::GspBootContext<'_, '_>,
) -> Result<Option<super::UnloadBundle>> {
mut ctx: super::GspBootContext<'_, 'gsp>,
) -> Result<Option<super::UnloadBundle<'gsp>>> {
let pdev = ctx.pdev;
let bar = ctx.bar;
let chipset = ctx.chipset;
let gsp_falcon = ctx.gsp_falcon;
let dev = pdev.as_ref();
@ -44,6 +42,11 @@ pub(crate) fn boot(
let gsp_fw = KBox::pin_init(GspFirmware::new(dev, chipset), GFP_KERNEL)?;
self.cmdq
.send_command_no_wait(commands::SetSystemInfo::new(pdev, chipset))?;
self.cmdq
.send_command_no_wait(commands::SetRegistry::new(ctx.vgpu.state())?)?;
// Perform the chipset-specific boot sequence, and retrieve the unload bundle.
let unload_bundle = hal.boot(&self, &mut ctx, &gsp_fw)?.or_else(|| {
dev_warn!(dev, "The GSP won't be able to unload properly on unbind.\n");
@ -73,11 +76,6 @@ pub(crate) fn boot(
dev_dbg!(pdev, "RISC-V active? {}\n", gsp_falcon.is_riscv_active(),);
self.cmdq
.send_command_no_wait(bar, commands::SetSystemInfo::new(pdev, chipset))?;
self.cmdq
.send_command_no_wait(bar, commands::SetRegistry::new(ctx.vgpu.state())?)?;
hal.post_boot(&self, ctx, &gsp_fw)?;
// Wait until GSP is fully initialized.
@ -88,13 +86,12 @@ pub(crate) fn boot(
/// Shut down the GSP and wait until it is offline.
fn shutdown_gsp(
cmdq: &Cmdq,
bar: Bar0<'_>,
cmdq: &Cmdq<'_>,
gsp_falcon: &Falcon<'_, Gsp>,
mode: commands::PowerStateLevel,
) -> Result {
// Command to shut the GSP down.
cmdq.send_command(bar, commands::UnloadingGuestDriver::new(mode))?;
cmdq.send_command(commands::UnloadingGuestDriver::new(mode))?;
// Wait until GSP signals it is suspended.
const LIBOS_INTERRUPT_PROCESSOR_SUSPENDED: u32 = bits::bit_u32(31);
@ -113,14 +110,13 @@ fn shutdown_gsp(
pub(crate) fn unload(
&self,
mut ctx: super::GspBootContext<'_, '_>,
unload_bundle: Option<super::UnloadBundle>,
unload_bundle: Option<super::UnloadBundle<'_>>,
) -> Result {
let dev = ctx.dev();
// Shut down the GSP. Keep going even in case of error.
let mut res = Self::shutdown_gsp(
&self.cmdq,
ctx.bar,
ctx.gsp_falcon,
commands::PowerStateLevel::Level0,
)

264
drivers/gpu/nova-core/gsp/cmdq.rs
View File

@ -2,13 +2,7 @@
mod continuation;
use core::{
mem,
sync::atomic::{
fence,
Ordering, //
},
};
use core::mem;
use kernel::{
device,
@ -26,10 +20,19 @@
prelude::*,
ptr,
sync::{
aref::ARef,
barrier::{
dma_mb,
Full,
Read,
Write, //
},
Mutex, //
},
time::Delta,
time::{
Delta,
Instant,
Monotonic, //
},
transmute::{
AsBytes,
FromBytes, //
@ -131,7 +134,9 @@ fn size(&self) -> usize {
/// Trait representing messages received from the GSP.
///
/// This trait tells [`Cmdq::receive_msg`] how it can receive a given type of message.
/// A reply that [`Cmdq::send_command`] waits for, or an event that [`Cmdq::await_msg`] waits for.
/// The receiver matches a message's function code against [`Self::FUNCTION`] and decodes the
/// message with [`Self::read`].
pub(crate) trait MessageFromGsp: Sized {
/// Function identifying this message from the GSP.
const FUNCTION: MsgFunction;
@ -230,19 +235,19 @@ unsafe impl FromBytes for GspMem {}
/// pointer and the GSP read pointer. This region is returned by [`Self::driver_write_area`].
/// * The driver owns (i.e. can read from) the part of the GSP message queue between the CPU read
/// pointer and the GSP write pointer. This region is returned by [`Self::driver_read_area`].
struct DmaGspMem(Coherent<GspMem>);
struct DmaGspMem<'a>(Coherent<'a, GspMem>);
impl DmaGspMem {
impl<'a> DmaGspMem<'a> {
/// Allocate a new instance and map it for `dev`.
fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
fn new(dev: &'a device::Device<device::Bound>) -> Result<Self> {
const MSGQ_SIZE: u32 = num::usize_into_u32::<{ size_of::<Msgq>() }>();
const RX_HDR_OFF: u32 = num::usize_into_u32::<{ mem::offset_of!(Msgq, rx) }>();
let mut gsp_mem = CoherentBox::<GspMem>::zeroed(dev, GFP_KERNEL)?;
let mut gsp_mem = CoherentBox::<'_, GspMem>::zeroed(dev, GFP_KERNEL)?;
gsp_mem.cpuq.tx = MsgqTxHeader::new(MSGQ_SIZE, RX_HDR_OFF, MSGQ_NUM_PAGES);
gsp_mem.cpuq.rx = MsgqRxHeader::new();
let gsp_mem: Coherent<_> = gsp_mem.into();
let gsp_mem: Coherent<'_, _> = gsp_mem.into();
PteArray::init(io_project!(gsp_mem, .ptes), gsp_mem.dma_address())?;
Ok(Self(gsp_mem))
@ -404,7 +409,12 @@ fn allocate_command(&mut self, size: usize, timeout: Delta) -> Result<GspCommand
//
// - The returned value is within `0..MSGQ_NUM_PAGES`.
fn gsp_write_ptr(&self) -> u32 {
MsgqTxHeader::write_ptr(io_project!(self.0, .gspq.tx)) % MSGQ_NUM_PAGES
let ptr = MsgqTxHeader::write_ptr(io_project!(self.0, .gspq.tx)) % MSGQ_NUM_PAGES;
// ORDERING: LOAD->LOAD ordering needed to order `gsp_write_ptr` read before data read.
dma_mb(Read);
ptr
}
// Returns the index of the memory page the GSP will read the next command from.
@ -413,7 +423,12 @@ fn gsp_write_ptr(&self) -> u32 {
//
// - The returned value is within `0..MSGQ_NUM_PAGES`.
fn gsp_read_ptr(&self) -> u32 {
MsgqRxHeader::read_ptr(io_project!(self.0, .gspq.rx)) % MSGQ_NUM_PAGES
let ptr = MsgqRxHeader::read_ptr(io_project!(self.0, .gspq.rx)) % MSGQ_NUM_PAGES;
// ORDERING: LOAD->STORE ordering needed to order `gsp_read_ptr` read before data write.
dma_mb(Full);
ptr
}
// Returns the index of the memory page the CPU can read the next message from.
@ -427,12 +442,11 @@ fn cpu_read_ptr(&self) -> u32 {
// Informs the GSP that it can send `elem_count` new pages into the message queue.
fn advance_cpu_read_ptr(&mut self, elem_count: u32) {
// ORDERING: LOAD->STORE ordering needed to order `cpu_read_ptr` write after data read.
dma_mb(Full);
let rx = io_project!(self.0, .cpuq.rx);
let rptr = MsgqRxHeader::read_ptr(rx).wrapping_add(elem_count) % MSGQ_NUM_PAGES;
// Ensure read pointer is properly ordered.
fence(Ordering::SeqCst);
MsgqRxHeader::set_read_ptr(rx, rptr)
}
@ -447,12 +461,12 @@ fn cpu_write_ptr(&self) -> u32 {
// Informs the GSP that it can process `elem_count` new pages from the command queue.
fn advance_cpu_write_ptr(&mut self, elem_count: u32) {
// ORDERING: STORE->STORE ordering needed to order `cpu_write_ptr` write after data write.
dma_mb(Write);
let tx = io_project!(self.0, .cpuq.tx);
let wptr = MsgqTxHeader::write_ptr(tx).wrapping_add(elem_count) % MSGQ_NUM_PAGES;
MsgqTxHeader::set_write_ptr(tx, wptr);
// Ensure all command data is visible before triggering the GSP read.
fence(Ordering::SeqCst);
}
}
@ -469,7 +483,7 @@ struct GspCommand<'a> {
/// A message ready to be processed from the message queue.
///
/// This is the type returned by [`Cmdq::wait_for_msg`].
/// This is the type returned by [`CmdqInner::wait_for_msg`].
struct GspMessage<'a> {
// Reference to the header of the message.
header: &'a GspMsgElement,
@ -483,15 +497,15 @@ struct GspMessage<'a> {
/// Provides the ability to send commands and receive messages from the GSP using a shared memory
/// area.
#[pin_data]
pub(crate) struct Cmdq {
pub(crate) struct Cmdq<'cmdq> {
/// Inner mutex-protected state.
#[pin]
inner: Mutex<CmdqInner>,
inner: Mutex<CmdqInner<'cmdq>>,
/// DMA address of the command queue's shared memory region.
pub(super) dma_addr: DmaAddress,
}
impl Cmdq {
impl<'cmdq> Cmdq<'cmdq> {
/// Offset of the data after the PTEs.
const POST_PTE_OFFSET: usize = core::mem::offset_of!(GspMem, cpuq);
@ -512,14 +526,18 @@ impl Cmdq {
pub(super) const RECEIVE_TIMEOUT: Delta = Delta::from_secs(5);
/// Creates a new command queue for `dev`.
pub(crate) fn new(dev: &device::Device<device::Bound>) -> impl PinInit<Self, Error> + '_ {
pub(crate) fn new(
dev: &'cmdq device::Device<device::Bound>,
bar: Bar0<'cmdq>,
) -> impl PinInit<Self, Error> + 'cmdq {
pin_init_scope(move || {
let gsp_mem = DmaGspMem::new(dev)?;
Ok(try_pin_init!(Self {
dma_addr: gsp_mem.0.dma_address(),
inner <- new_mutex!(CmdqInner {
dev: dev.into(),
dev,
bar,
gsp_mem,
seq: 0,
}),
@ -547,21 +565,21 @@ fn notify_gsp(bar: Bar0<'_>) {
/// Sends `command` to the GSP and waits for the reply.
///
/// Messages with non-matching function codes are silently consumed until the expected reply
/// arrives.
/// Events that arrive before the reply are logged and consumed.
///
/// The queue is locked for the entire send+receive cycle to ensure that no other command can
/// be interleaved.
///
/// # Errors
///
/// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
/// not received within the timeout.
/// - `ETIMEDOUT` if space does not become available to send the command, or if the reply does
/// not arrive within [`Self::RECEIVE_TIMEOUT`] of the send, however many events arrive
/// while waiting.
/// - `EIO` if the variable payload requested by the command has not been entirely
/// written to by its [`CommandToGsp::init_variable_payload`] method.
///
/// Error codes returned by the command and reply initializers are propagated as-is.
pub(crate) fn send_command<M>(&self, bar: Bar0<'_>, command: M) -> Result<M::Reply>
pub(crate) fn send_command<M>(&self, command: M) -> Result<M::Reply>
where
M: CommandToGsp,
M::Reply: MessageFromGsp,
@ -569,15 +587,9 @@ pub(crate) fn send_command<M>(&self, bar: Bar0<'_>, command: M) -> Result<M::Rep
Error: From<<M::Reply as MessageFromGsp>::InitError>,
{
let mut inner = self.inner.lock();
inner.send_command(bar, command)?;
inner.send_command(command)?;
loop {
match inner.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
Ok(reply) => break Ok(reply),
Err(ERANGE) => continue,
Err(e) => break Err(e),
}
}
inner.await_msg()
}
/// Sends `command` to the GSP without waiting for a reply.
@ -589,37 +601,63 @@ pub(crate) fn send_command<M>(&self, bar: Bar0<'_>, command: M) -> Result<M::Rep
/// written to by its [`CommandToGsp::init_variable_payload`] method.
///
/// Error codes returned by the command initializers are propagated as-is.
pub(crate) fn send_command_no_wait<M>(&self, bar: Bar0<'_>, command: M) -> Result
pub(crate) fn send_command_no_wait<M>(&self, command: M) -> Result
where
M: CommandToGsp<Reply = NoReply>,
Error: From<M::InitError>,
{
self.inner.lock().send_command(bar, command)
self.inner.lock().send_command(command)
}
/// Receive a message from the GSP.
/// Waits for an unsolicited GSP event of type `M`. Events that arrive before it are logged and
/// consumed.
///
/// See [`CmdqInner::receive_msg`] for details.
pub(crate) fn receive_msg<M: MessageFromGsp>(&self, timeout: Delta) -> Result<M>
/// The queue mutex is held for the whole wait, up to [`Self::RECEIVE_TIMEOUT`], so no other
/// caller can send a command or consume an event meanwhile.
///
/// # Errors
///
/// - `ETIMEDOUT` if the event does not arrive within [`Self::RECEIVE_TIMEOUT`] of the call,
/// however many other events arrive while waiting.
/// - `EIO` if a message fails framing or checksum validation.
///
/// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
where
// This allows all error types, including `Infallible`, to be used for `M::InitError`.
Error: From<M::InitError>,
{
self.inner.lock().receive_msg(timeout)
self.inner.lock().await_msg()
}
/// Logs and consumes every message the GSP has already posted, and returns without waiting for
/// more.
///
/// No caller is waiting for a reply while this holds the queue mutex, so every message is
/// logged as an event. See "Draining the GSP-to-CPU queue" in
/// `Documentation/gpu/nova/core/interrupts.rst`.
///
/// # Errors
///
/// `EIO` if a message fails framing or checksum validation.
pub(crate) fn drain(&self) -> Result {
self.inner.lock().drain()
}
}
/// Inner mutex protected state of [`Cmdq`].
struct CmdqInner {
struct CmdqInner<'a> {
/// Device this command queue belongs to.
dev: ARef<device::Device>,
dev: &'a device::Device,
/// MMIO mapping of PCI BAR0, for writing the GSP doorbell.
bar: Bar0<'a>,
/// Current command sequence number.
seq: u32,
/// Memory area shared with the GSP for communicating commands and messages.
gsp_mem: DmaGspMem,
gsp_mem: DmaGspMem<'a>,
}
impl CmdqInner {
impl CmdqInner<'_> {
/// Timeout for waiting for space on the command queue.
const ALLOCATE_TIMEOUT: Delta = Delta::from_secs(1);
@ -633,7 +671,7 @@ impl CmdqInner {
/// written to by its [`CommandToGsp::init_variable_payload`] method.
///
/// Error codes returned by the command initializers are propagated as-is.
fn send_single_command<M>(&mut self, bar: Bar0<'_>, command: M) -> Result
fn send_single_command<M>(&mut self, command: M) -> Result
where
M: CommandToGsp,
// This allows all error types, including `Infallible`, to be used for `M::InitError`.
@ -687,7 +725,7 @@ fn send_single_command<M>(&mut self, bar: Bar0<'_>, command: M) -> Result
let elem_count = dst.header.element_count();
self.seq += 1;
self.gsp_mem.advance_cpu_write_ptr(elem_count);
Cmdq::notify_gsp(bar);
Cmdq::notify_gsp(self.bar);
Ok(())
}
@ -703,19 +741,19 @@ fn send_single_command<M>(&mut self, bar: Bar0<'_>, command: M) -> Result
/// written to by its [`CommandToGsp::init_variable_payload`] method.
///
/// Error codes returned by the command initializers are propagated as-is.
fn send_command<M>(&mut self, bar: Bar0<'_>, command: M) -> Result
fn send_command<M>(&mut self, command: M) -> Result
where
M: CommandToGsp,
Error: From<M::InitError>,
{
match SplitState::new(command)? {
SplitState::Single(command) => self.send_single_command(bar, command),
SplitState::Single(command) => self.send_single_command(command),
SplitState::Split(command, mut continuations) => {
self.send_single_command(bar, command)?;
self.send_single_command(command)?;
while let Some(continuation) = continuations.next() {
// Turbofish needed because the compiler cannot infer M here.
self.send_single_command::<ContinuationRecord<'_>>(bar, continuation)?;
self.send_single_command::<ContinuationRecord<'_>>(continuation)?;
}
Ok(())
@ -805,8 +843,8 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
/// Receive a message from the GSP.
///
/// The expected message type is specified using the `M` generic parameter. If the pending
/// message has a different function code, `ERANGE` is returned and the message is consumed.
/// A message whose function code is `M::FUNCTION` is decoded and returned. Any other message
/// is logged as an event.
///
/// The read pointer is always advanced past the message, regardless of whether it matched.
///
@ -815,8 +853,7 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
/// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
/// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
/// message queue.
/// - `EINVAL` if the function code of the message was not recognized.
/// - `ERANGE` if the message had a recognized but non-matching function code.
/// - `ENOMSG` if the message was not the awaited reply.
///
/// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
@ -825,11 +862,11 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
Error: From<M::InitError>,
{
let message = self.wait_for_msg(timeout)?;
let function = message.header.function().map_err(|_| EINVAL)?;
let function = message.header.function();
let seq = message.header.sequence();
// Extract the message. Store the result as we want to advance the read pointer even in
// case of failure.
let result = if function == M::FUNCTION {
// An early return here would leave the read pointer on this message.
let result = if matches!(function, Ok(f) if f == M::FUNCTION) {
let (cmd, contents_1) = M::Message::from_bytes_prefix(message.contents.0).ok_or(EIO)?;
let mut sbuffer = SBufferIter::new_reader([contents_1, message.contents.1]);
@ -840,12 +877,14 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
dev_warn!(
&self.dev,
"GSP message {:?} has unprocessed data\n",
function
M::FUNCTION
);
}
})
} else {
Err(ERANGE)
self.log_event(function, seq);
Err(ENOMSG)
};
// Advance the read pointer past this message.
@ -855,4 +894,91 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
result
}
/// Receives a message of type `M`, waiting up to [`Cmdq::RECEIVE_TIMEOUT`] from the call.
///
/// Any other message that arrives first is logged as an event and does not extend the
/// deadline.
///
/// # Errors
///
/// - `ETIMEDOUT` if no message of type `M` arrives before the deadline, however many other
/// messages arrive while waiting.
/// - `EIO` if a message fails framing or checksum validation (see [`Self::wait_for_msg`]).
///
/// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
fn await_msg<M: MessageFromGsp>(&mut self) -> Result<M>
where
// This allows all error types, including `Infallible`, to be used for `M::InitError`.
Error: From<M::InitError>,
{
let deadline = Instant::<Monotonic>::now() + Cmdq::RECEIVE_TIMEOUT;
loop {
let remaining = deadline - Instant::<Monotonic>::now();
if remaining.is_negative() {
break Err(ETIMEDOUT);
}
match self.receive_msg::<M>(remaining) {
Ok(msg) => break Ok(msg),
Err(ENOMSG) => continue,
Err(e) => break Err(e),
}
}
}
/// Logs an event, meaning a message that no caller was waiting for.
///
/// An OS error or robust-channel record is logged at error level and an unknown function code
/// at warning level. Every other event is recorded only by the receive trace in
/// [`Self::wait_for_msg`].
fn log_event(&self, function: Result<MsgFunction, u32>, seq: u32) {
match function {
Ok(MsgFunction::OsErrorLog) => {
dev_err!(&self.dev, "GSP reported an OS error (seq {})\n", seq);
}
Ok(MsgFunction::RcTriggered) => {
dev_err!(
&self.dev,
"GSP triggered robust-channel recovery (seq {})\n",
seq
);
}
// Nothing to do for the remaining known function codes.
Ok(_) => {}
Err(raw) => {
dev_warn!(
&self.dev,
"unknown GSP message function {:#x} (seq {})\n",
raw,
seq
);
}
}
}
/// Logs and consumes every message the queue holds.
///
/// # Errors
///
/// `EIO` if a message fails framing or checksum validation, or if a message's page count
/// overflows a `u32`.
fn drain(&mut self) -> Result {
while !self.gsp_mem.driver_read_area().0.is_empty() {
// A message is available, so this returns without waiting.
let msg = self.wait_for_msg(Delta::ZERO)?;
let pages =
u32::try_from(msg.header.length().div_ceil(GSP_PAGE_SIZE)).map_err(|_| {
dev_err!(&self.dev, "GSP drain: message length overflow\n");
EIO
})?;
let function = msg.header.function();
let seq = msg.header.sequence();
self.gsp_mem.advance_cpu_read_ptr(pages);
self.log_event(function, seq);
}
Ok(())
}
}

6
drivers/gpu/nova-core/gsp/cmdq/continuation.rs
View File

@ -106,9 +106,9 @@ pub(super) fn new(command: C) -> Result<Self> {
if command.size() > MAX_CMD_SIZE {
let mut command_payload =
KVVec::<u8>::from_elem(0u8, payload_len.min(Self::MAX_FIRST_PAYLOAD), GFP_KERNEL)?;
KVVec::zeroed(payload_len.min(Self::MAX_FIRST_PAYLOAD), GFP_KERNEL)?;
let mut continuation_payload =
KVVec::<u8>::from_elem(0u8, payload_len - command_payload.len(), GFP_KERNEL)?;
KVVec::zeroed(payload_len - command_payload.len(), GFP_KERNEL)?;
let mut sbuffer = SBufferIter::new_writer([
command_payload.as_mut_slice(),
continuation_payload.as_mut_slice(),
@ -235,7 +235,7 @@ fn init_variable_payload(
fn read_payload(cmd: impl CommandToGsp) -> Result<KVVec<u8>> {
let len = cmd.variable_payload_len();
let mut buf = KVVec::from_elem(0u8, len, GFP_KERNEL)?;
let mut buf = KVVec::zeroed(len, GFP_KERNEL)?;
let mut sbuf = SBufferIter::new_writer([buf.as_mut_slice(), &mut []]);
cmd.init_variable_payload(&mut sbuf)?;
drop(sbuf);

48
drivers/gpu/nova-core/gsp/commands.rs
View File

@ -187,14 +187,8 @@ fn read(
}
/// Waits for GSP initialization to complete.
pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq) -> Result {
loop {
match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
Ok(_) => break Ok(()),
Err(ERANGE) => continue,
Err(e) => break Err(e),
}
}
pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq<'_>) -> Result {
cmdq.await_msg::<GspInitDone>().map(|_| ())
}
/// The `GetGspStaticInfo` command.
@ -212,10 +206,17 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> {
}
/// The reply from the GSP to the [`GetGspStaticInfo`] command.
pub(crate) struct GetGspStaticInfoReply {
pub struct GetGspStaticInfoReply {
gpu_name: [u8; 64],
gpu_short_name: [u8; 64],
/// The 16-byte SHA-1 based GPU identifier (GID) reported by GSP-RM.
pub gpu_gid: [u8; 16],
/// BAR1 Page Directory Entry base address.
pub(crate) bar1_pde_base: u64,
/// Usable FB (VRAM) regions for driver memory allocation.
pub(crate) usable_fb_regions: KVec<Range<u64>>,
/// Exclusive end of the FB physical address space.
pub(crate) total_fb_end: u64,
}
impl MessageFromGsp for GetGspStaticInfoReply {
@ -231,22 +232,26 @@ fn read(
for region in msg.usable_fb_regions() {
usable_fb_regions.push(region, GFP_KERNEL)?;
}
let total_fb_end = msg.total_fb_end().ok_or(EINVAL)?;
Ok(GetGspStaticInfoReply {
gpu_name: msg.gpu_name_str(),
gpu_short_name: msg.gpu_short_name_str(),
gpu_gid: msg.gpu_gid(),
bar1_pde_base: msg.bar1_pde_base(),
usable_fb_regions,
total_fb_end,
})
}
}
/// Error type for [`GetGspStaticInfoReply::gpu_name`].
#[derive(Debug)]
pub(crate) enum GpuNameError {
pub enum GpuNameError {
/// The GPU name string does not contain a null terminator.
NoNullTerminator(FromBytesUntilNulError),
/// The GPU name string contains invalid UTF-8.
#[expect(dead_code)]
InvalidUtf8(Utf8Error),
}
@ -255,12 +260,31 @@ impl GetGspStaticInfoReply {
///
/// Returns an error if the string given by the GSP does not contain a null terminator or
/// contains invalid UTF-8.
pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> {
pub fn gpu_name(&self) -> Result<&str, GpuNameError> {
CStr::from_bytes_until_nul(&self.gpu_name)
.map_err(GpuNameError::NoNullTerminator)?
.to_str()
.map_err(GpuNameError::InvalidUtf8)
}
/// Returns the short name of the GPU as a string.
///
/// Returns an error if the string given by the GSP does not contain a null terminator or
/// contains invalid UTF-8.
pub fn gpu_short_name(&self) -> core::result::Result<&str, GpuNameError> {
CStr::from_bytes_until_nul(&self.gpu_short_name)
.map_err(GpuNameError::NoNullTerminator)?
.to_str()
.map_err(GpuNameError::InvalidUtf8)
}
/// Returns the total usable VRAM size in bytes, i.e. the summed lengths of all usable FB
/// regions.
pub fn vram_size(&self) -> u64 {
self.usable_fb_regions.iter().fold(0, |size, region| {
size.saturating_add(region.end - region.start)
})
}
}
pub(crate) use fw::commands::PowerStateLevel;

18
drivers/gpu/nova-core/gsp/fw.rs
View File

@ -179,7 +179,7 @@ impl GspFwWprMeta {
/// Returns an initializer for a `GspFwWprMeta` suitable for booting `gsp_firmware` using the
/// framebuffer ranges `ranges`.
pub(crate) fn from_ranges<'a>(
gsp_firmware: &'a GspFirmware,
gsp_firmware: &'a GspFirmware<'_>,
ranges: &'a FbRanges,
) -> impl Init<Self> + 'a {
let init_inner = init!(bindings::GspFwWprMeta {
@ -187,7 +187,7 @@ pub(crate) fn from_ranges<'a>(
magic: bindings::GSP_FW_WPR_META_MAGIC as u64,
revision: u64::from(bindings::GSP_FW_WPR_META_REVISION),
sysmemAddrOfRadix3Elf: gsp_firmware.radix3_dma_address(),
sizeOfRadix3Elf: u64::from_safe_cast(gsp_firmware.size),
sizeOfRadix3Elf: u64::from_safe_cast(gsp_firmware.size()),
sysmemAddrOfBootloader: gsp_firmware.bootloader.ucode.dma_address(),
sizeOfBootloader: u64::from_safe_cast(gsp_firmware.bootloader.ucode.size()),
bootloaderCodeOffset: u64::from(gsp_firmware.bootloader.code_offset),
@ -205,7 +205,7 @@ pub(crate) fn from_ranges<'a>(
gspFwWprStart: ranges.wpr2.start,
gspFwHeapOffset: ranges.wpr2_heap.start,
gspFwHeapSize: ranges.wpr2_heap.len(),
gspFwOffset: ranges.elf.start,
gspFwOffset: ranges.fw_image.start,
bootBinOffset: ranges.boot.start,
frtsOffset: ranges.frts.start,
frtsSize: ranges.frts.len(),
@ -231,7 +231,7 @@ pub(crate) fn from_ranges<'a>(
///
/// The region offsets are left at zero: the ACR ucode computes them when it sets up WPR2.
pub(crate) fn from_sizes<'a>(
gsp_firmware: &'a GspFirmware,
gsp_firmware: &'a GspFirmware<'_>,
sizes: &'a FbSizes,
) -> impl Init<Self> + 'a {
/// VGA workspace size to reserve at the end of the framebuffer, in bytes.
@ -242,7 +242,7 @@ pub(crate) fn from_sizes<'a>(
magic: bindings::GSP_FW_WPR_META_MAGIC as u64,
revision: u64::from(bindings::GSP_FW_WPR_META_REVISION),
sysmemAddrOfRadix3Elf: gsp_firmware.radix3_dma_address(),
sizeOfRadix3Elf: u64::from_safe_cast(gsp_firmware.size),
sizeOfRadix3Elf: u64::from_safe_cast(gsp_firmware.size()),
sysmemAddrOfBootloader: gsp_firmware.bootloader.ucode.dma_address(),
sizeOfBootloader: u64::from_safe_cast(gsp_firmware.bootloader.ucode.size()),
bootloaderCodeOffset: u64::from(gsp_firmware.bootloader.code_offset),
@ -665,7 +665,7 @@ unsafe impl FromBytes for LibosMemoryRegionInitArgument {}
impl LibosMemoryRegionInitArgument {
pub(crate) fn new<'a, A: AsBytes + FromBytes + KnownSize + ?Sized>(
name: &'static str,
obj: &'a Coherent<A>,
obj: &'a Coherent<'_, A>,
) -> impl Init<Self> + 'a {
/// Generates the `ID8` identifier required for some GSP objects.
fn id8(name: &str) -> u64 {
@ -897,7 +897,7 @@ pub(crate) struct GspArgumentsCached {
impl GspArgumentsCached {
/// Creates the arguments for starting the GSP up using `cmdq` as its command queue.
pub(crate) fn new(cmdq: &Cmdq) -> impl Init<Self> + '_ {
pub(crate) fn new<'a, 'b>(cmdq: &'a Cmdq<'b>) -> impl Init<Self> + use<'a, 'b> {
let init_inner = init!(bindings::GSP_ARGUMENTS_CACHED {
messageQueueInitArguments <- MessageQueueInitArguments::new(cmdq),
bDmemStack: 1,
@ -924,7 +924,7 @@ pub(crate) struct GspArgumentsPadded {
}
impl GspArgumentsPadded {
pub(crate) fn new(cmdq: &Cmdq) -> impl Init<Self> + '_ {
pub(crate) fn new<'a, 'b>(cmdq: &'a Cmdq<'b>) -> impl Init<Self> + use<'a, 'b> {
init!(GspArgumentsPadded {
inner <- GspArgumentsCached::new(cmdq),
..Zeroable::init_zeroed()
@ -944,7 +944,7 @@ unsafe impl FromBytes for GspArgumentsPadded {}
impl MessageQueueInitArguments {
/// Creates a new init arguments structure for `cmdq`.
fn new(cmdq: &Cmdq) -> impl Init<Self> + '_ {
fn new<'a, 'b>(cmdq: &'a Cmdq<'b>) -> impl Init<Self> + use<'a, 'b> {
init!(MessageQueueInitArguments {
sharedMemPhysAddr: cmdq.dma_addr,
pageTableEntryCount: num::usize_into_u32::<{ Cmdq::NUM_PTES }>(),

28
drivers/gpu/nova-core/gsp/fw/commands.rs
View File

@ -131,6 +131,29 @@ impl GspStaticConfigInfo {
self.0.gpuNameString
}
/// Returns a bytes array containing the (hopefully) zero-terminated short name of this GPU.
pub(crate) fn gpu_short_name_str(&self) -> [u8; 64] {
self.0.gpuShortNameString
}
/// Returns the 16-byte SHA-1 GPU identifier supplied by GSP-RM.
///
/// GSP-RM reports the GID in binary SHA-1 form, which occupies the first 16 bytes of the
/// GID info payload.
pub(crate) fn gpu_gid(&self) -> [u8; 16] {
let mut gid = [0u8; 16];
gid.copy_from_slice(&self.0.gidInfo.data[..16]);
gid
}
/// Returns the BAR1 Page Directory Entry base address.
///
/// This is the root page table address for BAR1 virtual memory,
/// set up by GSP-RM firmware.
pub(crate) fn bar1_pde_base(&self) -> u64 {
self.0.bar1PdeBase
}
/// Returns an iterator over valid FB regions from GSP firmware data.
fn fb_regions(
&self,
@ -165,6 +188,11 @@ pub(crate) fn usable_fb_regions(&self) -> impl Iterator<Item = Range<u64>> + '_
}
})
}
/// Computes the exclusive end of the FB physical address space.
pub(crate) fn total_fb_end(&self) -> Option<u64> {
self.fb_regions().map(|reg| reg.limit).max()?.checked_add(1)
}
}
// SAFETY: Padding is explicit and will not contain uninitialized data.

16
drivers/gpu/nova-core/gsp/hal.rs
View File

@ -24,7 +24,7 @@
/// The GSP unload code might run in a situation where we cannot load firmware dynamically (e.g.
/// because we are in shutdown and the file system is not accessible anymore). Thus, the firmware
/// required for unloading is prepared at load time, and stored here until it needs to be run.
pub(super) trait UnloadBundle: Send {
pub(super) trait UnloadBundle: Send + Sync {
/// Performs the steps required to properly reset the GSP after it has been stopped.
fn run(&self, ctx: &mut GspBootContext<'_, '_>) -> Result;
}
@ -35,12 +35,12 @@ pub(super) trait GspHal: Send {
///
/// Upon success, returns the [`crate::gsp::UnloadBundle`] to use with [`Gsp::unload`], if one
/// could be created.
fn boot(
fn boot<'gpu>(
&self,
gsp: &Gsp,
ctx: &mut GspBootContext<'_, '_>,
gsp_fw: &GspFirmware,
) -> Result<Option<crate::gsp::UnloadBundle>>;
gsp: &Gsp<'gpu>,
ctx: &mut GspBootContext<'_, 'gpu>,
gsp_fw: &GspFirmware<'gpu>,
) -> Result<Option<super::UnloadBundle<'gpu>>>;
/// Performs HAL-specific post-GSP boot tasks.
///
@ -48,9 +48,9 @@ fn boot(
/// after the initialization commands have been pushed onto its queue.
fn post_boot(
&self,
_gsp: &Gsp,
_gsp: &Gsp<'_>,
_ctx: &mut GspBootContext<'_, '_>,
_gsp_fw: &GspFirmware,
_gsp_fw: &GspFirmware<'_>,
) -> Result {
Ok(())
}

16
drivers/gpu/nova-core/gsp/hal/gh100.rs
View File

@ -58,7 +58,7 @@ fn combined_addr(&self) -> u64 {
fn lockdown_released_or_error(
&self,
gsp_falcon: &Falcon<'_, GspEngine>,
fmc_boot_params: &Coherent<GspFmcBootParams>,
fmc_boot_params: &Coherent<'_, GspFmcBootParams>,
) -> bool {
// GSP-FMC normally clears the boot parameters address from the mailboxes early during
// boot. If the address is still there, keep polling rather than treating it as an error.
@ -75,7 +75,7 @@ fn lockdown_released_or_error(
fn wait_for_gsp_lockdown_release(
dev: &device::Device<device::Bound>,
gsp_falcon: &Falcon<'_, GspEngine>,
fmc_boot_params: &Coherent<GspFmcBootParams>,
fmc_boot_params: &Coherent<'_, GspFmcBootParams>,
) -> Result {
dev_dbg!(dev, "Waiting for GSP lockdown release\n");
@ -141,12 +141,12 @@ impl GspHal for Gh100 {
///
/// This path uses FSP to establish a chain of trust and boot GSP-FMC. FSP handles
/// the GSP boot internally - no manual GSP reset/boot is needed.
fn boot(
fn boot<'gpu>(
&self,
gsp: &Gsp,
ctx: &mut GspBootContext<'_, '_>,
gsp_fw: &GspFirmware,
) -> Result<Option<crate::gsp::UnloadBundle>> {
gsp: &Gsp<'gpu>,
ctx: &mut GspBootContext<'_, 'gpu>,
gsp_fw: &GspFirmware<'gpu>,
) -> Result<Option<crate::gsp::UnloadBundle<'gpu>>> {
let dev = ctx.dev();
let chipset = ctx.chipset;
let gsp_falcon = ctx.gsp_falcon;
@ -159,7 +159,7 @@ fn boot(
let args = FmcBootArgs::new(dev, chipset, wpr_meta, &gsp.libos, false)?;
let unload_bundle = crate::gsp::UnloadBundle(
KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>
KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle + 'gpu>
);
// Wait for the GSP RISC-V core to halt in case of error. We create this guard after `args`

41
drivers/gpu/nova-core/gsp/hal/tu102.rs
View File

@ -52,34 +52,33 @@
//
// Since there are two variants of the prepared firmware (with and without a bootloader), this type
// abstracts the difference.
enum FwsecUnloadFirmware {
enum FwsecUnloadFirmware<'a> {
WithoutBl(FwsecFirmware),
WithBl(FwsecFirmwareWithBl),
WithBl(FwsecFirmwareWithBl<'a>),
}
impl FwsecUnloadFirmware {
impl FwsecUnloadFirmware<'_> {
/// Runs the FWSEC SB firmware.
fn run(
&self,
dev: &device::Device<device::Bound>,
bar: Bar0<'_>,
gsp_falcon: &Falcon<'_, GspEngine>,
) -> Result {
match self {
Self::WithoutBl(fw) => fw.run(dev, gsp_falcon),
Self::WithBl(fw) => fw.run(dev, gsp_falcon, bar),
Self::WithBl(fw) => fw.run(dev, gsp_falcon),
}
}
}
// Contains the firmware required to fully reset GSP on chipsets where the GSP is started using
// FWSEC/Booter.
struct Sec2UnloadBundle {
fwsec_sb: FwsecUnloadFirmware,
struct Sec2UnloadBundle<'a> {
fwsec_sb: FwsecUnloadFirmware<'a>,
booter_unloader: BooterFirmware,
}
impl UnloadBundle for Sec2UnloadBundle {
impl UnloadBundle for Sec2UnloadBundle<'_> {
fn run(&self, ctx: &mut GspBootContext<'_, '_>) -> Result {
let dev = ctx.dev();
let bar = ctx.bar;
@ -88,7 +87,7 @@ fn run(&self, ctx: &mut GspBootContext<'_, '_>) -> Result {
// Log errors but keep going if it fails.
let fwsec_sb_res = self
.fwsec_sb
.run(dev, bar, ctx.gsp_falcon)
.run(dev, ctx.gsp_falcon)
.inspect_err(|e| dev_err!(dev, "FWSEC-SB failed to run: {:?}\n", e));
// Remove WPR2 region if set.
@ -168,7 +167,7 @@ fn run_fwsec_frts(
if self.needs_fwsec_bootloader {
let fwsec_frts_bl = FwsecFirmwareWithBl::new(fwsec_frts, dev, chipset)?;
// Load and run the bootloader, which will load FWSEC-FRTS and run it.
fwsec_frts_bl.run(dev, falcon, bar)?;
fwsec_frts_bl.run(dev, falcon)?;
} else {
// Load and run FWSEC-FRTS directly.
fwsec_frts.run(dev, falcon)?;
@ -213,14 +212,14 @@ fn run_fwsec_frts(
}
/// Load and prepare the resources required to properly reset the GSP after it has been stopped.
fn build_unload_bundle(
fn build_unload_bundle<'gpu>(
&self,
dev: &device::Device<device::Bound>,
dev: &'gpu device::Device<device::Bound>,
chipset: Chipset,
bios: &Vbios,
gsp_falcon: &Falcon<'_, GspEngine>,
sec2_falcon: &Falcon<'_, Sec2>,
) -> Result<crate::gsp::UnloadBundle> {
) -> Result<crate::gsp::UnloadBundle<'gpu>> {
// Load the FWSEC SB firmware, as well as its bootloader if required.
let fwsec_sb = FwsecFirmware::new(dev, gsp_falcon, bios, FwsecCommand::Sb)?;
let fwsec_sb = if self.needs_fwsec_bootloader {
@ -241,18 +240,18 @@ fn build_unload_bundle(
},
GFP_KERNEL,
)
.map(|b| crate::gsp::UnloadBundle(b))
.map(|b| crate::gsp::UnloadBundle(b as KBox<dyn UnloadBundle + 'gpu>))
.map_err(Into::into)
}
}
impl GspHal for Tu102 {
fn boot(
fn boot<'gpu>(
&self,
gsp: &Gsp,
ctx: &mut GspBootContext<'_, '_>,
gsp_fw: &GspFirmware,
) -> Result<Option<crate::gsp::UnloadBundle>> {
gsp: &Gsp<'gpu>,
ctx: &mut GspBootContext<'_, 'gpu>,
gsp_fw: &GspFirmware<'gpu>,
) -> Result<Option<crate::gsp::UnloadBundle<'gpu>>> {
let dev = ctx.dev();
let bar = ctx.bar;
let chipset = ctx.chipset;
@ -317,9 +316,9 @@ fn boot(
fn post_boot(
&self,
gsp: &Gsp,
gsp: &Gsp<'_>,
ctx: &mut GspBootContext<'_, '_>,
gsp_fw: &GspFirmware,
gsp_fw: &GspFirmware<'_>,
) -> Result {
GspSequencer::run(&gsp.cmdq, ctx, &gsp.libos, gsp_fw.bootloader.app_version)?;

9
drivers/gpu/nova-core/gsp/regs.rs
View File

@ -2,11 +2,16 @@
use kernel::io::register;
use crate::regs::NV_PBUS_SW_SCRATCH;
use crate::{
driver::NovaRegisters,
regs::NV_PBUS_SW_SCRATCH, //
};
// PGSP
register! {
base: NovaRegisters;
pub(super) NV_PGSP_QUEUE_HEAD(u32) @ 0x00110c00 {
31:0 address;
}
@ -15,6 +20,8 @@
// PBUS
register! {
base: NovaRegisters;
/// Scratch register 0xe used as FRTS firmware error code.
pub(super) NV_PBUS_SW_SCRATCH_0E_FRTS_ERR(u32) => NV_PBUS_SW_SCRATCH[0xe] {
31:16 frts_err_code;

14
drivers/gpu/nova-core/gsp/sequencer.rs
View File

@ -138,7 +138,7 @@ pub(crate) struct GspSequencer<'a> {
/// GSP falcon for core operations.
gsp_falcon: &'a Falcon<'a, Gsp>,
/// LibOS memory region init arguments.
libos: &'a Coherent<[LibosMemoryRegionInitArgument]>,
libos: &'a Coherent<'a, [LibosMemoryRegionInitArgument]>,
/// Bootloader application version.
bootloader_app_version: u32,
/// Device for logging.
@ -338,18 +338,12 @@ fn next(&mut self) -> Option<Self::Item> {
impl<'a> GspSequencer<'a> {
pub(crate) fn run(
cmdq: &Cmdq,
cmdq: &Cmdq<'_>,
ctx: &'a GspBootContext<'_, '_>,
libos: &'a Coherent<[LibosMemoryRegionInitArgument]>,
libos: &'a Coherent<'a, [LibosMemoryRegionInitArgument]>,
bootloader_app_version: u32,
) -> Result {
let seq_info = loop {
match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
Ok(seq_info) => break seq_info,
Err(ERANGE) => continue,
Err(e) => return Err(e),
}
};
let seq_info = cmdq.await_msg::<GspSequence>()?;
let sequencer = GspSequencer {
bar: ctx.bar,

29
drivers/gpu/nova-core/irq.rs Normal file
View File

@ -0,0 +1,29 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! GPU interrupt support.
//!
//! GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller. It latches
//! every interrupt source in a two-level register tree and delivers the tree to the CPU as a
//! message-signaled PCI interrupt.
//!
//! See `Documentation/gpu/nova/core/interrupts.rst`.
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
pub(crate) mod doorbell_test;
pub(crate) mod gsp;
mod hal;
pub(crate) mod interrupt_tree;
mod regs;
/// The message-signaled interrupt type that Linux granted.
///
/// nova-core never requests INTx, so this has no variant for it, unlike [`kernel::pci::IrqType`].
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum MsiType {
/// A single message, which every subtree raises.
Msi,
/// One table entry per subtree.
MsiX,
}

251
drivers/gpu/nova-core/irq/doorbell_test.rs Normal file
View File

@ -0,0 +1,251 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Interrupt delivery self-test.
//!
//! The test triggers the CPU doorbell vector from software, twice, and checks that each trigger
//! reaches a registered handler. It runs during probe under `CONFIG_NOVA_CORE_SELFTESTS`.
use core::pin::Pin;
use kernel::{
device::Bound,
io::poll::read_poll_timeout,
irq,
pci,
prelude::*,
sync::atomic::{
Acquire,
Atomic,
Relaxed,
Release, //
},
time::Delta, //
};
use super::interrupt_tree::{
GinVector,
LeafEnableGuard,
LeafMask,
Subtree,
TopEnableGuard,
Tree, //
};
use crate::{
driver::Bar0,
gpu::Chipset,
selftest_assert,
selftest_assert_eq, //
};
/// The CPU doorbell vector. Every supported GPU uses this number, so the test needs nothing from
/// GSP-RM, which is not running yet.
const DOORBELL_VECTOR: GinVector = GinVector::new::<129>();
/// The only subtree that this test services.
const DOORBELL_SUBTREE: Subtree = DOORBELL_VECTOR.subtree();
/// Time allowed for each delivery to arrive.
const DELIVERY_TIMEOUT: Delta = Delta::from_millis(1000);
/// The self-test's interrupt handler.
///
/// It clears only the doorbell's bit, rearms delivery, and never walks the tree. A missing rearm
/// shows up as a timeout on the second delivery.
#[pin_data]
struct DoorbellTestHandler<'a> {
tree: &'a Tree<'a>,
/// Deliveries that found the doorbell bit set.
irq_count: Atomic<u32>,
/// The doorbell leaf's pending bits, as read by the first delivery.
first_pending: Atomic<u32>,
/// The doorbell leaf's pending bits, as read by the second delivery.
second_pending: Atomic<u32>,
}
impl irq::Handler for DoorbellTestHandler<'_> {
fn handle(&self) -> irq::IrqReturn {
let leaf = self.tree.read_pending(DOORBELL_VECTOR.leaf_index());
let pending = leaf.vectors();
if !pending.contains(DOORBELL_VECTOR.leaf_mask()) {
self.tree.rearm_pci_irq(DOORBELL_SUBTREE);
return irq::IrqReturn::None;
}
leaf.clear_vectors(DOORBELL_VECTOR.leaf_mask());
// Rearm before counting the delivery, since the waiting thread triggers the next doorbell
// as soon as it observes the count.
self.tree.rearm_pci_irq(DOORBELL_SUBTREE);
match self.irq_count.load(Relaxed) {
0 => self.first_pending.store(pending.into_raw(), Relaxed),
1 => self.second_pending.store(pending.into_raw(), Relaxed),
_ => (),
}
self.irq_count.fetch_add(1, Release);
irq::IrqReturn::Handled
}
}
/// The self-test's handler registration and the enables that deliver to it.
///
/// Drops in the order that "Enabling the GSP event" in
/// `Documentation/gpu/nova/core/interrupts.rst` requires: the vector is disabled, then the
/// handler is freed, then the subtree is disabled.
struct SelftestResources<'a, 'r> {
_leaf_guard: LeafEnableGuard<'a>,
reg: Pin<KBox<irq::Registration<'r, DoorbellTestHandler<'a>>>>,
_top_guard: TopEnableGuard<'a>,
}
impl<'a> SelftestResources<'a, '_> {
fn handler(&self) -> &DoorbellTestHandler<'a> {
self.reg.handler()
}
/// Disables the doorbell vector and waits for a handler in flight on another CPU to finish.
///
/// The handler's counters and the leaf's pending bits are final on return.
fn quiesce_source(&self) {
self.handler()
.tree
.disable_leaf(DOORBELL_VECTOR.leaf_index(), DOORBELL_VECTOR.leaf_mask());
self.reg.synchronize();
}
}
/// Waits until `handler` has serviced `deliveries` interrupts, or [`DELIVERY_TIMEOUT`] elapses.
///
/// Returns `false` on timeout.
fn wait_for_deliveries(handler: &DoorbellTestHandler<'_>, deliveries: u32) -> bool {
const DELIVERY_POLL_INTERVAL: Delta = Delta::from_micros(100);
read_poll_timeout(
|| Ok(handler.irq_count.load(Acquire)),
|count| *count >= deliveries,
DELIVERY_POLL_INTERVAL,
DELIVERY_TIMEOUT,
)
.is_ok()
}
/// Runs the interrupt delivery self-test.
///
/// Call this only during probe, before GSP boot: it disables every vector in the tree and clears
/// every pending bit. On return, the doorbell's subtree is disabled at `TOP`, and the test's PCI
/// vectors and handler are released.
///
/// # Errors
///
/// `EINVAL` if `chipset` does not implement the doorbell's subtree. `ETIMEDOUT` if a delivery
/// does not arrive within [`DELIVERY_TIMEOUT`]. `EIO` if a self-test assertion fails.
/// Otherwise the error from allocating the PCI vectors or registering the handler.
pub(crate) fn run_selftest(pdev: &pci::Device<Bound>, bar: Bar0<'_>, chipset: Chipset) -> Result {
let tree = Tree::new(pdev, bar, chipset, DOORBELL_SUBTREE.into())?;
let tree_ref = &tree;
let request = tree.request_for(DOORBELL_SUBTREE)?;
let doorbell = DOORBELL_VECTOR.leaf_index();
let doorbell_mask = DOORBELL_VECTOR.leaf_mask();
dev_info!(
pdev,
"interrupt self-test: starting on vector {}, subtree {}, with {:?}\n",
DOORBELL_VECTOR.into_raw(),
DOORBELL_SUBTREE.index(),
tree.msi_type,
);
// GFW boot can leave vectors enabled and pending. Registering a handler unmasks the PCI
// interrupt, and they would be delivered to a handler that services only the doorbell.
tree.reset();
// A delivery proves nothing unless the doorbell bit starts out clear.
let pre_pending = tree.read_pending(doorbell).vectors();
selftest_assert!(
pdev,
!pre_pending.contains(doorbell_mask),
"vector {} already pending, leaf[{}] is {:#x}",
DOORBELL_VECTOR.into_raw(),
doorbell.get(),
pre_pending.into_raw()
);
let handler_init = try_pin_init!(DoorbellTestHandler {
tree: tree_ref,
irq_count: Atomic::new(0),
first_pending: Atomic::new(0),
second_pending: Atomic::new(0),
}? Error);
// Registration must precede any enable, or a delivery reaches no handler.
let reg = KBox::pin_init(
// SAFETY: this registration is dropped before the enclosing function returns, so its
// `Drop`, which calls `free_irq()`, always runs.
unsafe {
irq::Registration::new(
request,
irq::Flags::TRIGGER_NONE,
c"nova-core-selftest",
handler_init,
)
},
GFP_KERNEL,
)?;
let resources = SelftestResources {
_leaf_guard: tree.enable_leaf_guarded(doorbell, doorbell_mask),
_top_guard: tree.enable_top_guarded(),
reg,
};
let handler = resources.handler();
tree.trigger(DOORBELL_VECTOR)?;
let mut completed = wait_for_deliveries(handler, 1);
// The second trigger waits for the first delivery, or the two could coalesce.
if completed {
tree.trigger(DOORBELL_VECTOR)?;
completed = wait_for_deliveries(handler, 2);
}
resources.quiesce_source();
let count = handler.irq_count.load(Relaxed);
let first_pending = LeafMask::from_raw(handler.first_pending.load(Relaxed));
let second_pending = LeafMask::from_raw(handler.second_pending.load(Relaxed));
let residual = tree.read_pending(doorbell).vectors();
if !completed {
dev_err!(
pdev,
"interrupt self-test: only {} of 2 deliveries arrived within {} ms\n",
count,
DELIVERY_TIMEOUT.as_millis(),
);
return Err(ETIMEDOUT);
}
selftest_assert_eq!(pdev, count, 2, "delivery count");
// Every other vector in the leaf is disabled and was drained, so require the exact mask.
selftest_assert_eq!(pdev, first_pending, doorbell_mask, "first delivery");
selftest_assert_eq!(pdev, second_pending, doorbell_mask, "second delivery");
selftest_assert!(
pdev,
!residual.contains(doorbell_mask),
"vector {} still pending, leaf[{}] is {:#x}",
DOORBELL_VECTOR.into_raw(),
doorbell.get(),
residual.into_raw()
);
dev_info!(
pdev,
"interrupt self-test: passed, subtree {}, {} deliveries\n",
DOORBELL_SUBTREE.index(),
count,
);
Ok(())
}

195
drivers/gpu/nova-core/irq/gsp.rs Normal file
View File

@ -0,0 +1,195 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! The GSP event interrupt.
//!
//! The GSP posts messages to the GSP-to-CPU queue and raises SWGEN0, a software-generated cause
//! of its falcon. A threaded handler services it: the top half clears the tree and falcon state,
//! and the IRQ thread drains the queue.
//!
//! See "The GSP event" in `Documentation/gpu/nova/core/interrupts.rst`.
use kernel::{
device,
irq,
pci,
prelude::*, //
};
use super::interrupt_tree::{
GinVector,
LeafEnableGuard,
Subtree,
Tree, //
};
use crate::{
falcon::{
gsp::Gsp as GspFalcon,
Falcon, //
},
gsp::cmdq::Cmdq,
regs, //
};
/// The GSP event vector, which has the same number on every supported GPU.
const GSP_INTR_0_VECTOR: GinVector = GinVector::new::<155>();
/// The GSP event's subtree, the only one that nova-core services.
pub(crate) const GSP_SUBTREE: Subtree = GSP_INTR_0_VECTOR.subtree();
/// Clears the tree and falcon interrupt state that GSP boot leaves behind, and rearms PCI
/// interrupt delivery.
///
/// On return, no vector is enabled at its leaf, the serviced subtrees are disabled at `TOP`, and
/// the SWGEN0 latch is clear, so the next message that the GSP posts signals the tree.
pub(crate) fn quiesce(tree: &Tree<'_>, falcon: &Falcon<'_, GspFalcon>) {
tree.reset();
// The latch is cleared after the tree reset. The other order can leave the latch set with its
// leaf bit cleared. See "Enabling the GSP event" in interrupts.rst.
falcon.clear_swgen0_intr();
}
/// Threaded IRQ handler for the GSP event.
pub(crate) struct GspInterrupt<'a> {
falcon: &'a Falcon<'a, GspFalcon>,
cmdq: &'a Cmdq<'a>,
tree: &'a Tree<'a>,
/// For logging.
dev: &'a device::Device,
}
impl GspInterrupt<'_> {
/// Clears the latch of every host-routed cause in `status` other than SWGEN0, and logs them.
///
/// Returns the causes still set after the clear. A cause driven from outside the falcon stays
/// set, and only a device reset ends it.
fn clear_faults(
&self,
status: regs::NV_PFALCON_FALCON_IRQSTAT,
) -> regs::NV_PFALCON_FALCON_IRQSTAT {
let faults = status.with_swgen0(false);
if faults.into_raw() == 0 {
return faults;
}
dev_err!(
&self.dev,
"unserviceable GSP falcon interrupt, IRQSTAT {:#x}\n",
status.into_raw()
);
self.falcon.clear_intr(faults);
self.falcon.read_host_intr().with_swgen0(false)
}
}
impl irq::ThreadedHandler for GspInterrupt<'_> {
/// Top half, in hard interrupt context. Services the GSP vector only, so another vector
/// pending in the same leaf stays pending.
fn handle(&self) -> irq::ThreadedIrqReturn {
let leaf = self.tree.read_pending(GSP_INTR_0_VECTOR.leaf_index());
if !leaf.vectors().contains(GSP_INTR_0_VECTOR.leaf_mask()) {
self.tree.rearm_pci_irq(GSP_SUBTREE);
return irq::ThreadedIrqReturn::None;
}
leaf.clear_vectors(GSP_INTR_0_VECTOR.leaf_mask());
let status = self.falcon.take_host_intr();
let remaining_faults = self.clear_faults(status);
if remaining_faults.into_raw() == 0 {
self.falcon.retrigger_intr();
} else {
// Disabling the vector loses no notification: the falcon signals nothing further
// while a cause stays set. See "Retriggering a falcon" in interrupts.rst.
self.tree.disable_leaf(
GSP_INTR_0_VECTOR.leaf_index(),
GSP_INTR_0_VECTOR.leaf_mask(),
);
dev_err!(
&self.dev,
"GSP falcon cause {:#x} needs a device reset, GSP events are no longer serviced\n",
remaining_faults.into_raw()
);
}
self.tree.rearm_pci_irq(GSP_SUBTREE);
if status.swgen0() {
irq::ThreadedIrqReturn::WakeThread
} else {
irq::ThreadedIrqReturn::Handled
}
}
/// IRQ thread. Drains the GSP-to-CPU queue, which may sleep.
fn handle_threaded(&self) -> irq::IrqReturn {
if let Err(e) = self.cmdq.drain() {
// The failed message stays at the queue head, so every later drain fails the same way.
self.tree.disable_leaf(
GSP_INTR_0_VECTOR.leaf_index(),
GSP_INTR_0_VECTOR.leaf_mask(),
);
dev_err!(
&self.dev,
"GSP event drain failed ({:?}), the message queue is no longer serviced\n",
e
);
}
irq::IrqReturn::Handled
}
}
/// The registered GSP event handler and the enable of its vector.
///
/// The declaration order is the drop order, and it is required: the vector is disabled before
/// `free_irq` runs. See "Enabling the GSP event" in `Documentation/gpu/nova/core/interrupts.rst`.
#[pin_data]
pub(crate) struct GspIrq<'a> {
_leaf_guard: LeafEnableGuard<'a>,
#[pin]
reg: irq::ThreadedRegistration<'a, GspInterrupt<'a>>,
}
impl<'a> GspIrq<'a> {
/// Returns an initializer that registers the threaded handler and then enables the GSP
/// vector at its leaf.
///
/// An event that latched while the vector was disabled is delivered as soon as the GSP
/// subtree is enabled at `TOP`.
///
/// # Errors
///
/// `EINVAL` if `tree` does not service the GSP subtree. Otherwise the error from
/// `request_threaded_irq`.
///
/// # Safety
///
/// Callers must not `mem::forget()` the initialized `GspIrq` or otherwise prevent its [`Drop`]
/// implementation, which runs `free_irq`, from running.
pub(crate) unsafe fn new(
pdev: &'a pci::Device<device::Bound>,
tree: &'a Tree<'a>,
falcon: &'a Falcon<'a, GspFalcon>,
cmdq: &'a Cmdq<'a>,
) -> impl PinInit<Self, Error> + 'a {
let dev = pdev.as_ref();
try_pin_init!(Self {
// SAFETY: this function's caller must not leak the `GspIrq` that owns this
// registration, so the registration's `Drop` runs.
reg <- unsafe {
irq::ThreadedRegistration::new(
tree.request_for(GSP_SUBTREE)?,
irq::Flags::TRIGGER_NONE,
c"nova-core",
Ok(GspInterrupt { falcon, cmdq, tree, dev }),
)
},
_leaf_guard: tree.enable_leaf_guarded(
GSP_INTR_0_VECTOR.leaf_index(),
GSP_INTR_0_VECTOR.leaf_mask(),
),
})
}
}

67
drivers/gpu/nova-core/irq/hal.rs Normal file
View File

@ -0,0 +1,67 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Per-architecture differences in the GIN CPU interrupt tree.
//!
//! See "Per-architecture differences" in `Documentation/gpu/nova/core/interrupts.rst`.
mod gh100;
mod tu102;
use kernel::{
io::Io,
prelude::*, //
};
use crate::{
driver::Bar0,
gpu::{
Architecture,
Chipset, //
}, //
};
use super::{
interrupt_tree::{
LeafCount,
Subtree,
SubtreeSet, //
},
regs::*,
MsiType, //
};
/// Clears and then sets the `TOP` enables of `subtrees`, which produces a new delivery edge.
fn cycle_top_enables(bar: Bar0<'_>, subtrees: SubtreeSet) {
bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(subtrees));
bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(subtrees));
}
/// The leaf count and the rearm of the GIN CPU tree, which differ by GPU family.
pub(super) trait CpuInterruptHal: Send + Sync {
/// Returns the number of leaves the tree implements.
fn leaf_count(&self) -> LeafCount;
/// Rearms PCI interrupt delivery after a handler serviced `subtree`.
///
/// `serviced` is every subtree that nova-core services, for the Hopper-plus MSI form, which
/// cycles the `TOP` enables of them all. See "Rearming PCI interrupt delivery" in
/// `Documentation/gpu/nova/core/interrupts.rst`.
fn rearm_pci_irq(
&self,
bar: Bar0<'_>,
msi_type: MsiType,
serviced: SubtreeSet,
subtree: Subtree,
);
}
/// Returns the [`CpuInterruptHal`] for `chipset`'s architecture.
pub(super) fn cpu_interrupt_hal(chipset: Chipset) -> &'static dyn CpuInterruptHal {
match chipset.arch() {
Architecture::Turing | Architecture::Ampere | Architecture::Ada => tu102::TU102_HAL,
Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
gh100::GH100_HAL
}
}
}

40
drivers/gpu/nova-core/irq/hal/gh100.rs Normal file
View File

@ -0,0 +1,40 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
use crate::driver::Bar0;
use super::{
cycle_top_enables,
CpuInterruptHal,
LeafCount,
MsiType,
Subtree,
SubtreeSet, //
};
/// The CPU interrupt tree properties of Hopper and Blackwell.
struct Gh100;
impl CpuInterruptHal for Gh100 {
fn leaf_count(&self) -> LeafCount {
LeafCount::Sixteen
}
fn rearm_pci_irq(
&self,
bar: Bar0<'_>,
msi_type: MsiType,
serviced: SubtreeSet,
subtree: Subtree,
) {
let subtrees = match msi_type {
MsiType::Msi => serviced,
MsiType::MsiX => subtree.into(),
};
cycle_top_enables(bar, subtrees);
}
}
const GH100: Gh100 = Gh100;
pub(super) const GH100_HAL: &dyn CpuInterruptHal = &GH100;

43
drivers/gpu/nova-core/irq/hal/tu102.rs Normal file
View File

@ -0,0 +1,43 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
use kernel::io::Io;
use crate::{
driver::Bar0,
irq::regs::NV_XVE_CYA_2, //
};
use super::{
cycle_top_enables,
CpuInterruptHal,
LeafCount,
MsiType,
Subtree,
SubtreeSet, //
};
/// The CPU interrupt tree properties of Turing, Ampere, and Ada.
struct Tu102;
impl CpuInterruptHal for Tu102 {
fn leaf_count(&self) -> LeafCount {
LeafCount::Eight
}
fn rearm_pci_irq(
&self,
bar: Bar0<'_>,
msi_type: MsiType,
_serviced: SubtreeSet,
subtree: Subtree,
) {
match msi_type {
MsiType::Msi => bar.write(NV_XVE_CYA_2, 0u32.into()),
MsiType::MsiX => cycle_top_enables(bar, subtree.into()),
}
}
}
const TU102: Tu102 = Tu102;
pub(super) const TU102_HAL: &dyn CpuInterruptHal = &TU102;

682
drivers/gpu/nova-core/irq/interrupt_tree.rs Normal file
View File

@ -0,0 +1,682 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! The GIN CPU interrupt tree for one PCIe function.
//!
//! A [`GinVector`] names an interrupt source, a [`LeafIndex`] the leaf register that latches it,
//! a [`LeafMask`] a set of vectors within one leaf, and a [`Subtree`] one `TOP` bit. The types
//! keep the four from being confused with one another.
//!
//! Servicing a leaf requires reading its pending bits before clearing them. Only
//! [`Tree::read_pending`] produces a [`LeafPending`], and only a [`LeafPending`] clears a leaf,
//! so that the wrong order does not compile. This module does not serialize access to the tree.
//!
//! See `Documentation/gpu/nova/core/interrupts.rst`.
use kernel::{
device::Bound,
io::{
register::Array,
Io, //
},
irq,
num::Bounded,
pci::{
self,
IrqType, //
},
prelude::*, //
};
use crate::{
driver::Bar0,
gpu::Chipset,
num, //
};
use super::{
hal::{
cpu_interrupt_hal,
CpuInterruptHal, //
},
regs::*,
MsiType, //
};
/// Number of vectors one leaf register carries, one per bit.
const VECTORS_PER_LEAF: u32 = u32::BITS;
/// Number of leaves one subtree covers.
const LEAVES_PER_SUBTREE: u32 = 2;
/// Number of subtrees the widest supported tree implements.
const MAX_NUM_SUBTREES: u32 = 8;
/// Number of leaves the widest supported tree implements.
const MAX_NUM_LEAVES: u32 = MAX_NUM_SUBTREES * LEAVES_PER_SUBTREE;
/// Number of bits needed to address every vector in the widest supported tree.
const VECTOR_BITS: u32 = (MAX_NUM_LEAVES * VECTORS_PER_LEAF).ilog2();
/// Width of the vector field in the leaf trigger register.
const TRIGGER_VECTOR_BITS: u32 = {
let range = NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::VECTOR_RANGE;
num::u8_as_u32(*range.end() - *range.start() + 1)
};
/// Index of a leaf register within the widest supported tree. An 8-leaf tree implements only the
/// lower half of the range.
pub(super) type LeafIndex = Bounded<usize, { MAX_NUM_LEAVES.ilog2() }>;
/// Number of leaves a tree implements.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
#[repr(u32)]
pub(super) enum LeafCount {
/// Turing through Ada.
Eight = 8,
/// Hopper and later.
Sixteen = 16,
}
impl LeafCount {
pub(super) const fn into_u32(self) -> u32 {
// CAST: `LeafCount` is `repr(u32)`, so the cast is lossless.
self as u32
}
pub(super) const fn into_raw(self) -> usize {
num::u32_as_usize(self.into_u32())
}
/// Returns the number of subtrees a tree of this size implements.
pub(super) const fn subtree_count(self) -> u32 {
self.into_u32() / LEAVES_PER_SUBTREE
}
/// Returns the set of every subtree a tree of this size implements.
pub(super) const fn subtree_set(self) -> SubtreeSet {
SubtreeSet((1u32 << self.subtree_count()) - 1)
}
/// Returns the number of vectors a tree of this size carries.
pub(super) const fn vector_count(self) -> u32 {
self.into_u32() * VECTORS_PER_LEAF
}
/// Returns every leaf a tree of this size implements.
pub(super) fn iter(self) -> impl Iterator<Item = LeafIndex> {
(0..self.into_raw()).filter_map(LeafIndex::try_new)
}
}
// `VECTOR_BITS` and `LeafCount::Sixteen` are written separately. This assert keeps them in
// agreement about the widest supported tree.
static_assert!(1 << VECTOR_BITS == LeafCount::Sixteen.vector_count());
/// Set of vectors within one leaf, one bit per vector.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) struct LeafMask(u32);
impl LeafMask {
/// Returns the mask with every vector set.
pub(super) const fn all() -> Self {
Self(u32::MAX)
}
#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
pub(super) const fn from_raw(raw: u32) -> Self {
Self(raw)
}
#[cfg_attr(
not(any(CONFIG_NOVA_CORE_SELFTESTS, CONFIG_KUNIT = "y")),
expect(dead_code)
)]
pub(super) const fn into_raw(self) -> u32 {
self.0
}
pub(super) const fn is_empty(self) -> bool {
self.0 == 0
}
/// Returns whether every vector in `other` is also in this mask.
pub(super) const fn contains(self, other: Self) -> bool {
self.0 & other.0 == other.0
}
}
impl From<Bounded<u32, 32>> for LeafMask {
fn from(vectors: Bounded<u32, 32>) -> Self {
Self(vectors.get())
}
}
impl From<LeafMask> for Bounded<u32, 32> {
fn from(vectors: LeafMask) -> Self {
vectors.0.into()
}
}
/// One subtree, held as the `TOP` bit that covers it.
///
/// # Invariants
///
/// Exactly one bit is set.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) struct Subtree(u32);
impl Subtree {
/// Returns the subtree at index `idx`.
const fn new(idx: u32) -> Self {
// INVARIANT: shifting `1` left leaves exactly one bit set.
Self(1 << idx)
}
/// Returns this subtree's index within the tree.
pub(super) const fn index(self) -> u32 {
self.0.trailing_zeros()
}
pub(super) const fn into_raw(self) -> u32 {
self.0
}
}
/// Set of subtrees, one bit per subtree, in the layout of the `TOP` registers.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) struct SubtreeSet(u32);
impl SubtreeSet {
pub(super) const fn contains(self, subtree: Subtree) -> bool {
self.0 & subtree.into_raw() != 0
}
pub(super) const fn is_empty(self) -> bool {
self.0 == 0
}
pub(super) const fn intersection(self, other: Self) -> Self {
Self(self.0 & other.0)
}
/// Returns one more than the highest index in this set, or `0` for an empty set. An MSI-X
/// allocation that covers the set needs this many entries.
pub(super) const fn span(self) -> u32 {
u32::BITS - self.0.leading_zeros()
}
/// Returns the subtrees of this set, lowest index first.
pub(super) fn iter(self) -> impl Iterator<Item = Subtree> {
(0..u32::BITS)
.map(Subtree::new)
.filter(move |subtree| self.contains(*subtree))
}
}
impl From<Subtree> for SubtreeSet {
fn from(subtree: Subtree) -> Self {
Self(subtree.into_raw())
}
}
impl From<Bounded<u32, 32>> for SubtreeSet {
fn from(subtrees: Bounded<u32, 32>) -> Self {
Self(subtrees.get())
}
}
impl From<SubtreeSet> for Bounded<u32, 32> {
fn from(subtrees: SubtreeSet) -> Self {
subtrees.0.into()
}
}
/// A GIN interrupt vector, bounded to the widest tree that any supported chipset implements.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) struct GinVector(Bounded<u32, VECTOR_BITS>);
impl GinVector {
/// Returns vector number `VECTOR`.
///
/// Fails to compile if `VECTOR` is beyond the widest supported tree.
pub(super) const fn new<const VECTOR: u32>() -> Self {
Self(Bounded::<u32, VECTOR_BITS>::new::<VECTOR>())
}
#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
pub(super) const fn into_raw(self) -> u32 {
self.0.get()
}
/// Returns this vector's leaf.
pub(super) fn leaf_index(self) -> LeafIndex {
// CALC: `self.0 / VECTORS_PER_LEAF`.
self.0.shr::<{ VECTORS_PER_LEAF.ilog2() }, _>().cast()
}
/// Returns this vector's bit within its leaf.
pub(super) const fn leaf_mask(self) -> LeafMask {
LeafMask(1 << (self.0.get() % VECTORS_PER_LEAF))
}
/// Returns this vector's subtree.
pub(super) const fn subtree(self) -> Subtree {
Subtree::new(self.0.get() / (VECTORS_PER_LEAF * LEAVES_PER_SUBTREE))
}
/// Checks that a tree with `leaves` leaves implements this vector.
///
/// # Errors
///
/// `EINVAL` if it does not.
pub(super) const fn validate(self, leaves: LeafCount) -> Result {
if self.0.get() >= leaves.vector_count() {
Err(EINVAL)
} else {
Ok(())
}
}
}
impl From<GinVector> for Bounded<u32, TRIGGER_VECTOR_BITS> {
fn from(vector: GinVector) -> Self {
vector.0.extend()
}
}
/// Disables `vectors` in `leaf`.
fn clear_leaf_enables(bar: Bar0<'_>, leaf: LeafIndex, vectors: LeafMask) {
bar.write(
Array::at(*leaf),
NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR::zeroed().with_vectors(vectors),
);
}
/// Disables the subtrees in `serviced` at `TOP`.
fn clear_top_enables(bar: Bar0<'_>, serviced: SubtreeSet) {
bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(serviced));
}
/// The CPU tree of one PCIe function, the subtrees that nova-core services, and the PCI vectors
/// that deliver them.
///
/// A subtree may be enabled at `TOP` only once a handler is registered on the PCI vector that
/// delivers it. See "The serviced-subtree invariant" in
/// `Documentation/gpu/nova/core/interrupts.rst`.
pub(crate) struct Tree<'a> {
bar: Bar0<'a>,
hal: &'static dyn CpuInterruptHal,
serviced: SubtreeSet,
/// The interrupt type that Linux granted for `vectors`.
pub(super) msi_type: MsiType,
vectors: pci::IrqVectorRegistration<'a>,
}
impl<'a> Tree<'a> {
/// Creates the tree of `chipset` and allocates the PCI vectors for the subtrees in `serviced`.
///
/// Requests MSI-X entries `0` through the highest subtree in `serviced`, since an allocation
/// cannot be sparse, and falls back to a single MSI message for the whole tree.
///
/// # Errors
///
/// `EINVAL` if `serviced` is empty or names a subtree that `chipset` does not implement.
/// Otherwise, when neither interrupt type could be allocated, the error from the MSI request.
pub(crate) fn new(
pdev: &'a pci::Device<Bound>,
bar: Bar0<'a>,
chipset: Chipset,
serviced: SubtreeSet,
) -> Result<Self> {
let hal = cpu_interrupt_hal(chipset);
let num_leaves = hal.leaf_count();
if serviced.is_empty() || serviced.intersection(num_leaves.subtree_set()) != serviced {
return Err(EINVAL);
}
let entries = serviced.span();
let (vectors, msi_type) = pdev
.alloc_irq_vectors(entries, entries, IrqType::MsiX.into())
.map(|vectors| (vectors, MsiType::MsiX))
// Every subtree raises the one MSI message, so one MSI vector serves the whole tree.
// See "Delivery over PCI" in `Documentation/gpu/nova/core/interrupts.rst`.
.or_else(|_| {
pdev.alloc_irq_vectors(1, 1, IrqType::Msi.into())
.map(|vectors| (vectors, MsiType::Msi))
})?;
Ok(Self {
bar,
hal,
serviced,
msi_type,
vectors,
})
}
/// Returns the [`irq::IrqRequest`] for the PCI vector that delivers `subtree`.
///
/// # Errors
///
/// `EINVAL` if `subtree` is not one of the serviced subtrees.
pub(super) fn request_for(&self, subtree: Subtree) -> Result<irq::IrqRequest<'_>> {
if !self.serviced.contains(subtree) {
return Err(EINVAL);
}
let entry = match self.msi_type {
MsiType::MsiX => num::u32_as_usize(subtree.index()),
MsiType::Msi => 0,
};
self.vectors.index(entry).map(Into::into)
}
/// Rearms PCI interrupt delivery to the CPU after servicing `subtree`, the one subtree that
/// the calling handler serves.
///
/// A handler must call this before returning, or it receives no further interrupts.
pub(super) fn rearm_pci_irq(&self, subtree: Subtree) {
self.hal
.rearm_pci_irq(self.bar, self.msi_type, self.serviced, subtree);
}
/// Enables the serviced subtrees at `TOP`.
///
/// Each of them must have a handler registered on its PCI vector.
pub(super) fn enable_top(&self) {
self.bar.write_reg(
NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(self.serviced),
);
}
/// Disables the serviced subtrees at `TOP`.
pub(super) fn disable_top(&self) {
clear_top_enables(self.bar, self.serviced);
}
/// Enables the serviced subtrees at `TOP` until the returned guard drops.
pub(crate) fn enable_top_guarded(&self) -> TopEnableGuard<'a> {
self.enable_top();
TopEnableGuard {
bar: self.bar,
serviced: self.serviced,
}
}
/// Enables `vectors` in `leaf`.
pub(super) fn enable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
self.bar.write(
Array::at(*leaf),
NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET::zeroed().with_vectors(vectors),
);
}
/// Disables `vectors` in `leaf`.
pub(super) fn disable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
clear_leaf_enables(self.bar, leaf, vectors);
}
/// Enables `vectors` in `leaf` until the returned guard drops.
pub(super) fn enable_leaf_guarded(
&self,
leaf: LeafIndex,
vectors: LeafMask,
) -> LeafEnableGuard<'a> {
self.enable_leaf(leaf, vectors);
LeafEnableGuard {
bar: self.bar,
leaf,
vectors,
}
}
/// Reads the pending bits of `leaf`, and returns the handle that clears them.
pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
let pending = self
.bar
.read(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::at(*leaf))
.vectors();
LeafPending {
bar: self.bar,
leaf,
pending,
}
}
/// Sets the pending bit of `vector` in its leaf, as if the vector's source had raised it.
///
/// # Errors
///
/// `EINVAL` if this tree does not implement `vector`.
#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
pub(super) fn trigger(&self, vector: GinVector) -> Result {
vector.validate(self.hal.leaf_count())?;
self.bar.write_reg(
NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::zeroed().with_vector(vector),
);
Ok(())
}
/// Disables every vector in every implemented leaf, including the subtrees that nova-core does
/// not service. Call this only during probe.
pub(super) fn disable_all_leaves(&self) {
for leaf in self.hal.leaf_count().iter() {
self.disable_leaf(leaf, LeafMask::all());
}
}
/// Clears every pending bit in every implemented leaf, including the subtrees that nova-core
/// does not service.
///
/// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with no
/// interrupt handler registered.
pub(super) fn drain(&self) {
self.disable_top();
// A vector that latched while disabled does not show in `TOP`, so read every leaf rather
// than descending from it.
for leaf in self.hal.leaf_count().iter() {
self.read_pending(leaf).clear();
}
}
/// Disables every vector, clears every pending bit, and rearms PCI interrupt delivery.
///
/// The serviced subtrees are disabled at `TOP` on return. Call this only during probe, with
/// no interrupt handler registered.
pub(super) fn reset(&self) {
self.disable_all_leaves();
self.drain();
for subtree in self.serviced.iter() {
self.rearm_pci_irq(subtree);
}
// A `TOP`-enable rearm leaves the subtrees that it cycles enabled, and a subtree may be
// enabled only once a handler is registered on its vector.
self.disable_top();
}
}
/// The pending bits of one leaf as they were read, and the handle that clears them.
pub(super) struct LeafPending<'a> {
bar: Bar0<'a>,
leaf: LeafIndex,
pending: LeafMask,
}
impl LeafPending<'_> {
pub(super) fn vectors(&self) -> LeafMask {
self.pending
}
/// Clears the vectors that were pending at the read. A vector that latched since stays pending.
pub(super) fn clear(&self) {
self.clear_vectors(self.pending);
}
/// Clears `vectors` and no other bit.
pub(super) fn clear_vectors(&self, vectors: LeafMask) {
if !vectors.is_empty() {
self.bar.write(
Array::at(*self.leaf),
NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::zeroed().with_vectors(vectors),
);
}
}
}
/// Disables a set of vectors in one leaf when dropped.
pub(super) struct LeafEnableGuard<'a> {
bar: Bar0<'a>,
leaf: LeafIndex,
vectors: LeafMask,
}
impl Drop for LeafEnableGuard<'_> {
fn drop(&mut self) {
clear_leaf_enables(self.bar, self.leaf, self.vectors);
}
}
/// Disables the serviced subtrees at `TOP` when dropped.
pub(crate) struct TopEnableGuard<'a> {
bar: Bar0<'a>,
serviced: SubtreeSet,
}
impl Drop for TopEnableGuard<'_> {
fn drop(&mut self) {
clear_top_enables(self.bar, self.serviced);
}
}
#[kunit_tests(nova_core_gin_tree)]
mod tests {
use super::*;
/// A leaf index cannot name a leaf beyond the widest supported tree.
#[test]
fn leaf_index_bounds() {
assert!(LeafIndex::try_new(0).is_some());
assert!(LeafIndex::try_new(15).is_some());
assert!(LeafIndex::try_new(16).is_none());
}
/// The subtree count, the implemented-subtree set, and the vector count follow the leaf count.
#[test]
fn leaf_count_derives_subtrees_and_vectors() {
assert_eq!(LeafCount::Eight.subtree_count(), 4);
assert_eq!(
Bounded::<u32, 32>::from(LeafCount::Eight.subtree_set()).get(),
0x0f
);
assert_eq!(LeafCount::Eight.vector_count(), 256);
assert_eq!(LeafCount::Sixteen.subtree_count(), 8);
assert_eq!(
Bounded::<u32, 32>::from(LeafCount::Sixteen.subtree_set()).get(),
0xff
);
assert_eq!(LeafCount::Sixteen.vector_count(), 512);
}
/// A tree enumerates every leaf that it implements, in order, and no more.
#[test]
fn leaf_count_iter_covers_the_tree() {
for (count, expected) in [(LeafCount::Eight, 8usize), (LeafCount::Sixteen, 16)] {
let mut seen = 0;
for (index, leaf) in count.iter().enumerate() {
assert_eq!(leaf.get(), index);
seen += 1;
}
assert_eq!(seen, expected);
}
}
/// A vector maps to its leaf, its bit within that leaf, and its subtree. The doorbell (129)
/// and the GSP event (155) share a subtree.
#[test]
fn vector_maps_to_leaf_bit_and_subtree() {
let doorbell = GinVector::new::<129>();
let gsp = GinVector::new::<155>();
assert_eq!(doorbell.leaf_index().get(), 4);
assert_eq!(doorbell.leaf_mask().into_raw(), 1 << 1);
assert_eq!(doorbell.subtree().index(), 2);
assert_eq!(gsp.leaf_index().get(), 4);
assert_eq!(gsp.leaf_mask().into_raw(), 1 << 27);
assert_eq!(gsp.subtree().index(), 2);
assert_eq!(doorbell.subtree(), gsp.subtree());
}
/// Both fixed vectors are within the 8-leaf tree, so every supported chipset implements them.
#[test]
fn fixed_vectors_fit_the_narrowest_tree() {
assert!(GinVector::new::<129>().validate(LeafCount::Eight).is_ok());
assert!(GinVector::new::<155>().validate(LeafCount::Eight).is_ok());
// The first vector beyond an 8-leaf tree.
assert!(GinVector::new::<256>().validate(LeafCount::Eight).is_err());
assert!(GinVector::new::<256>().validate(LeafCount::Sixteen).is_ok());
}
/// A subtree set reports membership, intersection, and its span from subtree 0.
#[test]
fn subtree_set_operations() {
let gsp = GinVector::new::<155>().subtree();
assert!(LeafCount::Eight.subtree_set().contains(gsp));
assert!(!LeafCount::Eight.subtree_set().is_empty());
// The GSP needs no subtree above 2, so an MSI-X request covers entries 0 through 2.
assert_eq!(SubtreeSet::from(gsp).span(), 3);
// A 16-leaf tree implements every subtree that an 8-leaf tree does.
assert_eq!(
LeafCount::Sixteen
.subtree_set()
.intersection(LeafCount::Eight.subtree_set()),
LeafCount::Eight.subtree_set()
);
}
/// Iterating a subtree set yields each subtree once, lowest index first, and nothing for an
/// empty set.
#[test]
fn subtree_set_iterates_its_members() {
assert!(LeafCount::Eight
.subtree_set()
.iter()
.map(Subtree::index)
.eq([0u32, 1, 2, 3]));
let gsp = SubtreeSet::from(GinVector::new::<155>().subtree());
assert!(gsp.iter().map(Subtree::index).eq([2u32]));
let empty = SubtreeSet::from(Bounded::<u32, 32>::new::<0>());
assert_eq!(empty.iter().count(), 0);
}
/// Every supported chipset implements the subtree that carries the GSP event.
#[test]
fn gsp_subtree_is_implemented_everywhere() {
for &chipset in Chipset::ALL {
assert!(cpu_interrupt_hal(chipset)
.leaf_count()
.subtree_set()
.contains(crate::irq::gsp::GSP_SUBTREE));
}
}
}

88
drivers/gpu/nova-core/irq/regs.rs Normal file
View File

@ -0,0 +1,88 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
use kernel::io::register;
use crate::driver::NovaRegisters;
use super::interrupt_tree::{
LeafMask,
SubtreeSet, //
};
// The GIN CPU interrupt tree, reached through the `NV_VIRTUAL_FUNCTION_PRIV` aperture. See
// "Register naming" in `Documentation/gpu/nova/core/interrupts.rst`. The leaf arrays are declared
// with 16 entries, the widest tree that any supported chipset implements.
register! {
base: NovaRegisters;
/// Pending bits of one leaf, one per vector.
///
/// Vector `v` is bit `v % 32` of leaf `v / 32`. The bit is set when the vector's source
/// drives it, whether or not the vector is enabled. Writing a `1` clears the bit, and a `0`
/// leaves it as it was.
pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF(u32)[16] @ 0x00b81000 {
/// The vectors pending in this leaf.
31:0 vectors => LeafMask;
}
/// Enables vectors of one leaf.
///
/// A `1` enables the matching vector, and a `0` leaves it as it was.
pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET(u32)[16] @ 0x00b81200 {
/// Vectors to enable.
31:0 vectors => LeafMask;
}
/// Disables vectors of one leaf.
///
/// A `1` disables the matching vector, and a `0` leaves it as it was. A disabled vector still
/// latches in `LEAF`, and `TOP` does not show it.
pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR(u32)[16] @ 0x00b81400 {
/// Vectors to disable.
31:0 vectors => LeafMask;
}
/// Enables subtrees.
///
/// Bit `N` covers subtree `N`, which is leaves `2N` and `2N + 1`. A `1` enables the matching
/// subtree, and a `0` leaves it as it was.
///
/// The hardware headers declare a one-element array, so nova-core declares a scalar.
pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET(u32) @ 0x00b81608 {
/// Subtrees to enable.
31:0 subtrees => SubtreeSet;
}
/// Disables subtrees, with the bit layout of `TOP_EN_SET`.
///
/// A `1` disables the matching subtree, and a `0` leaves it as it was. A disabled subtree
/// delivers nothing, and `TOP` still reports it.
pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR(u32) @ 0x00b81610 {
/// Subtrees to disable.
31:0 subtrees => SubtreeSet;
}
/// Latches a vector from software. Write-only.
///
/// The write sets the pending bit of the vector in its `LEAF` register, as if the vector's
/// source had raised it, and the vector reaches the CPU under the same enables. Implemented on
/// every supported chipset.
pub(super) NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER(u32) @ 0x00b81640 {
/// Vector to latch.
11:0 vector;
}
}
// PCI configuration-space mirror in BAR0.
register! {
base: NovaRegisters;
/// MSI end-of-interrupt register. Writing any value rearms MSI delivery.
///
/// Only pre-Hopper MSI rearms through this register. See "Rearming PCI interrupt delivery"
/// in `Documentation/gpu/nova/core/interrupts.rst`.
pub(super) NV_XVE_CYA_2(u32) @ 0x00088704 {}
}

16
drivers/gpu/nova-core/mctp.rs
View File

@ -42,13 +42,25 @@ pub(crate) struct MctpHeader(u32) {
29:28 seq;
/// Source endpoint ID.
23:16 seid;
/// Destination endpoint ID.
15:8 deid;
/// MCTP header version.
3:0 version;
}
}
impl MctpHeader {
/// Builds a single-packet MCTP header (`SOM=1`, `EOM=1`, `SEQ=0`, `SEID=0`).
/// The MCTP header version that this driver uses.
const VERSION: u32 = 1;
/// Builds the MCTP header of a message that fits in one packet: `SOM` and `EOM` set, the
/// version set, and every other field zero.
pub(crate) fn single_packet() -> Self {
Self::zeroed().with_som(true).with_eom(true)
Self::zeroed()
.with_const_version::<{ Self::VERSION }>()
.with_const_deid::<0>()
.with_som(true)
.with_eom(true)
}
/// Returns whether this is a complete single-packet message (`SOM=1` and `EOM=1`).

335
drivers/gpu/nova-core/mm.rs Normal file
View File

@ -0,0 +1,335 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Memory management subsystems.
#![cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
/// Implements `From` conversions between a frame-number type and `Bounded<u64, N>`.
///
/// Each MMU version module should invoke this for the specific bit widths used by that version's
/// PTE/PDE bitfield definitions.
macro_rules! impl_frame_number_bounded {
($type:ty, $bits:literal) => {
impl From<Bounded<u64, $bits>> for $type {
fn from(val: Bounded<u64, $bits>) -> Self {
Self::new(val.get())
}
}
impl From<$type> for Bounded<u64, $bits> {
fn from(v: $type) -> Self {
Bounded::from_expr(v.raw() & ::kernel::bits::genmask_u64(0..=($bits - 1)))
}
}
};
}
/// Implements `From` conversions between [`Pfn`] and `Bounded<u64, N>` for bitfield interop.
macro_rules! impl_pfn_bounded {
($bits:literal) => {
impl_frame_number_bounded!(Pfn, $bits);
};
}
use core::{
fmt::LowerHex,
ops, //
};
use kernel::{
bitfield,
fmt,
gpu::buddy::{
GpuBuddy,
GpuBuddyParams, //
},
num::Bounded,
prelude::*,
ptr::{
Alignable,
Alignment, //
},
sizes::SZ_4K, //
};
use crate::{
driver::Bar0,
gpu::Chipset, //
};
pub(crate) use tlb::Tlb;
pub(crate) mod bar_user;
mod hal;
pub(super) mod pagetable;
mod pramin;
mod regs;
pub(super) mod tlb;
pub(super) mod vmm;
/// GPU Memory Manager - owns all core MM components.
///
/// Provides centralized ownership of memory management resources:
/// - [`GpuBuddy`] allocator for VRAM page table allocation.
/// - [`pramin::Pramin`] for direct VRAM access.
/// - [`Tlb`] manager for translation buffer flush operations.
pub(crate) struct GpuMm<'gpu> {
buddy: GpuBuddy,
pramin: pramin::Pramin<'gpu>,
tlb: Pin<KBox<Tlb<'gpu>>>,
}
impl<'gpu> GpuMm<'gpu> {
/// Creates the GPU memory manager.
pub(crate) fn new(
bar: Bar0<'gpu>,
chipset: Chipset,
buddy_params: GpuBuddyParams,
total_fb_end: VramAddress,
) -> Result<Self> {
// PRAMIN covers all physical VRAM (including GSP-reserved areas
// above the usable region, e.g. the BAR1 page directory).
let vram_region = VramAddress::ZERO..total_fb_end;
Ok(Self {
buddy: GpuBuddy::new(buddy_params)?,
pramin: pramin::Pramin::new(bar, chipset, vram_region)?,
tlb: KBox::pin_init(Tlb::new(bar), GFP_KERNEL)?,
})
}
/// Access the [`GpuBuddy`] allocator.
pub(crate) fn buddy(&self) -> &GpuBuddy {
&self.buddy
}
/// Access the [`pramin::Pramin`].
fn pramin_mut(&mut self) -> &mut pramin::Pramin<'gpu> {
&mut self.pramin
}
/// Access the [`Tlb`] manager.
pub(crate) fn tlb(&self) -> &Tlb<'gpu> {
self.tlb.as_ref().get_ref()
}
}
/// Page size in bytes (4 KiB).
pub(crate) const PAGE_SIZE: usize = SZ_4K;
/// Physical VRAM address in GPU video memory.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
#[repr(transparent)]
pub(crate) struct VramAddress(u64);
impl VramAddress {
/// The zero address.
pub(crate) const ZERO: Self = Self::from_raw(0);
/// Creates an address from a raw value.
pub(crate) const fn from_raw(addr: u64) -> Self {
Self(addr)
}
/// Returns the address as a raw value.
pub(crate) const fn into_raw(self) -> u64 {
self.0
}
/// Adds `rhs` to this address, returning [`None`] on overflow.
pub(crate) const fn checked_add(self, rhs: u64) -> Option<Self> {
match self.into_raw().checked_add(rhs) {
Some(addr) => Some(Self::from_raw(addr)),
None => None,
}
}
}
impl Alignable for VramAddress {
fn align_down(self, alignment: Alignment) -> Self {
Self::from_raw(self.into_raw().align_down(alignment))
}
fn align_up(self, alignment: Alignment) -> Option<Self> {
self.into_raw().align_up(alignment).map(Self::from_raw)
}
}
impl LowerHex for VramAddress {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
LowerHex::fmt(&self.into_raw(), f)
}
}
impl fmt::Debug for VramAddress {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_fmt(fmt!("{:#x}", self))
}
}
impl ops::Add<u64> for VramAddress {
type Output = Self;
fn add(self, rhs: u64) -> Self::Output {
Self::from_raw(self.into_raw() + rhs)
}
}
impl ops::Sub for VramAddress {
type Output = u64;
fn sub(self, rhs: Self) -> Self::Output {
self.into_raw() - rhs.into_raw()
}
}
impl From<Pfn> for VramAddress {
fn from(pfn: Pfn) -> Self {
Self::from_raw(pfn.raw() << 12)
}
}
bitfield! {
/// Virtual address in GPU address space.
pub(crate) struct VirtualAddress(u64) {
/// Offset within 4KB page.
11:0 offset;
/// Virtual frame number.
63:12 frame_number => Vfn;
}
}
impl VirtualAddress {
/// Create a new virtual address from a raw value.
#[expect(dead_code)]
pub(crate) const fn new(addr: u64) -> Self {
Self::from_raw(addr)
}
}
impl From<Vfn> for VirtualAddress {
fn from(vfn: Vfn) -> Self {
Self::zeroed().with_frame_number(vfn)
}
}
/// Physical Frame Number.
///
/// Represents a physical page in VRAM.
#[repr(transparent)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub(crate) struct Pfn(u64);
impl Pfn {
/// Create a new PFN from a frame number.
pub(crate) const fn new(frame_number: u64) -> Self {
Self(frame_number)
}
/// Get the raw frame number.
pub(crate) const fn raw(self) -> u64 {
self.0
}
}
impl From<VramAddress> for Pfn {
fn from(addr: VramAddress) -> Self {
Self::new(addr.into_raw() >> 12)
}
}
impl From<u64> for Pfn {
fn from(val: u64) -> Self {
Self(val)
}
}
impl From<Pfn> for u64 {
fn from(pfn: Pfn) -> Self {
pfn.0
}
}
impl_pfn_bounded!(52);
/// Virtual Frame Number.
///
/// Represents a virtual page in GPU address space.
#[repr(transparent)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub(crate) struct Vfn(u64);
impl Vfn {
/// Create a new VFN from a frame number.
pub(crate) const fn new(frame_number: u64) -> Self {
Self(frame_number)
}
/// Get the raw frame number.
pub(crate) const fn raw(self) -> u64 {
self.0
}
}
impl From<VirtualAddress> for Vfn {
fn from(addr: VirtualAddress) -> Self {
addr.frame_number()
}
}
impl From<u64> for Vfn {
fn from(val: u64) -> Self {
Self(val)
}
}
impl From<Vfn> for u64 {
fn from(vfn: Vfn) -> Self {
vfn.0
}
}
impl_frame_number_bounded!(Vfn, 52);
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
pub(crate) mod selftest {
use core::ops::Range;
use kernel::{
device,
sizes::SizeConstants, //
};
use super::*;
/// Run MM subsystem self-tests during probe.
pub(crate) fn run(
dev: &device::Device<device::Bound>,
mm: &mut GpuMm<'_>,
usable_fb_regions: &[Range<u64>],
bar_user: &bar_user::BarUser<'_>,
bar1_pdb: u64,
chipset: Chipset,
) -> Result {
// VRAM span the self-tests are free to overwrite, from the chosen test base.
const SELFTEST_SPAN: u64 = u64::SZ_64M;
let base = usable_fb_regions.iter().find_map(|region| {
// Tests rely on this being 8 byte aligned for checking misalignment handling.
let base = region.start.align_up(Alignment::new::<8>())?;
(base.checked_add(SELFTEST_SPAN)? <= region.end).then_some(base)
});
let Some(base) = base else {
dev_warn!(
dev,
"PRAMIN: skipping self-tests, no usable VRAM region of {:#x} bytes\n",
SELFTEST_SPAN
);
return Ok(());
};
pramin::selftest::run(dev, mm.pramin_mut(), VramAddress::from_raw(base))?;
bar_user::run_self_test(dev, mm, bar_user, bar1_pdb, chipset)
}
}

420
drivers/gpu/nova-core/mm/bar_user.rs Normal file
View File

@ -0,0 +1,420 @@
// SPDX-License-Identifier: GPL-2.0
//! BAR1 user interface for CPU access to GPU virtual memory. Used for USERD
//! for GPU work submission, and applications to access GPU buffers via mmap().
use kernel::{
io::Io,
new_mutex,
prelude::*,
sync::Mutex, //
};
use crate::{
driver::Bar1,
gpu::Chipset,
mm::{
vmm::{
MappedRange,
Vmm, //
},
GpuMm,
Pfn,
Vfn,
VirtualAddress,
VramAddress,
PAGE_SIZE, //
},
num::IntoSafeCast,
};
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
use kernel::device;
/// BAR1 user interface for virtual memory mappings.
///
/// Owns the [`Vmm`] for the BAR1 address space.
#[pin_data]
pub(crate) struct BarUser<'a> {
#[pin]
vmm: Mutex<Vmm>,
bar1: Bar1<'a>,
}
impl<'a> BarUser<'a> {
/// Create a pin-initializer for [`BarUser`].
pub(crate) fn new(
pdb_addr: VramAddress,
chipset: Chipset,
va_size: u64,
bar1: Bar1<'a>,
) -> impl PinInit<Self, Error> + 'a {
try_pin_init!(Self {
vmm <- new_mutex!(
Vmm::new(pdb_addr, chipset.mmu_version(), va_size)?,
"bar_user_vmm",
),
bar1,
})
}
/// Map physical pages to a contiguous BAR1 virtual range.
pub(crate) fn map(
&self,
mm: &mut GpuMm<'_>,
pfns: &[Pfn],
writable: bool,
) -> Result<BarUserAccess<'_>> {
if pfns.is_empty() {
return Err(EINVAL);
}
let mut vmm = self.vmm.lock();
let mapped = vmm.map_pages(mm, pfns, None, writable)?;
Ok(BarUserAccess {
bar_user: self,
mapped: Some(mapped),
})
}
}
/// Access object for a mapped BAR1 region.
///
/// Borrows the [`BarUser`] managing the mapping for the lifetime of the access object.
pub(crate) struct BarUserAccess<'a> {
bar_user: &'a BarUser<'a>,
/// [`BarUserAccess::release`] [`Option::take`]s this; `Some` at
/// drop time means `release()` was never called.
mapped: Option<MappedRange>,
}
#[expect(dead_code)]
impl BarUserAccess<'_> {
/// Tear down the BAR1 mapping.
pub(crate) fn release(mut self, mm: &mut GpuMm<'_>) -> Result {
let mapped = self.mapped.take().ok_or(EINVAL)?;
let mut vmm = self.bar_user.vmm.lock();
vmm.unmap_pages(mm, mapped)?;
Ok(())
}
/// Returns the active mapping.
fn mapped(&self) -> &MappedRange {
// `mapped` is only `None` after `take()` in `release`; hence unwrap()
// cannot panic here.
self.mapped.as_ref().unwrap()
}
/// Get the base virtual address of this mapping.
pub(crate) fn base(&self) -> VirtualAddress {
VirtualAddress::from(self.mapped().vfn_start)
}
/// Get the total size of the mapped region in bytes.
pub(crate) fn size(&self) -> usize {
self.mapped().num_pages * PAGE_SIZE
}
/// Get the starting virtual frame number.
pub(crate) fn vfn_start(&self) -> Vfn {
self.mapped().vfn_start
}
/// Get the number of pages in this mapping.
pub(crate) fn num_pages(&self) -> usize {
self.mapped().num_pages
}
/// Translate an offset within this mapping to a BAR1 aperture offset.
fn bar_offset(&self, offset: usize) -> Result<usize> {
if offset >= self.size() {
return Err(EINVAL);
}
let base_vfn: usize = self.mapped().vfn_start.raw().into_safe_cast();
let base = base_vfn.checked_mul(PAGE_SIZE).ok_or(EOVERFLOW)?;
base.checked_add(offset).ok_or(EOVERFLOW)
}
// Fallible accessors with runtime bounds checking.
/// Read a 32-bit value at the given offset.
pub(crate) fn try_read32(&self, offset: usize) -> Result<u32> {
let off = self.bar_offset(offset)?;
self.bar_user.bar1.try_read32(off)
}
/// Write a 32-bit value at the given offset.
pub(crate) fn try_write32(&self, value: u32, offset: usize) -> Result {
let off = self.bar_offset(offset)?;
self.bar_user.bar1.try_write32(value, off)
}
/// Read a 64-bit value at the given offset.
pub(crate) fn try_read64(&self, offset: usize) -> Result<u64> {
let off = self.bar_offset(offset)?;
self.bar_user.bar1.try_read64(off)
}
/// Write a 64-bit value at the given offset.
pub(crate) fn try_write64(&self, value: u64, offset: usize) -> Result {
let off = self.bar_offset(offset)?;
self.bar_user.bar1.try_write64(value, off)
}
}
impl Drop for BarUserAccess<'_> {
fn drop(&mut self) {
if self.mapped.is_some() {
kernel::pr_warn!(
"BarUserAccess dropped without calling release(). BarUser address space will leak.\n"
);
}
// The inner `MappedRange`'s own `MustUnmapGuard` will also fire,
// identifying the leaked VA range.
}
}
/// Run MM subsystem self-tests during probe.
///
/// Tests page table infrastructure and `BAR1` MMIO access using the `BAR1`
/// address space. Uses the `GpuMm`'s buddy allocator to allocate page tables
/// and test pages as needed.
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
pub(crate) fn run_self_test(
dev: &device::Device<device::Bound>,
mm: &mut GpuMm<'_>,
bar_user: &BarUser<'_>,
bar1_pdb: u64,
chipset: Chipset,
) -> Result {
use kernel::{
gpu::buddy::{
GpuBuddyAllocFlags,
GpuBuddyAllocMode, //
},
ptr::Alignment,
sizes::{
SZ_16K,
SZ_32K,
SZ_4K,
SZ_64K, //
},
};
// Test patterns.
const PATTERN_PRAMIN: u32 = 0xDEAD_BEEF;
const PATTERN_BAR1: u32 = 0xCAFE_BABE;
let bar1 = bar_user.bar1;
dev_info!(dev, "MM: Starting self-test...\n");
let pdb_addr = VramAddress::from_raw(bar1_pdb);
// Check if initial page tables are in VRAM.
if crate::mm::pagetable::check_pdb_valid(mm.pramin_mut(), pdb_addr, chipset).is_err() {
dev_info!(dev, "MM: Self-test SKIPPED - no valid VRAM page tables\n");
return Ok(());
}
// Set up a test page from the buddy allocator.
let test_page_blocks = KBox::pin_init(
mm.buddy().alloc_blocks(
GpuBuddyAllocMode::Simple,
SZ_4K.into_safe_cast(),
Alignment::new::<SZ_4K>(),
GpuBuddyAllocFlags::default(),
),
GFP_KERNEL,
)?;
let test_vram_offset = test_page_blocks.iter().next().ok_or(ENOMEM)?.offset();
let test_vram = VramAddress::from_raw(test_vram_offset);
let test_pfn = Pfn::from(test_vram);
// Create a VMM of size 64K to track virtual memory mappings.
let mut vmm = Vmm::new(pdb_addr, chipset.mmu_version(), SZ_64K.into_safe_cast())?;
// Create a test mapping.
let mapped = vmm.map_pages(mm, &[test_pfn], None, true)?;
let test_vfn = mapped.vfn_start;
// Pre-compute test addresses for the PRAMIN to BAR1 read test.
let vfn_offset: usize = test_vfn.raw().into_safe_cast();
let bar1_base_offset = vfn_offset.checked_mul(PAGE_SIZE).ok_or(EOVERFLOW)?;
let bar1_read_offset: usize = bar1_base_offset + 0x100;
let vram_read_addr = test_vram + 0x100;
// Test 1: Write via PRAMIN, read via BAR1.
mm.pramin_mut()
.window_at::<u32>(vram_read_addr)?
.view()
.write_val(PATTERN_PRAMIN);
// Read back via BAR1 aperture.
let bar1_value = bar1.try_read32(bar1_read_offset)?;
let test1_passed = if bar1_value == PATTERN_PRAMIN {
true
} else {
dev_err!(
dev,
"MM: Test 1 FAILED - Expected {:#010x}, got {:#010x}\n",
PATTERN_PRAMIN,
bar1_value
);
false
};
// Cleanup - invalidate PTE.
vmm.unmap_pages(mm, mapped)?;
// Test 2: Two-phase prepare/execute API.
let prepared = vmm.prepare_map(mm, 1, None)?;
let mapped2 = vmm.execute_map(mm, prepared, &[test_pfn], true)?;
let readback = vmm.read_mapping(mm, mapped2.vfn_start)?;
let test2_passed = if readback == Some(test_pfn) {
true
} else {
dev_err!(dev, "MM: Test 2 FAILED - Two-phase map readback mismatch\n");
false
};
vmm.unmap_pages(mm, mapped2)?;
// Test 3: Range-constrained allocation with a hole — exercises block.size()-driven
// BAR1 mapping. A 4K hole is punched at base+16K, then a single 32K allocation
// is requested within [base, base+36K). The buddy allocator must split around the
// hole, returning multiple blocks (expected: {16K, 4K, 8K, 4K} = 32K total).
// Each block is mapped into BAR1 and verified via PRAMIN read-back.
//
// Address layout (base = 0x10000):
// [ 16K ] [HOLE 4K] [4K] [ 8K ] [4K]
// 0x10000 0x14000 0x15000 0x16000 0x18000 0x19000
let range_base: u64 = SZ_64K.into_safe_cast();
let sz_4k: u64 = SZ_4K.into_safe_cast();
let sz_16k: u64 = SZ_16K.into_safe_cast();
let sz_32k_4k: u64 = (SZ_32K + SZ_4K).into_safe_cast();
// Punch a 4K hole at base+16K so the subsequent 32K allocation must split.
let _hole = KBox::pin_init(
mm.buddy().alloc_blocks(
GpuBuddyAllocMode::Range(range_base + sz_16k..range_base + sz_16k + sz_4k),
SZ_4K.into_safe_cast(),
Alignment::new::<SZ_4K>(),
GpuBuddyAllocFlags::default(),
),
GFP_KERNEL,
)?;
// Allocate 32K within [base, base+36K). The hole forces the allocator to return
// split blocks whose sizes are determined by buddy alignment.
let blocks = KBox::pin_init(
mm.buddy().alloc_blocks(
GpuBuddyAllocMode::Range(range_base..range_base + sz_32k_4k),
SZ_32K.into_safe_cast(),
Alignment::new::<SZ_4K>(),
GpuBuddyAllocFlags::default(),
),
GFP_KERNEL,
)?;
let mut test3_passed = true;
let mut total_size = 0usize;
for block in blocks.iter() {
total_size += IntoSafeCast::<usize>::into_safe_cast(block.size());
// Map all pages of this block.
let page_size: u64 = PAGE_SIZE.into_safe_cast();
let num_pages: usize = (block.size() / page_size).into_safe_cast();
let mut pfns = KVec::new();
for j in 0..num_pages {
let j_u64: u64 = j.into_safe_cast();
pfns.push(
Pfn::from(VramAddress::from_raw(
block.offset() + j_u64.checked_mul(page_size).ok_or(EOVERFLOW)?,
)),
GFP_KERNEL,
)?;
}
let mapped = vmm.map_pages(mm, &pfns, None, true)?;
let bar1_base_vfn: usize = mapped.vfn_start.raw().into_safe_cast();
let bar1_base = bar1_base_vfn.checked_mul(PAGE_SIZE).ok_or(EOVERFLOW)?;
for j in 0..num_pages {
let page_bar1_off = bar1_base + j * PAGE_SIZE;
let j_u64: u64 = j.into_safe_cast();
let page_phys = block.offset()
+ j_u64
.checked_mul(PAGE_SIZE.into_safe_cast())
.ok_or(EOVERFLOW)?;
bar1.try_write32(PATTERN_BAR1, page_bar1_off)?;
let pramin_val = mm
.pramin_mut()
.window_at::<u32>(VramAddress::from_raw(page_phys))?
.view()
.read_val();
if pramin_val != PATTERN_BAR1 {
dev_err!(
dev,
"MM: Test 3 FAILED block offset {:#x} page {} (val={:#x})\n",
block.offset(),
j,
pramin_val
);
test3_passed = false;
}
}
vmm.unmap_pages(mm, mapped)?;
}
// Verify aggregate: all returned block sizes must sum to allocation size.
if total_size != SZ_32K {
dev_err!(
dev,
"MM: Test 3 FAILED - total size {} != expected {}\n",
total_size,
SZ_32K
);
test3_passed = false;
}
// Release Tests 1-3's Vmm before Test 4 uses the GPU's BarUser on the same PDB.
drop(vmm);
// Test 4: Exercise `BarUser::map()` end-to-end.
let access = bar_user.map(mm, &[test_pfn], true)?;
// Write pattern via PRAMIN, read via BarUserAccess.
mm.pramin_mut()
.window_at::<u32>(test_vram)?
.view()
.write_val(PATTERN_BAR1);
let readback = access.try_read32(0)?;
let test4_passed = if readback == PATTERN_BAR1 {
true
} else {
dev_err!(
dev,
"MM: Test 4 FAILED - Expected {:#010x}, got {:#010x}\n",
PATTERN_BAR1,
readback
);
false
};
access.release(mm)?;
if test1_passed && test2_passed && test3_passed && test4_passed {
dev_info!(dev, "MM: All self-tests PASSED\n");
Ok(())
} else {
dev_err!(dev, "MM: Self-tests FAILED\n");
Err(EIO)
}
}

56
drivers/gpu/nova-core/mm/hal.rs Normal file
View File

@ -0,0 +1,56 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Memory management HAL.
use kernel::{
num::Bounded,
prelude::*, //
};
use crate::{
driver::Bar0,
gpu::{
Architecture,
Chipset, //
},
mm::VramAddress, //
};
mod gb100;
mod gh100;
mod tu102;
/// Trait implemented by per-architecture MM HALs.
///
/// `Sync` is required so that the `&'static dyn MmHal` references can be stored in `Send`
/// structures.
pub(super) trait MmHal: Sync {
/// Positions the PRAMIN window at `base`.
///
/// This fails if `base` is not aligned to the 64 KiB window alignment or is too large for
/// the receiving register.
fn write_pramin_window_base(&self, bar: Bar0<'_>, base: VramAddress) -> Result;
}
/// Returns the HAL corresponding to `chipset`.
pub(super) fn mm_hal(chipset: Chipset) -> &'static dyn MmHal {
match chipset.arch() {
Architecture::Turing | Architecture::Ampere | Architecture::Ada => tu102::TU102_HAL,
Architecture::Hopper => gh100::GH100_HAL,
Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => gb100::GB100_HAL,
}
}
/// Converts `base` into the value of the window-base register field.
///
/// Fails with [`EINVAL`] if `base` is not aligned to the window alignment required by the register
/// field's shift, or if the shifted value does not fit within `RES` bits.
fn window_base<const RES: u32>(base: VramAddress) -> Result<Bounded<u64, RES>> {
const WINDOW_BASE_SHIFT: u32 = 16;
Bounded::<u64, 64>::from(base.into_raw())
.shr_exact::<WINDOW_BASE_SHIFT, { 64 - WINDOW_BASE_SHIFT }>()
.and_then(Bounded::try_shrink)
.ok_or(EINVAL)
}

35
drivers/gpu/nova-core/mm/hal/gb100.rs Normal file
View File

@ -0,0 +1,35 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Blackwell GB10x/GB20x memory management HAL.
use kernel::{
io::Io,
prelude::*, //
};
use crate::{
driver::Bar0,
mm::{
hal::{
window_base,
MmHal, //
},
regs,
VramAddress, //
},
};
struct Gb100;
impl MmHal for Gb100 {
fn write_pramin_window_base(&self, bar: Bar0<'_>, base: VramAddress) -> Result {
bar.write_reg(
regs::gb100::NV_XAL_EP_BAR0_WINDOW::zeroed().with_base(window_base(base)?.cast()),
);
Ok(())
}
}
const GB100: Gb100 = Gb100;
pub(super) const GB100_HAL: &dyn MmHal = &GB100;

35
drivers/gpu/nova-core/mm/hal/gh100.rs Normal file
View File

@ -0,0 +1,35 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Hopper memory management HAL.
use kernel::{
io::Io,
prelude::*, //
};
use crate::{
driver::Bar0,
mm::{
hal::{
window_base,
MmHal, //
},
regs,
VramAddress, //
},
};
struct Gh100;
impl MmHal for Gh100 {
fn write_pramin_window_base(&self, bar: Bar0<'_>, base: VramAddress) -> Result {
bar.write_reg(
regs::gh100::NV_XAL_EP_BAR0_WINDOW::zeroed().with_base(window_base(base)?.cast()),
);
Ok(())
}
}
const GH100: Gh100 = Gh100;
pub(super) const GH100_HAL: &dyn MmHal = &GH100;

37
drivers/gpu/nova-core/mm/hal/tu102.rs Normal file
View File

@ -0,0 +1,37 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Turing, Ampere and Ada memory management HAL.
use kernel::{
io::Io,
prelude::*, //
};
use crate::{
driver::Bar0,
mm::{
hal::{
window_base,
MmHal, //
},
regs,
VramAddress, //
},
};
struct Tu102;
impl MmHal for Tu102 {
fn write_pramin_window_base(&self, bar: Bar0<'_>, base: VramAddress) -> Result {
bar.write_reg(
regs::NV_PBUS_BAR0_WINDOW::zeroed()
.with_target(regs::Bar0WindowTarget::VidMem)
.with_base(window_base(base)?.cast()),
);
Ok(())
}
}
const TU102: Tu102 = Tu102;
pub(super) const TU102_HAL: &dyn MmHal = &TU102;

424
drivers/gpu/nova-core/mm/pagetable.rs Normal file
View File

@ -0,0 +1,424 @@
// SPDX-License-Identifier: GPL-2.0
//! Common page table types shared between MMU v2 and v3.
//!
//! This module provides foundational types used by both MMU versions:
//! - Page table level hierarchy
//! - Memory aperture types for PDEs and PTEs
#![expect(dead_code)]
pub(super) mod map;
pub(super) mod ver2;
pub(super) mod ver3;
pub(super) mod walk;
use kernel::{
io::Io,
num::Bounded,
prelude::*, //
};
use crate::{
gpu::Architecture,
mm::{
pramin,
Pfn,
VirtualAddress,
VramAddress, //
},
};
/// Extracts the page table index at a given level from a virtual address.
pub(super) trait VaLevelIndex {
/// Return the page table index at `level` for this virtual address.
fn level_index(&self, level: u64) -> u64;
}
/// MMU version enumeration.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum MmuVersion {
/// MMU v2 for Turing/Ampere/Ada.
V2,
/// MMU v3 for Hopper and later.
V3,
}
impl From<Architecture> for MmuVersion {
fn from(arch: Architecture) -> Self {
match arch {
Architecture::Turing | Architecture::Ampere | Architecture::Ada => Self::V2,
Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
Self::V3
}
}
}
}
/// Page Table Level hierarchy for MMU v2/v3.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum PageTableLevel {
/// Level 0 - Page Directory Base (root).
Pdb,
/// Level 1 - Intermediate page directory.
L1,
/// Level 2 - Intermediate page directory.
L2,
/// Level 3 - Intermediate page directory or dual PDE (version-dependent).
L3,
/// Level 4 - PTE level for v2, intermediate page directory for v3.
L4,
/// Level 5 - PTE level used for MMU v3 only.
L5,
}
impl PageTableLevel {
/// Number of entries per page table (512 for 4KB pages).
pub(super) const ENTRIES_PER_TABLE: usize = 512;
/// Get the next level in the hierarchy.
pub(super) const fn next(&self) -> Option<PageTableLevel> {
match self {
Self::Pdb => Some(Self::L1),
Self::L1 => Some(Self::L2),
Self::L2 => Some(Self::L3),
Self::L3 => Some(Self::L4),
Self::L4 => Some(Self::L5),
Self::L5 => None,
}
}
/// Convert level to index.
pub(super) const fn as_index(&self) -> u64 {
match self {
Self::Pdb => 0,
Self::L1 => 1,
Self::L2 => 2,
Self::L3 => 3,
Self::L4 => 4,
Self::L5 => 5,
}
}
}
// Trait abstractions for page table operations.
/// Operations on Page Table Entries (`PTE`s).
pub(super) trait PteOps: Copy + core::fmt::Debug + Into<u64> {
/// Create a `PTE` from a raw `u64` value.
fn from_raw(val: u64) -> Self;
/// Create an invalid `PTE`.
fn invalid() -> Self;
/// Create a valid `PTE` for the given memory aperture.
fn new(aperture: AperturePte, pfn: Pfn, writable: bool) -> Self;
/// Check if this `PTE` is valid.
fn is_valid(&self) -> bool;
/// Get the physical frame number.
fn frame_number(&self) -> Pfn;
/// Read a `PTE` from VRAM.
fn read(pramin: &mut pramin::Pramin<'_>, addr: VramAddress) -> Result<Self> {
let val = pramin.window_at::<u64>(addr)?.view().read_val();
Ok(Self::from_raw(val))
}
/// Write this `PTE` to VRAM.
fn write(&self, pramin: &mut pramin::Pramin<'_>, addr: VramAddress) -> Result {
pramin
.window_at::<u64>(addr)?
.view()
.write_val((*self).into());
Ok(())
}
}
/// Operations on Page Directory Entries (`PDE`s).
pub(super) trait PdeOps: Copy + core::fmt::Debug + Into<u64> {
/// Create a `PDE` from a raw `u64` value.
fn from_raw(val: u64) -> Self;
/// Create a valid `PDE` pointing to a page table in the given aperture.
fn new(aperture: AperturePde, table_pfn: Pfn) -> Self;
/// Create an invalid `PDE`.
fn invalid() -> Self;
/// Check if this `PDE` is valid.
fn is_valid(&self) -> bool;
/// Get the memory aperture of this `PDE`.
fn aperture(&self) -> AperturePde;
/// Get the VRAM address of the page table.
fn table_vram_address(&self) -> VramAddress;
/// Read a `PDE` from VRAM.
fn read(pramin: &mut pramin::Pramin<'_>, addr: VramAddress) -> Result<Self> {
let val = pramin.window_at::<u64>(addr)?.view().read_val();
Ok(Self::from_raw(val))
}
/// Write this `PDE` to VRAM.
fn write(&self, pramin: &mut pramin::Pramin<'_>, addr: VramAddress) -> Result {
pramin
.window_at::<u64>(addr)?
.view()
.write_val((*self).into());
Ok(())
}
/// Check if this `PDE` is valid and points to video memory.
fn is_valid_vram(&self) -> bool {
self.is_valid() && self.aperture() == AperturePde::VideoMemory
}
}
/// Operations on Dual Page Directory Entries (128-bit `DualPde`s).
pub(super) trait DualPdeOps: Copy + core::fmt::Debug {
/// Create a `DualPde` from raw 128-bit value (two `u64`s).
fn from_raw(big: u64, small: u64) -> Self;
/// Create a `DualPde` with only the small page table pointer set.
fn new_small(table_pfn: Pfn) -> Self;
/// Check if the small page table pointer is valid.
fn has_small(&self) -> bool;
/// Get the small page table VRAM address.
fn small_vram_address(&self) -> VramAddress;
/// Get the raw `u64` value of the big PDE.
fn big_raw_u64(&self) -> u64;
/// Get the raw `u64` value of the small PDE.
fn small_raw_u64(&self) -> u64;
/// Read a dual PDE (128-bit) from VRAM.
fn read(pramin: &mut pramin::Pramin<'_>, addr: VramAddress) -> Result<Self> {
let lo = pramin.window_at::<u64>(addr)?.view().read_val();
let hi = pramin.window_at::<u64>(addr + 8)?.view().read_val();
Ok(Self::from_raw(lo, hi))
}
/// Write this dual PDE (128-bit) to VRAM.
fn write(&self, pramin: &mut pramin::Pramin<'_>, addr: VramAddress) -> Result {
pramin
.window_at::<u64>(addr)?
.view()
.write_val(self.big_raw_u64());
pramin
.window_at::<u64>(addr + 8)?
.view()
.write_val(self.small_raw_u64());
Ok(())
}
}
/// MMU configuration trait -- encodes version-specific constants and types.
pub(super) trait MmuConfig: 'static {
/// Page Table Entry type.
type Pte: PteOps;
/// Page Directory Entry type.
type Pde: PdeOps;
/// Dual Page Directory Entry type (128-bit).
type DualPde: DualPdeOps;
/// PDE levels (excluding PTE level) for page table walking.
const PDE_LEVELS: &'static [PageTableLevel];
/// PTE level for this MMU version.
const PTE_LEVEL: PageTableLevel;
/// Dual PDE level (128-bit entries) for this MMU version.
const DUAL_PDE_LEVEL: PageTableLevel;
/// Get the number of entries per page table page for a given level.
fn entries_per_page(level: PageTableLevel) -> usize;
/// Extract the page table index at `level` from `va`.
fn level_index(va: VirtualAddress, level: u64) -> u64;
/// Get the entry size in bytes for a given level.
fn entry_size(level: PageTableLevel) -> usize {
if level == Self::DUAL_PDE_LEVEL {
16 // 128-bit dual PDE
} else {
8 // 64-bit PDE/PTE
}
}
/// Compute upper bound on page table pages needed for `num_virt_pages`.
///
/// Walks from PTE level up through PDE levels, accumulating the tree.
fn pt_pages_upper_bound(num_virt_pages: usize) -> usize {
let mut total = 0;
// PTE pages at the leaf level.
let pte_epp = Self::entries_per_page(Self::PTE_LEVEL);
let mut pages_at_level = num_virt_pages.div_ceil(pte_epp);
total += pages_at_level;
// Walk PDE levels bottom-up (reverse of PDE_LEVELS).
for &level in Self::PDE_LEVELS.iter().rev() {
let epp = Self::entries_per_page(level);
// How many pages at this level do we need to point to
// the previous pages_at_level?
pages_at_level = pages_at_level.div_ceil(epp);
total += pages_at_level;
}
total
}
}
/// Marker struct for MMU v2 (Turing/Ampere/Ada).
pub(super) struct MmuV2;
impl MmuConfig for MmuV2 {
type Pte = ver2::Pte;
type Pde = ver2::Pde;
type DualPde = ver2::DualPde;
const PDE_LEVELS: &'static [PageTableLevel] = ver2::PDE_LEVELS;
const PTE_LEVEL: PageTableLevel = ver2::PTE_LEVEL;
const DUAL_PDE_LEVEL: PageTableLevel = ver2::DUAL_PDE_LEVEL;
fn entries_per_page(level: PageTableLevel) -> usize {
// TODO: Calculate these values from the bitfield dynamically
// instead of hardcoding them.
match level {
PageTableLevel::Pdb => 4, // PD3 root: bits [48:47] = 2 bits
PageTableLevel::L3 => 256, // PD0 dual: bits [28:21] = 8 bits
_ => 512, // PD2, PD1, PT: 9 bits each
}
}
fn level_index(va: VirtualAddress, level: u64) -> u64 {
ver2::VirtualAddressV2::new(va).level_index(level)
}
}
/// Marker struct for MMU v3 (Hopper and later).
pub(super) struct MmuV3;
impl MmuConfig for MmuV3 {
type Pte = ver3::Pte;
type Pde = ver3::Pde;
type DualPde = ver3::DualPde;
const PDE_LEVELS: &'static [PageTableLevel] = ver3::PDE_LEVELS;
const PTE_LEVEL: PageTableLevel = ver3::PTE_LEVEL;
const DUAL_PDE_LEVEL: PageTableLevel = ver3::DUAL_PDE_LEVEL;
fn entries_per_page(level: PageTableLevel) -> usize {
match level {
PageTableLevel::Pdb => 2, // PDE4 root: bit [56] = 1 bit, 2 entries
PageTableLevel::L4 => 256, // PDE0 dual: bits [28:21] = 8 bits
_ => 512, // PDE3, PDE2, PDE1, PT: 9 bits each
}
}
fn level_index(va: VirtualAddress, level: u64) -> u64 {
ver3::VirtualAddressV3::new(va).level_index(level)
}
}
/// Memory aperture for Page Table Entries (`PTE`s).
///
/// Determines which memory region the `PTE` points to.
#[repr(u8)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub(super) enum AperturePte {
/// Local video memory (VRAM).
#[default]
VideoMemory = 0,
/// Peer GPU's video memory.
PeerMemory = 1,
/// System memory with cache coherence.
SystemCoherent = 2,
/// System memory without cache coherence.
SystemNonCoherent = 3,
}
// TODO[FPRI]: Replace with `#[derive(FromPrimitive)]` when available.
impl From<Bounded<u64, 2>> for AperturePte {
fn from(val: Bounded<u64, 2>) -> Self {
match *val {
0 => Self::VideoMemory,
1 => Self::PeerMemory,
2 => Self::SystemCoherent,
3 => Self::SystemNonCoherent,
_ => Self::VideoMemory,
}
}
}
// TODO[FPRI]: Replace with `#[derive(ToPrimitive)]` when available.
impl From<AperturePte> for Bounded<u64, 2> {
fn from(val: AperturePte) -> Self {
Bounded::from_expr(val as u64 & 0x3)
}
}
/// Memory aperture for Page Directory Entries (`PDE`s).
///
/// Note: For `PDE`s, `Invalid` (0) means the entry is not valid.
#[repr(u8)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub(super) enum AperturePde {
/// Invalid/unused entry.
#[default]
Invalid = 0,
/// Page table is in video memory.
VideoMemory = 1,
/// Page table is in system memory with coherence.
SystemCoherent = 2,
/// Page table is in system memory without coherence.
SystemNonCoherent = 3,
}
// TODO[FPRI]: Replace with `#[derive(FromPrimitive)]` when available.
impl From<Bounded<u64, 2>> for AperturePde {
fn from(val: Bounded<u64, 2>) -> Self {
match *val {
1 => Self::VideoMemory,
2 => Self::SystemCoherent,
3 => Self::SystemNonCoherent,
_ => Self::Invalid,
}
}
}
// TODO[FPRI]: Replace with `#[derive(ToPrimitive)]` when available.
impl From<AperturePde> for Bounded<u64, 2> {
fn from(val: AperturePde) -> Self {
Bounded::from_expr(val as u64 & 0x3)
}
}
/// Check if the PDB has valid, VRAM-backed page tables.
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
fn check_pdb_inner<M: MmuConfig>(pramin: &mut pramin::Pramin<'_>, pdb_addr: VramAddress) -> Result {
let raw = pramin.window_at::<u64>(pdb_addr)?.view().read_val();
if !M::Pde::from_raw(raw).is_valid_vram() {
return Err(ENOENT);
}
Ok(())
}
/// Check if the PDB has valid, VRAM-backed page tables, dispatching by MMU version.
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
pub(super) fn check_pdb_valid(
pramin: &mut pramin::Pramin<'_>,
pdb_addr: VramAddress,
chipset: crate::gpu::Chipset,
) -> Result {
match MmuVersion::from(chipset.arch()) {
MmuVersion::V2 => check_pdb_inner::<MmuV2>(pramin, pdb_addr),
MmuVersion::V3 => check_pdb_inner::<MmuV3>(pramin, pdb_addr),
}
}

345
drivers/gpu/nova-core/mm/pagetable/map.rs Normal file
View File

@ -0,0 +1,345 @@
// SPDX-License-Identifier: GPL-2.0
//! Page table mapping operations for NVIDIA GPUs.
use core::marker::PhantomData;
use kernel::{
gpu::buddy::{
AllocatedBlocks,
GpuBuddyAllocFlags,
GpuBuddyAllocMode, //
},
io::io_write,
prelude::*,
ptr::Alignment,
rbtree::{
RBTree,
RBTreeNode, //
},
sizes::SZ_4K, //
};
use super::{
walk::{
PtWalkInner,
WalkPdeResult,
WalkResult, //
},
AperturePde,
AperturePte,
DualPdeOps,
MmuConfig,
MmuV2,
MmuV3,
MmuVersion,
PageTableLevel,
PdeOps,
PteOps, //
};
use crate::{
mm::{
GpuMm,
Pfn,
Vfn,
VramAddress,
PAGE_SIZE, //
},
num::{
IntoSafeCast, //
},
};
/// A pre-allocated and zeroed page table page.
///
/// Created during the mapping prepare phase and consumed during the execute phase.
/// Stored in an [`RBTree`] keyed by the PDE slot address (`install_addr`).
pub(in crate::mm) struct PreparedPtPage {
/// The allocated and zeroed page table page.
pub(in crate::mm) alloc: Pin<KBox<AllocatedBlocks>>,
/// Page table level -- needed to determine if this PT page is for a dual PDE.
pub(in crate::mm) level: PageTableLevel,
}
/// Page table mapper.
pub(in crate::mm) struct PtMapInner<M: MmuConfig> {
walker: PtWalkInner<M>,
pdb_addr: VramAddress,
_phantom: PhantomData<M>,
}
impl<M: MmuConfig> PtMapInner<M> {
/// Create a new [`PtMapInner`].
pub(super) fn new(pdb_addr: VramAddress) -> Self {
Self {
walker: PtWalkInner::<M>::new(pdb_addr),
pdb_addr,
_phantom: PhantomData,
}
}
/// Allocate and zero a physical page table page.
fn alloc_and_zero_page(mm: &mut GpuMm<'_>, level: PageTableLevel) -> Result<PreparedPtPage> {
let blocks = KBox::pin_init(
mm.buddy().alloc_blocks(
GpuBuddyAllocMode::Simple,
SZ_4K.into_safe_cast(),
Alignment::new::<SZ_4K>(),
GpuBuddyAllocFlags::default(),
),
GFP_KERNEL,
)?;
let page_vram = VramAddress::from_raw(blocks.iter().next().ok_or(ENOMEM)?.offset());
// Zero via PRAMIN.
let window = mm
.pramin_mut()
.window_at::<[u64; PAGE_SIZE / 8]>(page_vram)?;
for i in 0..PAGE_SIZE / 8 {
io_write!(window.view(), [build: i], 0);
}
Ok(PreparedPtPage {
alloc: blocks,
level,
})
}
/// Ensure all intermediate page table pages exist for a single VFN.
///
/// The mutable PRAMIN borrow ends before each allocation.
fn ensure_single_pte_path(
&self,
mm: &mut GpuMm<'_>,
vfn: Vfn,
pt_pages: &mut RBTree<VramAddress, PreparedPtPage>,
) -> Result {
let max_iter = 2 * M::PDE_LEVELS.len();
for _ in 0..max_iter {
let result = self
.walker
.walk_pde_levels(mm.pramin_mut(), vfn, |install_addr| {
pt_pages.get(&install_addr).and_then(|p| {
p.alloc
.iter()
.next()
.map(|b| VramAddress::from_raw(b.offset()))
})
})?;
match result {
WalkPdeResult::Complete { .. } => {
return Ok(());
}
WalkPdeResult::Missing {
install_addr,
level,
} => {
let page = Self::alloc_and_zero_page(mm, level)?;
let node = RBTreeNode::new(install_addr, page, GFP_KERNEL)?;
let old = pt_pages.insert(node);
if old.is_some() {
kernel::pr_warn_once!(
"VMM: duplicate install_addr in pt_pages (internal consistency error)\n"
);
return Err(EIO);
}
}
}
}
kernel::pr_warn!(
"VMM: ensure_pte_path: loop exhausted after {} iters (VFN {:?})\n",
max_iter,
vfn
);
Err(EIO)
}
/// Prepare page table resources for mapping `num_pages` pages starting at `vfn_start`.
///
/// Reserves capacity in `page_table_allocs`, then walks the hierarchy
/// per-VFN to prepare pages for all missing PDEs.
pub(super) fn prepare_map(
&self,
mm: &mut GpuMm<'_>,
vfn_start: Vfn,
num_pages: usize,
page_table_allocs: &mut KVec<Pin<KBox<AllocatedBlocks>>>,
pt_pages: &mut RBTree<VramAddress, PreparedPtPage>,
) -> Result {
// Pre-reserve so install_mappings() can use push_within_capacity (no alloc
// in fence signalling critical path).
let pt_upper_bound = M::pt_pages_upper_bound(num_pages);
page_table_allocs.reserve(pt_upper_bound, GFP_KERNEL)?;
// Walk the hierarchy per-VFN to prepare pages for all missing PDEs.
for i in 0..num_pages {
let i_u64: u64 = i.into_safe_cast();
let vfn = Vfn::new(vfn_start.raw() + i_u64);
self.ensure_single_pte_path(mm, vfn, pt_pages)?;
}
Ok(())
}
/// Install prepared PDEs and write PTEs, then flush TLB.
///
/// Drains `pt_pages` and moves allocations into `page_table_allocs`.
pub(super) fn install_mappings(
&self,
mm: &mut GpuMm<'_>,
pt_pages: &mut RBTree<VramAddress, PreparedPtPage>,
page_table_allocs: &mut KVec<Pin<KBox<AllocatedBlocks>>>,
vfn_start: Vfn,
pfns: &[Pfn],
writable: bool,
) -> Result {
{
let pramin = mm.pramin_mut();
// Drain prepared PT pages, install all pending PDEs.
let mut cursor = pt_pages.cursor_front_mut();
while let Some(c) = cursor {
let (next, node) = c.remove_current();
let (install_addr, page) = node.to_key_value();
let page_vram =
VramAddress::from_raw(page.alloc.iter().next().ok_or(ENOMEM)?.offset());
if page.level == M::DUAL_PDE_LEVEL {
let new_dpde = M::DualPde::new_small(Pfn::from(page_vram));
new_dpde.write(pramin, install_addr)?;
} else {
let new_pde = M::Pde::new(AperturePde::VideoMemory, Pfn::from(page_vram));
new_pde.write(pramin, install_addr)?;
}
page_table_allocs
.push_within_capacity(page.alloc)
.map_err(|_| ENOMEM)?;
cursor = next;
}
// Write PTEs (all PDEs now installed in HW).
for (i, &pfn) in pfns.iter().enumerate() {
let i_u64: u64 = i.into_safe_cast();
let vfn = Vfn::new(vfn_start.raw() + i_u64);
let result = self.walker.walk_to_pte_lookup_with_window(pramin, vfn)?;
match result {
WalkResult::Unmapped { pte_addr } | WalkResult::Mapped { pte_addr, .. } => {
let pte = M::Pte::new(AperturePte::VideoMemory, pfn, writable);
pte.write(pramin, pte_addr)?;
}
WalkResult::PageTableMissing => {
kernel::pr_warn_once!("VMM: page table missing for VFN {vfn:?}\n");
return Err(EIO);
}
}
}
}
// Flush TLB.
mm.tlb().flush(self.pdb_addr)
}
/// Invalidate PTEs for a range and flush TLB.
pub(super) fn invalidate_ptes(
&self,
mm: &mut GpuMm<'_>,
vfn_start: Vfn,
num_pages: usize,
) -> Result {
let invalid_pte = M::Pte::invalid();
{
let pramin = mm.pramin_mut();
for i in 0..num_pages {
let i_u64: u64 = i.into_safe_cast();
let vfn = Vfn::new(vfn_start.raw() + i_u64);
let result = self.walker.walk_to_pte_lookup_with_window(pramin, vfn)?;
match result {
WalkResult::Mapped { pte_addr, .. } | WalkResult::Unmapped { pte_addr } => {
invalid_pte.write(pramin, pte_addr)?;
}
WalkResult::PageTableMissing => {
continue;
}
}
}
}
mm.tlb().flush(self.pdb_addr)
}
}
macro_rules! pt_map_dispatch {
($self:expr, $method:ident ( $($arg:expr),* $(,)? )) => {
match $self {
PtMap::V2(inner) => inner.$method($($arg),*),
PtMap::V3(inner) => inner.$method($($arg),*),
}
};
}
/// Page table mapper dispatch.
pub(in crate::mm) enum PtMap {
/// MMU v2 (Turing/Ampere/Ada).
V2(PtMapInner<MmuV2>),
/// MMU v3 (Hopper+).
V3(PtMapInner<MmuV3>),
}
impl PtMap {
/// Create a new page table mapper for the given MMU version.
pub(in crate::mm) fn new(pdb_addr: VramAddress, version: MmuVersion) -> Self {
match version {
MmuVersion::V2 => Self::V2(PtMapInner::<MmuV2>::new(pdb_addr)),
MmuVersion::V3 => Self::V3(PtMapInner::<MmuV3>::new(pdb_addr)),
}
}
/// Prepare page table resources for a mapping.
pub(in crate::mm) fn prepare_map(
&self,
mm: &mut GpuMm<'_>,
vfn_start: Vfn,
num_pages: usize,
page_table_allocs: &mut KVec<Pin<KBox<AllocatedBlocks>>>,
pt_pages: &mut RBTree<VramAddress, PreparedPtPage>,
) -> Result {
pt_map_dispatch!(
self,
prepare_map(mm, vfn_start, num_pages, page_table_allocs, pt_pages)
)
}
/// Install prepared PDEs and write PTEs, then flush TLB.
pub(in crate::mm) fn install_mappings(
&self,
mm: &mut GpuMm<'_>,
pt_pages: &mut RBTree<VramAddress, PreparedPtPage>,
page_table_allocs: &mut KVec<Pin<KBox<AllocatedBlocks>>>,
vfn_start: Vfn,
pfns: &[Pfn],
writable: bool,
) -> Result {
pt_map_dispatch!(
self,
install_mappings(mm, pt_pages, page_table_allocs, vfn_start, pfns, writable)
)
}
/// Invalidate PTEs for a range and flush TLB.
pub(in crate::mm) fn invalidate_ptes(
&self,
mm: &mut GpuMm<'_>,
vfn_start: Vfn,
num_pages: usize,
) -> Result {
pt_map_dispatch!(self, invalidate_ptes(mm, vfn_start, num_pages))
}
}

275
drivers/gpu/nova-core/mm/pagetable/ver2.rs Normal file
View File

@ -0,0 +1,275 @@
// SPDX-License-Identifier: GPL-2.0
//! MMU v2 page table types for Turing, Ampere and Ada GPUs.
//!
//! This module defines MMU version 2 specific types (Turing, Ampere and Ada GPUs).
//!
//! Bit field layouts derived from the NVIDIA OpenRM documentation:
//! `open-gpu-kernel-modules/src/common/inc/swref/published/turing/tu102/dev_mmu.h`
#![allow(dead_code)]
use kernel::{
bitfield,
num::Bounded, //
};
use pin_init::Zeroable;
use super::{
AperturePde,
AperturePte,
DualPdeOps,
PageTableLevel,
PdeOps,
PteOps,
VaLevelIndex, //
};
use crate::mm::{
Pfn,
VirtualAddress,
VramAddress, //
};
// Bounded to version 2 Pfn bitfield conversions:
// 25 bits for video memory frame numbers (bits 32:8).
impl_pfn_bounded!(25);
// 46 bits for system memory frame numbers (bits 53:8).
impl_pfn_bounded!(46);
bitfield! {
/// MMU v2 49-bit virtual address layout.
pub(super) struct VirtualAddressV2(u64) {
/// Page offset [11:0].
11:0 offset;
/// PT index [20:12].
20:12 pt_idx;
/// PDE0 index [28:21].
28:21 pde0_idx;
/// PDE1 index [37:29].
37:29 pde1_idx;
/// PDE2 index [46:38].
46:38 pde2_idx;
/// PDE3 index [48:47].
48:47 pde3_idx;
}
}
impl VirtualAddressV2 {
/// Create a [`VirtualAddressV2`] from a [`VirtualAddress`].
pub(super) fn new(va: VirtualAddress) -> Self {
Self::from_raw(va.into_raw())
}
}
impl VaLevelIndex for VirtualAddressV2 {
fn level_index(&self, level: u64) -> u64 {
match level {
0 => *self.pde3_idx(),
1 => *self.pde2_idx(),
2 => *self.pde1_idx(),
3 => *self.pde0_idx(),
4 => *self.pt_idx(),
_ => 0,
}
}
}
/// `PDE` levels for MMU v2 (5-level hierarchy: `PDB` -> `L1` -> `L2` -> `L3` -> `L4`).
pub(super) const PDE_LEVELS: &[PageTableLevel] = &[
PageTableLevel::Pdb,
PageTableLevel::L1,
PageTableLevel::L2,
PageTableLevel::L3,
];
/// `PTE` level for MMU v2.
pub(super) const PTE_LEVEL: PageTableLevel = PageTableLevel::L4;
/// Dual `PDE` level for MMU v2 (128-bit entries).
pub(super) const DUAL_PDE_LEVEL: PageTableLevel = PageTableLevel::L3;
// Page Table Entry (PTE) for MMU v2 - 64-bit entry at level 4.
bitfield! {
/// Page Table Entry for MMU v2.
pub(in crate::mm) struct Pte(u64) {
/// Entry is valid.
0:0 valid;
/// Memory aperture type.
2:1 aperture => AperturePte;
/// Volatile (bypass L2 cache).
3:3 volatile;
/// Encryption enabled (Confidential Computing).
4:4 encrypted;
/// Privileged access only.
5:5 privilege;
/// Write protection.
6:6 read_only;
/// Atomic operations disabled.
7:7 atomic_disable;
/// Frame number for system memory.
53:8 frame_number_sys => Pfn;
/// Frame number for video memory.
32:8 frame_number_vid => Pfn;
/// Peer GPU ID for peer memory (0-7).
35:33 peer_id;
/// Compression tag line bits.
53:36 comptagline;
/// Surface kind/format.
63:56 kind;
}
}
impl PteOps for Pte {
fn from_raw(val: u64) -> Self {
Self::from_raw(val)
}
fn invalid() -> Self {
Self::zeroed()
}
fn new(aperture: AperturePte, pfn: Pfn, writable: bool) -> Self {
let base = Self::zeroed()
.with_valid(true)
.with_aperture(aperture)
.with_read_only(!writable);
match aperture {
AperturePte::VideoMemory => base.with_frame_number_vid(pfn),
// Sysmem PTEs use VOL=1 to bypass L2 for cache coherency.
AperturePte::SystemCoherent => base.with_frame_number_sys(pfn).with_volatile(true),
AperturePte::PeerMemory | AperturePte::SystemNonCoherent => {
kernel::pr_warn!("MMU v2 PTE aperture {:?} not supported\n", aperture);
Self::invalid()
}
}
}
fn is_valid(&self) -> bool {
self.valid().into_bool()
}
fn frame_number(&self) -> Pfn {
match self.aperture() {
AperturePte::VideoMemory => self.frame_number_vid(),
_ => self.frame_number_sys(),
}
}
}
// Page Directory Entry (PDE) for MMU v2 - 64-bit entry at levels 0-2.
bitfield! {
/// Page Directory Entry for MMU v2.
pub(in crate::mm) struct Pde(u64) {
/// Valid bit (inverted logic).
0:0 valid_inverted;
/// Memory aperture type.
2:1 aperture => AperturePde;
/// Volatile (bypass L2 cache).
3:3 volatile;
/// Disable Address Translation Services.
5:5 no_ats;
/// Table frame number for system memory.
53:8 table_frame_sys => Pfn;
/// Table frame number for video memory.
32:8 table_frame_vid => Pfn;
/// Peer GPU ID (0-7).
35:33 peer_id;
}
}
impl PdeOps for Pde {
fn from_raw(val: u64) -> Self {
Self::from_raw(val)
}
fn new(aperture: AperturePde, table_pfn: Pfn) -> Self {
let base = Self::zeroed()
.with_valid_inverted(false) // 0 = valid
.with_aperture(aperture);
match aperture {
AperturePde::VideoMemory => base.with_table_frame_vid(table_pfn),
// Sysmem PTEs use VOL=1 to bypass L2 for cache coherency.
AperturePde::SystemCoherent => base.with_table_frame_sys(table_pfn).with_volatile(true),
AperturePde::Invalid | AperturePde::SystemNonCoherent => {
kernel::pr_warn!("MMU v2 PDE aperture {:?} not supported\n", aperture);
Self::invalid()
}
}
}
fn invalid() -> Self {
Self::zeroed()
.with_valid_inverted(true)
.with_aperture(AperturePde::Invalid)
}
fn is_valid(&self) -> bool {
!self.valid_inverted().into_bool() && self.aperture() != AperturePde::Invalid
}
fn aperture(&self) -> AperturePde {
Pde::aperture(*self)
}
fn table_vram_address(&self) -> VramAddress {
debug_assert!(
Pde::aperture(*self) == AperturePde::VideoMemory,
"table_vram_address called on non-VRAM PDE (aperture: {:?})",
Pde::aperture(*self)
);
VramAddress::from(self.table_frame_vid())
}
}
/// Dual `PDE` at Level 3 - 128-bit entry of Large/Small Page Table pointers.
///
/// The dual `PDE` supports both large (64KB) and small (4KB) page tables.
#[repr(C)]
#[derive(Debug, Clone, Copy)]
pub(in crate::mm) struct DualPde {
/// Large/Big Page Table pointer (lower 64 bits).
pub(super) big: Pde,
/// Small Page Table pointer (upper 64 bits).
pub(super) small: Pde,
}
impl DualPde {
/// Check if the big page table pointer is valid.
fn has_big(&self) -> bool {
PdeOps::is_valid(&self.big)
}
}
impl DualPdeOps for DualPde {
fn from_raw(big: u64, small: u64) -> Self {
Self {
big: PdeOps::from_raw(big),
small: PdeOps::from_raw(small),
}
}
fn new_small(table_pfn: Pfn) -> Self {
Self {
big: PdeOps::from_raw(0),
small: PdeOps::new(AperturePde::VideoMemory, table_pfn),
}
}
fn has_small(&self) -> bool {
PdeOps::is_valid(&self.small)
}
fn small_vram_address(&self) -> VramAddress {
PdeOps::table_vram_address(&self.small)
}
fn big_raw_u64(&self) -> u64 {
self.big.into_raw()
}
fn small_raw_u64(&self) -> u64 {
self.small.into_raw()
}
}

421
drivers/gpu/nova-core/mm/pagetable/ver3.rs Normal file
View File

@ -0,0 +1,421 @@
// SPDX-License-Identifier: GPL-2.0
//! MMU v3 page table types for Hopper and later GPUs.
//!
//! This module defines MMU version 3 specific types (Hopper and later GPUs).
//!
//! Key differences from MMU v2:
//! - Unified 40-bit address field for all apertures (v2 had separate sys/vid fields).
//! - PCF (Page Classification Field) replaces separate privilege/RO/atomic/cache bits.
//! - KIND field is 4 bits (not 8).
//! - IS_PTE bit in PDE to support large pages directly.
//! - No COMPTAGLINE field (compression handled differently in v3).
//! - No separate ENCRYPTED bit.
//!
//! Bit field layouts derived from the NVIDIA OpenRM documentation:
//! `open-gpu-kernel-modules/src/common/inc/swref/published/hopper/gh100/dev_mmu.h`
#![allow(dead_code)]
use kernel::{
bitfield,
num::Bounded,
prelude::*, //
};
use pin_init::Zeroable;
use super::{
AperturePde,
AperturePte,
DualPdeOps,
PageTableLevel,
PdeOps,
PteOps,
VaLevelIndex, //
};
use crate::mm::{
Pfn,
VirtualAddress,
VramAddress, //
};
// Bounded to version 3 Pfn conversion.
impl_pfn_bounded!(40);
bitfield! {
/// MMU v3 57-bit virtual address layout.
pub(super) struct VirtualAddressV3(u64) {
/// Page offset [11:0].
11:0 offset;
/// PT index [20:12].
20:12 pt_idx;
/// PDE0 index [28:21].
28:21 pde0_idx;
/// PDE1 index [37:29].
37:29 pde1_idx;
/// PDE2 index [46:38].
46:38 pde2_idx;
/// PDE3 index [55:47].
55:47 pde3_idx;
/// PDE4 index [56].
56:56 pde4_idx;
}
}
impl VirtualAddressV3 {
/// Create a [`VirtualAddressV3`] from a [`VirtualAddress`].
pub(super) fn new(va: VirtualAddress) -> Self {
Self::from_raw(va.into_raw())
}
}
impl VaLevelIndex for VirtualAddressV3 {
fn level_index(&self, level: u64) -> u64 {
match level {
0 => *self.pde4_idx(),
1 => *self.pde3_idx(),
2 => *self.pde2_idx(),
3 => *self.pde1_idx(),
4 => *self.pde0_idx(),
5 => *self.pt_idx(),
_ => 0,
}
}
}
/// PDE levels for MMU v3 (6-level hierarchy).
pub(super) const PDE_LEVELS: &[PageTableLevel] = &[
PageTableLevel::Pdb,
PageTableLevel::L1,
PageTableLevel::L2,
PageTableLevel::L3,
PageTableLevel::L4,
];
/// PTE level for MMU v3.
pub(super) const PTE_LEVEL: PageTableLevel = PageTableLevel::L5;
/// Dual PDE level for MMU v3 (128-bit entries).
pub(super) const DUAL_PDE_LEVEL: PageTableLevel = PageTableLevel::L4;
bitfield! {
/// Page Classification Field for PTEs (5 bits) in MMU v3.
pub(in crate::mm) struct PtePcf(u8) {
/// Bypass L2 cache (0=cached, 1=bypass).
0:0 uncached;
/// Access counting disabled (0=enabled, 1=disabled).
1:1 acd;
/// Read-only access (0=read-write, 1=read-only).
2:2 read_only;
/// Atomics disabled (0=enabled, 1=disabled).
3:3 no_atomic;
/// Privileged access only (0=regular, 1=privileged).
4:4 privileged;
}
}
impl PtePcf {
/// Create PCF for read-write mapping (cached, no atomics, regular mode).
fn rw() -> Self {
Self::zeroed().with_no_atomic(true)
}
/// Create PCF for read-only mapping (cached, no atomics, regular mode).
fn ro() -> Self {
Self::zeroed().with_read_only(true).with_no_atomic(true)
}
/// Get the raw `u8` value.
fn raw_u8(&self) -> u8 {
self.into_raw()
}
}
impl From<Bounded<u64, 5>> for PtePcf {
fn from(val: Bounded<u64, 5>) -> Self {
Self::from_raw(u8::from(val))
}
}
impl From<PtePcf> for Bounded<u64, 5> {
fn from(pcf: PtePcf) -> Self {
Bounded::from_expr(u64::from(pcf.into_raw()) & 0x1F)
}
}
bitfield! {
/// Page Classification Field for PDEs (3 bits) in MMU v3.
///
/// Controls Address Translation Services (ATS) and caching.
pub(in crate::mm) struct PdePcf(u8) {
/// Bypass L2 cache (0=cached, 1=bypass).
0:0 uncached;
/// ATS disabled (0=enabled, 1=disabled).
1:1 no_ats;
}
}
impl PdePcf {
/// Create PCF for cached mapping with ATS enabled (default).
fn cached() -> Self {
Self::zeroed()
}
/// Get the raw `u8` value.
fn raw_u8(&self) -> u8 {
self.into_raw()
}
}
impl From<Bounded<u64, 3>> for PdePcf {
fn from(val: Bounded<u64, 3>) -> Self {
Self::from_raw(u8::from(val))
}
}
impl From<PdePcf> for Bounded<u64, 3> {
fn from(pcf: PdePcf) -> Self {
Bounded::from_expr(u64::from(pcf.into_raw()) & 0x7)
}
}
bitfield! {
/// Page Table Entry for MMU v3.
pub(in crate::mm) struct Pte(u64) {
/// Entry is valid.
0:0 valid;
/// Memory aperture type.
2:1 aperture => AperturePte;
/// Page Classification Field.
7:3 pcf => PtePcf;
/// Surface kind (4 bits, 0x0=pitch, 0xF=invalid).
11:8 kind;
/// Physical frame number (for all apertures).
51:12 frame_number => Pfn;
/// Peer GPU ID for peer memory (0-7).
63:61 peer_id;
}
}
impl PteOps for Pte {
fn from_raw(val: u64) -> Self {
Self::from_raw(val)
}
fn invalid() -> Self {
Self::zeroed()
}
fn new(aperture: AperturePte, pfn: Pfn, writable: bool) -> Self {
let pcf = match (aperture, writable) {
(AperturePte::VideoMemory, true) => PtePcf::rw(),
(AperturePte::VideoMemory, false) => PtePcf::ro(),
// Sysmem PTEs use uncached+no_atomic PCF for cache coherency.
(AperturePte::SystemCoherent, true) => {
PtePcf::zeroed().with_uncached(true).with_no_atomic(true)
}
(AperturePte::SystemCoherent, false) => PtePcf::zeroed()
.with_uncached(true)
.with_no_atomic(true)
.with_read_only(true),
(AperturePte::PeerMemory | AperturePte::SystemNonCoherent, _) => {
kernel::pr_warn!("MMU v3 PTE aperture {:?} not supported\n", aperture);
return Self::invalid();
}
};
Self::zeroed()
.with_valid(true)
.with_aperture(aperture)
.with_pcf(pcf)
.with_frame_number(pfn)
}
fn is_valid(&self) -> bool {
self.valid().into_bool()
}
fn frame_number(&self) -> Pfn {
Pte::frame_number(*self)
}
}
bitfield! {
/// Page Directory Entry for MMU v3 (Hopper+).
///
/// ## Note
///
/// v3 uses a unified 40-bit address field (v2 had separate sys/vid address fields).
pub(in crate::mm) struct Pde(u64) {
/// Entry is a PTE (0=PDE, 1=large page PTE).
0:0 is_pte;
/// Memory aperture type.
2:1 aperture => AperturePde;
/// Page Classification Field (3 bits for PDE).
5:3 pcf => PdePcf;
/// Table frame number (40-bit unified address).
51:12 table_frame => Pfn;
}
}
impl PdeOps for Pde {
fn from_raw(val: u64) -> Self {
Self::from_raw(val)
}
fn new(aperture: AperturePde, table_pfn: Pfn) -> Self {
match aperture {
AperturePde::VideoMemory => Self::zeroed()
.with_is_pte(false)
.with_aperture(aperture)
.with_table_frame(table_pfn),
AperturePde::Invalid | AperturePde::SystemCoherent | AperturePde::SystemNonCoherent => {
kernel::pr_warn!("MMU v3 PDE aperture {:?} not supported\n", aperture);
Self::invalid()
}
}
}
fn invalid() -> Self {
Self::zeroed().with_aperture(AperturePde::Invalid)
}
fn is_valid(&self) -> bool {
Pde::aperture(*self) != AperturePde::Invalid
}
fn aperture(&self) -> AperturePde {
Pde::aperture(*self)
}
fn table_vram_address(&self) -> VramAddress {
debug_assert!(
Pde::aperture(*self) == AperturePde::VideoMemory,
"table_vram_address called on non-VRAM PDE (aperture: {:?})",
Pde::aperture(*self)
);
VramAddress::from(self.table_frame())
}
}
bitfield! {
/// Big Page Table pointer in Dual PDE (MMU v3).
///
/// 64-bit lower word of the 128-bit Dual PDE.
pub(super) struct DualPdeBig(u64) {
/// Entry is a PTE (for large pages).
0:0 is_pte;
/// Memory aperture type.
2:1 aperture => AperturePde;
/// Page Classification Field.
5:3 pcf => PdePcf;
/// Table frame (table address 256-byte aligned).
51:8 table_frame;
}
}
impl DualPdeBig {
/// Create an invalid big page table pointer.
fn invalid() -> Self {
Self::zeroed().with_aperture(AperturePde::Invalid)
}
/// Create a valid big PDE pointing to a page table in the given aperture.
fn new(aperture: AperturePde, table_addr: VramAddress) -> Result<Self> {
// Big page table addresses must be 256-byte aligned (shift 8).
if table_addr.into_raw() & 0xFF != 0 {
return Err(EINVAL);
}
let table_frame = Bounded::from_expr(table_addr.into_raw() >> 8);
match aperture {
AperturePde::VideoMemory => Ok(Self::zeroed()
.with_is_pte(false)
.with_aperture(aperture)
.with_table_frame(table_frame)),
AperturePde::Invalid | AperturePde::SystemCoherent | AperturePde::SystemNonCoherent => {
kernel::pr_warn!("MMU v3 DualPdeBig aperture {:?} not supported\n", aperture);
Ok(Self::invalid())
}
}
}
/// Check if this big PDE is valid.
fn is_valid(&self) -> bool {
self.aperture() != AperturePde::Invalid
}
/// Get the VRAM address of the big page table.
fn table_vram_address(&self) -> VramAddress {
debug_assert!(
self.aperture() == AperturePde::VideoMemory,
"table_vram_address called on non-VRAM DualPdeBig (aperture: {:?})",
self.aperture()
);
VramAddress::from_raw(*self.table_frame() << 8)
}
}
/// Dual PDE at Level 4 for MMU v3 - 128-bit entry.
///
/// Contains both big (64KB) and small (4KB) page table pointers:
/// - Lower 64 bits: Big Page Table pointer.
/// - Upper 64 bits: Small Page Table pointer.
///
/// ## Note
///
/// The big and small page table pointers have different address layouts:
/// - Big address = field value << 8 (256-byte alignment).
/// - Small address = field value << 12 (4KB alignment).
///
/// This is why `DualPdeBig` is a separate type from `Pde`.
#[repr(C)]
#[derive(Debug, Clone, Copy)]
pub(in crate::mm) struct DualPde {
/// Big Page Table pointer.
pub(super) big: DualPdeBig,
/// Small Page Table pointer.
pub(super) small: Pde,
}
// SAFETY: Both `DualPdeBig` and `Pde` fields are `Zeroable` (bitfield types are Zeroable).
unsafe impl Zeroable for DualPde {}
impl DualPde {
/// Check if the big page table pointer is valid.
fn has_big(&self) -> bool {
self.big.is_valid()
}
}
impl DualPdeOps for DualPde {
fn from_raw(big: u64, small: u64) -> Self {
Self {
big: DualPdeBig::from_raw(big),
small: PdeOps::from_raw(small),
}
}
fn new_small(table_pfn: Pfn) -> Self {
Self {
big: DualPdeBig::invalid(),
small: PdeOps::new(AperturePde::VideoMemory, table_pfn),
}
}
fn has_small(&self) -> bool {
PdeOps::is_valid(&self.small)
}
fn small_vram_address(&self) -> VramAddress {
PdeOps::table_vram_address(&self.small)
}
fn big_raw_u64(&self) -> u64 {
self.big.into_raw()
}
fn small_raw_u64(&self) -> u64 {
self.small.into_raw()
}
}

244
drivers/gpu/nova-core/mm/pagetable/walk.rs Normal file
View File

@ -0,0 +1,244 @@
// SPDX-License-Identifier: GPL-2.0
//! Page table walker implementation for NVIDIA GPUs.
//!
//! This module provides page table walking functionality for MMU v2 and v3.
//! The walker traverses the page table hierarchy to resolve virtual addresses
//! to physical addresses or to find PTE locations.
//!
//! # Page Table Hierarchy
//!
//! ## MMU v2 (Turing/Ampere/Ada) - 5 levels
//!
//! ```text
//! +-------+ +-------+ +-------+ +---------+ +-------+
//! | PDB |---->| L1 |---->| L2 |---->| L3 Dual |---->| L4 |
//! | (L0) | | | | | | PDE | | (PTE) |
//! +-------+ +-------+ +-------+ +---------+ +-------+
//! 64-bit 64-bit 64-bit 128-bit 64-bit
//! PDE PDE PDE (big+small) PTE
//! ```
//!
//! ## MMU v3 (Hopper+) - 6 levels
//!
//! ```text
//! +-------+ +-------+ +-------+ +-------+ +---------+ +-------+
//! | PDB |---->| L1 |---->| L2 |---->| L3 |---->| L4 Dual |---->| L5 |
//! | (L0) | | | | | | | | PDE | | (PTE) |
//! +-------+ +-------+ +-------+ +-------+ +---------+ +-------+
//! 64-bit 64-bit 64-bit 64-bit 128-bit 64-bit
//! PDE PDE PDE PDE (big+small) PTE
//! ```
//!
//! # Result of a page table walk
//!
//! The walker returns a [`WalkResult`] indicating the outcome.
use core::marker::PhantomData;
use kernel::prelude::*;
use super::{
DualPdeOps,
MmuConfig,
MmuV2,
MmuV3,
MmuVersion,
PageTableLevel,
PdeOps,
PteOps, //
};
use crate::{
mm::{
pramin,
GpuMm,
Pfn,
Vfn,
VirtualAddress,
VramAddress, //
},
num::{
IntoSafeCast, //
},
};
/// Result of walking to a PTE.
#[derive(Debug, Clone, Copy)]
pub(in crate::mm) enum WalkResult {
/// Intermediate page tables are missing (only returned in lookup mode).
PageTableMissing,
/// PTE exists but is invalid (page not mapped).
Unmapped { pte_addr: VramAddress },
/// PTE exists and is valid (page is mapped).
Mapped { pte_addr: VramAddress, pfn: Pfn },
}
/// Result of walking PDE levels only.
///
/// Returned by [`PtWalkInner::walk_pde_levels()`] to indicate whether all PDE
/// levels resolved or a PDE is missing.
#[derive(Debug, Clone, Copy)]
pub(in crate::mm) enum WalkPdeResult {
/// All PDE levels resolved -- returns PTE page table address.
Complete {
/// VRAM address of the PTE-level page table.
pte_table: VramAddress,
},
/// A PDE is missing and no prepared page was provided by the closure.
Missing {
/// PDE slot address in the parent page table (where to install).
install_addr: VramAddress,
/// The page table level that is missing.
level: PageTableLevel,
},
}
/// Page table walker.
pub(in crate::mm) struct PtWalkInner<M: MmuConfig> {
pdb_addr: VramAddress,
_phantom: PhantomData<M>,
}
impl<M: MmuConfig> PtWalkInner<M> {
/// Calculate the VRAM address of an entry within a page table.
fn entry_addr(table: VramAddress, level: PageTableLevel, index: u64) -> VramAddress {
let entry_size: u64 = M::entry_size(level).into_safe_cast();
table + index * entry_size
}
/// Create a new page table walker.
pub(super) fn new(pdb_addr: VramAddress) -> Self {
Self {
pdb_addr,
_phantom: PhantomData,
}
}
/// Walk PDE levels with closure-based resolution for missing PDEs.
///
/// Traverses all PDE levels for the MMU version. At each level, reads the PDE.
/// If valid, extracts the child table address and continues. If missing, calls
/// `resolve_prepared(install_addr)` to resolve the missing PDE.
pub(super) fn walk_pde_levels(
&self,
pramin: &mut pramin::Pramin<'_>,
vfn: Vfn,
resolve_prepared: impl Fn(VramAddress) -> Option<VramAddress>,
) -> Result<WalkPdeResult> {
let va = VirtualAddress::from(vfn);
let mut cur_table = self.pdb_addr;
for &level in M::PDE_LEVELS {
let idx = M::level_index(va, level.as_index());
let install_addr = Self::entry_addr(cur_table, level, idx);
if level == M::DUAL_PDE_LEVEL {
// 128-bit dual PDE with big+small page table pointers.
let dpde = M::DualPde::read(pramin, install_addr)?;
if dpde.has_small() {
cur_table = dpde.small_vram_address();
continue;
}
} else {
// Regular 64-bit PDE. Use `is_valid_vram()` because
// `table_vram_address()` only reads the VRAM frame-number
// bitfield; system-memory PDEs store the address in a
// different (wider) field and would be silently truncated.
let pde = M::Pde::read(pramin, install_addr)?;
if pde.is_valid_vram() {
cur_table = pde.table_vram_address();
continue;
}
}
// PDE missing in HW. Ask caller for resolution.
if let Some(prepared_addr) = resolve_prepared(install_addr) {
cur_table = prepared_addr;
continue;
}
return Ok(WalkPdeResult::Missing {
install_addr,
level,
});
}
Ok(WalkPdeResult::Complete {
pte_table: cur_table,
})
}
/// Walk to PTE for lookup only (no allocation).
///
/// Returns [`WalkResult::PageTableMissing`] if intermediate tables don't exist.
pub(super) fn walk_to_pte_lookup(&self, mm: &mut GpuMm<'_>, vfn: Vfn) -> Result<WalkResult> {
self.walk_to_pte_lookup_with_window(mm.pramin_mut(), vfn)
}
/// Walk to PTE using a caller-provided PRAMIN manager (lookup only).
pub(super) fn walk_to_pte_lookup_with_window(
&self,
pramin: &mut pramin::Pramin<'_>,
vfn: Vfn,
) -> Result<WalkResult> {
match self.walk_pde_levels(pramin, vfn, |_| None)? {
WalkPdeResult::Complete { pte_table } => {
Self::read_pte_at_level(pramin, vfn, pte_table)
}
WalkPdeResult::Missing { .. } => Ok(WalkResult::PageTableMissing),
}
}
/// Read the PTE at the PTE level given the PTE table address.
fn read_pte_at_level(
pramin: &mut pramin::Pramin<'_>,
vfn: Vfn,
pte_table: VramAddress,
) -> Result<WalkResult> {
let va = VirtualAddress::from(vfn);
let pte_level = M::PTE_LEVEL;
let pte_idx = M::level_index(va, pte_level.as_index());
let pte_addr = Self::entry_addr(pte_table, pte_level, pte_idx);
let pte = M::Pte::read(pramin, pte_addr)?;
if pte.is_valid() {
return Ok(WalkResult::Mapped {
pte_addr,
pfn: pte.frame_number(),
});
}
Ok(WalkResult::Unmapped { pte_addr })
}
}
macro_rules! pt_walk_dispatch {
($self:expr, $method:ident ( $($arg:expr),* $(,)? )) => {
match $self {
PtWalk::V2(inner) => inner.$method($($arg),*),
PtWalk::V3(inner) => inner.$method($($arg),*),
}
};
}
/// Page table walker dispatch.
pub(in crate::mm) enum PtWalk {
/// MMU v2 (Turing/Ampere/Ada).
V2(PtWalkInner<MmuV2>),
/// MMU v3 (Hopper+).
V3(PtWalkInner<MmuV3>),
}
impl PtWalk {
/// Create a new page table walker for the given MMU version.
pub(in crate::mm) fn new(pdb_addr: VramAddress, version: MmuVersion) -> Self {
match version {
MmuVersion::V2 => Self::V2(PtWalkInner::<MmuV2>::new(pdb_addr)),
MmuVersion::V3 => Self::V3(PtWalkInner::<MmuV3>::new(pdb_addr)),
}
}
/// Walk to PTE for lookup.
pub(in crate::mm) fn walk_to_pte(&self, mm: &mut GpuMm<'_>, vfn: Vfn) -> Result<WalkResult> {
pt_walk_dispatch!(self, walk_to_pte_lookup(mm, vfn))
}
}

312
drivers/gpu/nova-core/mm/pramin.rs Normal file
View File

@ -0,0 +1,312 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Utilities for accessing VRAM through the PRAMIN window.
use core::ops::Range;
use kernel::{
io::{
io_project,
register,
register::OffsetLoc,
Io,
Mmio, //
},
prelude::*,
ptr::{
Alignable,
Alignment, //
},
sizes::{
SZ_1M,
SZ_64K, //
},
};
use crate::{
driver::{
Bar0,
NovaRegisters, //
},
gpu::Chipset,
mm::{
hal::{
self,
MmHal, //
},
VramAddress, //
},
num::IntoSafeCast, //
};
/// Size of the PRAMIN window (1 MiB).
const WINDOW_SIZE: usize = SZ_1M;
/// The PRAMIN window, which is a 1 MiB window into VRAM at a fixed BAR0 offset.
#[derive(FromBytes, IntoBytes)]
struct PraminWindow([u8; WINDOW_SIZE]);
register! {
base: NovaRegisters;
/// Location of the window inside BAR0.
PRAMIN: PraminWindow @ 0x700000;
}
/// Owner of the PRAMIN window state.
///
/// [`Pramin::window_at()`] repositions the window as needed and returns a typed MMIO view into
/// it, holding the manager borrowed for the lifetime of the view.
pub(super) struct Pramin<'gpu> {
bar: Bar0<'gpu>,
hal: &'static dyn MmHal,
/// MMIO view of the PRAMIN window in BAR0.
window: Mmio<'gpu, PraminWindow>,
/// VRAM range to keep the PRAMIN window inside.
vram_range: Range<VramAddress>,
/// Cached window position.
window_range: Range<VramAddress>,
}
/// Typed view of VRAM through the PRAMIN window.
///
/// Inserts an ordering point after previous writes through the window on drop. Views returned
/// by [`PraminAccess::view()`] cannot outlive this access, so the ordering point covers every
/// write made through them.
pub(super) struct PraminAccess<'a, T>
where
T: FromBytes + IntoBytes,
{
view: Mmio<'a, T>,
}
impl<T> PraminAccess<'_, T>
where
T: FromBytes + IntoBytes,
{
/// Returns the MMIO view of the accessed location.
pub(super) fn view(&self) -> Mmio<'_, T> {
self.view
}
}
impl<T> Drop for PraminAccess<'_, T>
where
T: FromBytes + IntoBytes,
{
fn drop(&mut self) {
// Insert an ordering point after previous writes through this window.
self.view.cast::<u8>().read_val();
}
}
impl<'gpu> Pramin<'gpu> {
/// Alignment required by the PRAMIN window.
const BASE_ALIGN: Alignment = Alignment::new::<SZ_64K>();
/// Creates the window manager for the given VRAM region.
pub(super) fn new(
bar: Bar0<'gpu>,
chipset: Chipset,
vram_range: Range<VramAddress>,
) -> Result<Self> {
let hal = hal::mm_hal(chipset);
let window = io_project!(bar, build: PRAMIN);
let base = vram_range.start.align_down(Self::BASE_ALIGN);
let window_range = Self::window_range(base)?;
hal.write_pramin_window_base(bar, base)?;
Ok(Self {
bar,
hal,
window,
vram_range,
window_range,
})
}
/// Returns the VRAM range a window based at `base` exposes.
fn window_range(base: VramAddress) -> Result<Range<VramAddress>> {
let end = base
.checked_add(WINDOW_SIZE.into_safe_cast())
.ok_or(EINVAL)?;
Ok(base..end)
}
/// Check the window covers `len` bytes at `addr`, moving it if needed.
///
/// Returns the window offset at which to perform the access.
fn window_offset(&mut self, addr: VramAddress, len: usize) -> Result<usize> {
let end = addr.checked_add(len.into_safe_cast()).ok_or(EINVAL)?;
let inside = |r: &Range<VramAddress>| r.contains(&addr) && end <= r.end;
if !inside(&self.vram_range) {
return Err(EINVAL);
}
// Reposition the window if the access falls outside it.
if !inside(&self.window_range) {
let base = addr.align_down(Self::BASE_ALIGN);
let window_range = Self::window_range(base)?;
if !inside(&window_range) {
return Err(EINVAL);
}
self.hal.write_pramin_window_base(self.bar, base)?;
self.window_range = window_range;
}
Ok((addr - self.window_range.start).into_safe_cast())
}
/// Return a typed MMIO view of a `T` at `vram_addr`.
///
/// Returns an error if `vram_addr` is not aligned to `T`'s alignment, or if
/// a `T` at `vram_addr` does not fit within the VRAM region.
pub(super) fn window_at<'a, T>(
&'a mut self,
vram_addr: VramAddress,
) -> Result<PraminAccess<'a, T>>
where
T: FromBytes + IntoBytes,
{
let offset = self.window_offset(vram_addr, size_of::<T>())?;
let view = io_project!(self.window, try: OffsetLoc::new(offset));
Ok(PraminAccess { view })
}
}
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
pub(super) mod selftest {
use kernel::{
device,
io::io_read,
sizes::SizeConstants, //
};
use super::*;
use crate::{
selftest_assert,
selftest_assert_eq, //
};
/// Test read/write at byte granularity, at unaligned addresses.
fn test_byte_readwrite(
dev: &device::Device<device::Bound>,
pramin: &mut Pramin<'_>,
base: VramAddress,
) -> Result {
for i in 0u8..4 {
let addr = base + 1 + u64::from(i);
pramin.window_at::<u8>(addr)?.view().write_val(0xA0 + i);
}
for i in 0u8..4 {
let addr = base + 1 + u64::from(i);
selftest_assert_eq!(
dev,
pramin.window_at::<u8>(addr)?.view().read_val(),
0xA0 + i
);
}
Ok(())
}
/// Test writing a `u32` and reading back as individual `u8`s.
fn test_u32_as_bytes(
dev: &device::Device<device::Bound>,
pramin: &mut Pramin<'_>,
base: VramAddress,
) -> Result {
let addr = base + 0x10;
let val: u32 = 0xDEADBEEF;
pramin.window_at::<u32>(addr)?.view().write_val(val);
let window = pramin.window_at::<[u8; 4]>(addr)?;
for (i, &expected) in val.to_le_bytes().iter().enumerate() {
selftest_assert_eq!(dev, io_read!(window.view(), [build: i]), expected);
}
Ok(())
}
/// Test window repositioning across 1 MiB boundaries.
fn test_window_reposition(
dev: &device::Device<device::Bound>,
pramin: &mut Pramin<'_>,
base: VramAddress,
) -> Result {
let addr_a = base;
let addr_b = base + u64::SZ_2M; // base + 2 MiB (different 1 MiB region).
let val_a: u32 = 0x11111111;
let val_b: u32 = 0x22222222;
pramin.window_at::<u32>(addr_a)?.view().write_val(val_a);
pramin.window_at::<u32>(addr_b)?.view().write_val(val_b);
selftest_assert_eq!(
dev,
pramin.window_at::<u32>(addr_a)?.view().read_val(),
val_a
);
selftest_assert_eq!(
dev,
pramin.window_at::<u32>(addr_b)?.view().read_val(),
val_b
);
Ok(())
}
/// Test that offsets outside the VRAM region are rejected.
fn test_invalid_offset(
dev: &device::Device<device::Bound>,
pramin: &mut Pramin<'_>,
vram_end: VramAddress,
) -> Result {
selftest_assert!(dev, pramin.window_at::<u32>(vram_end).is_err());
Ok(())
}
/// Test that misaligned accesses are rejected.
fn test_misaligned_access(
dev: &device::Device<device::Bound>,
pramin: &mut Pramin<'_>,
base: VramAddress,
) -> Result {
// `u16` at odd offset (not 2-byte aligned).
selftest_assert!(dev, pramin.window_at::<u16>(base + 0x21).is_err());
// `u32` at 2-byte-aligned (not 4-byte-aligned) offset.
selftest_assert!(dev, pramin.window_at::<u32>(base + 2).is_err());
// `u64` at a 4-byte-aligned (not 8-byte-aligned) address.
selftest_assert!(dev, pramin.window_at::<u64>(base + 0x44).is_err());
// A `u16` view at an even address is allowed.
pramin.window_at::<u16>(base + 0x22)?;
Ok(())
}
/// Run PRAMIN self-tests during probe.
///
/// `base` is the start of a driver-usable VRAM span that the tests are free to
/// overwrite.
pub(crate) fn run(
dev: &device::Device<device::Bound>,
pramin: &mut Pramin<'_>,
base: VramAddress,
) -> Result {
dev_dbg!(dev, "PRAMIN: starting self-tests\n");
let vram_end = pramin.vram_range.end;
test_byte_readwrite(dev, pramin, base)?;
test_u32_as_bytes(dev, pramin, base)?;
test_window_reposition(dev, pramin, base)?;
test_invalid_offset(dev, pramin, vram_end)?;
test_misaligned_access(dev, pramin, base)?;
dev_info!(dev, "PRAMIN: self-tests passed\n");
Ok(())
}
}

70
drivers/gpu/nova-core/mm/regs.rs Normal file
View File

@ -0,0 +1,70 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Registers used by the memory management subsystems: the BAR0 PRAMIN window.
use kernel::io::register;
use crate::{
bounded_enum,
driver::NovaRegisters, //
};
// PRAMIN window
bounded_enum! {
/// Target memory type for the BAR0 window register.
///
/// Only VRAM is needed by the driver. Pre-Hopper window registers also define
/// system-memory targets that are unused here; Hopper+ uses a separate register
/// without a target field.
#[derive(Debug, Copy, Clone)]
pub(super) enum Bar0WindowTarget with TryFrom<Bounded<u32, 2>> {
/// Video memory (GPU framebuffer memory).
VidMem = 0,
}
}
register! {
base: NovaRegisters;
/// BAR0 window control for PRAMIN access.
pub(super) NV_PBUS_BAR0_WINDOW(u32) @ 0x00001700 {
/// Target memory aperture for the window.
25:24 target ?=> Bar0WindowTarget;
/// PRAMIN window base bits 39:16.
23:0 base;
}
}
pub(super) mod gh100 {
use kernel::io::register;
use crate::driver::NovaRegisters;
register! {
base: NovaRegisters;
/// Hopper register for PRAMIN window.
pub(crate) NV_XAL_EP_BAR0_WINDOW(u32) @ 0x0010fd40 {
/// PRAMIN window base bits 37:16.
21:0 base;
}
}
}
pub(super) mod gb100 {
use kernel::io::register;
use crate::driver::NovaRegisters;
register! {
base: NovaRegisters;
/// Blackwell GB10x/GB20x register for PRAMIN window.
pub(crate) NV_XAL_EP_BAR0_WINDOW(u32) @ 0x0010fd40 {
/// PRAMIN window base bits 38:16.
22:0 base;
}
}
}

120
drivers/gpu/nova-core/mm/tlb.rs Normal file
View File

@ -0,0 +1,120 @@
// SPDX-License-Identifier: GPL-2.0
//! TLB (Translation Lookaside Buffer) flush support for GPU MMU.
//!
//! After modifying page table entries, the GPU's TLB must be flushed to
//! ensure the new mappings take effect. This module provides TLB flush
//! functionality for virtual memory managers.
//!
//! # Examples
//!
//! ```ignore
//! use crate::mm::tlb::Tlb;
//!
//! fn page_table_update(tlb: &Tlb, pdb_addr: VramAddress) -> Result<()> {
//! // ... modify page tables ...
//!
//! // Flush TLB to make changes visible (polls for completion).
//! tlb.flush(pdb_addr)?;
//!
//! Ok(())
//! }
//! ```
use kernel::{
io::poll::read_poll_timeout,
io::Io,
new_mutex,
prelude::*,
sync::Mutex,
time::Delta, //
};
use crate::{
bounded_enum,
driver::Bar0,
mm::VramAddress,
regs, //
};
bounded_enum! {
/// TLB invalidation acknowledgment scope.
///
/// Controls how far the hardware waits for the invalidation to propagate
/// before clearing the `trigger` bit of `NV_TLB_FLUSH_CTRL`.
#[derive(Debug, Copy, Clone, PartialEq, Eq)]
pub(crate) enum TlbAckMode with TryFrom<Bounded<u32, 2>> {
/// Fire-and-forget: no acknowledgment required.
None = 0,
/// Wait for acknowledgment from all consumers, including remote GPUs
/// reachable over NVLink.
///
/// Globally is strictly required only during unmap or permission
/// tightening, because the backing memory may be reassigned after the
/// flush returns and a stale TLB entry could let the GPU access freed
/// memory. For new mapping or relaxing permissions, a stale entry would
/// merely cause a redundant fault and retry, so [`TlbAckMode::None`]
/// would suffice.
Globally = 1,
/// Wait for acknowledgment from consumers within the local NVLink
/// fabric node only; skip cross-node ack.
Intranode = 2,
}
}
/// TLB manager for GPU translation buffer operations.
#[pin_data]
pub(crate) struct Tlb<'gpu> {
bar: Bar0<'gpu>,
/// TLB flush serialization lock: This lock is designed to be acquired during
/// the DMA fence signalling critical path. It should NEVER be held across any
/// reclaimable CPU memory allocations because the memory reclaim path can
/// call `dma_fence_wait()` (when implemented), which would deadlock if lock held.
#[pin]
lock: Mutex<()>,
}
impl<'gpu> Tlb<'gpu> {
/// Create a new TLB manager.
pub(super) fn new(bar: Bar0<'gpu>) -> impl PinInit<Self> {
pin_init!(Self {
bar,
lock <- new_mutex!((), "tlb_flush"),
})
}
/// Flush the GPU TLB for a specific page directory base.
///
/// This invalidates all TLB entries associated with the given PDB address.
/// Must be called after modifying page table entries to ensure the GPU sees
/// the updated mappings.
pub(super) fn flush(&self, pdb_addr: VramAddress) -> Result {
let _guard = self.lock.lock();
// Write PDB address.
self.bar.write_reg(regs::NV_TLB_FLUSH_PDB_LO::from_pdb_addr(
pdb_addr.into_raw(),
));
self.bar.write_reg(regs::NV_TLB_FLUSH_PDB_HI::from_pdb_addr(
pdb_addr.into_raw(),
));
// Trigger flush.
self.bar.write_reg(
regs::NV_TLB_FLUSH_CTRL::zeroed()
.with_all_va(true)
.with_ack(TlbAckMode::None)
.with_trigger(true),
);
// Poll for completion.
read_poll_timeout(
|| Ok(self.bar.read(regs::NV_TLB_FLUSH_CTRL)),
|ctrl: &regs::NV_TLB_FLUSH_CTRL| !ctrl.trigger(),
Delta::ZERO,
Delta::from_secs(2),
)?;
Ok(())
}
}

346
drivers/gpu/nova-core/mm/vmm.rs Normal file
View File

@ -0,0 +1,346 @@
// SPDX-License-Identifier: GPL-2.0
//! Virtual Memory Manager for NVIDIA GPU page table management.
//!
//! The [`Vmm`] provides high-level page mapping and unmapping operations for GPU
//! virtual address spaces (Channels, BAR1, BAR2).
use kernel::{
gpu::buddy::AllocatedBlocks,
maple_tree::MapleTreeAlloc,
prelude::*,
rbtree::RBTree, //
};
use core::{
cell::Cell,
ops::Range, //
};
use crate::{
mm::{
pagetable::{
map::{
PtMap, //
},
walk::{
PtWalk,
WalkResult, //
},
MmuVersion, //
},
GpuMm,
Pfn,
Vfn,
VramAddress,
PAGE_SIZE, //
},
num::{
IntoSafeCast, //
},
};
/// Multi-page prepared mapping -- VA range allocated, ready for execute.
///
/// Produced by [`Vmm::prepare_map()`], consumed by [`Vmm::execute_map()`].
/// The VA space allocation is tracked in the [`Vmm`]'s maple tree and freed
/// on error or via [`Vmm::unmap_pages()`].
///
/// Dropping without calling [`Vmm::execute_map()`] logs a warning and leaks
/// the VA range in the maple tree.
pub(crate) struct PreparedMapping {
vfn_start: Vfn,
num_pages: usize,
/// Logs a warning if dropped without executing.
_drop_guard: MustExecuteGuard,
}
/// Result of a mapping operation -- tracks the active mapped range.
///
/// Returned by [`Vmm::execute_map()`] and [`Vmm::map_pages()`].
/// Callers must call [`Vmm::unmap_pages()`] before dropping to invalidate
/// PTEs and free the VA range. Dropping without unmapping logs a warning
/// and leaks the VA range in the maple tree.
pub(crate) struct MappedRange {
pub(super) vfn_start: Vfn,
pub(super) num_pages: usize,
/// Logs a warning if dropped without unmapping.
_drop_guard: MustUnmapGuard,
}
/// Guard that logs a warning if a [`PreparedMapping`] is dropped without
/// being consumed by [`Vmm::execute_map()`].
struct MustExecuteGuard {
armed: Cell<bool>,
}
impl MustExecuteGuard {
const fn new() -> Self {
Self {
armed: Cell::new(true),
}
}
fn disarm(&self) {
self.armed.set(false);
}
}
impl Drop for MustExecuteGuard {
fn drop(&mut self) {
if self.armed.get() {
kernel::pr_warn!("PreparedMapping dropped without calling execute_map()\n");
}
}
}
/// Guard that logs a warning if a [`MappedRange`] is dropped without
/// calling [`Vmm::unmap_pages()`].
struct MustUnmapGuard {
armed: Cell<bool>,
}
impl MustUnmapGuard {
const fn new() -> Self {
Self {
armed: Cell::new(true),
}
}
fn disarm(&self) {
self.armed.set(false);
}
}
impl Drop for MustUnmapGuard {
fn drop(&mut self) {
if self.armed.get() {
kernel::pr_warn!("MappedRange dropped without calling unmap_pages()\n");
}
}
}
/// Virtual Memory Manager for a GPU address space.
///
/// Each [`Vmm`] instance manages a single address space identified by its Page
/// Directory Base (`PDB`) address. Used for Channel, BAR1 and BAR2 mappings.
pub(crate) struct Vmm {
/// Page Directory Base address for this address space.
#[expect(dead_code)]
pdb_addr: VramAddress,
/// Page table walker for reading existing mappings.
pt_walk: PtWalk,
/// Page table mapper for prepare/execute operations.
pt_map: PtMap,
/// Page table allocations required for mappings.
page_table_allocs: KVec<Pin<KBox<AllocatedBlocks>>>,
/// Maple tree allocator for virtual address range tracking.
virt_alloc: Pin<KBox<MapleTreeAlloc<()>>>,
/// Total number of pages in the virtual address space.
va_pages: usize,
/// Prepared PT pages pending PDE installation, keyed by `install_addr`.
///
/// Populated during prepare phase and drained in execute phase. Shared by all
/// pending maps, preventing races on the same PDE slot.
pt_pages: RBTree<VramAddress, super::pagetable::map::PreparedPtPage>,
}
impl Vmm {
/// Create a new [`Vmm`] for the given Page Directory Base address.
///
/// The [`Vmm`] will manage a virtual address space of `va_size` bytes.
pub(crate) fn new(
pdb_addr: VramAddress,
mmu_version: MmuVersion,
va_size: u64,
) -> Result<Self> {
let page_size: u64 = PAGE_SIZE.into_safe_cast();
let va_pages: usize = (va_size / page_size).into_safe_cast();
let virt_alloc = KBox::pin_init(MapleTreeAlloc::<()>::new(), GFP_KERNEL)?;
Ok(Self {
pdb_addr,
pt_walk: PtWalk::new(pdb_addr, mmu_version),
pt_map: PtMap::new(pdb_addr, mmu_version),
page_table_allocs: KVec::new(),
virt_alloc,
va_pages,
pt_pages: RBTree::new(),
})
}
/// Allocate a contiguous virtual frame number range.
fn alloc_vfn_range(&self, num_pages: usize, va_range: Option<Range<u64>>) -> Result<Vfn> {
let page_size: u64 = PAGE_SIZE.into_safe_cast();
let start_vfn = match va_range {
Some(r) => {
let num_pages_u64: u64 = num_pages.into_safe_cast();
let size = num_pages_u64.checked_mul(page_size).ok_or(EOVERFLOW)?;
let range_size = r.end.checked_sub(r.start).ok_or(EOVERFLOW)?;
if range_size != size {
return Err(EINVAL);
}
let start_vfn: usize = (r.start / page_size).into_safe_cast();
let end_vfn: usize = (r.end / page_size).into_safe_cast();
self.virt_alloc
.insert_range(start_vfn..end_vfn, (), GFP_KERNEL)?;
start_vfn
}
None => self
.virt_alloc
.alloc_range(num_pages, (), ..self.va_pages, GFP_KERNEL)?,
};
Ok(Vfn::new(start_vfn.into_safe_cast()))
}
/// Free a virtual frame number range back to the maple tree.
fn free_vfn(&self, vfn: Vfn) {
let vfn_index: usize = vfn.raw().into_safe_cast();
if self.virt_alloc.erase(vfn_index).is_none() {
kernel::pr_warn!("free_vfn: VFN {} not found in maple tree\n", vfn_index);
}
}
/// Read the [`Pfn`] for a mapped [`Vfn`] if one is mapped.
pub(super) fn read_mapping(&self, mm: &mut GpuMm<'_>, vfn: Vfn) -> Result<Option<Pfn>> {
match self.pt_walk.walk_to_pte(mm, vfn)? {
WalkResult::Mapped { pfn, .. } => Ok(Some(pfn)),
WalkResult::Unmapped { .. } | WalkResult::PageTableMissing => Ok(None),
}
}
/// Prepare resources for mapping `num_pages` pages.
///
/// Allocates a contiguous VA range, then walks the hierarchy per-VFN to prepare pages
/// for all missing PDEs. Returns a [`PreparedMapping`] with the VA allocation.
///
/// If `va_range` is not `None`, the VA range is constrained to the given range. Safe
/// to call outside the fence signalling critical path.
pub(crate) fn prepare_map(
&mut self,
mm: &mut GpuMm<'_>,
num_pages: usize,
va_range: Option<Range<u64>>,
) -> Result<PreparedMapping> {
if num_pages == 0 {
return Err(EINVAL);
}
// Allocate contiguous VA range.
let vfn_start = self.alloc_vfn_range(num_pages, va_range)?;
if let Err(e) = self.pt_map.prepare_map(
mm,
vfn_start,
num_pages,
&mut self.page_table_allocs,
&mut self.pt_pages,
) {
self.free_vfn(vfn_start);
return Err(e);
}
Ok(PreparedMapping {
vfn_start,
num_pages,
_drop_guard: MustExecuteGuard::new(),
})
}
/// Execute a prepared multi-page mapping.
///
/// Installs all prepared PDEs and writes PTEs into the page table, then flushes TLB.
pub(crate) fn execute_map(
&mut self,
mm: &mut GpuMm<'_>,
prepared: PreparedMapping,
pfns: &[Pfn],
writable: bool,
) -> Result<MappedRange> {
if pfns.len() != prepared.num_pages {
self.free_vfn(prepared.vfn_start);
return Err(EINVAL);
}
let PreparedMapping {
vfn_start,
num_pages,
_drop_guard,
} = prepared;
_drop_guard.disarm();
if let Err(e) = self.pt_map.install_mappings(
mm,
&mut self.pt_pages,
&mut self.page_table_allocs,
vfn_start,
pfns,
writable,
) {
self.free_vfn(vfn_start);
return Err(e);
}
Ok(MappedRange {
vfn_start,
num_pages,
_drop_guard: MustUnmapGuard::new(),
})
}
/// Map pages doing prepare and execute in the same call.
///
/// This is a convenience wrapper for callers outside the fence signalling critical
/// path (e.g., BAR mappings). For DRM usecases, [`Vmm::prepare_map()`] and
/// [`Vmm::execute_map()`] will be called separately.
pub(crate) fn map_pages(
&mut self,
mm: &mut GpuMm<'_>,
pfns: &[Pfn],
va_range: Option<Range<u64>>,
writable: bool,
) -> Result<MappedRange> {
if pfns.is_empty() {
return Err(EINVAL);
}
// Check if provided VA range is sufficient (if provided).
if let Some(ref range) = va_range {
let required: u64 = pfns
.len()
.checked_mul(PAGE_SIZE)
.ok_or(EOVERFLOW)?
.into_safe_cast();
let available = range.end.checked_sub(range.start).ok_or(EINVAL)?;
if available < required {
return Err(EINVAL);
}
}
let prepared = self.prepare_map(mm, pfns.len(), va_range)?;
self.execute_map(mm, prepared, pfns, writable)
}
/// Unmap all pages in a [`MappedRange`] with a single TLB flush.
pub(crate) fn unmap_pages(&mut self, mm: &mut GpuMm<'_>, range: MappedRange) -> Result {
let result = self
.pt_map
.invalidate_ptes(mm, range.vfn_start, range.num_pages);
// TODO: Internal page table pages (PDE, PTE pages) are still kept around.
// This is by design as repeated maps/unmaps will be fast. As a future TODO,
// we can add a reclaimer here to reclaim if VRAM is short. For now, the PT
// pages are dropped once the `Vmm` is dropped.
// Free the VA range regardless of PTE invalidation success, so that the VA
// range is recovered even on failure (PTEs may be stale, but that is better
// than leaking both PTEs and VA range).
self.free_vfn(range.vfn_start);
// Unmap complete, safe to drop `MappedRange`.
range._drop_guard.disarm();
result
}
}

5
drivers/gpu/nova-core/nova_core.rs
View File

@ -10,6 +10,7 @@
InPlaceModule, //
};
pub mod api;
mod driver;
mod falcon;
mod fb;
@ -17,11 +18,15 @@
mod fsp;
mod gpu;
mod gsp;
mod irq;
mod mctp;
mod mm;
#[macro_use]
mod num;
mod regs;
mod sbuffer;
#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
mod selftest;
mod vbios;
mod vgpu;

2
drivers/gpu/nova-core/num.rs
View File

@ -263,7 +263,7 @@ fn try_from(
) -> kernel::error::Result<Self> {
match value.get() {
$(
$value => Ok($enum_type::$variant),
value if value == $value => Ok($enum_type::$variant),
)*
_ => Err(kernel::error::code::EINVAL),
}

326
drivers/gpu/nova-core/regs.rs
View File

@ -4,110 +4,37 @@
use kernel::{
io::{
register,
register::WithBase,
Io, //
Io,
Mmio, //
},
prelude::*,
sizes::SizeConstants,
time, //
};
use pin_init::Zeroable;
use crate::{
driver::Bar0,
driver::NovaRegisters,
falcon::{
DmaTrfCmdSize,
FalconCoreRev,
FalconCoreRevSubversion,
FalconEngine,
FalconFbifMemType,
FalconFbifTarget,
FalconMem,
FalconModSelAlgo,
FalconSecurityModel,
PFalcon2Base,
PFalconBase,
PFalcon2Registers,
PFalconRegisters,
PeregrineCoreSelect, //
},
gpu::{
Architecture,
Chipset, //
},
mm::tlb::TlbAckMode, //
};
// PMC
register! {
/// Basic revision information about the GPU.
pub(crate) NV_PMC_BOOT_0(u32) @ 0x00000000 {
/// Lower bits of the architecture.
28:24 architecture_0;
/// Implementation version of the architecture.
23:20 implementation;
/// MSB of the architecture.
8:8 architecture_1;
/// Major revision of the chip.
7:4 major_revision;
/// Minor revision of the chip.
3:0 minor_revision;
}
/// Extended architecture information.
pub(crate) NV_PMC_BOOT_42(u32) @ 0x00000a00 {
/// Architecture value.
29:24 architecture ?=> Architecture;
/// Implementation version of the architecture.
23:20 implementation;
/// Major revision of the chip.
19:16 major_revision;
/// Minor revision of the chip.
15:12 minor_revision;
}
}
impl NV_PMC_BOOT_0 {
pub(crate) fn is_older_than_fermi(self) -> bool {
// From https://github.com/NVIDIA/open-gpu-doc/tree/master/manuals :
const NV_PMC_BOOT_0_ARCHITECTURE_GF100: u32 = 0xc;
// Older chips left arch1 zeroed out. That, combined with an arch0 value that is less than
// GF100, means "older than Fermi".
self.architecture_1() == 0 && self.architecture_0() < NV_PMC_BOOT_0_ARCHITECTURE_GF100
}
}
impl NV_PMC_BOOT_42 {
/// Combines `architecture` and `implementation` to obtain a code unique to the chipset.
pub(crate) fn chipset(self) -> Result<Chipset> {
self.architecture()
.map(|arch| {
((arch as u32) << Self::IMPLEMENTATION_RANGE.len())
| u32::from(self.implementation())
})
.and_then(Chipset::try_from)
}
/// Returns the raw architecture value from the register.
fn architecture_raw(self) -> u8 {
((self.into_raw() >> Self::ARCHITECTURE_RANGE.start())
& ((1 << Self::ARCHITECTURE_RANGE.len()) - 1)) as u8
}
}
impl kernel::fmt::Display for NV_PMC_BOOT_42 {
fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result {
write!(
f,
"boot42 = 0x{:08x} (architecture 0x{:x}, implementation 0x{:x})",
self.inner,
self.architecture_raw(),
self.implementation()
)
}
}
// PBUS
register! {
base: NovaRegisters;
pub(crate) NV_PBUS_SW_SCRATCH(u32)[64] @ 0x00001400 {}
}
@ -121,6 +48,8 @@ fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result {
// number.
register! {
base: NovaRegisters;
/// Boot Sequence Interface (BSI) register used to determine
/// if GSP reload/resume has completed during the boot process.
pub(crate) NV_PGC6_BSI_SECURE_SCRATCH_14(u32) @ 0x001180f8 {
@ -175,6 +104,8 @@ pub(crate) fn usable_fb_size(self) -> u64 {
pub(crate) const NV_FUSE_OPT_FPF_SIZE: usize = 16;
register! {
base: NovaRegisters;
pub(crate) NV_FUSE_OPT_FPF_NVDEC_UCODE1_VERSION(u32)[NV_FUSE_OPT_FPF_SIZE] @ 0x00824100 {
15:0 data => u16;
}
@ -191,30 +122,46 @@ pub(crate) fn usable_fb_size(self) -> u64 {
// PFALCON
register! {
pub(crate) NV_PFALCON_FALCON_IRQSCLR(u32) @ PFalconBase + 0x00000004 {
base: PFalconRegisters;
/// Clears the latch of every cause whose bit is written as `1`. Write-only.
///
/// The write ends the latch and not the source, so a cause driven from outside the falcon
/// stays set. "Retriggering a falcon" in `Documentation/gpu/nova/core/interrupts.rst` names
/// those causes.
pub(crate) NV_PFALCON_FALCON_IRQSCLR(u32) @ 0x00000004 {
6:6 swgen0 => bool;
4:4 halt => bool;
}
pub(crate) NV_PFALCON_FALCON_MAILBOX0(u32) @ PFalconBase + 0x00000040 {
/// Interrupt causes latched in the falcon, one bit per cause, whichever target each is routed
/// to.
///
/// The causes routed to the host are the ones also set in `NV_PRISCV_RISCV_IRQMASK` and
/// `NV_PRISCV_RISCV_IRQDEST`.
pub(crate) NV_PFALCON_FALCON_IRQSTAT(u32) @ 0x00000008 {
6:6 swgen0 => bool;
}
pub(crate) NV_PFALCON_FALCON_MAILBOX0(u32) @ 0x00000040 {
31:0 value => u32;
}
pub(crate) NV_PFALCON_FALCON_MAILBOX1(u32) @ PFalconBase + 0x00000044 {
pub(crate) NV_PFALCON_FALCON_MAILBOX1(u32) @ 0x00000044 {
31:0 value => u32;
}
/// Used to store version information about the firmware running
/// on the Falcon processor.
pub(crate) NV_PFALCON_FALCON_OS(u32) @ PFalconBase + 0x00000080 {
pub(crate) NV_PFALCON_FALCON_OS(u32) @ 0x00000080 {
31:0 value => u32;
}
pub(crate) NV_PFALCON_FALCON_RM(u32) @ PFalconBase + 0x00000084 {
pub(crate) NV_PFALCON_FALCON_RM(u32) @ 0x00000084 {
31:0 value => u32;
}
pub(crate) NV_PFALCON_FALCON_HWCFG2(u32) @ PFalconBase + 0x000000f4 {
pub(crate) NV_PFALCON_FALCON_HWCFG2(u32) @ 0x000000f4 {
/// Signal indicating that reset is completed (GA102+).
31:31 reset_ready => bool;
/// RISC-V branch privilege lockdown bit.
@ -224,17 +171,17 @@ pub(crate) fn usable_fb_size(self) -> u64 {
10:10 riscv => bool;
}
pub(crate) NV_PFALCON_FALCON_CPUCTL(u32) @ PFalconBase + 0x00000100 {
pub(crate) NV_PFALCON_FALCON_CPUCTL(u32) @ 0x00000100 {
6:6 alias_en => bool;
4:4 halted => bool;
1:1 startcpu => bool;
}
pub(crate) NV_PFALCON_FALCON_BOOTVEC(u32) @ PFalconBase + 0x00000104 {
pub(crate) NV_PFALCON_FALCON_BOOTVEC(u32) @ 0x00000104 {
31:0 value => u32;
}
pub(crate) NV_PFALCON_FALCON_DMACTL(u32) @ PFalconBase + 0x0000010c {
pub(crate) NV_PFALCON_FALCON_DMACTL(u32) @ 0x0000010c {
7:7 secure_stat => bool;
6:3 dmaq_num;
2:2 imem_scrubbing => bool;
@ -242,15 +189,15 @@ pub(crate) fn usable_fb_size(self) -> u64 {
0:0 require_ctx => bool;
}
pub(crate) NV_PFALCON_FALCON_DMATRFBASE(u32) @ PFalconBase + 0x00000110 {
pub(crate) NV_PFALCON_FALCON_DMATRFBASE(u32) @ 0x00000110 {
31:0 base => u32;
}
pub(crate) NV_PFALCON_FALCON_DMATRFMOFFS(u32) @ PFalconBase + 0x00000114 {
pub(crate) NV_PFALCON_FALCON_DMATRFMOFFS(u32) @ 0x00000114 {
23:0 offs;
}
pub(crate) NV_PFALCON_FALCON_DMATRFCMD(u32) @ PFalconBase + 0x00000118 {
pub(crate) NV_PFALCON_FALCON_DMATRFCMD(u32) @ 0x00000118 {
16:16 set_dmtag;
14:12 ctxdma;
10:8 size ?=> DmaTrfCmdSize;
@ -261,15 +208,15 @@ pub(crate) fn usable_fb_size(self) -> u64 {
0:0 full => bool;
}
pub(crate) NV_PFALCON_FALCON_DMATRFFBOFFS(u32) @ PFalconBase + 0x0000011c {
pub(crate) NV_PFALCON_FALCON_DMATRFFBOFFS(u32) @ 0x0000011c {
31:0 offs => u32;
}
pub(crate) NV_PFALCON_FALCON_DMATRFBASE1(u32) @ PFalconBase + 0x00000128 {
pub(crate) NV_PFALCON_FALCON_DMATRFBASE1(u32) @ 0x00000128 {
8:0 base;
}
pub(crate) NV_PFALCON_FALCON_HWCFG1(u32) @ PFalconBase + 0x0000012c {
pub(crate) NV_PFALCON_FALCON_HWCFG1(u32) @ 0x0000012c {
/// Core revision subversion.
7:6 core_rev_subversion => FalconCoreRevSubversion;
/// Security model.
@ -278,12 +225,12 @@ pub(crate) fn usable_fb_size(self) -> u64 {
3:0 core_rev ?=> FalconCoreRev;
}
pub(crate) NV_PFALCON_FALCON_CPUCTL_ALIAS(u32) @ PFalconBase + 0x00000130 {
pub(crate) NV_PFALCON_FALCON_CPUCTL_ALIAS(u32) @ 0x00000130 {
1:1 startcpu => bool;
}
/// IMEM access control register. Up to 4 ports are available for IMEM access.
pub(crate) NV_PFALCON_FALCON_IMEMC(u32)[4, stride = 16] @ PFalconBase + 0x00000180 {
pub(crate) NV_PFALCON_FALCON_IMEMC(u32)[4, stride = 16] @ 0x00000180 {
/// Access secure IMEM.
28:28 secure => bool;
/// Auto-increment on write.
@ -294,17 +241,17 @@ pub(crate) fn usable_fb_size(self) -> u64 {
/// IMEM data register. Reading/writing this register accesses IMEM at the address
/// specified by the corresponding IMEMC register.
pub(crate) NV_PFALCON_FALCON_IMEMD(u32)[4, stride = 16] @ PFalconBase + 0x00000184 {
pub(crate) NV_PFALCON_FALCON_IMEMD(u32)[4, stride = 16] @ 0x00000184 {
31:0 data;
}
/// IMEM tag register. Used to set the tag for the current IMEM block.
pub(crate) NV_PFALCON_FALCON_IMEMT(u32)[4, stride = 16] @ PFalconBase + 0x00000188 {
pub(crate) NV_PFALCON_FALCON_IMEMT(u32)[4, stride = 16] @ 0x00000188 {
15:0 tag;
}
/// DMEM access control register. Up to 8 ports are available for DMEM access.
pub(crate) NV_PFALCON_FALCON_DMEMC(u32)[8, stride = 8] @ PFalconBase + 0x000001c0 {
pub(crate) NV_PFALCON_FALCON_DMEMC(u32)[8, stride = 8] @ 0x000001c0 {
/// Auto-increment on write.
24:24 aincw => bool;
/// DMEM block and word offset.
@ -313,29 +260,39 @@ pub(crate) fn usable_fb_size(self) -> u64 {
/// DMEM data register. Reading/writing this register accesses DMEM at the address
/// specified by the corresponding DMEMC register.
pub(crate) NV_PFALCON_FALCON_DMEMD(u32)[8, stride = 8] @ PFalconBase + 0x000001c4 {
pub(crate) NV_PFALCON_FALCON_DMEMD(u32)[8, stride = 8] @ 0x000001c4 {
31:0 data;
}
/// Actually known as `NV_PSEC_FALCON_ENGINE` and `NV_PGSP_FALCON_ENGINE` depending on the
/// falcon instance.
pub(crate) NV_PFALCON_FALCON_ENGINE(u32) @ PFalconBase + 0x000003c0 {
pub(crate) NV_PFALCON_FALCON_ENGINE(u32) @ 0x000003c0 {
0:0 reset => bool;
}
pub(crate) NV_PFALCON_FBIF_TRANSCFG(u32)[8] @ PFalconBase + 0x00000600 {
/// Makes the falcon re-emit its host-routed causes into the interrupt tree. Write-only.
///
/// Present from GA100 on. See "Retriggering a falcon" in
/// `Documentation/gpu/nova/core/interrupts.rst`.
///
/// The hardware headers declare two elements, and OpenRM writes only the first.
pub(crate) NV_PFALCON_FALCON_INTR_RETRIGGER(u32)[2] @ 0x000003e8 {
0:0 trigger => bool;
}
pub(crate) NV_PFALCON_FBIF_TRANSCFG(u32)[8] @ 0x00000600 {
2:2 mem_type => FalconFbifMemType;
1:0 target ?=> FalconFbifTarget;
}
pub(crate) NV_PFALCON_FBIF_CTL(u32) @ PFalconBase + 0x00000624 {
pub(crate) NV_PFALCON_FBIF_CTL(u32) @ 0x00000624 {
7:7 allow_phys_no_ctx => bool;
}
// Falcon EMEM PIO registers (used by FSP on Hopper/Blackwell).
// These provide the falcon external memory communication interface.
pub(crate) NV_PFALCON_FALCON_EMEMC(u32) @ PFalconBase + 0x00000ac0 {
pub(crate) NV_PFALCON_FALCON_EMEMC(u32) @ 0x00000ac0 {
/// EMEM byte offset (4-byte aligned) within the block.
7:2 offs;
/// EMEM block to access.
@ -346,7 +303,7 @@ pub(crate) fn usable_fb_size(self) -> u64 {
25:25 aincr => bool;
}
pub(crate) NV_PFALCON_FALCON_EMEMD(u32) @ PFalconBase + 0x00000ac4 {
pub(crate) NV_PFALCON_FALCON_EMEMD(u32) @ 0x00000ac4 {
31:0 data => u32;
}
}
@ -372,13 +329,13 @@ pub(crate) fn with_falcon_mem(self, mem: FalconMem) -> Self {
impl NV_PFALCON_FALCON_ENGINE {
/// Resets the falcon
pub(crate) fn reset_engine<E: FalconEngine>(bar: Bar0<'_>) {
bar.update(Self::of::<E>(), |r| r.with_reset(true));
pub(crate) fn reset_engine(pfalcon: Mmio<'_, PFalconRegisters>) {
pfalcon.update(NV_PFALCON_FALCON_ENGINE, |r| r.with_reset(true));
// TIMEOUT: falcon engine should not take more than 10us to reset.
time::delay::fsleep(time::Delta::from_micros(10));
bar.update(Self::of::<E>(), |r| r.with_reset(false));
pfalcon.update(NV_PFALCON_FALCON_ENGINE, |r| r.with_reset(false));
}
}
@ -392,21 +349,23 @@ pub(crate) fn mem_scrubbing_done(self) -> bool {
/* PFALCON2 */
register! {
pub(crate) NV_PFALCON2_FALCON_MOD_SEL(u32) @ PFalcon2Base + 0x00000180 {
base: PFalcon2Registers;
pub(crate) NV_PFALCON2_FALCON_MOD_SEL(u32) @ 0x00000180 {
7:0 algo ?=> FalconModSelAlgo;
}
pub(crate) NV_PFALCON2_FALCON_BROM_CURR_UCODE_ID(u32) @ PFalcon2Base + 0x00000198 {
pub(crate) NV_PFALCON2_FALCON_BROM_CURR_UCODE_ID(u32) @ 0x00000198 {
7:0 ucode_id => u8;
}
pub(crate) NV_PFALCON2_FALCON_BROM_ENGIDMASK(u32) @ PFalcon2Base + 0x0000019c {
pub(crate) NV_PFALCON2_FALCON_BROM_ENGIDMASK(u32) @ 0x0000019c {
31:0 value => u32;
}
/// OpenRM defines this as a register array, but doesn't specify its size and only uses its
/// first element. Be conservative until we know the actual size or need to use more registers.
pub(crate) NV_PFALCON2_FALCON_BROM_PARAADDR(u32)[1] @ PFalcon2Base + 0x00000210 {
pub(crate) NV_PFALCON2_FALCON_BROM_PARAADDR(u32)[1] @ 0x00000210 {
31:0 value => u32;
}
}
@ -414,21 +373,23 @@ pub(crate) fn mem_scrubbing_done(self) -> bool {
// PRISCV
register! {
base: PFalcon2Registers;
/// RISC-V status register for debug (Turing and GA100 only).
/// Reflects current RISC-V core status.
pub(crate) NV_PRISCV_RISCV_CORE_SWITCH_RISCV_STATUS(u32) @ PFalcon2Base + 0x00000240 {
pub(crate) NV_PRISCV_RISCV_CORE_SWITCH_RISCV_STATUS(u32) @ 0x00000240 {
/// RISC-V core active/inactive status.
0:0 active_stat => bool;
}
/// GA102 and later.
pub(crate) NV_PRISCV_RISCV_CPUCTL(u32) @ PFalcon2Base + 0x00000388 {
pub(crate) NV_PRISCV_RISCV_CPUCTL(u32) @ 0x00000388 {
7:7 active_stat => bool;
4:4 halted => bool;
}
/// GA102 and later.
pub(crate) NV_PRISCV_RISCV_BCR_CTRL(u32) @ PFalcon2Base + 0x00000668 {
pub(crate) NV_PRISCV_RISCV_BCR_CTRL(u32) @ 0x00000668 {
8:8 br_fetch => bool;
4:4 core_select => PeregrineCoreSelect;
0:0 valid => bool;
@ -439,6 +400,8 @@ pub(crate) fn mem_scrubbing_done(self) -> bool {
// These registers manage falcon EMEM communication queues.
register! {
base: NovaRegisters;
pub(crate) NV_PFSP_QUEUE_HEAD(u32)[8] @ 0x008f2c00 {
31:0 address => u32;
}
@ -462,35 +425,92 @@ pub(crate) fn mem_scrubbing_done(self) -> bool {
pub(crate) mod gm107 {
use kernel::io::register;
use crate::driver::NovaRegisters;
// FUSE
register! {
base: NovaRegisters;
pub(crate) NV_FUSE_STATUS_OPT_DISPLAY(u32) @ 0x00021c04 {
0:0 display_disabled => bool;
}
}
}
pub(crate) mod tu102 {
use kernel::io::register;
use crate::falcon::PFalcon2Registers;
// The RISC-V interrupt routing registers, at the offsets that Turing and GA100 use.
register! {
base: PFalcon2Registers;
/// Enabled causes, one bit per cause. Read-only to the host.
pub(crate) NV_PRISCV_RISCV_IRQMASK(u32) @ 0x000002b4 {
31:0 value => u32;
}
/// Causes routed to the host, one bit per cause. A clear bit routes the cause to the
/// RISC-V core.
pub(crate) NV_PRISCV_RISCV_IRQDEST(u32) @ 0x000002b8 {
31:0 value => u32;
}
}
}
pub(crate) mod ga100 {
use kernel::io::register;
use crate::driver::NovaRegisters;
// FUSE
register! {
base: NovaRegisters;
pub(crate) NV_FUSE_STATUS_OPT_DISPLAY(u32) @ 0x00820c04 {
0:0 display_disabled => bool;
}
}
}
pub(crate) mod ga102 {
use kernel::io::register;
use crate::falcon::PFalcon2Registers;
// The RISC-V interrupt routing registers, at the offsets that GA102 and later use.
register! {
base: PFalcon2Registers;
/// Same as [`super::tu102::NV_PRISCV_RISCV_IRQMASK`], at the GA102 offset.
pub(crate) NV_PRISCV_RISCV_IRQMASK(u32) @ 0x00000528 {
31:0 value => u32;
}
/// Same as [`super::tu102::NV_PRISCV_RISCV_IRQDEST`], at the GA102 offset.
pub(crate) NV_PRISCV_RISCV_IRQDEST(u32) @ 0x0000052c {
31:0 value => u32;
}
}
}
pub(crate) const NV_THERM_I2CS_SCRATCH_FSP_BOOT_COMPLETE_STATUS_SUCCESS: u32 = 0xff;
pub(crate) mod gh100 {
use kernel::io::register;
use crate::driver::NovaRegisters;
// PTHERM
register! {
base: NovaRegisters;
pub(crate) NV_THERM_I2CS_SCRATCH(u32) @ 0x000200bc {
31:0 data;
}
@ -505,9 +525,13 @@ pub(crate) mod gh100 {
pub(crate) mod gb202 {
use kernel::io::register;
use crate::driver::NovaRegisters;
// PTHERM
register! {
base: NovaRegisters;
pub(crate) NV_THERM_I2CS_SCRATCH(u32) @ 0x00ad00bc {
31:0 data;
}
@ -518,3 +542,69 @@ pub(crate) mod gb202 {
}
}
}
// MMU TLB
register! {
base: NovaRegisters;
/// TLB flush register: PDB address lower bits.
pub(crate) NV_TLB_FLUSH_PDB_LO(u32) @ 0x00b830a0 {
/// PDB address bits [39:8].
31:0 pdb_lo => u32;
}
/// TLB flush register: PDB address higher bits.
pub(crate) NV_TLB_FLUSH_PDB_HI(u32) @ 0x00b830a4 {
/// PDB address bits [47:40].
7:0 pdb_hi => u8;
}
/// TLB flush control register.
pub(crate) NV_TLB_FLUSH_CTRL(u32) @ 0x00b830b0 {
/// Invalidate every VA in the PDB selected by `NV_TLB_FLUSH_PDB_LO/HI`.
0:0 all_va => bool;
/// Invalidate TLBs for all PDBs (ignores `NV_TLB_FLUSH_PDB_LO/HI`).
1:1 all_pdb => bool;
/// Restrict the flush to the HUB MMU's TLBs; skip broadcasting to the
/// per-GPC L2 TLBs.
///
/// The GPU MMU has a two-level TLB hierarchy:
/// 1. The *HUB MMU* sits at the top and serves memory requests from
/// "host-side" engines: the host/channel interface, copy engines,
/// display, and BAR1/BAR2 accesses.
/// 2. Each GPC (Graphics Processing Cluster — the block that houses
/// shader cores / SMs) has its own L2 TLB that serves requests from
/// the compute and graphics engines inside the cluster.
///
/// When set, only the HUB TLBs are invalidated. This is a performance
/// optimization for flushes that only affect HUB-side mappings (e.g.
/// BAR1/BAR2 windows), where fanning the invalidation out to every
/// GPC's L2 TLB would be wasted work. Must be false when flushing
/// mappings that may be cached by compute/graphics engines.
2:2 hubtlb_only => bool;
/// Invalidation acknowledgment scope. See [`TlbAckMode`] for details.
8:7 ack ?=> TlbAckMode;
/// Write 1 to kick off the flush. Hardware clears this bit when the
/// flush completes; reads as 1 while the flush is in progress.
31:31 trigger => bool;
}
}
impl NV_TLB_FLUSH_PDB_LO {
/// Create a register value from a PDB address.
///
/// Extracts bits [39:8] of the address and shifts it right by 8 bits.
pub(crate) fn from_pdb_addr(addr: u64) -> Self {
Self::zeroed().with_pdb_lo(((addr >> 8) & 0xFFFF_FFFF) as u32)
}
}
impl NV_TLB_FLUSH_PDB_HI {
/// Create a register value from a PDB address.
///
/// Extracts bits [47:40] of the address and shifts it right by 40 bits.
pub(crate) fn from_pdb_addr(addr: u64) -> Self {
Self::zeroed().with_pdb_hi(((addr >> 40) & 0xFF) as u8)
}
}

64
drivers/gpu/nova-core/selftest.rs Normal file
View File

@ -0,0 +1,64 @@
// SPDX-License-Identifier: GPL-2.0
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
//! Assertion macros for driver self-tests.
//!
//! Self-tests run against live hardware during probe, so a failed assertion should not panic. These
//! macros log the failure on the device and fail the enclosing test by returning
//! [`EIO`](kernel::error::code::EIO) instead.
/// Like [`assert!`], but logs the failure via `dev` and fails the enclosing test instead of
/// panicking.
///
/// As with [`assert!`], a custom message with format arguments can follow the condition.
#[macro_export]
macro_rules! selftest_assert {
($dev:expr, $cond:expr $(,)?) => {
$crate::selftest_assert!($dev, $cond, "assertion failed: {}", ::core::stringify!($cond))
};
($dev:expr, $cond:expr, $($arg:tt)+) => {{
if !$cond {
::kernel::dev_err!(
$dev,
"Selftest: {}:{}: {}\n",
::core::file!(),
::core::line!(),
::kernel::prelude::fmt!($($arg)+)
);
return Err(::kernel::error::code::EIO);
}
}};
}
/// Like [`assert_eq!`], but logs the failure via `dev` and fails the enclosing test instead of
/// panicking.
///
/// As with [`assert_eq!`], a custom message with format arguments can follow the compared values.
#[macro_export]
macro_rules! selftest_assert_eq {
($dev:expr, $left:expr, $right:expr $(,)?) => {
match (&$left, &$right) {
(left, right) => $crate::selftest_assert!(
$dev,
left == right,
"assertion `{} == {}` failed: left {:?}, right {:?}",
::core::stringify!($left),
::core::stringify!($right),
left,
right
),
}
};
($dev:expr, $left:expr, $right:expr, $($arg:tt)+) => {
match (&$left, &$right) {
(left, right) => $crate::selftest_assert!(
$dev,
left == right,
"assertion `left == right` failed: {}: left {:?}, right {:?}",
::kernel::prelude::fmt!($($arg)+),
left,
right
),
}
};
}

11
drivers/gpu/nova-core/vbios.rs
View File

@ -16,7 +16,10 @@
};
use crate::{
driver::Bar0,
driver::{
Bar0,
NovaRegisters, //
},
firmware::{
fwsec::Bcrt30Rsa3kSignature,
FalconUCodeDesc,
@ -92,12 +95,16 @@ impl<'a> VbiosIterator<'a> {
fn rom_offset(dev: &device::Device, bar0: Bar0<'_>) -> Result<usize> {
// IFR Header in VBIOS.
register! {
base: NovaRegisters;
NV_PBUS_IFR_FMT_FIXED0(u32) @ 0x300000 {
31:0 signature;
}
}
register! {
base: NovaRegisters;
NV_PBUS_IFR_FMT_FIXED1(u32) @ 0x300004 {
30:16 fixed_data_size;
15:8 version => u8;
@ -105,6 +112,8 @@ fn rom_offset(dev: &device::Device, bar0: Bar0<'_>) -> Result<usize> {
}
register! {
base: NovaRegisters;
NV_PBUS_IFR_FMT_FIXED2(u32) @ 0x300008 {
19:0 total_data_size;
}

146
include/uapi/drm/nova_drm.h
View File

@ -18,26 +18,56 @@
extern "C" {
#endif
/*
* NOVA_GETPARAM_VRAM_BAR_SIZE
*
* Query the VRAM BAR size in bytes.
/**
* enum drm_nova_architecture - GPU architecture identifier
*/
#define NOVA_GETPARAM_VRAM_BAR_SIZE 0x1
enum drm_nova_architecture {
NOVA_DRM_ARCHITECTURE_TURING = 0x16,
NOVA_DRM_ARCHITECTURE_AMPERE = 0x17,
NOVA_DRM_ARCHITECTURE_HOPPER = 0x18,
NOVA_DRM_ARCHITECTURE_ADA = 0x19,
NOVA_DRM_ARCHITECTURE_BLACKWELL_GB10X = 0x1a,
NOVA_DRM_ARCHITECTURE_BLACKWELL_GB20X = 0x1b,
};
/**
* struct drm_nova_getparam - query GPU and driver metadata
* enum drm_nova_chipid - opaque GPU chip identifier
*
* These values identify the chip a GPU is based on. They are otherwise
* opaque: userspace must not assume the values carry any particular meaning
* or encoding, only that they may be compared against this enum.
*/
struct drm_nova_getparam {
/**
* @param: The identifier of the parameter to query.
*/
__u64 param;
/**
* @value: The value for the specified parameter.
*/
__u64 value;
enum drm_nova_chipid {
/* Turing */
NOVA_DRM_CHIPID_TU102 = 0x162,
NOVA_DRM_CHIPID_TU104 = 0x164,
NOVA_DRM_CHIPID_TU106 = 0x166,
NOVA_DRM_CHIPID_TU117 = 0x167,
NOVA_DRM_CHIPID_TU116 = 0x168,
/* Ampere */
NOVA_DRM_CHIPID_GA100 = 0x170,
NOVA_DRM_CHIPID_GA102 = 0x172,
NOVA_DRM_CHIPID_GA103 = 0x173,
NOVA_DRM_CHIPID_GA104 = 0x174,
NOVA_DRM_CHIPID_GA106 = 0x176,
NOVA_DRM_CHIPID_GA107 = 0x177,
/* Hopper */
NOVA_DRM_CHIPID_GH100 = 0x180,
/* Ada */
NOVA_DRM_CHIPID_AD102 = 0x192,
NOVA_DRM_CHIPID_AD103 = 0x193,
NOVA_DRM_CHIPID_AD104 = 0x194,
NOVA_DRM_CHIPID_AD106 = 0x196,
NOVA_DRM_CHIPID_AD107 = 0x197,
/* Blackwell GB10x */
NOVA_DRM_CHIPID_GB100 = 0x1a0,
NOVA_DRM_CHIPID_GB102 = 0x1a2,
/* Blackwell GB20x */
NOVA_DRM_CHIPID_GB202 = 0x1b2,
NOVA_DRM_CHIPID_GB203 = 0x1b3,
NOVA_DRM_CHIPID_GB205 = 0x1b5,
NOVA_DRM_CHIPID_GB206 = 0x1b6,
NOVA_DRM_CHIPID_GB207 = 0x1b7,
};
/**
@ -80,14 +110,92 @@ struct drm_nova_gem_info {
__u64 size;
};
#define DRM_NOVA_GETPARAM 0x00
/**
* struct drm_nova_info - query device information
*/
struct drm_nova_info {
/**
* @id: The identifier of the information to query.
*/
__u32 id;
/**
* @size: The amount of space allocated by userspace at @data. The kernel
* will return the number of bytes it wrote, or the size of the queried
* information structure if @data is NULL.
*/
__u32 size;
/**
* @data: Pointer to the userspace buffer into which the queried
* information will be written. May be NULL, in which case nothing is
* written and @size is set to the size of the information structure so
* userspace can allocate a suitably sized buffer.
*/
__u64 data;
};
/**
* DRM_NOVA_INFO_GPU
*
* Query GPU information. The result is returned in a
* &struct drm_nova_info_gpu.
*/
#define DRM_NOVA_INFO_GPU 0x00
/**
* struct drm_nova_info_gpu - GPU information
*/
struct drm_nova_info_gpu {
/**
* @architecture: GPU architecture identifier.
*
* See &enum drm_nova_architecture for currently known architectures.
*/
__u32 architecture;
/**
* @chipid: Opaque GPU chip identifier.
*
* See &enum drm_nova_chipid for currently known chips.
*/
__u32 chipid;
/**
* @vram_size: Amount of usable FB, excluding GSP carveouts and protected
* regions.
*/
__u64 vram_size;
/**
* @vram_bar_size: Size of the PCI BAR aperture used to access VRAM, in bytes.
*/
__u64 vram_bar_size;
/**
* @gpu_name: NUL-terminated full GPU name.
*/
__u8 gpu_name[64];
/**
* @gpu_short_name: NUL-terminated short GPU name.
*/
__u8 gpu_short_name[64];
/**
* @gpu_gid: 16-byte SHA-1 GPU identifier supplied by GSP-RM.
*/
__u8 gpu_gid[16];
};
#define DRM_NOVA_INFO 0x00
#define DRM_NOVA_GEM_CREATE 0x01
#define DRM_NOVA_GEM_INFO 0x02
/* Note: this is an enum so that it can be resolved by Rust bindgen. */
enum {
DRM_IOCTL_NOVA_GETPARAM = DRM_IOWR(DRM_COMMAND_BASE + DRM_NOVA_GETPARAM,
struct drm_nova_getparam),
DRM_IOCTL_NOVA_INFO = DRM_IOWR(DRM_COMMAND_BASE + DRM_NOVA_INFO,
struct drm_nova_info),
DRM_IOCTL_NOVA_GEM_CREATE = DRM_IOWR(DRM_COMMAND_BASE + DRM_NOVA_GEM_CREATE,
struct drm_nova_gem_create),
DRM_IOCTL_NOVA_GEM_INFO = DRM_IOWR(DRM_COMMAND_BASE + DRM_NOVA_GEM_INFO,

1
rust/bindings/bindings_helper.h
View File

@ -53,6 +53,7 @@
#include <linux/debugfs.h>
#include <linux/device/faux.h>
#include <linux/dma-direction.h>
#include <linux/dma-fence.h>
#include <linux/dma-mapping.h>
#include <linux/dma-resv.h>
#include <linux/errname.h>

49
rust/helpers/dma_fence.c Normal file
View File

@ -0,0 +1,49 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/dma-fence.h>
__rust_helper void rust_helper_dma_fence_get(struct dma_fence *f)
{
dma_fence_get(f);
}
__rust_helper void rust_helper_dma_fence_put(struct dma_fence *f)
{
dma_fence_put(f);
}
__rust_helper bool rust_helper_dma_fence_begin_signalling(void)
{
return dma_fence_begin_signalling();
}
__rust_helper void rust_helper_dma_fence_end_signalling(bool cookie)
{
dma_fence_end_signalling(cookie);
}
__rust_helper bool rust_helper_dma_fence_is_signaled(struct dma_fence *f)
{
return dma_fence_is_signaled(f);
}
__rust_helper bool rust_helper_dma_fence_test_signaled_flag(struct dma_fence *f)
{
return dma_fence_test_signaled_flag(f);
}
__rust_helper void rust_helper_dma_fence_lock_irqsave(struct dma_fence *f, unsigned long *flags)
{
dma_fence_lock_irqsave(f, *flags);
}
__rust_helper void rust_helper_dma_fence_unlock_irqrestore(struct dma_fence *f,
unsigned long *flags)
{
dma_fence_unlock_irqrestore(f, *flags);
}
__rust_helper void rust_helper_dma_fence_set_error(struct dma_fence *f, int error)
{
dma_fence_set_error(f, error);
}

1
rust/helpers/helpers.c
View File

@ -58,6 +58,7 @@
#include "device.c"
#include "dma.c"
#ifdef CONFIG_DMA_SHARED_BUFFER
#include "dma_fence.c"
#include "dma-resv.c"
#endif
#include "drm.c"

6
rust/helpers/pci.c
View File

@ -19,6 +19,12 @@ __rust_helper resource_size_t rust_helper_pci_resource_len(struct pci_dev *pdev,
return pci_resource_len(pdev, bar);
}
__rust_helper unsigned long rust_helper_pci_resource_flags(const struct pci_dev *pdev,
int bar)
{
return pci_resource_flags(pdev, bar);
}
__rust_helper bool rust_helper_dev_is_pci(const struct device *dev)
{
return dev_is_pci(dev);

17
rust/kernel/auxiliary.rs
View File

@ -305,6 +305,10 @@ unsafe fn registration_data_pinned<F: ForLt + 'static>(&self) -> Result<Pin<&F::
/// `F` is the [`ForLt`](trait@ForLt) encoding of the data type. The closure receives a pinned
/// reference to the registration data.
///
/// The outer reference carries the `&self` lifetime while the inner type carries the HRTB
/// lifetime `'a`, implying `'a` outlives `&self`. This allows the closure to coerce covariant
/// sub-fields (e.g. `&'a T` to the caller's lifetime) and return them directly in `R`.
///
/// For covariant types that implement [`trait@CovariantForLt`], prefer
/// [`registration_data`](Self::registration_data) which returns a direct reference.
///
@ -314,13 +318,14 @@ unsafe fn registration_data_pinned<F: ForLt + 'static>(&self) -> Result<Pin<&F::
/// Returns [`ENOENT`] if no registration data has been set, e.g. when the device was
/// registered by a C driver.
#[inline]
pub fn registration_data_with<F: ForLt + 'static, R>(
&self,
f: impl for<'a> FnOnce(Pin<&'a F::Of<'a>>) -> R,
pub fn registration_data_with<'this, F: ForLt + 'static, R>(
&'this self,
f: impl for<'a> FnOnce(Pin<&'this F::Of<'a>>) -> R,
) -> Result<R> {
// SAFETY: The HRTB closure prevents the caller from smuggling in references with a
// concrete short lifetime, making the round-trip from `'static` sound regardless of
// variance.
// SAFETY: The HRTB on the inner type prevents the caller from exploiting a specific
// choice of `'a`. Covariant sub-fields can be safely coerced to `'this`, while
// invariant fields cannot be coerced and thus cannot escape with an incorrect
// lifetime.
let pinned = unsafe { self.registration_data_pinned::<F>()? };
Ok(f(pinned))

9
rust/kernel/bitfield.rs
View File

@ -346,6 +346,15 @@ fn from(val: $storage) -> $name {
Self::from_raw(val)
}
}
// SAFETY: `$name` is transparent over `$storage` and `$storage` has no interior mutability.
unsafe impl $crate::mem::AsRepr for $name {
// Normalize `$storage` to the canonical repr type in case it is signed.
type Repr = <$storage as $crate::mem::AsRepr>::Repr;
}
// SAFETY: `$name` is transparent over `$storage`.
unsafe impl $crate::mem::AsReprMut for $name {}
};
// Definitions requiring knowledge of individual fields: private and public field accessors,

26
rust/kernel/debugfs.rs
View File

@ -538,7 +538,7 @@ pub fn dir<'dir2>(&'dir2 self, name: &CStr) -> ScopedDir<'data, 'dir2> {
}
}
fn create_file<T: Sync>(&self, name: &CStr, data: &'data T, vtable: &'static FileOps<T>) {
fn create_file<T: Sync>(&self, name: &CStr, data: &'data T, vtable: &FileOps<T>) {
#[cfg(CONFIG_DEBUG_FS)]
core::mem::forget(Entry::file(name, &self.entry, data, vtable));
}
@ -550,7 +550,7 @@ fn create_file<T: Sync>(&self, name: &CStr, data: &'data T, vtable: &'static Fil
/// This function does not produce an owning handle to the file. The created
/// file is removed when the [`Scope`] that this directory belongs
/// to is dropped.
pub fn read_only_file<T: Writer + Send + Sync + 'static>(&self, name: &CStr, data: &'data T) {
pub fn read_only_file<T: Writer + Send + Sync>(&self, name: &CStr, data: &'data T) {
self.create_file(name, data, &T::FILE_OPS)
}
@ -560,11 +560,7 @@ pub fn read_only_file<T: Writer + Send + Sync + 'static>(&self, name: &CStr, dat
///
/// This function does not produce an owning handle to the file. The created file is removed
/// when the [`Scope`] that this directory belongs to is dropped.
pub fn read_binary_file<T: BinaryWriter + Send + Sync + 'static>(
&self,
name: &CStr,
data: &'data T,
) {
pub fn read_binary_file<T: BinaryWriter + Send + Sync>(&self, name: &CStr, data: &'data T) {
self.create_file(name, data, &T::FILE_OPS)
}
@ -596,11 +592,7 @@ pub fn read_callback_file<T, F>(&self, name: &CStr, data: &'data T, _f: &'static
/// This function does not produce an owning handle to the file. The created
/// file is removed when the [`Scope`] that this directory belongs
/// to is dropped.
pub fn read_write_file<T: Writer + Reader + Send + Sync + 'static>(
&self,
name: &CStr,
data: &'data T,
) {
pub fn read_write_file<T: Writer + Reader + Send + Sync>(&self, name: &CStr, data: &'data T) {
let vtable = &<T as ReadWriteFile<_>>::FILE_OPS;
self.create_file(name, data, vtable)
}
@ -612,7 +604,7 @@ pub fn read_write_file<T: Writer + Reader + Send + Sync + 'static>(
///
/// This function does not produce an owning handle to the file. The created file is removed
/// when the [`Scope`] that this directory belongs to is dropped.
pub fn read_write_binary_file<T: BinaryWriter + BinaryReader + Send + Sync + 'static>(
pub fn read_write_binary_file<T: BinaryWriter + BinaryReader + Send + Sync>(
&self,
name: &CStr,
data: &'data T,
@ -655,7 +647,7 @@ pub fn read_write_callback_file<T, F, W>(
/// This function does not produce an owning handle to the file. The created
/// file is removed when the [`Scope`] that this directory belongs
/// to is dropped.
pub fn write_only_file<T: Reader + Send + Sync + 'static>(&self, name: &CStr, data: &'data T) {
pub fn write_only_file<T: Reader + Send + Sync>(&self, name: &CStr, data: &'data T) {
let vtable = &<T as WriteFile<_>>::FILE_OPS;
self.create_file(name, data, vtable)
}
@ -666,11 +658,7 @@ pub fn write_only_file<T: Reader + Send + Sync + 'static>(&self, name: &CStr, da
///
/// This function does not produce an owning handle to the file. The created file is removed
/// when the [`Scope`] that this directory belongs to is dropped.
pub fn write_binary_file<T: BinaryReader + Send + Sync + 'static>(
&self,
name: &CStr,
data: &'data T,
) {
pub fn write_binary_file<T: BinaryReader + Send + Sync>(&self, name: &CStr, data: &'data T) {
self.create_file(name, data, &T::FILE_OPS)
}

4
rust/kernel/debugfs/entry.rs
View File

@ -74,7 +74,7 @@ pub(crate) unsafe fn dynamic_file<T>(
parent.as_ptr(),
core::ptr::from_ref(data) as *mut c_void,
core::ptr::null(),
&**file_ops,
file_ops.fops(),
)
};
@ -127,7 +127,7 @@ pub(crate) fn file<T>(
parent.as_ptr(),
core::ptr::from_ref(data) as *mut c_void,
core::ptr::null(),
&**file_ops,
file_ops.fops(),
)
};

29
rust/kernel/debugfs/file_ops.rs
View File

@ -20,9 +20,6 @@
use core::marker::PhantomData;
#[cfg(CONFIG_DEBUG_FS)]
use core::ops::Deref;
/// # Invariant
///
/// `FileOps<T>` will always contain an `operations` which is safe to use for a file backed
@ -30,7 +27,7 @@
/// into a reference.
pub(super) struct FileOps<T> {
#[cfg(CONFIG_DEBUG_FS)]
operations: bindings::file_operations,
operations: &'static bindings::file_operations,
#[cfg(CONFIG_DEBUG_FS)]
mode: u16,
_phantom: PhantomData<T>,
@ -41,7 +38,7 @@ impl<T> FileOps<T> {
///
/// The caller asserts that the provided `operations` is safe to use for a file whose
/// inode has a pointer to `T` in its private data that is safe to convert into a reference.
const unsafe fn new(operations: bindings::file_operations, mode: u16) -> Self {
const unsafe fn new(operations: &'static bindings::file_operations, mode: u16) -> Self {
Self {
#[cfg(CONFIG_DEBUG_FS)]
operations,
@ -65,11 +62,11 @@ pub(super) const fn adapt(&self) -> &FileOps<T::Inner> {
}
#[cfg(CONFIG_DEBUG_FS)]
impl<T> Deref for FileOps<T> {
type Target = bindings::file_operations;
fn deref(&self) -> &Self::Target {
&self.operations
impl<T> FileOps<T> {
/// Returns a `'static` reference to the inner `file_operations`.
#[inline]
pub(crate) fn fops(&self) -> &'static bindings::file_operations {
self.operations
}
}
@ -130,7 +127,7 @@ pub(crate) trait ReadFile<T> {
impl<T: Writer + Sync> ReadFile<T> for T {
const FILE_OPS: FileOps<T> = {
let operations = bindings::file_operations {
let operations = &bindings::file_operations {
read: Some(bindings::seq_read),
llseek: Some(bindings::seq_lseek),
release: Some(bindings::single_release),
@ -181,7 +178,7 @@ pub(crate) trait ReadWriteFile<T> {
impl<T: Writer + Reader + Sync> ReadWriteFile<T> for T {
const FILE_OPS: FileOps<T> = {
let operations = bindings::file_operations {
let operations = &bindings::file_operations {
open: Some(writer_open::<T>),
read: Some(bindings::seq_read),
write: Some(write::<T>),
@ -238,7 +235,7 @@ pub(crate) trait WriteFile<T> {
impl<T: Reader + Sync> WriteFile<T> for T {
const FILE_OPS: FileOps<T> = {
let operations = bindings::file_operations {
let operations = &bindings::file_operations {
open: Some(write_only_open),
write: Some(write_only_write::<T>),
llseek: Some(bindings::noop_llseek),
@ -290,7 +287,7 @@ pub(crate) trait BinaryReadFile<T> {
impl<T: BinaryWriter + Sync> BinaryReadFile<T> for T {
const FILE_OPS: FileOps<T> = {
let operations = bindings::file_operations {
let operations = &bindings::file_operations {
read: Some(blob_read::<T>),
llseek: Some(bindings::default_llseek),
open: Some(bindings::simple_open),
@ -344,7 +341,7 @@ pub(crate) trait BinaryWriteFile<T> {
impl<T: BinaryReader + Sync> BinaryWriteFile<T> for T {
const FILE_OPS: FileOps<T> = {
let operations = bindings::file_operations {
let operations = &bindings::file_operations {
write: Some(blob_write::<T>),
llseek: Some(bindings::default_llseek),
open: Some(bindings::simple_open),
@ -368,7 +365,7 @@ pub(crate) trait BinaryReadWriteFile<T> {
impl<T: BinaryWriter + BinaryReader + Sync> BinaryReadWriteFile<T> for T {
const FILE_OPS: FileOps<T> = {
let operations = bindings::file_operations {
let operations = &bindings::file_operations {
read: Some(blob_read::<T>),
write: Some(blob_write::<T>),
llseek: Some(bindings::default_llseek),

3
rust/kernel/device_id.rs
View File

@ -146,8 +146,7 @@ impl<T: RawDeviceId, const N: usize> IdArray<T, (), N> {
/// If the device implements [`RawDeviceIdIndex`], consider using [`IdArray::new`] instead.
pub const fn new_without_index(ids: [T; N]) -> Self {
// SAFETY: `T` is layout-wise compatible with `T::RawType`, so is the array of them.
let raw_ids: [MaybeUninit<T::RawType>; N] = unsafe { core::mem::transmute_copy(&ids) };
core::mem::forget(ids);
let raw_ids: [MaybeUninit<T::RawType>; N] = unsafe { crate::mem::transmute(ids) };
Self {
ids: raw_ids,

141
rust/kernel/dma.rs
View File

@ -24,7 +24,6 @@
},
prelude::*,
ptr::KnownSize,
sync::aref::ARef,
transmute::{
AsBytes,
FromBytes, //
@ -223,7 +222,7 @@ pub const fn value(&self) -> u64 {
///
/// # fn test(dev: &Device<Bound>) -> Result {
/// let attribs = DMA_ATTR_FORCE_CONTIGUOUS | DMA_ATTR_NO_WARN;
/// let c: Coherent<[u64]> =
/// let c: Coherent<'_, [u64]> =
/// Coherent::zeroed_slice_with_attrs(dev, 4, GFP_KERNEL, attribs)?;
/// # Ok::<(), Error>(()) }
/// ```
@ -390,9 +389,9 @@ fn from(direction: DataDirection) -> Self {
/// };
///
/// # fn test(dev: &Device<Bound>) -> Result {
/// let mut dmem: CoherentBox<u64> = CoherentBox::zeroed(dev, GFP_KERNEL)?;
/// let mut dmem: CoherentBox<'_, u64> = CoherentBox::zeroed(dev, GFP_KERNEL)?;
/// *dmem = 42;
/// let dmem: Coherent<u64> = dmem.into();
/// let dmem: Coherent<'_, u64> = dmem.into();
/// # Ok::<(), Error>(()) }
/// ```
///
@ -410,18 +409,18 @@ fn from(direction: DataDirection) -> Self {
/// };
///
/// # fn test(dev: &Device<Bound>) -> Result {
/// let mut dmem: CoherentBox<[u64]> = CoherentBox::zeroed_slice(dev, 4, GFP_KERNEL)?;
/// let mut dmem: CoherentBox<'_, [u64]> = CoherentBox::zeroed_slice(dev, 4, GFP_KERNEL)?;
/// dmem.fill(42);
/// let dmem: Coherent<[u64]> = dmem.into();
/// let dmem: Coherent<'_, [u64]> = dmem.into();
/// # Ok::<(), Error>(()) }
/// ```
pub struct CoherentBox<T: KnownSize + ?Sized>(Coherent<T>);
pub struct CoherentBox<'a, T: KnownSize + ?Sized>(Coherent<'a, T>);
impl<T: AsBytes + FromBytes> CoherentBox<[T]> {
impl<'a, T: AsBytes + FromBytes> CoherentBox<'a, [T]> {
/// [`CoherentBox`] variant of [`Coherent::zeroed_slice_with_attrs`].
#[inline]
pub fn zeroed_slice_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
count: usize,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
@ -432,7 +431,7 @@ pub fn zeroed_slice_with_attrs(
/// Same as [CoherentBox::zeroed_slice_with_attrs], but with `dma::Attrs(0)`.
#[inline]
pub fn zeroed_slice(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
count: usize,
gfp_flags: kernel::alloc::Flags,
) -> Result<Self> {
@ -480,14 +479,14 @@ pub fn init_at<E>(&mut self, i: usize, init: impl Init<T, E>) -> Result
///
/// # fn test(dev: &Device<Bound>) -> Result {
/// let data = [0u8, 1u8, 2u8, 3u8];
/// let c: CoherentBox<[u8]> =
/// let c: CoherentBox<'_, [u8]> =
/// CoherentBox::from_slice_with_attrs(dev, &data, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
///
/// assert_eq!(c.deref(), &data);
/// # Ok::<(), Error>(()) }
/// ```
pub fn from_slice_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
data: &[T],
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
@ -512,7 +511,7 @@ pub fn from_slice_with_attrs(
/// `dma_attrs` is 0 by default.
#[inline]
pub fn from_slice(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
data: &[T],
gfp_flags: kernel::alloc::Flags,
) -> Result<Self>
@ -523,11 +522,11 @@ pub fn from_slice(
}
}
impl<T: AsBytes + FromBytes> CoherentBox<T> {
impl<'a, T: AsBytes + FromBytes> CoherentBox<'a, T> {
/// Same as [`CoherentBox::zeroed_slice_with_attrs`], but for a single element.
#[inline]
pub fn zeroed_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
) -> Result<Self> {
@ -536,12 +535,12 @@ pub fn zeroed_with_attrs(
/// Same as [`CoherentBox::zeroed_slice`], but for a single element.
#[inline]
pub fn zeroed(dev: &device::Device<Bound>, gfp_flags: kernel::alloc::Flags) -> Result<Self> {
pub fn zeroed(dev: &'a device::Device<Bound>, gfp_flags: kernel::alloc::Flags) -> Result<Self> {
Self::zeroed_with_attrs(dev, gfp_flags, Attrs(0))
}
}
impl<T: KnownSize + ?Sized> Deref for CoherentBox<T> {
impl<T: KnownSize + ?Sized> Deref for CoherentBox<'_, T> {
type Target = T;
#[inline]
@ -554,7 +553,7 @@ fn deref(&self) -> &Self::Target {
}
}
impl<T: AsBytes + FromBytes + KnownSize + ?Sized> DerefMut for CoherentBox<T> {
impl<T: AsBytes + FromBytes + KnownSize + ?Sized> DerefMut for CoherentBox<'_, T> {
#[inline]
fn deref_mut(&mut self) -> &mut Self::Target {
// SAFETY:
@ -565,9 +564,9 @@ fn deref_mut(&mut self) -> &mut Self::Target {
}
}
impl<T: AsBytes + FromBytes + KnownSize + ?Sized> From<CoherentBox<T>> for Coherent<T> {
impl<'a, T: AsBytes + FromBytes + KnownSize + ?Sized> From<CoherentBox<'a, T>> for Coherent<'a, T> {
#[inline]
fn from(value: CoherentBox<T>) -> Self {
fn from(value: CoherentBox<'a, T>) -> Self {
value.0
}
}
@ -588,26 +587,20 @@ fn from(value: CoherentBox<T>) -> Self {
/// to an allocated region of coherent memory and `dma_addr` is the DMA address base of the
/// region.
/// - The size in bytes of the allocation is equal to size information via pointer.
// TODO
//
// DMA allocations potentially carry device resources (e.g.IOMMU mappings), hence for soundness
// reasons DMA allocation would need to be embedded in a `Devres` container, in order to ensure
// that device resources can never survive device unbind.
//
// However, it is neither desirable nor necessary to protect the allocated memory of the DMA
// allocation from surviving device unbind; it would require RCU read side critical sections to
// access the memory, which may require subsequent unnecessary copies.
//
// Hence, find a way to revoke the device resources of a `Coherent`, but not the
// entire `Coherent` including the allocated memory itself.
pub struct Coherent<T: KnownSize + ?Sized> {
dev: ARef<device::Device>,
// The lifetime parameter ties DMA allocations to the device's bound scope, ensuring they are freed
// before the device is unbound under normal circumstances. However, if a `Coherent` is leaked (e.g.
// via `mem::forget`), device resources such as IOMMU mappings will not be released. Making all
// constructors `unsafe` to prevent this is considered too restrictive for the common case; this
// soundness hole is accepted for now.
pub struct Coherent<'a, T: KnownSize + ?Sized> {
dev: &'a device::Device<Bound>,
dma_addr: DmaAddress,
cpu_addr: NonNull<T>,
dma_attrs: Attrs,
}
impl<T: KnownSize + ?Sized> Coherent<T> {
impl<T: KnownSize + ?Sized> Coherent<'_, T> {
/// Returns the size in bytes of this allocation.
#[inline]
pub fn size(&self) -> usize {
@ -663,10 +656,10 @@ pub unsafe fn as_mut(&self) -> &mut T {
}
}
impl<T: AsBytes + FromBytes> Coherent<T> {
impl<'a, T: AsBytes + FromBytes> Coherent<'a, T> {
/// Allocates a region of `T` of coherent memory.
fn alloc_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
) -> Result<Self> {
@ -692,9 +685,9 @@ fn alloc_with_attrs(
// INVARIANT:
// - We just successfully allocated a coherent region which is adequately sized for `T`,
// hence the cpu address is valid.
// - We also hold a refcounted reference to the device.
// - `dev` is a valid reference to a bound device that outlives this allocation.
Ok(Self {
dev: dev.into(),
dev,
dma_addr,
cpu_addr,
dma_attrs,
@ -716,13 +709,13 @@ fn alloc_with_attrs(
/// };
///
/// # fn test(dev: &Device<Bound>) -> Result {
/// let c: Coherent<[u64; 4]> =
/// let c: Coherent<'_, [u64; 4]> =
/// Coherent::zeroed_with_attrs(dev, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
/// # Ok::<(), Error>(()) }
/// ```
#[inline]
pub fn zeroed_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
) -> Result<Self> {
@ -732,14 +725,14 @@ pub fn zeroed_with_attrs(
/// Performs the same functionality as [`Coherent::zeroed_with_attrs`], except the
/// `dma_attrs` is 0 by default.
#[inline]
pub fn zeroed(dev: &device::Device<Bound>, gfp_flags: kernel::alloc::Flags) -> Result<Self> {
pub fn zeroed(dev: &'a device::Device<Bound>, gfp_flags: kernel::alloc::Flags) -> Result<Self> {
Self::zeroed_with_attrs(dev, gfp_flags, Attrs(0))
}
/// Same as [`Coherent::zeroed_with_attrs`], but instead of a zero-initialization the memory is
/// initialized with `init`.
pub fn init_with_attrs<E>(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
init: impl Init<T, E>,
@ -764,7 +757,7 @@ pub fn init_with_attrs<E>(
/// with `init`.
#[inline]
pub fn init<E>(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
gfp_flags: kernel::alloc::Flags,
init: impl Init<T, E>,
) -> Result<Self>
@ -776,11 +769,11 @@ pub fn init<E>(
/// Allocates a region of `[T; len]` of coherent memory.
fn alloc_slice_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
len: usize,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
) -> Result<Coherent<[T]>> {
) -> Result<Coherent<'a, [T]>> {
const {
assert!(
core::mem::size_of::<T>() > 0,
@ -809,9 +802,9 @@ fn alloc_slice_with_attrs(
// INVARIANT:
// - We just successfully allocated a coherent region which is adequately sized for
// `[T; len]`, hence the cpu address is valid.
// - We also hold a refcounted reference to the device.
// - `dev` is a valid reference to a bound device that outlives this allocation.
Ok(Coherent {
dev: dev.into(),
dev,
dma_addr,
cpu_addr,
dma_attrs,
@ -836,17 +829,17 @@ fn alloc_slice_with_attrs(
/// };
///
/// # fn test(dev: &Device<Bound>) -> Result {
/// let c: Coherent<[u64]> =
/// let c: Coherent<'_, [u64]> =
/// Coherent::zeroed_slice_with_attrs(dev, 4, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
/// # Ok::<(), Error>(()) }
/// ```
#[inline]
pub fn zeroed_slice_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
len: usize,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
) -> Result<Coherent<[T]>> {
) -> Result<Coherent<'a, [T]>> {
Coherent::alloc_slice_with_attrs(dev, len, gfp_flags | __GFP_ZERO, dma_attrs)
}
@ -854,10 +847,10 @@ pub fn zeroed_slice_with_attrs(
/// `dma_attrs` is 0 by default.
#[inline]
pub fn zeroed_slice(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
len: usize,
gfp_flags: kernel::alloc::Flags,
) -> Result<Coherent<[T]>> {
) -> Result<Coherent<'a, [T]>> {
Self::zeroed_slice_with_attrs(dev, len, gfp_flags, Attrs(0))
}
@ -876,18 +869,18 @@ pub fn zeroed_slice(
/// # fn test(dev: &Device<Bound>) -> Result {
/// let data = [0u8, 1u8, 2u8, 3u8];
/// // `c` has the same content as `data`.
/// let c: Coherent<[u8]> =
/// let c: Coherent<'_, [u8]> =
/// Coherent::from_slice_with_attrs(dev, &data, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
///
/// # Ok::<(), Error>(()) }
/// ```
#[inline]
pub fn from_slice_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
data: &[T],
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
) -> Result<Coherent<[T]>>
) -> Result<Coherent<'a, [T]>>
where
T: Copy,
{
@ -898,10 +891,10 @@ pub fn from_slice_with_attrs(
/// `dma_attrs` is 0 by default.
#[inline]
pub fn from_slice(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
data: &[T],
gfp_flags: kernel::alloc::Flags,
) -> Result<Coherent<[T]>>
) -> Result<Coherent<'a, [T]>>
where
T: Copy,
{
@ -909,7 +902,7 @@ pub fn from_slice(
}
}
impl<T> Coherent<[T]> {
impl<T> Coherent<'_, [T]> {
/// Returns the number of elements `T` in this allocation.
///
/// Note that this is not the size of the allocation in bytes, which is provided by
@ -922,10 +915,10 @@ pub fn len(&self) -> usize {
}
/// Note that the device configured to do DMA must be halted before this object is dropped.
impl<T: KnownSize + ?Sized> Drop for Coherent<T> {
impl<T: KnownSize + ?Sized> Drop for Coherent<'_, T> {
fn drop(&mut self) {
let size = T::size(self.cpu_addr.as_ptr());
// SAFETY: Device pointer is guaranteed as valid by the type invariant on `Device`.
// SAFETY: Device pointer is guaranteed as valid by the lifetime of this `Coherent`.
// The cpu address, and the dma address are valid due to the type invariants on
// `Coherent`.
unsafe {
@ -942,15 +935,15 @@ fn drop(&mut self) {
// SAFETY: It is safe to send a `Coherent` to another thread if `T`
// can be sent to another thread.
unsafe impl<T: KnownSize + Send + ?Sized> Send for Coherent<T> {}
unsafe impl<T: KnownSize + Send + ?Sized> Send for Coherent<'_, T> {}
// SAFETY: Sharing `&Coherent` across threads is safe if `T` is `Sync`, because all
// methods that access the buffer contents (`field_read`, `field_write`, `as_slice`,
// `as_slice_mut`) are `unsafe`, and callers are responsible for ensuring no data races occur.
// The safe methods only return metadata or raw pointers whose use requires `unsafe`.
unsafe impl<T: KnownSize + ?Sized + AsBytes + FromBytes + Sync> Sync for Coherent<T> {}
unsafe impl<T: KnownSize + ?Sized + AsBytes + FromBytes + Sync> Sync for Coherent<'_, T> {}
impl<T: KnownSize + AsBytes + ?Sized> debugfs::BinaryWriter for Coherent<T> {
impl<T: KnownSize + AsBytes + ?Sized> debugfs::BinaryWriter for Coherent<'_, T> {
fn write_to_slice(
&self,
writer: &mut UserSliceWriter,
@ -996,15 +989,15 @@ fn write_to_slice(
/// - `size` is the allocation size in bytes as passed to `dma_alloc_attrs`.
/// - `dma_attrs` contains the attributes used for the allocation, always including
/// `DMA_ATTR_NO_KERNEL_MAPPING`.
pub struct CoherentHandle {
dev: ARef<device::Device>,
pub struct CoherentHandle<'a> {
dev: &'a device::Device<Bound>,
dma_addr: DmaAddress,
cpu_handle: NonNull<c_void>,
size: usize,
dma_attrs: Attrs,
}
impl CoherentHandle {
impl<'a> CoherentHandle<'a> {
/// Allocates `size` bytes of coherent DMA memory without creating a kernel virtual mapping.
///
/// Additional DMA attributes may be passed via `dma_attrs`; `DMA_ATTR_NO_KERNEL_MAPPING` is
@ -1012,7 +1005,7 @@ impl CoherentHandle {
///
/// Returns `EINVAL` if `size` is zero, `ENOMEM` if the allocation fails.
pub fn alloc_with_attrs(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
size: usize,
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
@ -1038,9 +1031,9 @@ pub fn alloc_with_attrs(
// INVARIANT: `cpu_handle` is the opaque handle from a successful `dma_alloc_attrs` call
// with `DMA_ATTR_NO_KERNEL_MAPPING`, `dma_addr` is the corresponding DMA address,
// and we hold a refcounted reference to the device.
// and `dev` is a valid reference to a bound device that outlives this allocation.
Ok(Self {
dev: dev.into(),
dev,
dma_addr,
cpu_handle,
size,
@ -1051,7 +1044,7 @@ pub fn alloc_with_attrs(
/// Allocates `size` bytes of coherent DMA memory without creating a kernel virtual mapping.
#[inline]
pub fn alloc(
dev: &device::Device<Bound>,
dev: &'a device::Device<Bound>,
size: usize,
gfp_flags: kernel::alloc::Flags,
) -> Result<Self> {
@ -1073,7 +1066,7 @@ pub fn size(&self) -> usize {
}
}
impl Drop for CoherentHandle {
impl Drop for CoherentHandle<'_> {
fn drop(&mut self) {
// SAFETY: All values are valid by the type invariants on `CoherentHandle`.
// `cpu_handle` is the opaque handle from `dma_alloc_attrs` and is passed back unchanged.
@ -1091,12 +1084,12 @@ fn drop(&mut self) {
// SAFETY: `CoherentHandle` only holds a device reference, a DMA address, an opaque CPU handle,
// and a size. None of these are tied to a specific thread.
unsafe impl Send for CoherentHandle {}
unsafe impl Send for CoherentHandle<'_> {}
// SAFETY: `CoherentHandle` provides no CPU access to the underlying allocation. The only
// operations on `&CoherentHandle` are reading the DMA address and size, both of which are
// plain `Copy` values.
unsafe impl Sync for CoherentHandle {}
unsafe impl Sync for CoherentHandle<'_> {}
/// View type for `Coherent`.
///
@ -1236,7 +1229,7 @@ fn as_view(self) -> CoherentView<'a, Self::Target> {
}
}
impl<'a, T: ?Sized + KnownSize> IoBase<'a> for &'a Coherent<T> {
impl<'a, T: ?Sized + KnownSize> IoBase<'a> for &'a Coherent<'_, T> {
type Backend = CoherentIoBackend;
type Target = T;

1022
rust/kernel/dma_buf/dma_fence.rs Normal file
View File

File diff suppressed because it is too large Load Diff

14
rust/kernel/dma_buf/mod.rs Normal file
View File

@ -0,0 +1,14 @@
// SPDX-License-Identifier: GPL-2.0 OR MIT
//! DMA-buf subsystem abstractions.
pub mod dma_fence;
pub use self::dma_fence::{
DriverFence,
Fence,
FenceCallback,
FenceCallbackRegistration,
FenceContext,
FenceContextOps, //
};

75
rust/kernel/drm/driver.rs
View File

@ -12,7 +12,10 @@
prelude::*,
sync::aref::ARef, //
};
use core::ptr::NonNull;
use core::{
marker::PhantomPinned,
ptr::NonNull, //
};
/// Driver use the GEM memory manager. This should be set for all modern drivers.
pub(crate) const FEAT_GEM: u32 = bindings::drm_driver_feature_DRIVER_GEM;
@ -143,9 +146,15 @@ pub trait Driver {
/// The registration type of a `drm::Device`.
///
/// Once the `Registration` structure is dropped, the device is unregistered.
#[pin_data(PinnedDrop)]
pub struct Registration<'a, T: Driver> {
drm: ARef<drm::Device<T>>,
_reg_data: Pin<KBox<T::RegistrationData<'a>>>,
#[pin]
data: T::RegistrationData<'a>,
/// Even if `T::RegistrationData` is `Unpin`, the registration must not be moved because a
/// pointer to `data` is stored in the DRM device.
#[pin]
_pin: PhantomPinned,
}
impl<'a, T: Driver> Registration<'a, T> {
@ -159,39 +168,40 @@ impl<'a, T: Driver> Registration<'a, T> {
pub unsafe fn new<E>(
dev: &'a device::Device<device::Bound>,
drm: drm::UnregisteredDevice<T>,
reg_data: impl PinInit<T::RegistrationData<'a>, E>,
flags: usize,
) -> Result<Self>
data: impl PinInit<T::RegistrationData<'a>, E>,
) -> impl PinInit<Self, Error>
where
Error: From<E>,
{
let parent = drm.as_ref();
if parent.as_ref().as_raw() != dev.as_raw() {
return Err(EINVAL);
}
try_pin_init!(Self {
_: {
let parent = drm.as_ref();
if parent.as_ref().as_raw() != dev.as_raw() {
return Err(EINVAL);
}
},
let reg_data: Pin<KBox<T::RegistrationData<'a>>> = KBox::pin_init(reg_data, GFP_KERNEL)?;
// Store the registration data pointer in the device before registration, so that it is
// visible once ioctls can be called.
let ptr: NonNull<T::RegistrationData<'static>> =
NonNull::from(Pin::get_ref(reg_data.as_ref())).cast();
// SAFETY: No concurrent access; the device is not yet registered.
unsafe { *drm.registration_data.get() = ptr };
// SAFETY: `drm` is a valid, initialized but not yet registered DRM device.
let ret = unsafe { bindings::drm_dev_register(drm.as_raw(), flags) };
if let Err(e) = to_result(ret) {
// SAFETY: `drm_dev_register()` synchronizes SRCU on failure, so no concurrent
// access to `registration_data` is possible at this point.
unsafe { *drm.registration_data.get() = NonNull::dangling() };
return Err(e);
}
Ok(Self {
data <- data,
drm: (&*drm).into(),
_reg_data: reg_data,
_pin: PhantomPinned,
_: {
// Store the registration data pointer in the device before registration, so
// that it is visible once ioctls can be called.
let ptr: NonNull<T::RegistrationData<'static>> =
NonNull::from(data.as_ref().get_ref()).cast();
// SAFETY: No concurrent access; the device is not yet registered.
unsafe { *drm.registration_data.get() = ptr };
// SAFETY: `drm` is a valid, initialized but not yet registered DRM device.
let ret = unsafe { bindings::drm_dev_register(drm.as_raw(), 0) };
to_result(ret).inspect_err(|_| {
// SAFETY: `drm_dev_register()` synchronizes SRCU on failure, so no
// concurrent access to `registration_data` is possible at this point.
unsafe { *drm.registration_data.get() = NonNull::dangling() };
})?;
},
})
}
@ -208,8 +218,9 @@ unsafe impl<T: Driver> Sync for Registration<'_, T> {}
// SAFETY: Registration with and unregistration from the DRM subsystem can happen from any thread.
unsafe impl<T: Driver> Send for Registration<'_, T> {}
impl<T: Driver> Drop for Registration<'_, T> {
fn drop(&mut self) {
#[pinned_drop]
impl<T: Driver> PinnedDrop for Registration<'_, T> {
fn drop(self: Pin<&mut Self>) {
// Use `drm_dev_unplug` rather than `drm_dev_unregister` to ensure that existing
// `drm_dev_enter()` critical sections complete before unregistration proceeds. This
// is required for the safety of `RegistrationGuard`, which relies on the SRCU barrier in

220
rust/kernel/io.rs
View File

@ -11,6 +11,10 @@
use crate::{
bindings,
mem::{
AsRepr,
AsReprMut, //
},
prelude::*,
ptr::{
Alignment,
@ -226,6 +230,17 @@ fn io_view<'a, IO: Io<'a>, U>(
Ok(unsafe { IO::Backend::project_view(view, projected_ptr) })
}
/// Returns the primitive view of a I/O view.
#[inline]
fn io_view_as_repr<'a, IO: Io<'a, Target = T>, T: AsRepr>(
this: IO,
) -> <IO::Backend as IoBackend>::View<'a, T::Repr> {
let view = this.as_view();
// SAFETY: `AsRepr` guarantees layout compatibility.
unsafe { IO::Backend::project_view(view, IO::Backend::as_ptr(view).cast::<T::Repr>()) }
}
/// I/O backends.
///
/// This is an abstract representation to be implemented by arbitrary I/O
@ -353,15 +368,12 @@ fn copy_write<T: IntoBytes>(view: Self::View<'_, T>, value: T) {
///
/// - The valid `Base` to operate on. For most registers, this should be [`Region`].
/// - The offset to access (returned by [`IoLoc::offset`]),
/// - The width of the access (determined by [`IoLoc::IoType`]),
/// - The type `T` in which the raw data is returned or provided.
/// - The type `T` in which the data is returned or provided.
///
/// `T` and `IoLoc::IoType` may differ: for instance, a typed register has `T` = the register type
/// with its bitfields, and `IoType` = its backing primitive (e.g. `u32`).
/// `T` is not necessarily the type for underlying I/O operation. Methods that take `IoLoc` have `T:
/// AsRepr` bound and the `<T as AsRepr>::Repr` type would be used to perform I/O and converted to
/// `T` instead.
pub trait IoLoc<Base: ?Sized, T> {
/// Size ([`u8`], [`u16`], etc) of the I/O performed on the returned [`offset`](IoLoc::offset).
type IoType: Into<T> + From<T>;
/// Consumes `self` and returns the offset of this location.
fn offset(self) -> usize;
}
@ -372,8 +384,6 @@ macro_rules! impl_usize_ioloc {
($($ty:ty),*) => {
$(
impl<const SIZE: usize> IoLoc<Region<SIZE>, $ty> for usize {
type IoType = $ty;
#[inline(always)]
fn offset(self) -> usize {
self
@ -437,6 +447,45 @@ fn is_empty<T>(self) -> bool
self.len() == 0
}
/// Convert into a different typed I/O view.
///
/// The target type must be known (statically) to be of the same or smaller size to current
/// type, and the current view must be properly aligned for the target type.
///
/// # Examples
///
/// ```no_run
/// use kernel::io::{
/// io_project,
/// Mmio,
/// Io,
/// Region,
/// };
/// #[derive(FromBytes, IntoBytes)]
/// #[repr(C)]
/// struct MyStruct { field: u32, }
///
/// # fn test(mmio: &Mmio<'_, Region<0x1000>>) {
/// // let mmio: Mmio<'_, Region<0x1000>>;
/// let whole: Mmio<'_, MyStruct> = mmio.cast();
/// # }
/// ```
#[inline]
fn cast<U>(self) -> <Self::Backend as IoBackend>::View<'a, U>
where
Self::Target: FromBytes + IntoBytes,
U: FromBytes + IntoBytes,
{
let view = self.as_view();
let ptr = Self::Backend::as_ptr(view);
const_assert!(size_of::<U>() <= Self::Target::MIN_SIZE);
const_assert!(align_of::<U>() <= Self::Target::MIN_ALIGN.as_usize());
// SAFETY: We have checked bounds and alignment, so this is a valid projection.
unsafe { Self::Backend::project_view(view, ptr.cast()) }
}
/// Try to convert into a different typed I/O view.
///
/// A runtime check is performed to ensure that the target type is of same or smaller size to
@ -498,10 +547,10 @@ fn try_cast<U>(self) -> Result<<Self::Backend as IoBackend>::View<'a, U>>
#[inline]
fn read_val(self) -> Self::Target
where
Self::Backend: IoCapable<Self::Target>,
Self::Target: Sized,
Self::Target: AsReprMut,
Self::Backend: IoCapable<<Self::Target as AsRepr>::Repr>,
{
Self::Backend::io_read(self.as_view())
Self::Target::from_repr(Self::Backend::io_read(io_view_as_repr(self)))
}
/// Write a value to I/O.
@ -520,10 +569,10 @@ fn read_val(self) -> Self::Target
#[inline]
fn write_val(self, value: Self::Target)
where
Self::Backend: IoCapable<Self::Target>,
Self::Target: Sized,
Self::Target: AsRepr,
Self::Backend: IoCapable<<Self::Target as AsRepr>::Repr>,
{
Self::Backend::io_write(self.as_view(), value)
Self::Backend::io_write(io_view_as_repr(self), Self::Target::into_repr(value))
}
/// Copy-read from I/O memory.
@ -645,7 +694,7 @@ fn copy_to_slice(self, data: &mut [u8])
#[inline(always)]
fn try_read8(self, offset: usize) -> Result<u8>
where
usize: IoLoc<Self::Target, u8, IoType = u8>,
usize: IoLoc<Self::Target, u8>,
Self::Backend: IoCapable<u8>,
{
self.try_read(offset)
@ -655,7 +704,7 @@ fn try_read8(self, offset: usize) -> Result<u8>
#[inline(always)]
fn try_read16(self, offset: usize) -> Result<u16>
where
usize: IoLoc<Self::Target, u16, IoType = u16>,
usize: IoLoc<Self::Target, u16>,
Self::Backend: IoCapable<u16>,
{
self.try_read(offset)
@ -665,7 +714,7 @@ fn try_read16(self, offset: usize) -> Result<u16>
#[inline(always)]
fn try_read32(self, offset: usize) -> Result<u32>
where
usize: IoLoc<Self::Target, u32, IoType = u32>,
usize: IoLoc<Self::Target, u32>,
Self::Backend: IoCapable<u32>,
{
self.try_read(offset)
@ -675,7 +724,7 @@ fn try_read32(self, offset: usize) -> Result<u32>
#[inline(always)]
fn try_read64(self, offset: usize) -> Result<u64>
where
usize: IoLoc<Self::Target, u64, IoType = u64>,
usize: IoLoc<Self::Target, u64>,
Self::Backend: IoCapable<u64>,
{
self.try_read(offset)
@ -685,7 +734,7 @@ fn try_read64(self, offset: usize) -> Result<u64>
#[inline(always)]
fn try_write8(self, value: u8, offset: usize) -> Result
where
usize: IoLoc<Self::Target, u8, IoType = u8>,
usize: IoLoc<Self::Target, u8>,
Self::Backend: IoCapable<u8>,
{
self.try_write(offset, value)
@ -695,7 +744,7 @@ fn try_write8(self, value: u8, offset: usize) -> Result
#[inline(always)]
fn try_write16(self, value: u16, offset: usize) -> Result
where
usize: IoLoc<Self::Target, u16, IoType = u16>,
usize: IoLoc<Self::Target, u16>,
Self::Backend: IoCapable<u16>,
{
self.try_write(offset, value)
@ -705,7 +754,7 @@ fn try_write16(self, value: u16, offset: usize) -> Result
#[inline(always)]
fn try_write32(self, value: u32, offset: usize) -> Result
where
usize: IoLoc<Self::Target, u32, IoType = u32>,
usize: IoLoc<Self::Target, u32>,
Self::Backend: IoCapable<u32>,
{
self.try_write(offset, value)
@ -715,7 +764,7 @@ fn try_write32(self, value: u32, offset: usize) -> Result
#[inline(always)]
fn try_write64(self, value: u64, offset: usize) -> Result
where
usize: IoLoc<Self::Target, u64, IoType = u64>,
usize: IoLoc<Self::Target, u64>,
Self::Backend: IoCapable<u64>,
{
self.try_write(offset, value)
@ -727,7 +776,7 @@ fn try_write64(self, value: u64, offset: usize) -> Result
#[inline(always)]
fn read8(self, offset: usize) -> u8
where
usize: IoLoc<Self::Target, u8, IoType = u8>,
usize: IoLoc<Self::Target, u8>,
Self::Backend: IoCapable<u8>,
{
self.read(offset)
@ -739,7 +788,7 @@ fn read8(self, offset: usize) -> u8
#[inline(always)]
fn read16(self, offset: usize) -> u16
where
usize: IoLoc<Self::Target, u16, IoType = u16>,
usize: IoLoc<Self::Target, u16>,
Self::Backend: IoCapable<u16>,
{
self.read(offset)
@ -751,7 +800,7 @@ fn read16(self, offset: usize) -> u16
#[inline(always)]
fn read32(self, offset: usize) -> u32
where
usize: IoLoc<Self::Target, u32, IoType = u32>,
usize: IoLoc<Self::Target, u32>,
Self::Backend: IoCapable<u32>,
{
self.read(offset)
@ -763,7 +812,7 @@ fn read32(self, offset: usize) -> u32
#[inline(always)]
fn read64(self, offset: usize) -> u64
where
usize: IoLoc<Self::Target, u64, IoType = u64>,
usize: IoLoc<Self::Target, u64>,
Self::Backend: IoCapable<u64>,
{
self.read(offset)
@ -775,7 +824,7 @@ fn read64(self, offset: usize) -> u64
#[inline(always)]
fn write8(self, value: u8, offset: usize)
where
usize: IoLoc<Self::Target, u8, IoType = u8>,
usize: IoLoc<Self::Target, u8>,
Self::Backend: IoCapable<u8>,
{
self.write(offset, value)
@ -787,7 +836,7 @@ fn write8(self, value: u8, offset: usize)
#[inline(always)]
fn write16(self, value: u16, offset: usize)
where
usize: IoLoc<Self::Target, u16, IoType = u16>,
usize: IoLoc<Self::Target, u16>,
Self::Backend: IoCapable<u16>,
{
self.write(offset, value)
@ -799,7 +848,7 @@ fn write16(self, value: u16, offset: usize)
#[inline(always)]
fn write32(self, value: u32, offset: usize)
where
usize: IoLoc<Self::Target, u32, IoType = u32>,
usize: IoLoc<Self::Target, u32>,
Self::Backend: IoCapable<u32>,
{
self.write(offset, value)
@ -811,7 +860,7 @@ fn write32(self, value: u32, offset: usize)
#[inline(always)]
fn write64(self, value: u64, offset: usize)
where
usize: IoLoc<Self::Target, u64, IoType = u64>,
usize: IoLoc<Self::Target, u64>,
Self::Backend: IoCapable<u64>,
{
self.write(offset, value)
@ -843,11 +892,11 @@ fn write64(self, value: u64, offset: usize)
#[inline(always)]
fn try_read<T, L>(self, location: L) -> Result<T>
where
T: AsReprMut,
L: IoLoc<Self::Target, T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
{
let view = io_view::<Self, L::IoType>(self, location.offset())?;
Ok(Self::Backend::io_read(view).into())
Ok(io_read!(self, try: location))
}
/// Generic fallible write with runtime bounds check.
@ -876,12 +925,11 @@ fn try_read<T, L>(self, location: L) -> Result<T>
#[inline(always)]
fn try_write<T, L>(self, location: L, value: T) -> Result
where
T: AsRepr,
L: IoLoc<Self::Target, T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
{
let view = io_view::<Self, L::IoType>(self, location.offset())?;
let io_value = value.into();
Self::Backend::io_write(view, io_value);
io_write!(self, try: location, value);
Ok(())
}
@ -900,6 +948,8 @@ fn try_write<T, L>(self, location: L, value: T) -> Result
/// };
///
/// register! {
/// base: Region;
///
/// VERSION(u32) @ 0x100 {
/// 15:8 major;
/// 7:0 minor;
@ -920,9 +970,10 @@ fn try_write<T, L>(self, location: L, value: T) -> Result
#[inline(always)]
fn try_write_reg<T, L, V>(self, value: V) -> Result
where
T: AsRepr,
L: IoLoc<Self::Target, T>,
V: LocatedRegister<Self::Target, Location = L, Value = T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
{
let (location, value) = value.into_io_op();
@ -954,16 +1005,13 @@ fn try_write_reg<T, L, V>(self, value: V) -> Result
#[inline(always)]
fn try_update<T, L, F>(self, location: L, f: F) -> Result
where
T: AsReprMut,
L: IoLoc<Self::Target, T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
F: FnOnce(T) -> T,
{
let view = io_view::<Self, L::IoType>(self, location.offset())?;
let value: T = Self::Backend::io_read(view).into();
let io_value = f(value).into();
Self::Backend::io_write(view, io_value);
let view = io_project!(self, try: location);
view.write_val(f(view.read_val()));
Ok(())
}
@ -991,11 +1039,11 @@ fn try_update<T, L, F>(self, location: L, f: F) -> Result
#[inline(always)]
fn read<T, L>(self, location: L) -> T
where
T: AsReprMut,
L: IoLoc<Self::Target, T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
{
let view = io_view_assert::<Self, L::IoType>(self, location.offset());
Self::Backend::io_read(view).into()
io_read!(self, build: location)
}
/// Generic infallible write with compile-time bounds check.
@ -1022,12 +1070,11 @@ fn read<T, L>(self, location: L) -> T
#[inline(always)]
fn write<T, L>(self, location: L, value: T)
where
T: AsRepr,
L: IoLoc<Self::Target, T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
{
let view = io_view_assert::<Self, L::IoType>(self, location.offset());
let io_value = value.into();
Self::Backend::io_write(view, io_value);
io_write!(self, build: location, value);
}
/// Generic infallible write of a fully-located register value.
@ -1045,6 +1092,8 @@ fn write<T, L>(self, location: L, value: T)
/// };
///
/// register! {
/// base: Region<0x1000>;
///
/// VERSION(u32) @ 0x100 {
/// 15:8 major;
/// 7:0 minor;
@ -1064,9 +1113,10 @@ fn write<T, L>(self, location: L, value: T)
#[inline(always)]
fn write_reg<T, L, V>(self, value: V)
where
T: AsRepr,
L: IoLoc<Self::Target, T>,
V: LocatedRegister<Self::Target, Location = L, Value = T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
{
let (location, value) = value.into_io_op();
@ -1098,14 +1148,13 @@ fn write_reg<T, L, V>(self, value: V)
#[inline(always)]
fn update<T, L, F>(self, location: L, f: F)
where
T: AsReprMut,
L: IoLoc<Self::Target, T>,
Self::Backend: IoCapable<L::IoType>,
Self::Backend: IoCapable<<T as AsRepr>::Repr>,
F: FnOnce(T) -> T,
{
let view = io_view_assert::<Self, L::IoType>(self, location.offset());
let value: T = Self::Backend::io_read(view).into();
let io_value = f(value).into();
Self::Backend::io_write(view, io_value);
let view = io_project!(self, build: location);
view.write_val(f(view.read_val()));
}
}
@ -1649,6 +1698,25 @@ pub unsafe fn project_view<U: ?Sized + KnownSize>(
// SAFETY: Per safety requirement.
unsafe { T::Backend::project_view::<T::Target, _>(self.0, ptr) }
}
#[inline(always)]
pub fn try_project_loc<U, L>(
self,
location: L,
) -> Result<<T::Backend as IoBackend>::View<'a, U>>
where
L: IoLoc<T::Target, U>,
{
io_view::<_, U>(self.0, location.offset())
}
#[inline(always)]
pub fn project_loc<U, L>(self, location: L) -> <T::Backend as IoBackend>::View<'a, U>
where
L: IoLoc<T::Target, U>,
{
io_view_assert::<_, U>(self.0, location.offset())
}
}
/// Project an I/O type to a subview of it.
@ -1656,26 +1724,54 @@ pub unsafe fn project_view<U: ?Sized + KnownSize>(
/// The syntax is of form `io_project!(io, proj)` where `io` is an expression to a type that
/// implements [`Io`] and `proj` is a [projection specification](kernel::ptr::project!).
///
/// `io_project!` can also project to a subview of registers defined with [`register!`] macro.
/// Register projection has syntax `io_project!(io, try: REGISTER)` for fallible projection and
/// `io_project!(io, build: REGISTER)` for infallible projection.
///
/// # Examples
///
/// ```
/// use kernel::io::{
/// io_project,
/// register,
/// Mmio,
/// };
/// #[repr(C)]
/// struct MyStruct { field: u32, }
///
/// register! {
/// base: MyStruct;
/// FIELD(u32) @ 0 {
/// 31:0 val;
/// }
/// }
///
/// # fn test(mmio: Mmio<'_, [MyStruct]>) -> Result {
/// // let mmio: Mmio<[MyStruct]>;
/// let field: Mmio<'_, u32> = io_project!(mmio, [try: 1].field);
/// let whole: Mmio<'_, MyStruct> = io_project!(mmio, [try: 2]);
/// let nested: Mmio<'_, u32> = io_project!(whole, .field);
/// let reg: Mmio<'_, FIELD> = io_project!(whole, build: FIELD);
/// # Ok::<(), Error>(()) }
/// ```
#[macro_export]
#[doc(hidden)]
macro_rules! io_project {
// Register projection
($io:expr, try: $ioloc:expr) => {{
#[allow(unused)]
use $crate::io::IoBase as _;
let view = $crate::io::ProjectHelper($io.as_view());
view.try_project_loc($ioloc)?
}};
($io:expr, build: $ioloc:expr) => {{
#[allow(unused)]
use $crate::io::IoBase as _;
let view = $crate::io::ProjectHelper($io.as_view());
view.project_loc($ioloc)
}};
// Field or index projection
($io:expr, $($proj:tt)*) => {{
#[allow(unused)]
use $crate::io::IoBase as _;
@ -1746,6 +1842,12 @@ macro_rules! io_write {
(@parse [$io:expr] [$($proj:tt)*] [[$flavor:ident: $index:expr] $($rest:tt)*]) => {
$crate::io_write!(@parse [$io] [$($proj)* [$flavor: $index]] [$($rest)*])
};
(@parse [$io:expr] [] [try: $ioloc:expr, $($rest:tt)*]) => {
$crate::io_write!(@parse [$io] [try: $ioloc] [, $($rest)*])
};
(@parse [$io:expr] [] [build: $ioloc:expr, $($rest:tt)*]) => {
$crate::io_write!(@parse [$io] [build: $ioloc] [, $($rest)*])
};
($io:expr, $($rest:tt)*) => {
$crate::io_write!(@parse [$io] [] [$($rest)*])
};

803
rust/kernel/io/register.rs
View File

@ -8,14 +8,19 @@
//!
//! Note: most of the items in this module are public so they can be referenced by the macro, but
//! most are not to be used directly by users. Outside of the `register!` macro itself, the only
//! items you might want to import from this module are [`WithBase`] and [`Array`].
//! item you might want to import from this module is [`Array`].
//!
//! # Simple example
//!
//! ```no_run
//! use kernel::io::register;
//! use kernel::io::{
//! register,
//! Region,
//! };
//!
//! register! {
//! base: Region<0x1000>;
//!
//! /// Basic information about the chip.
//! pub BOOT_0(u32) @ 0x00000100 {
//! /// Vendor ID.
@ -55,11 +60,14 @@
//! register,
//! Io,
//! IoLoc,
//! Region,
//! },
//! num::Bounded,
//! };
//! # use kernel::io::{Mmio, Region};
//! # use kernel::io::Mmio;
//! # register! {
//! # base: Region<0x1000>;
//! #
//! # pub BOOT_0(u32) @ 0x00000100 {
//! # 15:8 vendor_id;
//! # 7:4 major_revision;
@ -113,149 +121,50 @@
io::IoLoc, //
};
use super::Region;
/// Allows `()` to be used as the `location` parameter of [`Io::write`](super::Io::write) when
/// passing a [`FixedIoLoc`] value.
impl<Base: ?Sized, T> IoLoc<Base, T> for ()
where
T: FixedIoLoc<Base>,
{
#[inline(always)]
fn offset(self) -> usize {
T::LOCATION.offset()
}
}
/// Trait implemented by all registers.
pub trait Register: Sized {
/// Backing primitive type of the register.
type Storage: Into<Self> + From<Self>;
// Provides a `IoLoc` impl that for a fixed offset.
#[doc(hidden)]
pub struct OffsetLoc<Base: ?Sized, T>(usize, PhantomData<(T, Base)>);
impl<Base: ?Sized, T> OffsetLoc<Base, T> {
#[inline]
pub const fn new(offset: usize) -> Self {
Self(offset, PhantomData)
}
#[inline]
pub const fn const_offset(self) -> usize {
self.0
}
}
impl<Base: ?Sized, T> IoLoc<Base, T> for OffsetLoc<Base, T> {
#[inline(always)]
fn offset(self) -> usize {
self.0
}
}
/// Trait implemented by arrays of registers.
pub trait RegisterArray: Sized {
/// Base type for this register.
type Base: ?Sized;
/// Start offset of the register.
///
/// The interpretation of this offset depends on the type of the register.
const OFFSET: usize;
}
/// Trait implemented by registers with a fixed offset.
pub trait FixedRegister: Register {}
/// Allows `()` to be used as the `location` parameter of [`Io::write`](super::Io::write) when
/// passing a [`FixedRegister`] value.
impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for ()
where
T: FixedRegister,
{
type IoType = T::Storage;
#[inline(always)]
fn offset(self) -> usize {
T::OFFSET
}
}
/// A [`FixedRegister`] carries its location in its type. Thus `FixedRegister` values can be used
/// as an [`IoLoc`].
impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for T
where
T: FixedRegister,
{
type IoType = T::Storage;
#[inline(always)]
fn offset(self) -> usize {
T::OFFSET
}
}
/// Location of a fixed register.
pub struct FixedRegisterLoc<T: FixedRegister>(PhantomData<T>);
impl<T: FixedRegister> FixedRegisterLoc<T> {
/// Returns the location of `T`.
#[inline(always)]
// We do not implement `Default` so we can be const.
#[expect(clippy::new_without_default)]
pub const fn new() -> Self {
Self(PhantomData)
}
}
impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for FixedRegisterLoc<T>
where
T: FixedRegister,
{
type IoType = T::Storage;
#[inline(always)]
fn offset(self) -> usize {
T::OFFSET
}
}
/// Trait providing a base address to be added to the offset of a relative register to obtain
/// its actual offset.
///
/// The `T` generic argument is used to distinguish which base to use, in case a type provides
/// several bases. It is given to the `register!` macro to restrict the use of the register to
/// implementors of this particular variant.
pub trait RegisterBase<T> {
/// Base address to which register offsets are added.
const BASE: usize;
}
/// Trait implemented by all registers that are relative to a base.
pub trait WithBase {
/// Family of bases applicable to this register.
type BaseFamily;
/// Returns the absolute location of this type when using `B` as its base.
#[inline(always)]
fn of<B: RegisterBase<Self::BaseFamily>>() -> RelativeRegisterLoc<Self, B>
where
Self: Register,
{
RelativeRegisterLoc::new()
}
}
/// Trait implemented by relative registers.
pub trait RelativeRegister: Register + WithBase {}
/// Location of a relative register.
///
/// This can either be an immediately accessible regular [`RelativeRegister`], or a
/// [`RelativeRegisterArray`] that needs one additional resolution through
/// [`RelativeRegisterLoc::at`].
pub struct RelativeRegisterLoc<T: WithBase, B: ?Sized>(PhantomData<T>, PhantomData<B>);
impl<T, B> RelativeRegisterLoc<T, B>
where
T: Register + WithBase,
B: RegisterBase<T::BaseFamily> + ?Sized,
{
/// Returns the location of a relative register or register array.
#[inline(always)]
// We do not implement `Default` so we can be const.
#[expect(clippy::new_without_default)]
pub const fn new() -> Self {
Self(PhantomData, PhantomData)
}
// Returns the absolute offset of the relative register using base `B`.
//
// This is implemented as a private const method so it can be reused by the [`IoLoc`]
// implementations of both [`RelativeRegisterLoc`] and [`RelativeRegisterArrayLoc`].
#[inline]
const fn offset(self) -> usize {
B::BASE + T::OFFSET
}
}
impl<const SIZE: usize, T, B> IoLoc<Region<SIZE>, T> for RelativeRegisterLoc<T, B>
where
T: RelativeRegister,
B: RegisterBase<T::BaseFamily> + ?Sized,
{
type IoType = T::Storage;
#[inline(always)]
fn offset(self) -> usize {
RelativeRegisterLoc::offset(self)
}
}
/// Trait implemented by arrays of registers.
pub trait RegisterArray: Register {
/// Number of elements in the registers array.
const SIZE: usize;
/// Number of bytes between the start of elements in the registers array.
@ -285,12 +194,10 @@ pub fn try_new(idx: usize) -> Option<Self> {
}
}
impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for RegisterArrayLoc<T>
impl<Base: ?Sized, T> IoLoc<Base, T> for RegisterArrayLoc<T>
where
T: RegisterArray,
T: RegisterArray<Base = Base>,
{
type IoType = T::Storage;
#[inline(always)]
fn offset(self) -> usize {
T::OFFSET + self.0 * T::STRIDE
@ -318,71 +225,15 @@ fn try_at(idx: usize) -> Option<RegisterArrayLoc<Self>>
}
}
/// Trait implemented by arrays of relative registers.
pub trait RelativeRegisterArray: RegisterArray + WithBase {}
/// Trait implemented by types that indicate there is a fixed I/O location for this given type.
///
/// Implementors can be used with [`Io::write_reg`](super::Io::write_reg).
pub trait FixedIoLoc<Base: ?Sized>: Sized {
/// Type of [`FixedIoLoc::LOCATION`].
type Location: IoLoc<Base, Self>;
/// Location of a relative array register.
pub struct RelativeRegisterArrayLoc<
T: RelativeRegisterArray,
B: RegisterBase<T::BaseFamily> + ?Sized,
>(RelativeRegisterLoc<T, B>, usize);
impl<T, B> RelativeRegisterArrayLoc<T, B>
where
T: RelativeRegisterArray,
B: RegisterBase<T::BaseFamily> + ?Sized,
{
/// Returns the location of register `T` from the base `B` at index `idx`, with build-time
/// validation.
#[inline(always)]
pub fn new(idx: usize) -> Self {
build_assert!(idx < T::SIZE);
Self(RelativeRegisterLoc::new(), idx)
}
/// Attempts to return the location of register `T` from the base `B` at index `idx`, with
/// runtime validation.
#[inline(always)]
pub fn try_new(idx: usize) -> Option<Self> {
if idx < T::SIZE {
Some(Self(RelativeRegisterLoc::new(), idx))
} else {
None
}
}
}
/// Methods exclusive to [`RelativeRegisterLoc`]s created with a [`RelativeRegisterArray`].
impl<T, B> RelativeRegisterLoc<T, B>
where
T: RelativeRegisterArray,
B: RegisterBase<T::BaseFamily> + ?Sized,
{
/// Returns the location of the register at position `idx`, with build-time validation.
#[inline(always)]
pub fn at(self, idx: usize) -> RelativeRegisterArrayLoc<T, B> {
RelativeRegisterArrayLoc::new(idx)
}
/// Returns the location of the register at position `idx`, with runtime validation.
#[inline(always)]
pub fn try_at(self, idx: usize) -> Option<RelativeRegisterArrayLoc<T, B>> {
RelativeRegisterArrayLoc::try_new(idx)
}
}
impl<const SIZE: usize, T, B> IoLoc<Region<SIZE>, T> for RelativeRegisterArrayLoc<T, B>
where
T: RelativeRegisterArray,
B: RegisterBase<T::BaseFamily> + ?Sized,
{
type IoType = T::Storage;
#[inline(always)]
fn offset(self) -> usize {
self.0.offset() + self.1 * T::STRIDE
}
/// Location of this type within given base.
const LOCATION: Self::Location;
}
/// Trait implemented by items that contain both a register value and the absolute I/O location at
@ -390,8 +241,8 @@ fn offset(self) -> usize {
///
/// Implementors can be used with [`Io::write_reg`](super::Io::write_reg).
pub trait LocatedRegister<Base: ?Sized> {
/// Register value to write.
type Value: Register;
/// Value to write.
type Value;
/// Full location information at which to write the value.
type Location: IoLoc<Base, Self::Value>;
@ -400,27 +251,38 @@ pub trait LocatedRegister<Base: ?Sized> {
fn into_io_op(self) -> (Self::Location, Self::Value);
}
impl<const SIZE: usize, T> LocatedRegister<Region<SIZE>> for T
impl<Base: ?Sized, T> LocatedRegister<Base> for T
where
T: FixedRegister,
T: FixedIoLoc<Base>,
{
type Location = FixedRegisterLoc<Self::Value>;
type Location = T::Location;
type Value = T;
#[inline(always)]
fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
(FixedRegisterLoc::new(), self)
fn into_io_op(self) -> (T::Location, T) {
(T::LOCATION, self)
}
}
/// Helper function for register element alias implementation.
///
/// This is used to enforce base matching and provide bounds checking.
#[doc(hidden)]
#[inline(always)] // for const eval only
pub const fn element_alias_offset<Base: ?Sized, Alias: RegisterArray<Base = Base>>(
idx: usize,
) -> usize {
assert!(idx < Alias::SIZE);
Alias::OFFSET + idx * Alias::STRIDE
}
/// Defines a dedicated type for a register, including getter and setter methods for its fields and
/// methods to read and write it from an [`Io`](kernel::io::Io) region.
///
/// This documentation focuses on how to declare registers. See the [module-level
/// documentation](mod@kernel::io::register) for examples of how to access them.
///
/// There are 4 possible kinds of registers: fixed offset registers, relative registers, arrays of
/// registers, and relative arrays of registers.
/// Registers can either be fixed offset registers or arrays of registers.
///
/// ## Fixed offset registers
///
@ -444,11 +306,14 @@ fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
/// io::{
/// register,
/// Io,
/// Region,
/// },
/// };
/// # use kernel::io::{Mmio, Region};
/// # use kernel::io::Mmio;
///
/// register! {
/// base: Region<0x1000>;
///
/// FIXED_REG(u32) @ 0x100 {
/// 15:8 high_byte;
/// 7:0 low_byte;
@ -479,9 +344,14 @@ fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
/// the context:
///
/// ```no_run
/// use kernel::io::register;
/// use kernel::io::{
/// register,
/// Region,
/// };
///
/// register! {
/// base: Region<0x1000>;
///
/// /// Scratch register.
/// pub SCRATCH(u32) @ 0x00000200 {
/// 31:0 value;
@ -497,113 +367,45 @@ fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
/// In this example, `SCRATCH_BOOT_STATUS` uses the same I/O address as `SCRATCH`, while providing
/// its own `completed` field.
///
/// ## Relative registers
///
/// Relative registers can be instantiated several times at a relative offset of a group of bases.
/// For instance, imagine the following I/O space:
///
/// ```text
/// +-----------------------------+
/// | ... |
/// | |
/// 0x100--->+------------CPU0-------------+
/// | |
/// 0x110--->+-----------------------------+
/// | CPU_CTL |
/// +-----------------------------+
/// | ... |
/// | |
/// | |
/// 0x200--->+------------CPU1-------------+
/// | |
/// 0x210--->+-----------------------------+
/// | CPU_CTL |
/// +-----------------------------+
/// | ... |
/// +-----------------------------+
/// ```
///
/// `CPU0` and `CPU1` both have a `CPU_CTL` register that starts at offset `0x10` of their I/O
/// space segment. Since both instances of `CPU_CTL` share the same layout, we don't want to define
/// them twice and would prefer a way to select which one to use from a single definition.
///
/// This can be done using the `Base + Offset` syntax when specifying the register's address:
///
/// ```ignore
/// register! {
/// pub RELATIVE_REG(u32) @ Base + 0x80 {
/// ...
/// }
/// }
/// ```
///
/// This creates a register with an offset of `0x80` from a given base.
///
/// `Base` is an arbitrary type (typically a ZST) to be used as a generic parameter of the
/// [`RegisterBase`] trait to provide the base as a constant, i.e. each type providing a base for
/// this register needs to implement `RegisterBase<Base>`.
///
/// The location of relative registers can be built using the [`WithBase::of`] method to specify
/// its base. All relative registers implement [`WithBase`].
///
/// Here is the above layout translated into code:
/// If you do not wish to have a bitfield defined, you can also create a register using an existing
/// type.
///
/// ```no_run
/// use kernel::{
/// io::{
/// register,
/// register::{
/// RegisterBase,
/// WithBase,
/// },
/// Io,
/// },
/// };
/// # use kernel::io::{Mmio, Region};
///
/// // Type used to identify the base.
/// pub struct CpuCtlBase;
///
/// // ZST describing `CPU0`.
/// struct Cpu0;
/// impl RegisterBase<CpuCtlBase> for Cpu0 {
/// const BASE: usize = 0x100;
/// }
///
/// // ZST describing `CPU1`.
/// struct Cpu1;
/// impl RegisterBase<CpuCtlBase> for Cpu1 {
/// const BASE: usize = 0x200;
/// }
///
/// // This makes `CPU_CTL` accessible from all implementors of `RegisterBase<CpuCtlBase>`.
/// # use kernel::io::*;
/// register! {
/// /// CPU core control.
/// pub CPU_CTL(u32) @ CpuCtlBase + 0x10 {
/// 0:0 start;
/// base: Region<0x1000>;
///
/// /// UART RX register.
/// pub UART_RX: u8 @ 0x100;
/// }
/// ```
///
/// In case there is a fixed register associated with a specific type in the base, you can apply
/// `#[unique]` attribute which enables `write_reg` shorthand. This is automatically applied to
/// bitfields instantiated via the `register!` macro.
///
/// This should only be used when types meaningfully represent a register. For example, in the
/// previous `UART_RX` example, even if only a single register is defined with `u8` type, it is a
/// bad idea to annotate it with `#[unique]`.
///
/// ```no_run
/// # use kernel::{bitfield, io::*};
///
/// bitfield! {
/// pub struct Reset(u32) {
/// 0:0 reset;
/// }
/// }
///
/// # fn test(io: Mmio<'_, Region<0x1000>>) {
/// // Read the status of `Cpu0`.
/// let cpu0_started = io.read(CPU_CTL::of::<Cpu0>());
///
/// // Stop `Cpu0`.
/// io.write(WithBase::of::<Cpu0>(), CPU_CTL::zeroed());
/// # }
///
/// // Aliases can also be defined for relative register.
/// register! {
/// /// Alias to CPU core control.
/// pub CPU_CTL_ALIAS(u32) => CpuCtlBase + CPU_CTL {
/// /// Start the aliased CPU core.
/// 1:1 alias_start;
/// }
/// base: Region<0x1000>;
///
/// pub RESET: #[unique] Reset @ 0x100;
/// }
///
/// # fn test2(io: Mmio<'_, Region<0x1000>>) {
/// // Start the aliased `CPU0`, leaving its other fields untouched.
/// io.update(CPU_CTL_ALIAS::of::<Cpu0>(), |r| r.with_alias_start(true));
/// # fn test(mmio: Mmio<'_, Region<0x1000>>) {
/// // let mmio: Mmio<'_, Region<0x1000>>;
/// mmio.write_reg(Reset::zeroed().with_const_reset::<1>());
/// # }
/// ```
///
@ -636,15 +438,18 @@ fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
/// register,
/// register::Array,
/// Io,
/// Region,
/// },
/// };
/// # use kernel::io::{Mmio, Region};
/// # use kernel::io::Mmio;
/// # fn get_scratch_idx() -> usize {
/// # 0x15
/// # }
///
/// // Array of 64 consecutive registers with the same layout starting at offset `0x80`.
/// register! {
/// base: Region<0x1000>;
///
/// /// Scratch registers.
/// pub SCRATCH(u32)[64] @ 0x00000080 {
/// 31:0 value;
@ -670,6 +475,8 @@ fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
/// // Alias to a specific register in an array.
/// // Here `SCRATCH[8]` is used to convey the firmware exit code.
/// register! {
/// base: Region<0x1000>;
///
/// /// Firmware exit status code.
/// pub FIRMWARE_STATUS(u32) => SCRATCH[8] {
/// 7:0 status;
@ -682,6 +489,8 @@ fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
/// // Here, each of the 16 registers of the array is separated by 8 bytes, meaning that the
/// // registers of the two declarations below are interleaved.
/// register! {
/// base: Region<0x1000>;
///
/// /// Scratch registers bank 0.
/// pub SCRATCH_INTERLEAVED_0(u32)[16, stride = 8] @ 0x000000c0 {
/// 31:0 value;
@ -696,332 +505,88 @@ fn into_io_op(self) -> (FixedRegisterLoc<T>, T) {
/// # }
/// ```
///
/// ## Relative arrays of registers
/// ## Relative registers
///
/// Combining the two features described in the sections above, arrays of registers accessible from
/// a base can also be defined:
/// There are cases where a register region is subdivided into small subregions, and you may wish to
/// have your register definition be relative to these subregions. This may be needed, for example,
/// if these subregions are instantiated several times, or you just want it for encapsulation
/// purpose.
///
/// ```ignore
/// register! {
/// pub RELATIVE_REGISTER_ARRAY(u8)[10, stride = 4] @ Base + 0x100 {
/// ...
/// }
/// }
/// For instance, imagine the following I/O space:
///
/// ```text
/// +-----------------------------+
/// | ... |
/// | |
/// 0x100--->+------------CPU0-------------+
/// | |
/// 0x110--->+-----------------------------+
/// | CPU_CTL |
/// +-----------------------------+
/// | ... |
/// | |
/// | |
/// 0x200--->+------------CPU1-------------+
/// | |
/// 0x210--->+-----------------------------+
/// | CPU_CTL |
/// +-----------------------------+
/// | ... |
/// +-----------------------------+
/// ```
///
/// Like relative registers, they implement the [`WithBase`] trait. However the return value of
/// [`WithBase::of`] cannot be used directly as a location and must be further specified using the
/// [`at`](RelativeRegisterLoc::at) method.
/// `CPU0` and `CPU1` both have a `CPU_CTL` register that starts at offset `0x10` of their I/O
/// space segment. Since both instances of `CPU_CTL` share the same layout, we don't want to define
/// them twice and would prefer a way to select which one to use from a single definition.
///
/// This can be done by defining a new type for the subregion, and then defining registers that use
/// the new type as the base:
///
/// ```no_run
/// use kernel::{
/// io::{
/// io_project,
/// register,
/// register::{
/// RegisterBase,
/// WithBase,
/// },
/// Io,
/// Region,
/// },
/// };
/// # use kernel::io::{Mmio, Region};
/// # fn get_scratch_idx() -> usize {
/// # 0x15
/// # }
/// # use kernel::io::Mmio;
///
/// // Type used as parameter of `RegisterBase` to specify the base.
/// pub struct CpuCtlBase;
/// // Subregion type. Make sure it has adequate size and alignment.
/// #[repr(align(4))]
/// #[derive(FromBytes, IntoBytes)]
/// pub struct CpuCtl([u8; 0x100]);
///
/// // ZST describing `CPU0`.
/// struct Cpu0;
/// impl RegisterBase<CpuCtlBase> for Cpu0 {
/// const BASE: usize = 0x100;
/// }
///
/// // ZST describing `CPU1`.
/// struct Cpu1;
/// impl RegisterBase<CpuCtlBase> for Cpu1 {
/// const BASE: usize = 0x200;
/// }
///
/// // 64 per-cpu scratch registers, arranged as a contiguous array.
/// register! {
/// /// Per-CPU scratch registers.
/// pub CPU_SCRATCH(u32)[64] @ CpuCtlBase + 0x00000080 {
/// 31:0 value;
/// }
/// base: Region<0x1000>;
///
/// // Subregions can just be defined like normal registers.
/// CPU0: CpuCtl @ 0x100;
/// CPU1: CpuCtl @ 0x200;
/// }
///
/// # fn test(io: Mmio<'_, Region<0x1000>>) -> Result<(), Error> {
/// // Read scratch register 0 of CPU0.
/// let scratch = io.read(CPU_SCRATCH::of::<Cpu0>().at(0));
///
/// // Write the retrieved value into scratch register 15 of CPU1.
/// io.write(WithBase::of::<Cpu1>().at(15), scratch);
///
/// // This won't build.
/// // let cpu0_scratch_128 = io.read(CPU_SCRATCH::of::<Cpu0>().at(128)).value();
///
/// // Runtime-obtained array index.
/// let scratch_idx = get_scratch_idx();
/// // Access on a runtime index returns an error if it is out-of-bounds.
/// let cpu0_scratch = io.read(
/// CPU_SCRATCH::of::<Cpu0>().try_at(scratch_idx).ok_or(EINVAL)?
/// ).value();
/// # Ok(())
/// # }
///
/// // Alias to `SCRATCH[8]` used to convey the firmware exit code.
/// // Then you can define new registers on the subregion.
/// register! {
/// /// Per-CPU firmware exit status code.
/// pub CPU_FIRMWARE_STATUS(u32) => CpuCtlBase + CPU_SCRATCH[8] {
/// 7:0 status;
/// base: CpuCtl;
///
/// /// CPU core control.
/// pub CPU_CTL(u32) @ 0x10 {
/// 0:0 start;
/// }
/// }
///
/// // Non-contiguous relative register arrays can be defined by adding a stride parameter.
/// // Here, each of the 16 registers of the array is separated by 8 bytes, meaning that the
/// // registers of the two declarations below are interleaved.
/// register! {
/// /// Scratch registers bank 0.
/// pub CPU_SCRATCH_INTERLEAVED_0(u32)[16, stride = 8] @ CpuCtlBase + 0x00000d00 {
/// 31:0 value;
/// }
/// # fn test(io: Mmio<'_, Region<0x1000>>) {
/// // Read the status of `Cpu0`.
/// let cpu0_started = io_project!(io, build: CPU0).read(CPU_CTL);
///
/// /// Scratch registers bank 1.
/// pub CPU_SCRATCH_INTERLEAVED_1(u32)[16, stride = 8] @ CpuCtlBase + 0x00000d04 {
/// 31:0 value;
/// }
/// }
///
/// # fn test2(io: Mmio<'_, Region<0x1000>>) -> Result<(), Error> {
/// let cpu0_status = io.read(CPU_FIRMWARE_STATUS::of::<Cpu0>()).status();
/// # Ok(())
/// // Stop `Cpu0`.
/// io_project!(io, build: CPU0).write_reg(CPU_CTL::zeroed());
/// # }
/// ```
#[macro_export]
macro_rules! register {
// Entry point for the macro, allowing multiple registers to be defined in one call.
// It matches all possible register declaration patterns to dispatch them to corresponding
// `@reg` rule that defines a single register.
//
// TODO: change `alias:ident` to `alias:path` once relative registers are replaced by I/O
// projections.
(
$(
$(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)
$([ $size:expr $(, stride = $stride:expr)? ])?
$(@ $($base:ident +)? $offset:literal)?
$(=> $alias:ident $(+ $alias_offset:ident)? $([$alias_idx:expr])? )?
{ $($fields:tt)* }
)*
) => {
$(
$crate::register!(
@reg $(#[$attr])* $vis $name ($storage) $([$size $(, stride = $stride)?])?
$(@ $($base +)? $offset)?
$(=> $alias $(+ $alias_offset)? $([$alias_idx])? )?
{ $($fields)* }
);
)*
};
// All the rules below are private helpers.
// Creates a register at a fixed offset of the MMIO space.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) @ $offset:literal
{ $($fields:tt)* }
) => {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
$crate::register!(@io_fixed $(#[$attr])* $vis $name);
};
// Creates an alias register of fixed offset register `alias` with its own fields.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $alias:path
{ $($fields:tt)* }
) => {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(
@io_base $name($storage) @
<$alias as $crate::io::register::Register>::OFFSET
);
$crate::register!(@io_fixed $(#[$attr])* $vis $name);
};
// Creates a register at a relative offset from a base address provider.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) @ $base:ident + $offset:literal
{ $($fields:tt)* }
) => {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
$crate::register!(@io_relative $name @ $base);
};
// Creates an alias register of relative offset register `alias` with its own fields.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $base:ident + $alias:ident
{ $($fields:tt)* }
) => {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(
@io_base $name($storage) @ <$alias as $crate::io::register::Register>::OFFSET
);
$crate::register!(@io_relative $name @ $base);
};
// Creates an array of registers at a fixed offset of the MMIO space.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)
[ $size:expr, stride = $stride:expr ] @ $offset:literal { $($fields:tt)* }
) => {
$crate::build_assert::static_assert!(::core::mem::size_of::<$storage>() <= $stride);
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
$crate::register!(@io_array $name [ $size, stride = $stride ]);
};
// Shortcut for contiguous array of registers (stride == size of element).
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) [ $size:expr ] @ $offset:literal
{ $($fields:tt)* }
) => {
$crate::register!(
@reg $(#[$attr])* $vis $name($storage)
[ $size, stride = ::core::mem::size_of::<$storage>() ]
@ $offset { $($fields)* }
);
};
// Creates an alias of register `idx` of array of registers `alias` with its own fields.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $alias:path [ $idx:expr ]
{ $($fields:tt)* }
) => {
$crate::build_assert::static_assert!(
$idx < <$alias as $crate::io::register::RegisterArray>::SIZE
);
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(
@io_base $name($storage) @
<$alias as $crate::io::register::Register>::OFFSET
+ $idx * <$alias as $crate::io::register::RegisterArray>::STRIDE
);
$crate::register!(@io_fixed $(#[$attr])* $vis $name);
};
// Creates an array of registers at a relative offset from a base address provider.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)
[ $size:expr, stride = $stride:expr ]
@ $base:ident + $offset:literal { $($fields:tt)* }
) => {
$crate::build_assert::static_assert!(::core::mem::size_of::<$storage>() <= $stride);
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
$crate::register!(@io_relative_array $name [ $size, stride = $stride ] @ $base);
};
// Shortcut for contiguous array of relative registers (stride == size of element).
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) [ $size:expr ]
@ $base:ident + $offset:literal { $($fields:tt)* }
) => {
$crate::register!(
@reg $(#[$attr])* $vis $name($storage)
[ $size, stride = ::core::mem::size_of::<$storage>() ]
@ $base + $offset { $($fields)* }
);
};
// Creates an alias of register `idx` of relative array of registers `alias` with its own
// fields.
(
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)
=> $base:ident + $alias:ident [ $idx:expr ] { $($fields:tt)* }
) => {
$crate::build_assert::static_assert!(
$idx < <$alias as $crate::io::register::RegisterArray>::SIZE
);
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(
@io_base $name($storage) @
<$alias as $crate::io::register::Register>::OFFSET +
$idx * <$alias as $crate::io::register::RegisterArray>::STRIDE
);
$crate::register!(@io_relative $name @ $base);
};
// Generates the bitfield for the register.
//
// `#[allow(non_camel_case_types)]` is added since register names typically use
// `SCREAMING_CASE`.
(
@bitfield $(#[$attr:meta])* $vis:vis struct $name:ident($storage:ty) { $($fields:tt)* }
) => {
$crate::bitfield!(
#[allow(non_camel_case_types)]
$(#[$attr])* $vis struct $name($storage) { $($fields)* }
);
};
// Implementations shared by all registers types.
(@io_base $name:ident($storage:ty) @ $offset:expr) => {
impl $crate::io::register::Register for $name {
type Storage = $storage;
const OFFSET: usize = $offset;
}
};
// Implementations of fixed registers.
(@io_fixed $(#[$attr:meta])* $vis:vis $name:ident) => {
impl $crate::io::register::FixedRegister for $name {}
$(#[$attr])*
$vis const $name: $crate::io::register::FixedRegisterLoc<$name> =
$crate::io::register::FixedRegisterLoc::<$name>::new();
};
// Implementations of relative registers.
(@io_relative $name:ident @ $base:ident) => {
impl $crate::io::register::WithBase for $name {
type BaseFamily = $base;
}
impl $crate::io::register::RelativeRegister for $name {}
};
// Implementations of register arrays.
(@io_array $name:ident [ $size:expr, stride = $stride:expr ]) => {
impl $crate::io::register::Array for $name {}
impl $crate::io::register::RegisterArray for $name {
const SIZE: usize = $size;
const STRIDE: usize = $stride;
}
};
// Implementations of relative array registers.
(
@io_relative_array $name:ident [ $size:expr, stride = $stride:expr ] @ $base:ident
) => {
impl $crate::io::register::WithBase for $name {
type BaseFamily = $base;
}
impl $crate::io::register::RegisterArray for $name {
const SIZE: usize = $size;
const STRIDE: usize = $stride;
}
impl $crate::io::register::RelativeRegisterArray for $name {}
($($tt:tt)*) => {
$crate::macros::register!($($tt)*);
};
}

8
rust/kernel/io/resource.rs
View File

@ -226,10 +226,18 @@ impl Flags {
/// Resource represents a memory region that must be ioremaped using `ioremap_np`.
pub const IORESOURCE_MEM_NONPOSTED: Flags = Flags::new(bindings::IORESOURCE_MEM_NONPOSTED);
/// Memory region uses a 64-bit address (consumes two consecutive PCI resource slots).
pub const IORESOURCE_MEM_64: Flags = Flags::new(bindings::IORESOURCE_MEM_64);
// Always inline to optimize out error path of `build_assert`.
#[inline(always)]
const fn new(value: u32) -> Self {
build_assert!(value as u64 <= c_ulong::MAX as u64);
Flags(value as c_ulong)
}
/// Wrap a raw `c_ulong` value returned by a C API into [`Flags`].
pub(crate) const fn from_raw(value: c_ulong) -> Self {
Flags(value)
}
}

3
rust/kernel/lib.rs
View File

@ -67,6 +67,8 @@
pub mod device_id;
pub mod devres;
pub mod dma;
#[cfg(CONFIG_DMA_SHARED_BUFFER)]
pub mod dma_buf;
pub mod driver;
#[cfg(CONFIG_DRM = "y")]
pub mod drm;
@ -98,6 +100,7 @@
pub mod kunit;
pub mod list;
pub mod maple_tree;
pub mod mem;
pub mod miscdevice;
pub mod mm;
pub mod module;

30
rust/kernel/maple_tree.rs
View File

@ -16,7 +16,11 @@
alloc::Flags,
error::to_result,
prelude::*,
types::{ForeignOwnable, Opaque},
types::{
ForeignOwnable,
NotThreadSafe,
Opaque, //
},
};
/// A maple tree optimized for storing non-overlapping ranges.
@ -240,7 +244,10 @@ pub fn lock(&self) -> MapleGuard<'_, T> {
unsafe { bindings::spin_lock(self.ma_lock()) };
// INVARIANT: We just took the spinlock.
MapleGuard(self)
MapleGuard {
tree: self,
_not_send: NotThreadSafe,
}
}
#[inline]
@ -302,19 +309,30 @@ fn drop(mut self: Pin<&mut Self>) {
}
}
// SAFETY: `MapleTree<T>` is `Send` if `T` is `Send` because `MapleTree` owns its elements.
unsafe impl<T: ForeignOwnable + Send> Send for MapleTree<T> {}
// SAFETY: `&MapleTree<T>` allows inserting and erasing entries from any thread, so `T: Send` is
// required, and shared borrows of entries require `T: Sync`.
unsafe impl<T: ForeignOwnable + Send + Sync> Sync for MapleTree<T> {}
/// A reference to a [`MapleTree`] that owns the inner lock.
///
/// # Invariants
///
/// This guard owns the inner spinlock.
#[must_use = "if unused, the lock will be immediately unlocked"]
pub struct MapleGuard<'tree, T: ForeignOwnable>(&'tree MapleTree<T>);
pub struct MapleGuard<'tree, T: ForeignOwnable> {
tree: &'tree MapleTree<T>,
// A held spinlock must be released on the same CPU that acquired it.
_not_send: NotThreadSafe,
}
impl<'tree, T: ForeignOwnable> Drop for MapleGuard<'tree, T> {
#[inline]
fn drop(&mut self) {
// SAFETY: By the type invariants, we hold this spinlock.
unsafe { bindings::spin_unlock(self.0.ma_lock()) };
unsafe { bindings::spin_unlock(self.tree.ma_lock()) };
}
}
@ -323,7 +341,7 @@ impl<'tree, T: ForeignOwnable> MapleGuard<'tree, T> {
pub fn ma_state(&mut self, first: usize, end: usize) -> MaState<'_, T> {
// SAFETY: The `MaState` borrows this `MapleGuard`, so it can also borrow the `MapleGuard`s
// read/write permissions to the maple tree.
unsafe { MaState::new_raw(self.0, first, end) }
unsafe { MaState::new_raw(self.tree, first, end) }
}
/// Load the value at the given index.
@ -375,7 +393,7 @@ pub fn ma_state(&mut self, first: usize, end: usize) -> MaState<'_, T> {
#[inline]
pub fn load(&mut self, index: usize) -> Option<T::BorrowedMut<'_>> {
// SAFETY: `self.tree` contains a valid maple tree.
let ret = unsafe { bindings::mtree_load(self.0.tree.get(), index) };
let ret = unsafe { bindings::mtree_load(self.tree.tree.get(), index) };
if ret.is_null() {
return None;
}

234
rust/kernel/mem.rs Normal file
View File

@ -0,0 +1,234 @@
// SPDX-License-Identifier: GPL-2.0
//! Basic utilities for dealing with memory, values, and types.
use crate::prelude::*;
/// Transmute between two types.
///
/// Use this instead of [`core::mem::transmute`] when it is known that sizes are identical but this
/// cannot be proven by the compiler.
///
/// This is equivalent to Rust's `transmute_unchecked` intrinsics.
///
/// # Safety
///
/// All safety requirements of [`core::mem::transmute`] apply, plus that the size `Src` and `Dst`
/// must match.
///
/// # Examples
///
/// This can be used when types are known to have the same size, but only at runtime.
///
/// ```no_run
/// # use core::any::TypeId;
/// fn to_u32<T: 'static>(v: T) -> Option<u32> {
/// if TypeId::of::<T>() != TypeId::of::<u32>() {
/// return None;
/// }
///
/// // `core::mem::transmute` won't work here.
/// // SAFETY: We've checked that `T` is `u32`!
/// Some(unsafe { kernel::mem::transmute_unchecked(v) })
/// }
///
/// to_u32(1u32);
/// ```
#[inline(always)]
pub const unsafe fn transmute_unchecked<Src, Dst>(val: Src) -> Dst {
// SAFETY: This is identical to `transmute` except that we bypassed the size check; which is
// true per safety requirement.
unsafe { core::mem::transmute_copy(&core::mem::ManuallyDrop::new(val)) }
}
/// Version of `transmute` that performs size check at monomorphization-time.
///
/// Use this instead of [`core::mem::transmute`] when it is known that sizes are identical but this
/// cannot be proven by the compiler during type checking and can be proven during monomorphization.
///
/// The signature is equivalent to Rust standard library's unstable `transmute_neo` and that of
/// [RFC 3844](https://github.com/rust-lang/rfcs/pull/3844).
///
/// # Safety
///
/// Same as [`core::mem::transmute`].
///
/// # Examples
///
/// This is typically used in generic code where it's known that type will have the same size, but
/// the compiler cannot prove it generically.
///
/// ```no_run
/// trait IsU32 {}
/// impl IsU32 for u32 {}
///
/// fn to_u32<T: IsU32>(v: T) -> u32 {
/// // `core::mem::transmute` won't work here.
/// // SAFETY: We know that `v` is u32!
/// unsafe { kernel::mem::transmute(v) }
/// }
///
/// to_u32(1u32);
/// ```
#[inline(always)]
pub const unsafe fn transmute<Src, Dst>(val: Src) -> Dst {
const_assert!(size_of::<Src>() == size_of::<Dst>());
// SAFETY: Size is checked above. Other safety requirements follow those of the function.
unsafe { transmute_unchecked(val) }
}
/// Safely transmutes a value of one type to a value of another type of the same size.
///
/// The sizes are checked during monomorphization.
///
/// This can be considered as generic version of [`zerocopy::transmute!`] macro that defers the size
/// check and thus can be used in more cases.
///
/// # Examples
///
/// ```no_run
/// fn to_u32<T: FromBytes + IntoBytes>(v: T) -> u32 {
/// // `zerocopy::transmute!` won't work here.
/// kernel::mem::safe_transmute(v)
/// }
///
/// to_u32(1i32);
/// ```
#[inline(always)]
pub const fn safe_transmute<Src: IntoBytes, Dst: FromBytes>(val: Src) -> Dst {
// SAFETY: `transmute` is safe with `IntoBytes` and `FromBytes` bounds.
unsafe { transmute(val) }
}
/// Type that is layout-compatible with a primitive representation.
///
/// # Safety
///
/// - [`Self`] must have the same size and alignment as [`Self::Repr`].
/// - [`Self`] must be [transmutable] to [`Self::Repr`].
/// - Neither [`Self`] nor [`Self::Repr`] contains interior mutability.
///
/// The above basically says that `&Self` can be transmuted to `&Self::Repr`.
///
/// [transmutable]: core::mem::transmute
pub unsafe trait AsRepr: Sized {
/// Primitive representation of this type.
type Repr;
/// Convert from [`&Self`](Self) to [`&Self::Repr`](AsRepr::Repr).
#[inline(always)]
fn as_repr(this: &Self) -> &Self::Repr {
// SAFETY: Per safety requirement of the trait.
unsafe { core::mem::transmute(this) }
}
/// Convert from [`Self`] to [`Self::Repr`].
#[inline(always)]
fn into_repr(this: Self) -> Self::Repr {
// SAFETY: Per safety requirement of the trait.
unsafe { transmute(this) }
}
/// Convert from [`Self::Repr`] to [`Self`].
///
/// # Safety
///
/// `repr` must be a valid bit pattern of [`Self`] and satisfy type-specific invariants of it.
///
/// Alternatively, if `repr` is previously obtained using [`Self::into_repr`], and each
/// `from_repr_unchecked` should correspond to a unique `into_repr` call, then it is safe to
/// call as well (this means that we're undoing a `into_repr` call getting the exact bytes
/// back).
///
/// No guarantee is made if the result of a `into_repr` is passed to multiple
/// `from_repr_unchecked` (i.e. copies are made), to allow for cases where `Repr` is a pointer
/// and the user of the API wants ownership transfer. Users that want the ability to call
/// `from_repr_unchecked` after copying can require `Copy` bound explicitly.
#[inline(always)]
unsafe fn from_repr_unchecked(repr: Self::Repr) -> Self {
// SAFETY: Per safety requirement, `repr` is valid repr of `Self`, or it is previously from
// `into_repr`, in which case we're undoing the transmute so it is also safe.
unsafe { transmute(repr) }
}
}
/// Type that is bi-directionally transmutable with a primitive representation.
///
/// # Safety
///
/// - [`Self`] must be [transmutable] from [`Self::Repr`].
///
/// [transmutable]: core::mem::transmute
/// [`Self::Repr`]: AsRepr::Repr
pub unsafe trait AsReprMut: AsRepr {
/// Convert from `&mut Self` to [`&mut Self::Repr`](AsRepr::Repr).
#[inline(always)]
fn as_repr_mut(this: &mut Self) -> &mut Self::Repr {
// SAFETY: Per safety requirement of the trait.
unsafe { core::mem::transmute(this) }
}
/// Convert from [`Self::Repr`](AsRepr::Repr) to `Self`.
#[inline(always)]
fn from_repr(repr: Self::Repr) -> Self {
// SAFETY: Per safety requirement of the trait.
unsafe { transmute(repr) }
}
}
// SAFETY: `bool` has the same size and alignment as `u8`, and Rust guarantees that `bool` has
// only two valid bit patterns: 0 (`false`) and 1 (`true`). Thus `bool` can be transmuted to `u8`.
// Neither types contain interior mutability.
unsafe impl AsRepr for bool {
type Repr = u8;
}
// SAFETY: `*mut T` has the same size and alignment with `*const c_void`, and thus `*mut T` is
// transmutable to `*const c_void`. Neither types contain interior mutability.
unsafe impl<T> AsRepr for *mut T {
type Repr = *const c_void;
}
// SAFETY: `*mut T` is transmutable from `*const c_void`.
unsafe impl<T> AsReprMut for *mut T {}
// SAFETY: `*const T` has the same size and alignment with `*const c_void`, and is transmutable to
// `*const c_void`. Neither types contain interior mutability.
unsafe impl<T> AsRepr for *const T {
type Repr = *const c_void;
}
// SAFETY: `*const T` is transmutable from `*const c_void`.
unsafe impl<T> AsReprMut for *const T {}
macro_rules! int_impl {
($($unsigned:ident $signed:ident ,)*) => {$(
// SAFETY: `$unsigned` has the same size and alignment with itself, and is transmutable to
// itself. It does not contain interior mutability.
unsafe impl AsRepr for $unsigned {
type Repr = $unsigned;
}
// SAFETY: `$unsigned` is transmutable from itself.
unsafe impl AsReprMut for $unsigned {}
// SAFETY: `$signed` has the same size and alignment with `$unsigned`, and is transmutable
// to it Neither types contain interior mutability.
unsafe impl AsRepr for $signed {
type Repr = $unsigned;
}
// SAFETY: `$signed` is transmutable from `$unsigned`.
unsafe impl AsReprMut for $signed {}
)*};
}
int_impl! {
u8 i8,
u16 i16,
u32 i32,
u64 i64,
// `usize` is not normalized to particular integer for portability.
usize isize,
}

14
rust/kernel/pci.rs
View File

@ -17,6 +17,7 @@
from_result,
to_result, //
},
io::resource,
prelude::*,
str::CStr,
types::Opaque,
@ -439,6 +440,19 @@ pub fn resource_len(&self, bar: u32) -> Result<bindings::resource_size_t> {
Ok(unsafe { bindings::pci_resource_len(self.as_raw(), bar.try_into()?) })
}
/// Returns the resource flags (`IORESOURCE_*`) of the given PCI BAR.
pub fn resource_flags(&self, bar: u32) -> Result<resource::Flags> {
if !Bar::index_is_valid(bar) {
return Err(EINVAL);
}
// SAFETY:
// - `bar` is a valid bar number, as guaranteed by the above call to `Bar::index_is_valid`,
// - by its type invariant `self.as_raw` is always a valid pointer to a `struct pci_dev`.
let raw = unsafe { bindings::pci_resource_flags(self.as_raw(), bar.try_into()?) };
Ok(resource::Flags::from_raw(raw))
}
/// Returns the PCI class as a `Class` struct.
#[inline]
pub fn pci_class(&self) -> Class {

68
rust/kernel/pci/irq.rs
View File

@ -13,27 +13,26 @@
};
use core::num::NonZero;
/// IRQ type flags for PCI interrupt allocation.
#[derive(Debug, Clone, Copy)]
pub enum IrqType {
/// INTx interrupts.
Intx,
/// Message Signaled Interrupts (MSI).
Msi,
/// Extended Message Signaled Interrupts (MSI-X).
MsiX,
}
crate::impl_flags!(
/// Set of IRQ types that can be used for PCI interrupt allocation.
#[derive(Debug, Clone, Copy, Default)]
pub struct IrqTypes(u32);
/// IRQ type flags for PCI interrupt allocation.
#[derive(Debug, Clone, Copy)]
pub enum IrqType {
/// INTx interrupts.
Intx = bindings::PCI_IRQ_INTX,
/// Message Signaled Interrupts (MSI).
Msi = bindings::PCI_IRQ_MSI,
/// Extended Message Signaled Interrupts (MSI-X).
MsiX = bindings::PCI_IRQ_MSIX,
}
);
impl IrqType {
/// Convert to the corresponding kernel flags.
const fn as_raw(self) -> u32 {
match self {
IrqType::Intx => bindings::PCI_IRQ_INTX,
IrqType::Msi => bindings::PCI_IRQ_MSI,
IrqType::MsiX => bindings::PCI_IRQ_MSIX,
}
}
/// Construct from raw value.
#[inline]
const fn from_raw(raw: u32) -> Self {
@ -45,33 +44,10 @@ const fn from_raw(raw: u32) -> Self {
}
}
/// Set of IRQ types that can be used for PCI interrupt allocation.
#[derive(Debug, Clone, Copy, Default)]
pub struct IrqTypes(u32);
impl IrqTypes {
/// Create a set containing all IRQ types (MSI-X, MSI, and INTx).
pub const fn all() -> Self {
Self(bindings::PCI_IRQ_ALL_TYPES)
}
/// Build a set of IRQ types.
///
/// # Examples
///
/// ```ignore
/// // Create a set with only MSI and MSI-X (no INTx interrupts).
/// let msi_only = IrqTypes::default()
/// .with(IrqType::Msi)
/// .with(IrqType::MsiX);
/// ```
pub const fn with(self, irq_type: IrqType) -> Self {
Self(self.0 | irq_type.as_raw())
}
/// Get the raw flags value.
const fn as_raw(self) -> u32 {
self.0
Self(Self::all_bits())
}
}
@ -205,9 +181,7 @@ impl Device<device::Bound> {
/// let vectors = dev.alloc_irq_vectors(1, 32, pci::IrqTypes::all())?;
///
/// // Allocate MSI or MSI-X only (no INTx interrupts).
/// let msi_only = pci::IrqTypes::default()
/// .with(pci::IrqType::Msi)
/// .with(pci::IrqType::MsiX);
/// let msi_only = pci::IrqType::Msi | pci::IrqType::MsiX;
/// let vectors = dev.alloc_irq_vectors(4, 16, msi_only)?;
/// # Ok(())
/// # }
@ -224,7 +198,7 @@ pub fn alloc_irq_vectors(
// - `pci_alloc_irq_vectors` internally validates all other parameters
// and returns error codes.
let ret = unsafe {
bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, irq_types.as_raw())
bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, u32::from(irq_types))
};
to_result(ret)?;

4
rust/kernel/sync/atomic.rs
View File

@ -140,7 +140,7 @@ pub unsafe trait AtomicAdd<Rhs = Self>: AtomicType {
const fn into_repr<T: AtomicType>(v: T) -> T::Repr {
// SAFETY: Per the safety requirement of `AtomicType`, `T` is round-trip transmutable to
// `T::Repr`, therefore the transmute operation is sound.
unsafe { core::mem::transmute_copy(&v) }
unsafe { crate::mem::transmute(v) }
}
/// # Safety
@ -149,7 +149,7 @@ const fn into_repr<T: AtomicType>(v: T) -> T::Repr {
#[inline(always)]
const unsafe fn from_repr<T: AtomicType>(r: T::Repr) -> T {
// SAFETY: Per the safety requirement of the function, the transmute operation is sound.
unsafe { core::mem::transmute_copy(&r) }
unsafe { crate::mem::transmute(r) }
}
impl<T: AtomicType> Atomic<T> {

Some files were not shown because too many files have changed in this diff Show More