github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-10-22 12:52:02 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Rust changes for v7.4

Browse Source 
Toolchain and infrastructure:
 
  - Update vendored 'zerocopy' crate to v0.8.62 (released 2026-10-08).
 
    This update advances 8 releases, with 3 of them carrying most of the
    changes. It contains new features as well as fixes. However, almost
    nothing has changed that requires updates on our side (a move of our
    non-ASCII change and the addition of an SPDX license identifier).
 
  - Generate Rust KUnit test suites for doctests reusing the
    '#[kunit_tests]' and '#[test]' attributes instead of C FFI
    functions, which allows to remove the generated C glue file.
 
    Relatedly, simplify 'kunit_assert!' by using 'file!()' and 'line!()'
    inside (which doctests override as needed), which allows to remove
    the 'file' helper and 'CONFIG_RUSTC_HAS_SPAN_FILE'.
 
  - Trim the function names in doctests generated by 'rustdoc' < 1.95.0
    when building with 'O=', which were not reproducible.
 
  - Fix the 'VM_MAYSHARE' value in the generated bindings and its
    blocklist entry, due to typos.
 
  - Build 'core' with '--cfg no_io_statics' to ensure 'core' does not
    introduce any use of 'core::sync::atomic' into the kernel image
    since upstream Rust is moving 'std::io::Error' into 'core', which
    uses a global atomic pointer.
 
    Upstream Rust merged a PR to add a test checking that builds of
    'core' are warning-free with that 'cfg' enabled.
 
  - Enable the unstable 'proc_macro_span' feature in 'proc-macro2',
    which provides 'Span::join' and thus improves the quality of the
    diagnostics of all our procedural macros, notably 'pin-init'. This
    is only used for producing diagnostics, thus there is no hard
    reliance on the feature.
 
  - Globally allow 'clippy::comparison_chain', since its suggestions are
    not always ideal and upstream downgraded it to 'pedantic' too.
 
 'kernel' crate:
 
  - New 'mem' module: add the 'Align<N>' zero-sized type so that
    alignment can be specified via const generics ('repr(align(...))'
    only works with integer literals), e.g.:
 
        const N: usize = 128;
 
        struct S {
            _align: Align<N>,
        }
 
        assert_eq!(align_of::<S>(), N);
 
  - New 'ratelimit' module: add the 'Ratelimit' abstraction for 'struct
    ratelimit_state' and, based on it, the 'pr_*_ratelimited!' printing
    macros, which can be used for printing that may be triggered by
    userspace.
 
    In addition, convert Rust Binder to use them.
 
  - 'sync' module: to construct a global 'struct ratelimit_state', add a
    'const' constructor for 'raw_spinlock_t'.
 
    In addition, move the lockdep types into a new 'sync::lockdep'
    submodule, since there were starting to be too many of them.
 
  - 'num' module: add the 'cv!' macro to create constant values ("cv")
    from an integer constant expression, with validity checked at
    compile time, inferring the target type from the context. It works
    for any type that implements the new 'FromConst' trait, e.g.:
 
        let a: Alignment = cv!(4096);
        let b: Bounded<u32, 4> = cv!(15);
 
        // Any integer constant expression works, not only literals.
        let m: NonZero<usize> = cv!(usize::MAX);
 
        // Checked narrowing, including in `const` items.
        const SMALL: u8 = cv!(200u32);
 
        // The target type can be given explicitly.
        let e = cv!(200u32 => u8);
 
        // With an explicit primitive target, the expression can use
        // generic parameters.
        const fn as_u64<const KEY: u16>() -> u64 {
            cv!(KEY => u64)
        }
 
    This avoids the verbose turbofish syntax which disincentivized using
    these types at interface boundaries (e.g. 'Alignment:🆕:<8>()').
 
  - 'alloc' module:
 
      - Allow different error types in 'KBox::pin_slice()', enabling use
        of infallible initializers.
 
      - Add 'NumaNode::id()' accessor to retrieve the underlying NUMA
        node identifier.
 
  - 'time' module:
 
      - Add 'Delta::to_jiffies_timeout()' for use in timeout arguments.
 
      - Add 'Delta::as_jiffies_unsigned()'.
 
      - Make 'Delta::ZERO' generic over the time unit.
 
      - Warn and assert when passing out-of-range inputs to 'fsleep' and
        'udelay'.
 
      - Rename 'ClockSource' to 'ClockId' to better align with naming
        conventions in C kernel.
 
      - Make 'CONFIG_HZ' value available as 'bindings::HZ'.
 
      - Add implementation detail documentation for 'HrTimer'.
 
  - 'bitmap' module:
 
      - Add contiguous area operations ('next_zero_area{,_off}', 'set'
        and 'clear'), with extra checks compared to the underlying C
        code, plus additional ones under 'CONFIG_RUST_BITMAP_HARDENED'.
        It includes tests demonstrating the edge cases.
 
      - Restrict bitmap length to at most 'i32::MAX', matching
        'BitmapVec::MAX_LEN' and simplifying a few things. This also
        fixes 'u32' casts truncating in 'copy_and_extend', which could
        otherwise lead to OOB writes.
 
  - 'bug' module:
 
      - Fix 'warn_on!' on UML which called 'warn_slowpath_fmt()' with
        the wrong arguments; and fix its '// SAFETY:' comment too.
 
      - Add a KUnit test for 'warn_on!'.
 
  - 'id_pool' module: add contiguous ID reservation via the new
    'reserve_ids' and 'release_ids' methods, built on top of the new
    bitmap area operations.
 
  - 'sizes' module: add the sub-1K size constants ('SZ_1' to 'SZ_512')
    and implement 'SizeConstants' for 'Alignment', so that one can write
    e.g. 'Alignment::SZ_8'.
 
 'macros' crate:
 
  - Add '#[optional]' attribute for optional methods in '#[vtable]' to
    replace the 'build_error!(VTABLE_DEFAULT_ERROR)' boilerplate, e.g.:
 
        #[vtable]
        pub trait Operations: Sized {
            #[optional]
            fn f(&self) -> Result;
        }
 
    In addition, apply it to miscdevice and hide generated 'HAS_'
    constants for required methods.
 
  - Allow non-ASCII characters in the 'authors' field of the 'module!'
    macro, which was unintentionally restricted when multiple authors
    support was introduced.
 
 'pin-init' crate:
 
   [ Extra context from Gary's pull request:
      "This is a cycle packed with feature updates. Two of the most
       anticipated features, tuple struct support and safe self
       reference creation, has been included in this pull request.
 
       Safe self reference is a big one and is pushed a bit late in the
       cycle. The initial plan was to include a minimum subset for this
       cycle because I was worried about potential soundness issues;
       however during Plumbers it was agreed that the features should be
       upstreamed at once. This is because the initial users (Nova,
       JobQueue) would all need the advanced features, so it is better
       to ensure that the feature get put into use for real, and it can
       iterate with users, rather than being fully designed on paper.
       The existing safety checks are already very comprehensive, and
       Benno and I are confident that if new ones are discovered, it
       wouldn't affect the overall design.
 
       That said, the the usage of the advanced features (invariance,
       existential lifetime) shall be limited to the pre-agreed users
       only, so even in case where the feature needs to be reworked, the
       blast radius is limited.
 
       Another theme in this cycle is to improve the developer
       experience when things are used wrongly. A lot of work has been
       done to use proper spans from user-provided tokens so error
       message point to things that cause them. Many common misuse
       scenarios also gain additional custom diagnostics." ]
 
  - User visible changes:
 
      - Support tuple structs.
 
      - Allow safe creation of self references.
 
      - Vastly improve diagnostics for common misuse scenarios.
 
  - Other changes:
 
      - Use 'mixed_site' hygiene consistently.
 
 'MAINTAINERS':
 
  - Add 'RUST [BUG]' entry for the Rust 'bug' module (i.e. 'warn_on!'),
    which will be maintained by FUJITA Tomonori -- thank you!
 
  - Add Alice Ryhl as co-maintainer for 'Rust [ALLOC]'.
 
 And some other fixes, cleanups and improvements.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEPjU5OPd5QIZ9jqqOGXyLc2htIW0FAmrU9HoACgkQGXyLc2ht
 IW2t/A/8CWBNGMU1HlM23gKmG9z3UuOp9RdebQMQxcjKcW5+odHHLtLn5ytuKfds
 TwGHoPXBlm9iyVBTCbklJfRmVqXwGgg/Dr5PyUpFSMLPKfeTDANnseJmFzcnNtkr
 3ChqB4D1J91OWZuUbS2YqNyXYuQQHhPCfu4idi/HjqdDPDZr8bAG3RdYhydTHITu
 ahlbeisR6fyFO2ctWkxCFbbsAJ6nX37TVDF8lTKu5vxbJp5b7r4u9u2cQ2OEWrNc
 sh/n3tv7JZSoIWapOAlsDpRXX00MsPFqiO1Ytuf8JOb3kwoVDDT0Ww3dDbmkiVKN
 NEcEkYUcFRkAltscPHSos8MuElVqXQyY+ZrjCykXrFsxwKWeGXHYwBS9HBVPs49Q
 AYzdbNe+hsUfqNJo4lP/9R8U9SQTv4/0T0cvgg9W1sdYJCePuzOwuTgaFDPs6zUL
 vz/RZ6tIA0UbRQVnLnjoD8xWne+kpfDaZVinTCGqHpOBmEvPSxWwsHKj9Q5qJsC8
 1UakcpFQ3gsvXn4Od7VAL2ODrc/gxSHEG7J1Zj0625CbrSd81Ga0fvE3VxtjxpRH
 AQF9W/fnZ17pV4xBlqYZP2TR0/QNCHH8b/JCevOg1Rkafe3eqf2FjarfbrhVyOTT
 wrm9gdDVwv40j0K1tVVnOAo+ey3wjlh+Q5jqVrnu15FpMsfzxlE=
 =nyUg
 -----END PGP SIGNATURE-----

Merge tag 'rust-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/ojeda/linux

Pull Rust updates from Miguel Ojeda:
 "Toolchain and infrastructure:

   - Update vendored 'zerocopy' crate to v0.8.62 (released 2026-10-08).

     This update advances 8 releases, with 3 of them carrying most of
     the changes. It contains new features as well as fixes. However,
     almost nothing has changed that requires updates on our side (a
     move of our non-ASCII change and the addition of an SPDX license
     identifier).

   - Generate Rust KUnit test suites for doctests reusing the
     '#[kunit_tests]' and '#[test]' attributes instead of C FFI
     functions, which allows to remove the generated C glue file.

     Relatedly, simplify 'kunit_assert!' by using 'file!()' and
     'line!()' inside (which doctests override as needed), which allows
     to remove the 'file' helper and 'CONFIG_RUSTC_HAS_SPAN_FILE'.

   - Trim the function names in doctests generated by 'rustdoc' < 1.95.0
     when building with 'O=', which were not reproducible.

   - Fix the 'VM_MAYSHARE' value in the generated bindings and its
     blocklist entry, due to typos.

   - Build 'core' with '--cfg no_io_statics' to ensure 'core' does not
     introduce any use of 'core::sync::atomic' into the kernel image
     since upstream Rust is moving 'std::io::Error' into 'core', which
     uses a global atomic pointer.

     Upstream Rust merged a PR to add a test checking that builds of
     'core' are warning-free with that 'cfg' enabled.

   - Enable the unstable 'proc_macro_span' feature in 'proc-macro2',
     which provides 'Span::join' and thus improves the quality of the
     diagnostics of all our procedural macros, notably 'pin-init'. This
     is only used for producing diagnostics, thus there is no hard
     reliance on the feature.

   - Globally allow 'clippy::comparison_chain', since its suggestions
     are not always ideal and upstream downgraded it to 'pedantic' too.

  'kernel' crate:

   - New 'mem' module: add the 'Align<N>' zero-sized type so that
     alignment can be specified via const generics ('repr(align(...))'
     only works with integer literals), e.g.:

         const N: usize = 128;

         struct S {
             _align: Align<N>,
         }

         assert_eq!(align_of::<S>(), N);

   - New 'ratelimit' module: add the 'Ratelimit' abstraction for 'struct
     ratelimit_state' and, based on it, the 'pr_*_ratelimited!' printing
     macros, which can be used for printing that may be triggered by
     userspace.

     In addition, convert Rust Binder to use them.

   - 'sync' module: to construct a global 'struct ratelimit_state', add
     a 'const' constructor for 'raw_spinlock_t'.

     In addition, move the lockdep types into a new 'sync::lockdep'
     submodule, since there were starting to be too many of them.

   - 'num' module: add the 'cv!' macro to create constant values ("cv")
     from an integer constant expression, with validity checked at
     compile time, inferring the target type from the context. It works
     for any type that implements the new 'FromConst' trait, e.g.:

         let a: Alignment = cv!(4096);
         let b: Bounded<u32, 4> = cv!(15);

         // Any integer constant expression works, not only literals.
         let m: NonZero<usize> = cv!(usize::MAX);

         // Checked narrowing, including in `const` items.
         const SMALL: u8 = cv!(200u32);

         // The target type can be given explicitly.
         let e = cv!(200u32 => u8);

         // With an explicit primitive target, the expression can use
         // generic parameters.
         const fn as_u64<const KEY: u16>() -> u64 {
             cv!(KEY => u64)
         }

     This avoids the verbose turbofish syntax which disincentivized
     using these types at interface boundaries (e.g.
     'Alignment:🆕:<8>()').

   - 'alloc' module:

       - Allow different error types in 'KBox::pin_slice()', enabling use
         of infallible initializers.

       - Add 'NumaNode::id()' accessor to retrieve the underlying NUMA
         node identifier.

   - 'time' module:

       - Add 'Delta::to_jiffies_timeout()' for use in timeout arguments.

       - Add 'Delta::as_jiffies_unsigned()'.

       - Make 'Delta::ZERO' generic over the time unit.

       - Warn and assert when passing out-of-range inputs to 'fsleep' and
         'udelay'.

       - Rename 'ClockSource' to 'ClockId' to better align with naming
         conventions in C kernel.

       - Make 'CONFIG_HZ' value available as 'bindings::HZ'.

       - Add implementation detail documentation for 'HrTimer'.

   - 'bitmap' module:

       - Add contiguous area operations ('next_zero_area{,_off}', 'set'
         and 'clear'), with extra checks compared to the underlying C
         code, plus additional ones under 'CONFIG_RUST_BITMAP_HARDENED'.
         It includes tests demonstrating the edge cases.

       - Restrict bitmap length to at most 'i32::MAX', matching
         'BitmapVec::MAX_LEN' and simplifying a few things. This also
         fixes 'u32' casts truncating in 'copy_and_extend', which could
         otherwise lead to OOB writes.

   - 'bug' module:

       - Fix 'warn_on!' on UML which called 'warn_slowpath_fmt()' with
         the wrong arguments; and fix its '// SAFETY:' comment too.

       - Add a KUnit test for 'warn_on!'.

   - 'id_pool' module: add contiguous ID reservation via the new
     'reserve_ids' and 'release_ids' methods, built on top of the new
     bitmap area operations.

   - 'sizes' module: add the sub-1K size constants ('SZ_1' to 'SZ_512')
     and implement 'SizeConstants' for 'Alignment', so that one can
     write e.g. 'Alignment::SZ_8'.

  'macros' crate:

   - Add '#[optional]' attribute for optional methods in '#[vtable]' to
     replace the 'build_error!(VTABLE_DEFAULT_ERROR)' boilerplate, e.g.:

         #[vtable]
         pub trait Operations: Sized {
             #[optional]
             fn f(&self) -> Result;
         }

     In addition, apply it to miscdevice and hide generated 'HAS_'
     constants for required methods.

   - Allow non-ASCII characters in the 'authors' field of the 'module!'
     macro, which was unintentionally restricted when multiple authors
     support was introduced.

  'pin-init' crate:

    [ Extra context from Gary's pull request:
       "This is a cycle packed with feature updates. Two of the most
        anticipated features, tuple struct support and safe self
        reference creation, has been included in this pull request.

        Safe self reference is a big one and is pushed a bit late in the
        cycle. The initial plan was to include a minimum subset for this
        cycle because I was worried about potential soundness issues;
        however during Plumbers it was agreed that the features should be
        upstreamed at once. This is because the initial users (Nova,
        JobQueue) would all need the advanced features, so it is better
        to ensure that the feature get put into use for real, and it can
        iterate with users, rather than being fully designed on paper.
        The existing safety checks are already very comprehensive, and
        Benno and I are confident that if new ones are discovered, it
        wouldn't affect the overall design.

        That said, the the usage of the advanced features (invariance,
        existential lifetime) shall be limited to the pre-agreed users
        only, so even in case where the feature needs to be reworked, the
        blast radius is limited.

        Another theme in this cycle is to improve the developer
        experience when things are used wrongly. A lot of work has been
        done to use proper spans from user-provided tokens so error
        message point to things that cause them. Many common misuse
        scenarios also gain additional custom diagnostics." ]

   - User visible changes:

       - Support tuple structs.

       - Allow safe creation of self references.

       - Vastly improve diagnostics for common misuse scenarios.

   - Other changes:

       - Use 'mixed_site' hygiene consistently.

  'MAINTAINERS':

   - Add 'RUST [BUG]' entry for the Rust 'bug' module (i.e. 'warn_on!'),
     which will be maintained by FUJITA Tomonori -- thank you!

   - Add Alice Ryhl as co-maintainer for 'Rust [ALLOC]'.

  And some other fixes, cleanups and improvements.

* tag 'rust-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/ojeda/linux: (91 commits)
  rust: zerocopy: update to v0.8.62
  rust: zerocopy: fix link to upstream `README.md`
  rust: id_pool: add contiguous ID reservation
  rust: bitmap: add contiguous area operations
  rust: sizes: implement SizeConstants for Alignment
  rust: sizes: add sub-1K size constants
  rust: bitmap: restrict bitmap length to at most i32::MAX
  rust: bitmap: use function-level cfg on kunit test
  rust: num: add cv! macro to create values from constant expressions
  rust: bug: update the SAFETY comment for warn_slowpath_fmt() on UML
  rust: time: add Delta::as_jiffies_unsigned()
  rust: time: make Delta::ZERO generic over the time unit
  rust_binder: use pr_*_ratelimited! for printing
  rust: add pr_*_ratelimited! macros for printing
  rust: sync: add const constructor for raw_spinlock_t
  rust: sync: move lockdep types to rust/kernel/sync/lockdep.rs
  rust: pin-init: internal: suppress `explicit_outlives_requirements`
  MAINTAINERS: add entry for the Rust `bug` module
  rust: pin-init: internal: pin_data: support existential lifetimes
  rust: pin-init: internal: pin_data: support shared projection
  ...
This commit is contained in:
Linus Torvalds 2026-10-21 10:20:36 -10:00
commit 12d5fc96eb
248 changed files with 17273 additions and 3901 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Documentation/rust/general-information.rst
View File

@ -157,5 +157,5 @@ numerical comparisons, one may define a new Kconfig symbol:
.. code-block:: kconfig
config RUSTC_HAS_SPAN_FILE
def_bool RUSTC_VERSION >= 108800
config RUSTC_HAS_FILE_AS_C_STR
def_bool RUSTC_VERSION >= 109100

11
MAINTAINERS
View File

@ -15318,6 +15318,7 @@ F: rust/helpers/mutex.c
F: rust/helpers/spinlock.c
F: rust/kernel/sync/lock.rs
F: rust/kernel/sync/lock/
F: rust/kernel/sync/lockdep.rs
F: rust/kernel/sync/locked_by.rs
X: kernel/locking/locktorture.c
@ -24022,6 +24023,7 @@ K: \b(?i:rust)\b
RUST [ALLOC]
M: Danilo Krummrich <dakr@kernel.org>
M: Alice Ryhl <aliceryhl@google.com>
R: Lorenzo Stoakes <ljs@kernel.org>
R: Vlastimil Babka <vbabka@kernel.org>
R: Liam R. Howlett <liam@infradead.org>
@ -24039,6 +24041,15 @@ L: rust-for-linux@vger.kernel.org
S: Maintained
F: rust/kernel/bitfield.rs
RUST [BUG]
M: FUJITA Tomonori <fujita.tomonori@gmail.com>
L: rust-for-linux@vger.kernel.org
S: Maintained
F: rust/helpers/bug.c
F: rust/kernel/bug.rs
F: rust/kernel/generated_arch_reachable_asm.rs.S
F: rust/kernel/generated_arch_warn_asm.rs.S
RUST [INTEROP]
M: Joel Fernandes <joelagnelf@nvidia.com>
M: Alexandre Courbot <acourbot@nvidia.com>

2
Makefile
View File

@ -476,6 +476,7 @@ KBUILD_USERLDFLAGS := $(USERLDFLAGS)
# These flags apply to all Rust code in the tree, including the kernel and
# host programs.
#
# `-Aclippy::comparison_chain`: the lint has been demoted since 1.87.
# `-Aclippy::unwrap_or_default`: the lint is buggy [1] and ignores our
# MSRV. It can trigger depending on the optimization level.
# [1] https://github.com/rust-lang/rust-clippy/issues/17379
@ -494,6 +495,7 @@ export rust_common_flags := --edition=2021 \
-Wclippy::cast_lossless \
-Aclippy::collapsible_if \
-Aclippy::collapsible_match \
-Aclippy::comparison_chain \
-Wclippy::ignored_unit_patterns \
-Aclippy::incompatible_msrv \
-Wclippy::mut_mut \

4
drivers/android/binder/allocation.rs
View File

@ -261,7 +261,7 @@ fn drop(&mut self) {
let view = AllocationView::new(self, offsets.start);
for i in offsets.step_by(size_of::<u64>()) {
if view.cleanup_object(i).is_err() {
pr_warn!("Error cleaning up object at offset {}\n", i)
pr_warn_ratelimited!("Error cleaning up object at offset {}\n", i)
}
}
}
@ -286,7 +286,7 @@ fn drop(&mut self) {
if info.clear_on_free {
if let Err(e) = self.fill_zero() {
pr_warn!("Failed to clear data on free: {:?}", e);
pr_warn_ratelimited!("Failed to clear data on free: {:?}\n", e);
}
}
}

6
drivers/android/binder/context.rs
View File

@ -80,7 +80,7 @@ pub(crate) fn deregister(self: &Arc<Self>) {
pub(crate) fn register_process(self: &Arc<Self>, proc: Arc<Process>) -> Result {
if !Arc::ptr_eq(self, &proc.ctx) {
pr_err!("Context::register_process called on the wrong context.");
pr_err_ratelimited!("Context::register_process called on the wrong context.\n");
return Err(EINVAL);
}
self.manager.lock().all_procs.push(proc, GFP_KERNEL)?;
@ -89,7 +89,7 @@ pub(crate) fn register_process(self: &Arc<Self>, proc: Arc<Process>) -> Result {
pub(crate) fn deregister_process(self: &Arc<Self>, proc: &Arc<Process>) {
if !Arc::ptr_eq(self, &proc.ctx) {
pr_err!("Context::deregister_process called on the wrong context.");
pr_err_ratelimited!("Context::deregister_process called on the wrong context.\n");
return;
}
let mut manager = self.manager.lock();
@ -110,7 +110,7 @@ pub(crate) fn deregister_process(self: &Arc<Self>, proc: &Arc<Process>) {
pub(crate) fn set_manager_node(&self, node_ref: NodeRef) -> Result {
let mut manager = self.manager.lock();
if manager.node.is_some() {
pr_warn!("BINDER_SET_CONTEXT_MGR already set");
pr_warn_ratelimited!("BINDER_SET_CONTEXT_MGR already set\n");
return Err(EBUSY);
}
security::binder_set_context_mgr(&node_ref.node.owner.cred)?;

4
drivers/android/binder/debug.rs
View File

@ -53,7 +53,7 @@ macro_rules! binder_debug {
// Rule to explicitly specify a PID (used in kworkers).
(pid=$pid:expr, $mask:ident, $($arg:tt)*) => {
if $crate::debug::debug_mask_enabled($crate::debug::DebugMask::$mask) {
kernel::pr_info!(
kernel::pr_info_ratelimited!(
"{}: {}\n",
$pid,
kernel::prelude::fmt!($($arg)*)
@ -65,7 +65,7 @@ macro_rules! binder_debug {
($mask:ident, $($arg:tt)*) => {
if $crate::debug::debug_mask_enabled($crate::debug::DebugMask::$mask) {
let thread = kernel::current!();
kernel::pr_info!(
kernel::pr_info_ratelimited!(
"{}:{} {}\n",
thread.tgid(),
thread.pid(),

2
drivers/android/binder/freeze.rs
View File

@ -412,7 +412,7 @@ fn find_freeze_recipients(&self) -> Result<KVVec<(DArc<Node>, Arc<Process>)>, Al
recipients
.push_within_capacity(node_proc_pair)
.map_err(|_| {
pr_err!(
pr_err_ratelimited!(
"push_within_capacity failed even though we checked the capacity\n"
);
AllocError

4
drivers/android/binder/node.rs
View File

@ -405,7 +405,7 @@ pub(crate) fn update_refcount_locked(
!is_dead && !state.has_count
} else {
if state.count < count {
pr_err!("Failure: refcount underflow!");
pr_err_ratelimited!("Failure: refcount underflow!\n");
return None;
}
state.count -= count;
@ -693,7 +693,7 @@ pub(crate) fn remove_freeze_listener(&self, p: &Process) -> KVVec<Arc<Process>>
.freeze_list
.retain(|proc| !core::ptr::eq::<Process>(&**proc, p));
if len == inner.freeze_list.len() {
pr_warn!(
pr_warn_ratelimited!(
"Could not remove freeze listener for {}\n",
p.pid_in_current_ns()
);

12
drivers/android/binder/page_range.rs
View File

@ -212,7 +212,7 @@ unsafe fn set_page(me: *mut PageInfo, page: Page) {
// SAFETY: The pointer is valid for writing, so also valid for reading.
if unsafe { (*ptr).is_some() } {
pr_err!("set_page called when there is already a page");
pr_err_ratelimited!("set_page called when there is already a page\n");
// SAFETY: We will initialize the page again below.
unsafe { ptr::drop_in_place(ptr) };
}
@ -300,11 +300,11 @@ pub(crate) fn register_with_vma(&self, vma: &virt::VmaNew) -> Result<usize> {
let num_pages = num_bytes >> PAGE_SHIFT;
if !ptr::eq::<Mm>(&*self.mm, &**vma.mm()) {
pr_debug!("Failed to register with vma: invalid vma->vm_mm");
pr_debug_ratelimited!("Failed to register with vma: invalid vma->vm_mm\n");
return Err(EINVAL);
}
if num_pages == 0 {
pr_debug!("Failed to register with vma: size zero");
pr_debug_ratelimited!("Failed to register with vma: size zero\n");
return Err(EINVAL);
}
@ -325,7 +325,7 @@ pub(crate) fn register_with_vma(&self, vma: &virt::VmaNew) -> Result<usize> {
let mut inner = self.lock.lock();
if inner.size > 0 {
pr_debug!("Failed to register with vma: already registered");
pr_debug_ratelimited!("Failed to register with vma: already registered\n");
drop(inner);
return Err(EBUSY);
}
@ -380,7 +380,7 @@ pub(crate) fn use_range(&self, start: usize, end: usize) -> Result<()> {
match unsafe { self.use_page_slow(i) } {
Ok(()) => {}
Err(err) => {
pr_warn!("Error in use_page_slow: {:?}", err);
pr_warn_ratelimited!("Error in use_page_slow: {:?}\n", err);
return Err(err);
}
}
@ -529,7 +529,7 @@ unsafe fn iterate<T>(&self, mut offset: usize, mut size: usize, mut cb: T) -> Re
// duration of this call to `iterate`, so nobody will change the page.
let page = unsafe { PageInfo::get_page(page_info) };
if page.is_none() {
pr_warn!("Page is null!");
pr_warn_ratelimited!("Page is null!\n");
}
let page = page.ok_or(EFAULT)?;
cb(page, offset, available)?;

22
drivers/android/binder/process.rs
View File

@ -324,7 +324,7 @@ pub(crate) fn death_delivered(&mut self, death: DArc<NodeDeath>) {
if let Some(death) = ListArc::try_from_arc_or_drop(death) {
self.delivered_deaths.push_back(death);
} else {
pr_warn!("Notification added to `delivered_deaths` twice.");
pr_warn_ratelimited!("Notification added to `delivered_deaths` twice.\n");
}
}
@ -701,7 +701,7 @@ fn get_current_thread(self: ArcBorrow<'_, Self>) -> Result<Arc<Thread>> {
let id = {
let current = kernel::current!();
if self.task != current.group_leader() {
pr_err!("get_current_thread was called from the wrong process.");
pr_err_ratelimited!("get_current_thread was called from the wrong process.\n");
return Err(EINVAL);
}
current.pid()
@ -725,7 +725,7 @@ fn get_current_thread(self: ArcBorrow<'_, Self>) -> Result<Arc<Thread>> {
Ok(ta)
}
rbtree::Entry::Occupied(_entry) => {
pr_err!("Cannot create two threads with the same id.");
pr_err_ratelimited!("Cannot create two threads with the same id.\n");
Err(EINVAL)
}
}
@ -861,7 +861,9 @@ pub(crate) fn insert_or_update_handle(
match refs.by_handle.entry(res.as_u32()) {
rbtree::Entry::Vacant(entry) => break (res, entry),
rbtree::Entry::Occupied(_) => {
pr_err!("Detected mismatch between handle_is_present and by_handle");
pr_err_ratelimited!(
"Detected mismatch between handle_is_present and by_handle\n"
);
res.acquire();
kernel::warn_on!(true);
return Err(EINVAL);
@ -1101,7 +1103,7 @@ pub(crate) fn buffer_alloc(
) {
Ok(()) => {}
Err(err) => {
pr_warn!("use_range failure {:?}", err);
pr_warn_ratelimited!("use_range failure {:?}\n", err);
return Err(err.into());
}
}
@ -1132,7 +1134,7 @@ pub(crate) fn buffer_raw_free(&self, ptr: usize) {
let freed_range = match mapping.alloc.reservation_abort(offset) {
Ok(freed_range) => freed_range,
Err(_) => {
pr_warn!(
pr_warn_ratelimited!(
"Pointer {:x} failed to free, base = {:x}\n",
ptr,
mapping.address
@ -1154,7 +1156,7 @@ pub(crate) fn buffer_make_freeable(&self, offset: usize, mut data: Option<Alloca
let mut inner = self.inner.lock();
if let Some(ref mut mapping) = &mut inner.mapping {
if mapping.alloc.reservation_commit(offset, &mut data).is_err() {
pr_warn!("Offset {} failed to be marked freeable\n", offset);
pr_warn_ratelimited!("Offset {} failed to be marked freeable\n", offset);
}
}
}
@ -1516,7 +1518,7 @@ pub(crate) fn drop_outstanding_txn(&self) {
let wake = {
let mut inner = self.inner.lock();
if inner.outstanding_txns == 0 {
pr_err!("outstanding_txns underflow");
pr_err_ratelimited!("outstanding_txns underflow\n");
return;
}
inner.outstanding_txns -= 1;
@ -1836,7 +1838,7 @@ fn new(thread: &'a Arc<Thread>, guard: &mut Guard<'_, ProcessInner, SpinLockBack
// It is an error to hit this branch, and it should not be reachable. We try to do
// something reasonable when the failure path happens. Most likely, the thread in
// question will sleep forever.
pr_err!("Same thread registered with `ready_threads` twice.");
pr_err_ratelimited!("Same thread registered with `ready_threads` twice.\n");
}
Self { thread }
}
@ -1861,7 +1863,7 @@ impl Drop for WithNodes<'_> {
fn drop(&mut self) {
core::mem::swap(&mut self.nodes, &mut self.inner.nodes);
if self.nodes.iter().next().is_some() {
pr_err!("nodes array was modified while using lock_with_nodes\n");
pr_err_ratelimited!("nodes array was modified while using lock_with_nodes\n");
}
}
}

26
drivers/android/binder/range_alloc/tree.rs
View File

@ -166,7 +166,7 @@ pub(crate) fn reserve_new(
let (found_size, found_off, tree_node, free_tree_node) = match self.find_best_match(size) {
None => {
pr_warn!("ENOSPC from range_alloc.reserve_new - size: {}", size);
pr_warn_ratelimited!("ENOSPC from range_alloc.reserve_new - size: {}\n", size);
return Err(ENOSPC);
}
Some(desc) => {
@ -208,8 +208,8 @@ pub(crate) fn reserve_new(
pub(crate) fn reservation_abort(&mut self, offset: usize) -> Result<FreedRange> {
let mut cursor = self.tree.cursor_lower_bound_mut(&offset).ok_or_else(|| {
pr_warn!(
"EINVAL from range_alloc.reservation_abort - offset: {}",
pr_warn_ratelimited!(
"EINVAL from range_alloc.reservation_abort - offset: {}\n",
offset
);
EINVAL
@ -218,8 +218,8 @@ pub(crate) fn reservation_abort(&mut self, offset: usize) -> Result<FreedRange>
let (_, desc) = cursor.current_mut();
if desc.offset != offset {
pr_warn!(
"EINVAL from range_alloc.reservation_abort - offset: {}",
pr_warn_ratelimited!(
"EINVAL from range_alloc.reservation_abort - offset: {}\n",
offset
);
return Err(EINVAL);
@ -230,15 +230,15 @@ pub(crate) fn reservation_abort(&mut self, offset: usize) -> Result<FreedRange>
(None, Ok((reservation, free_node_res)))
}
None => {
pr_warn!(
"EINVAL from range_alloc.reservation_abort - offset: {}",
pr_warn_ratelimited!(
"EINVAL from range_alloc.reservation_abort - offset: {}\n",
offset
);
(None, Err(EINVAL))
}
allocated => {
pr_warn!(
"EPERM from range_alloc.reservation_abort - offset: {}",
pr_warn_ratelimited!(
"EPERM from range_alloc.reservation_abort - offset: {}\n",
offset
);
(allocated, Err(EPERM))
@ -327,8 +327,8 @@ pub(crate) fn reservation_commit(&mut self, offset: usize, data: &mut Option<T>)
/// Returns the size of the existing entry and the data associated with it.
pub(crate) fn reserve_existing(&mut self, offset: usize) -> Result<(usize, usize, Option<T>)> {
let desc = self.tree.get_mut(&offset).ok_or_else(|| {
pr_warn!(
"ENOENT from range_alloc.reserve_existing - offset: {}",
pr_warn_ratelimited!(
"ENOENT from range_alloc.reserve_existing - offset: {}\n",
offset
);
ENOENT
@ -344,8 +344,8 @@ pub(crate) fn reserve_existing(&mut self, offset: usize) -> Result<(usize, usize
)
}
other => {
pr_warn!(
"ENOENT from range_alloc.reserve_existing - offset: {}",
pr_warn_ratelimited!(
"ENOENT from range_alloc.reserve_existing - offset: {}\n",
offset
);
(other, Err(ENOENT))

48
drivers/android/binder/thread.rs
View File

@ -160,8 +160,8 @@ fn validate_parent_fixup(
let sg_entry = match self.sg_entries.get(sg_idx) {
Some(sg_entry) => sg_entry,
None => {
pr_err!(
"self.ancestors[{}] is {}, but self.sg_entries.len() is {}",
pr_err_ratelimited!(
"self.ancestors[{}] is {}, but self.sg_entries.len() is {}\n",
ancestors_i,
sg_idx,
self.sg_entries.len()
@ -170,8 +170,8 @@ fn validate_parent_fixup(
}
};
if sg_entry.fixup_min_offset > parent_offset {
pr_warn!(
"validate_parent_fixup: fixup_min_offset={}, parent_offset={}",
pr_warn_ratelimited!(
"validate_parent_fixup: fixup_min_offset={}, parent_offset={}\n",
sg_entry.fixup_min_offset,
parent_offset
);
@ -179,8 +179,8 @@ fn validate_parent_fixup(
}
let new_min_offset = parent_offset.checked_add(length).ok_or(EINVAL)?;
if new_min_offset > sg_entry.length {
pr_warn!(
"validate_parent_fixup: new_min_offset={}, sg_entry.length={}",
pr_warn_ratelimited!(
"validate_parent_fixup: new_min_offset={}, sg_entry.length={}\n",
new_min_offset,
sg_entry.length
);
@ -339,7 +339,7 @@ fn push_reply_work(&mut self, code: u32) -> PushWorkRes {
work.set_error_code(code);
self.push_work(work)
} else {
pr_warn!("Thread reply work is already in use.");
pr_warn_ratelimited!("Thread reply work is already in use.\n");
PushWorkRes::Ok
}
}
@ -350,7 +350,7 @@ fn push_return_work(&mut self, reply: u32) {
// Not notifying: Reply to current thread.
let _ = self.push_work(work);
} else {
pr_warn!("Thread return work is already in use.");
pr_warn_ratelimited!("Thread return work is already in use.\n");
}
}
@ -809,8 +809,8 @@ fn translate_object(
let parent_entry = match sg_state.sg_entries.get_mut(info.parent_sg_index) {
Some(parent_entry) => parent_entry,
None => {
pr_err!(
"validate_parent_fixup returned index out of bounds for sg.entries"
pr_err_ratelimited!(
"validate_parent_fixup returned index out of bounds for sg.entries\n"
);
return Err(EINVAL.into());
}
@ -856,8 +856,8 @@ fn translate_object(
let parent_entry = match sg_state.sg_entries.get_mut(info.parent_sg_index) {
Some(parent_entry) => parent_entry,
None => {
pr_err!(
"validate_parent_fixup returned index out of bounds for sg.entries"
pr_err_ratelimited!(
"validate_parent_fixup returned index out of bounds for sg.entries\n"
);
return Err(EINVAL.into());
}
@ -890,7 +890,9 @@ fn translate_object(
.read_all(&mut fda_bytes, GFP_KERNEL)?;
if fds_len != fda_bytes.len() {
pr_err!("UserSlice::read_all returned wrong length in BINDER_TYPE_FDA");
pr_err_ratelimited!(
"UserSlice::read_all returned wrong length in BINDER_TYPE_FDA\n"
);
return Err(EINVAL.into());
}
@ -1005,7 +1007,11 @@ pub(crate) fn copy_transaction_data(
let ctx = match security::SecurityCtx::from_secid(secid) {
Ok(ctx) => ctx,
Err(err) => {
pr_warn!("Failed to get security ctx for id {}: {:?}", secid, err);
pr_warn_ratelimited!(
"Failed to get security ctx for id {}: {:?}\n",
secid,
err
);
return Err(err.into());
}
};
@ -1239,7 +1245,7 @@ fn top_of_transaction_stack(&self) -> Result<Option<DArc<Transaction>>> {
let inner = self.inner.lock();
if let Some(cur) = &inner.current_transaction {
if core::ptr::eq(self, cur.from.as_ref()) {
pr_warn!("got new transaction with bad transaction stack");
pr_warn_ratelimited!("got new transaction with bad transaction stack\n");
return Err(EINVAL);
}
Ok(Some(cur.clone()))
@ -1568,7 +1574,7 @@ fn read(self: &Arc<Self>, req: &mut BinderWriteRead, wait: bool) -> Result {
let mut has_noop_placeholder = false;
if req.read_consumed == 0 {
if let Err(err) = writer.write_code(BR_NOOP) {
pr_warn!("Failure when writing BR_NOOP at beginning of buffer.");
pr_warn_ratelimited!("Failure when writing BR_NOOP at beginning of buffer.\n");
return Err(err);
}
has_noop_placeholder = true;
@ -1591,7 +1597,7 @@ fn read(self: &Arc<Self>, req: &mut BinderWriteRead, wait: bool) -> Result {
Err(err) => {
// Propagate the error if we haven't written anything else.
if err != EINTR && err != EAGAIN {
pr_warn!("Failure in work getter: {:?}", err);
pr_warn_ratelimited!("Failure in work getter: {:?}\n", err);
}
if initial_len == writer.len() {
return Err(err);
@ -1626,8 +1632,8 @@ pub(crate) fn write_read(self: &Arc<Self>, data: UserSlice, wait: bool) -> Resul
ret = self.write(&mut req);
crate::trace::trace_write_done(ret);
if let Err(err) = ret {
pr_warn!(
"Write failure {:?} in pid:{}",
pr_warn_ratelimited!(
"Write failure {:?} in pid:{}\n",
err,
self.process.pid_in_current_ns()
);
@ -1643,8 +1649,8 @@ pub(crate) fn write_read(self: &Arc<Self>, data: UserSlice, wait: bool) -> Resul
ret = self.read(&mut req, wait);
crate::trace::trace_read_done(ret);
if ret.is_err() && ret != Err(EINTR) {
pr_warn!(
"Read failure {:?} in pid:{}",
pr_warn_ratelimited!(
"Read failure {:?} in pid:{}\n",
ret,
self.process.pid_in_current_ns()
);

6
drivers/android/binder/transaction.rs
View File

@ -83,7 +83,7 @@ pub(crate) fn is_oneway(&self) -> bool {
pub(crate) fn report_netlink(&self, reply: u32, ctx: &crate::Context) {
if let Err(err) = self.report_netlink_inner(reply, ctx) {
pr_warn!(
pr_warn_ratelimited!(
"{}:{} netlink report failed: {err:?}\n",
self.from_pid,
self.from_tid
@ -183,7 +183,7 @@ pub(crate) fn new(
};
if info.is_oneway() {
if from_parent.is_some() {
pr_warn!("Oneway transaction should not be in a transaction stack.");
pr_warn_ratelimited!("Oneway transaction should not be in a transaction stack.\n");
return Err(EINVAL.into());
}
alloc.set_info_oneway_node(node_ref.node.clone());
@ -384,7 +384,7 @@ pub(crate) fn submit(self: DLArc<Self>, info: &mut TransactionInfo) -> BinderRes
return Ok(());
}
} else {
pr_err!("Failed to submit oneway transaction to node.");
pr_err_ratelimited!("Failed to submit oneway transaction to node.\n");
}
}

4
include/linux/spinlock_types_raw.h
View File

@ -11,6 +11,10 @@
#include <linux/lockdep_types.h>
/*
* Keep in sync with rust/kernel/sync/lock/spinlock.rs
*/
context_lock_struct(raw_spinlock) {
arch_spinlock_t raw_lock;
#ifdef CONFIG_DEBUG_SPINLOCK

3
init/Kconfig
View File

@ -183,9 +183,6 @@ config LD_CAN_USE_KEEP_IN_OVERLAY
# https://github.com/llvm/llvm-project/pull/130661
def_bool LD_IS_BFD || LLD_VERSION >= 210000
config RUSTC_HAS_SPAN_FILE
def_bool RUSTC_VERSION >= 108800
config RUSTC_HAS_UNNECESSARY_TRANSMUTES
def_bool RUSTC_VERSION >= 108800

1
rust/.gitignore vendored
View File

@ -3,7 +3,6 @@
bindings_generated.rs
bindings_helpers_generated.rs
doctests_kernel_generated.rs
doctests_kernel_generated_kunit.c
uapi_generated.rs
exports_*_generated.h
doc/

14
rust/Makefile
View File

@ -39,10 +39,8 @@ obj-$(CONFIG_RUST) += exports.o
always-$(CONFIG_RUST) += host/libproc_macro2.rlib host/libquote.rlib host/libsyn.rlib
always-$(CONFIG_RUST_KERNEL_DOCTESTS) += doctests_kernel_generated.rs
always-$(CONFIG_RUST_KERNEL_DOCTESTS) += doctests_kernel_generated_kunit.c
obj-$(CONFIG_RUST_KERNEL_DOCTESTS) += doctests_kernel_generated.o
obj-$(CONFIG_RUST_KERNEL_DOCTESTS) += doctests_kernel_generated_kunit.o
always-$(subst y,$(CONFIG_RUST),$(CONFIG_JUMP_LABEL)) += kernel/generated_arch_static_branch_asm.rs
ifndef CONFIG_UML
@ -53,7 +51,6 @@ endif
ifdef CONFIG_RUST
procmacro-name = $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name $(1) --crate-type proc-macro - </dev/null)
procmacro-extension := $(patsubst libname.%,%,$(call procmacro-name,name))
libzerocopy_derive_name := $(call procmacro-name,zerocopy_derive)
libmacros_name := $(call procmacro-name,macros)
@ -75,7 +72,8 @@ endif
cfgs-to-flags = $(patsubst %,--cfg='%',$1)
core-cfgs := \
no_fp_fmt_parse
no_fp_fmt_parse \
no_io_statics
core-edition := $(if $(call rustc-min-version,108700),2024,2021)
@ -85,6 +83,7 @@ core-skip_flags := \
core-flags := \
--edition=$(core-edition) \
-Aunused_variables \
$(call cfgs-to-flags,$(core-cfgs))
zerocopy-cfgs := \
@ -95,11 +94,12 @@ zerocopy-flags := \
$(call cfgs-to-flags,$(zerocopy-cfgs))
zerocopy-envs := \
CARGO_PKG_VERSION=0.8.54
CARGO_PKG_VERSION=0.8.62
proc_macro2-cfgs := \
feature="proc-macro" \
wrap_proc_macro \
proc_macro_span \
$(if $(call rustc-min-version,108800),proc_macro_span_file proc_macro_span_location)
proc_macro2-flags := \
@ -400,7 +400,7 @@ quiet_cmd_rustdoc_test_kernel = RUSTDOC TK $<
$< $(rustdoc_test_kernel_quiet); \
$(objtree)/scripts/rustdoc_test_gen
%/doctests_kernel_generated.rs %/doctests_kernel_generated_kunit.c: \
%/doctests_kernel_generated.rs: \
$(src)/kernel/lib.rs $(obj)/kernel.o \
$(objtree)/scripts/rustdoc_test_builder \
$(objtree)/scripts/rustdoc_test_gen FORCE
@ -620,7 +620,7 @@ quiet_cmd_rustc_procmacro = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET))
-Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \
--emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
--crate-type proc-macro -L$(objtree)/$(obj)/host \
--crate-name $(patsubst lib%.$(procmacro-extension),%,$(notdir $@)) \
--crate-name $(basename $(@F:lib%=%)) \
@$(objtree)/include/generated/rustc_cfg $<
# Procedural macros can only be used with the `rustc` that compiled it.

2
rust/bindgen_parameters
View File

@ -50,7 +50,7 @@
--blocklist-item VM_MAYREAD
--blocklist-item VM_MAYWRITE
--blocklist-item VM_MAYEXEC
--blocklist-item VM_MAYEXEC
--blocklist-item VM_MAYSHARE
--blocklist-item VM_PFNMAP
--blocklist-item VM_IO
--blocklist-item VM_DONTCOPY

2
rust/bindings/bindings_helper.h
View File

@ -138,7 +138,7 @@ const vm_flags_t RUST_CONST_HELPER_VM_SHARED = VM_SHARED;
const vm_flags_t RUST_CONST_HELPER_VM_MAYREAD = VM_MAYREAD;
const vm_flags_t RUST_CONST_HELPER_VM_MAYWRITE = VM_MAYWRITE;
const vm_flags_t RUST_CONST_HELPER_VM_MAYEXEC = VM_MAYEXEC;
const vm_flags_t RUST_CONST_HELPER_VM_MAYSHARE = VM_MAYEXEC;
const vm_flags_t RUST_CONST_HELPER_VM_MAYSHARE = VM_MAYSHARE;
const vm_flags_t RUST_CONST_HELPER_VM_PFNMAP = VM_PFNMAP;
const vm_flags_t RUST_CONST_HELPER_VM_IO = VM_IO;
const vm_flags_t RUST_CONST_HELPER_VM_DONTCOPY = VM_DONTCOPY;

30
rust/bindings/lib.rs
View File

@ -83,3 +83,33 @@ mod bindings_helper {
None
}
};
// `bindgen` keeps the first definition of a macro and ignores a later `#undef`.
// The generated `HZ` is therefore `__USER_HZ` from
// `include/uapi/asm-generic/param.h`, not `CONFIG_HZ`.
pub const HZ: u32 = CONFIG_HZ;
// Explicitly list architectures where this logic is checked correct.
#[cfg(any(
CONFIG_ARM,
CONFIG_ARM64,
CONFIG_LOONGARCH,
CONFIG_PPC,
CONFIG_RISCV,
CONFIG_S390,
CONFIG_UML_X86,
CONFIG_X86,
))]
pub const __ARCH_SPIN_LOCK_UNLOCKED: arch_spinlock_t = {
// SAFETY: The `arch_spinlock_t` type can be zeroed.
#[allow(unused_mut)]
let mut lock: arch_spinlock_t = unsafe { core::mem::zeroed() };
#[cfg(not(CONFIG_SMP))]
#[cfg(CONFIG_DEBUG_SPINLOCK)]
{
lock.slock = 1;
}
lock
};

2
rust/helpers/helpers.c
View File

@ -73,6 +73,7 @@
#include "kunit.c"
#include "list.c"
#include "maple_tree.c"
#include "math.c"
#include "mm.c"
#include "mutex.c"
#include "net/genetlink.c"
@ -85,6 +86,7 @@
#include "processor.c"
#include "property.c"
#include "pwm.c"
#include "ratelimit.c"
#include "rbtree.c"
#include "rcu.c"
#include "refcount.c"

8
rust/helpers/math.c Normal file
View File

@ -0,0 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/math64.h>
__rust_helper u64 rust_helper_mul_u64_add_u64_div_u64(u64 a, u64 b, u64 c, u64 d)
{
return mul_u64_add_u64_div_u64(a, b, c, d);
}

14
rust/helpers/ratelimit.c Normal file
View File

@ -0,0 +1,14 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/ratelimit.h>
__rust_helper void rust_helper_ratelimit_state_init(struct ratelimit_state *rs,
int interval, int burst)
{
ratelimit_state_init(rs, interval, burst);
}
__rust_helper void rust_helper_ratelimit_state_exit(struct ratelimit_state *rs)
{
ratelimit_state_exit(rs);
}

21
rust/kernel/Kconfig.test
View File

@ -83,4 +83,25 @@ config RUST_BITFIELD_KUNIT_TEST
If unsure, say N.
config RUST_BUG_KUNIT_TEST
bool "KUnit tests for the Rust BUG/WARN functionality" if !KUNIT_ALL_TESTS
depends on BUG
default KUNIT_ALL_TESTS
help
This option enables KUnit tests for the Rust BUG/WARN functionality.
These are only for development and testing, not for regular
kernel use cases.
If unsure, say N.
config RUST_TIME_KUNIT_TEST
bool "KUnit tests for the Rust time API" if !KUNIT_ALL_TESTS
default KUNIT_ALL_TESTS
help
This option enables KUnit tests for the Rust time API.
These are only for development and testing, not for regular
kernel use cases.
If unsure, say N.
endif

6
rust/kernel/alloc.rs
View File

@ -130,6 +130,12 @@ pub fn new(node: i32) -> Result<Self> {
}
Ok(Self(node))
}
/// Returns the raw NUMA node identifier as an `i32`.
#[inline]
pub fn id(&self) -> i32 {
self.0
}
}
/// Specify necessary constant to pass the information to Allocator that the caller doesn't care

7
rust/kernel/alloc/kbox.rs
View File

@ -345,7 +345,7 @@ pub fn pin(x: T, flags: Flags) -> Result<Pin<Box<T, A>>, AllocError>
/// }
///
/// // Allocate a boxed slice of 10 `Example`s.
/// let s = KBox::pin_slice(
/// let s = KBox::pin_slice::<_, _, Error, _>(
/// | _i | Example::new(),
/// 10,
/// GFP_KERNEL
@ -355,15 +355,16 @@ pub fn pin(x: T, flags: Flags) -> Result<Pin<Box<T, A>>, AllocError>
/// assert_eq!(s[3].d.lock().a, 20);
/// # Ok::<(), Error>(())
/// ```
pub fn pin_slice<Func, Item, E>(
pub fn pin_slice<Func, Item, E, E2>(
mut init: Func,
len: usize,
flags: Flags,
) -> Result<Pin<Box<[T], A>>, E>
where
Func: FnMut(usize) -> Item,
Item: PinInit<T, E>,
Item: PinInit<T, E2>,
E: From<AllocError>,
E: From<E2>,
{
let mut buffer = super::Vec::<T, A>::with_capacity(len, flags)?;
for i in 0..len {

1
rust/kernel/bitfield.rs
View File

@ -295,7 +295,6 @@ macro_rules! bitfield {
$(#[$attr:meta])* $vis:vis struct $name:ident($storage:ty) { $($fields:tt)* }
) => {
$crate::bitfield!(@core
#[allow(non_camel_case_types)]
$(#[$attr])* $vis $name $storage
);
$crate::bitfield!(@fields $vis $name $storage { $($fields)* });

336
rust/kernel/bitmap.rs
View File

@ -10,31 +10,70 @@
use crate::bindings;
#[cfg(not(CONFIG_RUST_BITMAP_HARDENED))]
use crate::pr_err;
use core::ptr::NonNull;
use crate::ptr::Alignment;
use core::{
num::NonZero,
ptr::NonNull, //
};
/// Represents a C bitmap. Wraps underlying C bitmap API.
///
/// # Invariants
///
/// Must reference a `[c_ulong]` long enough to fit `data.len()` bits.
/// Must not be longer than [`i32::MAX`] bits, so offsets and lengths used with
/// [`Bitmap`] functions fit in the `int` and `unsigned int` arguments of the C bitmap API.
/// This also matches [`BitmapVec::MAX_LEN`].
#[cfg_attr(CONFIG_64BIT, repr(align(8)))]
#[cfg_attr(not(CONFIG_64BIT), repr(align(4)))]
pub struct Bitmap {
data: [()],
}
macro_rules! bitmap_assert {
($cond:expr, $($arg:tt)+) => {
#[cfg(CONFIG_RUST_BITMAP_HARDENED)]
assert!($cond, $($arg)*);
}
}
macro_rules! bitmap_assert_return {
($cond:expr, $($arg:tt)+) => {
#[cfg(CONFIG_RUST_BITMAP_HARDENED)]
assert!($cond, $($arg)*);
#[cfg(not(CONFIG_RUST_BITMAP_HARDENED))]
if !($cond) {
pr_err!($($arg)*);
return
}
}
}
impl Bitmap {
/// Borrows a C bitmap.
///
/// # Panics
///
/// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and `nbits` exceeds [`i32::MAX`].
///
/// # Safety
///
/// * `ptr` holds a non-null address of an initialized array of `unsigned long`
/// that is large enough to hold `nbits` bits.
/// * `nbits` must not exceed [`i32::MAX`].
/// * the array must not be freed for the lifetime of this [`Bitmap`]
/// * concurrent access only happens through atomic operations
pub unsafe fn from_raw<'a>(ptr: *const usize, nbits: usize) -> &'a Bitmap {
bitmap_assert!(
nbits <= i32::MAX as usize,
"`nbits` must be <= {}, was {}",
i32::MAX,
nbits
);
let data: *const [()] = core::ptr::slice_from_raw_parts(ptr.cast(), nbits);
// INVARIANT: `data` references an initialized array that can hold `nbits` bits.
// INVARIANT: The caller guarantees that `nbits` does not exceed `i32::MAX`.
// SAFETY:
// The caller guarantees that `data` (derived from `ptr` and `nbits`)
// points to a valid, initialized, and appropriately sized memory region
@ -51,15 +90,27 @@ pub unsafe fn from_raw<'a>(ptr: *const usize, nbits: usize) -> &'a Bitmap {
/// Borrows a C bitmap exclusively.
///
/// # Panics
///
/// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and `nbits` exceeds [`i32::MAX`].
///
/// # Safety
///
/// * `ptr` holds a non-null address of an initialized array of `unsigned long`
/// that is large enough to hold `nbits` bits.
/// * `nbits` must not exceed [`i32::MAX`].
/// * the array must not be freed for the lifetime of this [`Bitmap`]
/// * no concurrent access may happen.
pub unsafe fn from_raw_mut<'a>(ptr: *mut usize, nbits: usize) -> &'a mut Bitmap {
bitmap_assert!(
nbits <= i32::MAX as usize,
"`nbits` must be <= {}, was {}",
i32::MAX,
nbits
);
let data: *mut [()] = core::ptr::slice_from_raw_parts_mut(ptr.cast(), nbits);
// INVARIANT: `data` references an initialized array that can hold `nbits` bits.
// INVARIANT: The caller guarantees that `nbits` does not exceed `i32::MAX`.
// SAFETY:
// The caller guarantees that `data` (derived from `ptr` and `nbits`)
// points to a valid, initialized, and appropriately sized memory region
@ -96,26 +147,6 @@ union BitmapRepr {
ptr: NonNull<usize>,
}
macro_rules! bitmap_assert {
($cond:expr, $($arg:tt)+) => {
#[cfg(CONFIG_RUST_BITMAP_HARDENED)]
assert!($cond, $($arg)*);
}
}
macro_rules! bitmap_assert_return {
($cond:expr, $($arg:tt)+) => {
#[cfg(CONFIG_RUST_BITMAP_HARDENED)]
assert!($cond, $($arg)*);
#[cfg(not(CONFIG_RUST_BITMAP_HARDENED))]
if !($cond) {
pr_err!($($arg)*);
return
}
}
}
/// Represents an owned bitmap.
///
/// Wraps underlying C bitmap API. See [`Bitmap`] for available
@ -415,7 +446,8 @@ pub fn clear_bit_atomic(&self, index: usize) {
#[inline]
pub fn copy_and_extend(&mut self, src: &Bitmap) {
let len = core::cmp::min(src.len(), self.len());
// SAFETY: access to `self` and `src` is within bounds.
// SAFETY: Access to `self` and `src` is within bounds. Both lengths fit in `u32`
// because a `Bitmap` is at most `i32::MAX` bits, so the casts are lossless.
unsafe {
bindings::bitmap_copy_and_extend(
self.as_mut_ptr(),
@ -510,13 +542,159 @@ pub fn next_zero_bit(&self, start: usize) -> Option<usize> {
Some(index)
}
}
/// Finds a contiguous area of `nbits` zero bits at or after `start`, where the area plus
/// `align_offset` is aligned to `align`.
///
/// Returns the bit index of the start of the area, or [`None`] if no such area fitting in
/// the bitmap exists.
///
/// The returned index plus `align_offset` is a multiple of `align`.
///
/// # Panics
///
/// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and `start` is out of bounds.
#[inline]
pub fn next_zero_area_off(
&self,
start: usize,
nbits: NonZero<usize>,
align: Alignment,
align_offset: usize,
) -> Option<usize> {
bitmap_assert!(
start < self.len(),
"`start` must be < {}, was {}",
self.len(),
start
);
let nr = u32::try_from(nbits.get()).ok()?;
let align_mask = align.as_usize() - 1;
// The C alignment and end arithmetic must not overflow, or it can read out of bounds.
// Overflow is only possible on 32-bit.
#[cfg(not(CONFIG_64BIT))]
align_mask
.checked_add(self.len())?
.checked_add(nbits.get())?;
// SAFETY: `bitmap_find_next_zero_area_off` is safe to use with an out of bounds `start`
// value and, given the overflow check above, never reads beyond `self.len()` bits.
let index = unsafe {
bindings::bitmap_find_next_zero_area_off(
self.as_ptr().cast_mut(),
self.len(),
start,
nr,
align_mask,
align_offset,
)
};
(index < self.len()).then_some(index)
}
/// Finds a contiguous area of `nbits` zero bits at or after `start`, aligned to `align`.
///
/// Returns the bit index of the start of the area, or [`None`] if no such area fitting in
/// the bitmap exists.
///
/// The returned index is a multiple of `align`.
///
/// # Panics
///
/// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and `start` is out of bounds.
///
/// # Examples
///
/// ```
/// use kernel::{
/// alloc::{AllocError, flags::GFP_KERNEL},
/// bitmap::BitmapVec,
/// ptr::Alignment,
/// sizes::SizeConstants, //
/// };
///
/// let mut b = BitmapVec::new(64, GFP_KERNEL)?;
///
/// assert_eq!(Some(0), b.next_zero_area(0, cv!(8), Alignment::SZ_1));
/// b.set(0, cv!(5));
/// assert_eq!(Some(5), b.next_zero_area(0, cv!(8), Alignment::SZ_1));
/// assert_eq!(Some(8), b.next_zero_area(0, cv!(8), Alignment::SZ_8));
/// assert_eq!(None, b.next_zero_area(0, cv!(65), Alignment::SZ_1));
/// # Ok::<(), AllocError>(())
/// ```
#[inline]
pub fn next_zero_area(
&self,
start: usize,
nbits: NonZero<usize>,
align: Alignment,
) -> Option<usize> {
self.next_zero_area_off(start, nbits, align, 0)
}
/// Sets a contiguous area of `nbits` bits starting at `start`.
///
/// If CONFIG_RUST_BITMAP_HARDENED is not enabled and the area `start..start + nbits` is out of
/// bounds, does nothing.
///
/// # Panics
///
/// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and the area `start..start + nbits` is out
/// of bounds.
#[inline]
pub fn set(&mut self, start: usize, nbits: NonZero<usize>) {
bitmap_assert_return!(
start
.checked_add(nbits.get())
.is_some_and(|end| end <= self.len()),
"Area `start..start + nbits` ({}..{}) must be within bounds {}",
start,
start.saturating_add(nbits.get()),
self.len()
);
// SAFETY: The area `start..start + nbits` is within bounds and a `Bitmap` is at most
// `i32::MAX` bits, so the casts are lossless.
unsafe { bindings::__bitmap_set(self.as_mut_ptr(), start as u32, nbits.get() as i32) };
}
/// Clears a contiguous area of `nbits` bits starting at `start`.
///
/// If CONFIG_RUST_BITMAP_HARDENED is not enabled and the area `start..start + nbits` is out of
/// bounds, does nothing.
///
/// # Panics
///
/// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and the area `start..start + nbits` is out
/// of bounds.
#[inline]
pub fn clear(&mut self, start: usize, nbits: NonZero<usize>) {
bitmap_assert_return!(
start
.checked_add(nbits.get())
.is_some_and(|end| end <= self.len()),
"Area `start..start + nbits` ({}..{}) must be within bounds {}",
start,
start.saturating_add(nbits.get()),
self.len()
);
// SAFETY: The area `start..start + nbits` is within bounds and a `Bitmap` is at most
// `i32::MAX` bits, so the casts are lossless.
unsafe { bindings::__bitmap_clear(self.as_mut_ptr(), start as u32, nbits.get() as i32) };
}
}
#[cfg(CONFIG_RUST_BITMAP_KUNIT_TEST)]
#[macros::kunit_tests(rust_kernel_bitmap)]
mod tests {
use super::*;
use kernel::alloc::flags::GFP_KERNEL;
use kernel::{
alloc::flags::GFP_KERNEL,
num::cv,
sizes::SizeConstants, //
};
#[test]
fn bitmap_borrow() {
@ -585,24 +763,21 @@ fn bitmap_set_clear_find() -> Result<(), AllocError> {
}
#[test]
#[cfg(not(CONFIG_RUST_BITMAP_HARDENED))]
fn owned_bitmap_out_of_bounds() -> Result<(), AllocError> {
// TODO: Kunit #[test]s do not support `cfg` yet,
// so we add it here in the body.
#[cfg(not(CONFIG_RUST_BITMAP_HARDENED))]
{
let mut b = BitmapVec::new(128, GFP_KERNEL)?;
b.set_bit(2048);
b.set_bit_atomic(2048);
b.clear_bit(2048);
b.clear_bit_atomic(2048);
assert_eq!(None, b.next_bit(2048));
assert_eq!(None, b.next_zero_bit(2048));
assert_eq!(None, b.last_bit());
}
let mut b = BitmapVec::new(128, GFP_KERNEL)?;
b.set_bit(2048);
b.set_bit_atomic(2048);
b.clear_bit(2048);
b.clear_bit_atomic(2048);
assert_eq!(None, b.next_bit(2048));
assert_eq!(None, b.next_zero_bit(2048));
assert_eq!(None, b.last_bit());
Ok(())
}
// TODO: uncomment once kunit supports [should_panic] and `cfg`.
// TODO: uncomment once kunit supports `#[should_panic]`.
// #[cfg(CONFIG_RUST_BITMAP_HARDENED)]
// #[test]
// #[should_panic]
@ -630,4 +805,91 @@ fn bitmap_copy_and_extend() -> Result<(), AllocError> {
assert_eq!(Some(17), long_bitmap.last_bit());
Ok(())
}
#[test]
fn bitmap_area_set_clear_find() -> Result<(), AllocError> {
let mut b = BitmapVec::new(128, GFP_KERNEL)?;
assert_eq!(Some(0), b.next_zero_area(0, cv!(5), Alignment::SZ_1));
b.set(0, cv!(5)); // Now contains {[0, 5)}.
assert_eq!(Some(0), b.next_bit(0));
assert_eq!(Some(4), b.next_bit(4));
assert_eq!(Some(5), b.next_zero_bit(0));
assert_eq!(Some(5), b.next_zero_area(0, cv!(5), Alignment::SZ_1));
assert_eq!(Some(8), b.next_zero_area(0, cv!(5), Alignment::SZ_8));
b.set(8, cv!(8)); // Now contains {[0, 5), [8, 16)}.
assert_eq!(Some(16), b.next_zero_area(0, cv!(4), Alignment::SZ_16));
assert_eq!(Some(16), b.next_zero_area(0, cv!(4), Alignment::SZ_1));
b.clear(0, cv!(5)); // Now contains {[8, 16)}.
assert_eq!(Some(0), b.next_zero_area(0, cv!(5), Alignment::SZ_1));
assert_eq!(Some(8), b.next_bit(0));
assert_eq!(Some(15), b.last_bit());
b.set(60, cv!(10)); // Now contains {[8, 16), [60, 70)}.
assert_eq!(Some(60), b.next_bit(16));
assert_eq!(Some(69), b.last_bit());
assert_eq!(Some(16), b.next_zero_area(9, cv!(40), Alignment::SZ_1));
assert_eq!(Some(70), b.next_zero_area(0, cv!(45), Alignment::SZ_1));
b.clear(62, cv!(6)); // Now contains {[8, 16), [60, 62), [68, 70)}.
assert_eq!(Some(62), b.next_zero_area(60, cv!(6), Alignment::SZ_1));
assert_eq!(Some(61), b.next_bit(61));
assert_eq!(Some(69), b.last_bit());
Ok(())
}
#[test]
fn bitmap_area_exhaustion() -> Result<(), AllocError> {
let mut b = BitmapVec::new(64, GFP_KERNEL)?;
assert_eq!(None, b.next_zero_area(0, cv!(65), Alignment::SZ_1));
assert_eq!(None, b.next_zero_area(0, cv!(usize::MAX), Alignment::SZ_1));
assert_eq!(None, b.next_zero_area(1, cv!(usize::MAX), Alignment::SZ_1));
b.set_bit(0); // Now contains {[0, 1)}.
assert_eq!(None, b.next_zero_area(0, cv!(usize::MAX), Alignment::SZ_1));
b.set(0, cv!(61)); // Now contains {[0, 61)}.
assert_eq!(None, b.next_zero_area(0, cv!(4), Alignment::SZ_1));
assert_eq!(Some(61), b.next_zero_area(0, cv!(3), Alignment::SZ_1));
assert_eq!(None, b.next_zero_area(0, cv!(1), Alignment::SZ_64));
Ok(())
}
#[test]
fn bitmap_area_off() -> Result<(), AllocError> {
let mut b = BitmapVec::new(64, GFP_KERNEL)?;
b.set(0, cv!(5)); // Now contains {[0, 5)}.
// The area plus align_offset starts at a multiple of the alignment.
assert_eq!(Some(7), b.next_zero_area_off(0, cv!(8), Alignment::SZ_8, 1));
assert_eq!(Some(5), b.next_zero_area_off(0, cv!(8), Alignment::SZ_8, 3));
// A zero offset behaves like next_zero_area().
assert_eq!(
b.next_zero_area(0, cv!(8), Alignment::SZ_8),
b.next_zero_area_off(0, cv!(8), Alignment::SZ_8, 0)
);
Ok(())
}
#[test]
#[cfg(not(CONFIG_RUST_BITMAP_HARDENED))]
fn owned_bitmap_area_out_of_bounds() -> Result<(), AllocError> {
let mut b = BitmapVec::new(64, GFP_KERNEL)?;
// Should be ignored since out of bounds.
b.set(64, cv!(4));
b.set(62, cv!(8));
b.set(usize::MAX, cv!(1));
b.clear(usize::MAX, cv!(1));
b.clear(2048, cv!(8));
assert_eq!(None, b.next_bit(0));
assert_eq!(None, b.next_zero_area(64, cv!(1), Alignment::SZ_1));
Ok(())
}
}

39
rust/kernel/bug.rs
View File

@ -85,15 +85,18 @@ macro_rules! warn_flags {
($file:expr, $flags:expr) => {
if false {
_ = $file;
_ = $flags;
}
// SAFETY: It is always safe to call `warn_slowpath_fmt()`
// with a valid null-terminated string.
// SAFETY:
// - The file name is a null-terminated string with `'static` lifetime.
// - `TAINT_WARN` is a valid taint number.
// - `fmt` can be NULL, and NULL is passed.
unsafe {
$crate::bindings::warn_slowpath_fmt(
$crate::str::CStrExt::as_char_ptr($crate::c_str!(::core::file!())),
line!() as $crate::ffi::c_int,
$flags as $crate::ffi::c_uint,
$crate::bindings::TAINT_WARN,
::core::ptr::null(),
);
}
@ -152,3 +155,33 @@ macro_rules! warn_on {
cond
}};
}
#[cfg(CONFIG_RUST_BUG_KUNIT_TEST)]
#[macros::kunit_tests(rust_kernel_bug)]
mod tests {
// The counter is incremented by the kernel warning path, not by `warn_on!`
// itself. A count of one means the warning was really reported.
#[test]
fn test_warn_on() {
// SAFETY: `kunit_get_current_test()` is always safe to call (it has
// fallbacks for when no KUnit test is running).
let test = unsafe { bindings::kunit_get_current_test() };
// SAFETY: This function runs only as a KUnit test case, so `test` is a
// valid pointer to the running test.
let handle = unsafe { bindings::kunit_start_suppress_warning(test) };
assert!(!warn_on!(false));
assert!(warn_on!(true));
// SAFETY: `kunit_suppressed_warning_count()` accepts any value returned by
// `kunit_start_suppress_warning()`.
let suppressed_count = unsafe { bindings::kunit_suppressed_warning_count(handle) };
// SAFETY: `test` is valid as above. `kunit_end_suppress_warning()` accepts any
// value returned by `kunit_start_suppress_warning()`.
unsafe { bindings::kunit_end_suppress_warning(test, handle) };
assert_eq!(suppressed_count, 1);
}
}

32
rust/kernel/id_pool.rs
View File

@ -4,8 +4,14 @@
//! Rust API for an ID pool backed by a [`BitmapVec`].
use core::{
num::NonZero,
ops::Range, //
};
use crate::alloc::{AllocError, Flags};
use crate::bitmap::BitmapVec;
use crate::ptr::Alignment;
/// Represents a dynamic ID pool backed by a [`BitmapVec`].
///
@ -253,6 +259,32 @@ pub fn find_unused_id(&mut self, offset: usize) -> Option<UnusedId<'_>> {
pub fn release_id(&mut self, id: usize) {
self.map.clear_bit(id);
}
/// Reserves a contiguous area of `count` IDs at or after `offset`.
///
/// The start of the returned area is a multiple of `align`.
///
/// Returns the reserved range upon success, or [`None`] if no such area could be found.
#[inline]
#[must_use]
pub fn reserve_ids(
&mut self,
offset: usize,
count: NonZero<usize>,
align: Alignment,
) -> Option<Range<usize>> {
let start = self.map.next_zero_area(offset, count, align)?;
self.map.set(start, count);
Some(start..start + count.get())
}
/// Releases a contiguous area of IDs.
#[inline]
pub fn release_ids(&mut self, range: &Range<usize>) {
if let Some(nbits) = NonZero::new(range.len()) {
self.map.clear(range.start, nbits);
}
}
}
/// Represents an unused id in an [`IdPool`].

12
rust/kernel/kunit.rs
View File

@ -58,15 +58,17 @@ pub fn info(args: fmt::Arguments<'_>) {
#[doc(hidden)]
#[macro_export]
macro_rules! kunit_assert {
($name:literal, $file:literal, $diff:expr, $condition:expr $(,)?) => {
($name:literal, $condition:expr $(,)?) => {
'out: {
// Do nothing if the condition is `true`.
if $condition {
break 'out;
}
static FILE: &'static $crate::str::CStr = $file;
static LINE: i32 = ::core::line!() as i32 - $diff;
// Use `file!()` instead of `::core::file!()` here so it can be overridden.
static FILE: &'static $crate::str::CStr = $crate::c_str!(file!());
// Use `line!()` instead of `::core::line!()` here so it can be overridden.
static LINE: i32 = line!() as i32;
static CONDITION: &'static $crate::str::CStr = $crate::c_str!(stringify!($condition));
// SAFETY: FFI call without safety requirements.
@ -164,10 +166,10 @@ unsafe impl Sync for UnaryAssert {}
#[doc(hidden)]
#[macro_export]
macro_rules! kunit_assert_eq {
($name:literal, $file:literal, $diff:expr, $left:expr, $right:expr $(,)?) => {{
($name:literal, $left:expr, $right:expr $(,)?) => {{
// For the moment, we just forward to the expression assert because, for binary asserts,
// KUnit supports only a few types (e.g. integers).
$crate::kunit_assert!($name, $file, $diff, $left == $right);
$crate::kunit_assert!($name, $left == $right);
}};
}

2
rust/kernel/lib.rs
View File

@ -98,6 +98,7 @@
pub mod kunit;
pub mod list;
pub mod maple_tree;
pub mod mem;
pub mod miscdevice;
pub mod mm;
pub mod module;
@ -119,6 +120,7 @@
pub mod ptr;
#[cfg(CONFIG_RUST_PWM_ABSTRACTIONS)]
pub mod pwm;
pub mod ratelimit;
pub mod rbtree;
pub mod regulator;
pub mod revocable;

70
rust/kernel/mem.rs Normal file
View File

@ -0,0 +1,70 @@
// SPDX-License-Identifier: GPL-2.0
//! Basic utilities for dealing with memory, values, and types.
// TODO: `Alignment` should be moved here as well, Rust moved it upstream in 1.96.
pub use crate::ptr::Alignment;
/// Trait indicating that [`Align<N>`] is a valid alignment.
pub trait ValidAlign {
#[doc(hidden)]
type Repr: Copy;
}
/// Zero-sized type that provides the alignment specified via the generic parameter.
///
/// # Examples
///
/// ```
/// # use kernel::mem::Align;
/// const ALIGN: usize = 128;
/// struct MyStruct {
/// _align: Align<ALIGN>,
/// }
/// assert_eq!(align_of::<MyStruct>(), ALIGN);
/// ```
#[repr(transparent)]
#[derive(Default, Clone, Copy)]
pub struct Align<const N: usize>([<Self as ValidAlign>::Repr; 0])
where
Self: ValidAlign;
impl<const N: usize> Align<N>
where
Self: ValidAlign,
{
/// Create a new [`Align<N>`].
#[inline]
pub const fn new() -> Self {
Align([])
}
}
macro_rules! impl_align {
() => {};
($a:literal $($rest:literal)*) => {
const _: () = {
#[repr(align($a))]
#[derive(Clone, Copy)]
pub struct Repr;
impl ValidAlign for Align<$a> {
type Repr = Repr;
}
};
impl_align!($($rest)*);
}
}
impl_align!(1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536);
impl<const N: usize> core::fmt::Debug for Align<N>
where
Self: ValidAlign,
{
#[inline]
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
write!(f, "Align<{N}>")
}
}

35
rust/kernel/miscdevice.rs
View File

@ -11,10 +11,7 @@
use crate::{
bindings,
device::Device,
error::{
to_result,
VTABLE_DEFAULT_ERROR, //
},
error::to_result,
fs::{
File,
Kiocb, //
@ -145,37 +142,33 @@ fn release(device: Self::Ptr, _file: &File) {
/// the VMA initialization by calling methods of `vma`. If the function does not return an
/// error, the kernel will complete initialization of the VMA according to the properties of
/// `vma`.
#[optional]
fn mmap(
_device: <Self::Ptr as ForeignOwnable>::Borrowed<'_>,
_file: &File,
_vma: &VmaNew,
) -> Result {
build_error!(VTABLE_DEFAULT_ERROR)
}
) -> Result;
/// Read from this miscdevice.
fn read_iter(_kiocb: Kiocb<'_, Self::Ptr>, _iov: &mut IovIterDest<'_>) -> Result<usize> {
build_error!(VTABLE_DEFAULT_ERROR)
}
#[optional]
fn read_iter(_kiocb: Kiocb<'_, Self::Ptr>, _iov: &mut IovIterDest<'_>) -> Result<usize>;
/// Write to this miscdevice.
fn write_iter(_kiocb: Kiocb<'_, Self::Ptr>, _iov: &mut IovIterSource<'_>) -> Result<usize> {
build_error!(VTABLE_DEFAULT_ERROR)
}
#[optional]
fn write_iter(_kiocb: Kiocb<'_, Self::Ptr>, _iov: &mut IovIterSource<'_>) -> Result<usize>;
/// Handler for ioctls.
///
/// The `cmd` argument is usually manipulated using the utilities in [`kernel::ioctl`].
///
/// [`kernel::ioctl`]: mod@crate::ioctl
#[optional]
fn ioctl(
_device: <Self::Ptr as ForeignOwnable>::Borrowed<'_>,
_file: &File,
_cmd: u32,
_arg: usize,
) -> Result<isize> {
build_error!(VTABLE_DEFAULT_ERROR)
}
) -> Result<isize>;
/// Handler for ioctls.
///
@ -185,23 +178,21 @@ fn ioctl(
/// that have different layout on 32-bit and 64-bit userspace. If no implementation is
/// provided, then `compat_ptr_ioctl` will be used instead.
#[cfg(CONFIG_COMPAT)]
#[optional]
fn compat_ioctl(
_device: <Self::Ptr as ForeignOwnable>::Borrowed<'_>,
_file: &File,
_cmd: u32,
_arg: usize,
) -> Result<isize> {
build_error!(VTABLE_DEFAULT_ERROR)
}
) -> Result<isize>;
/// Show info for this fd.
#[optional]
fn show_fdinfo(
_device: <Self::Ptr as ForeignOwnable>::Borrowed<'_>,
_m: &SeqFile,
_file: &File,
) {
build_error!(VTABLE_DEFAULT_ERROR)
}
);
}
/// A vtable for the file operations of a Rust miscdevice.

149
rust/kernel/num.rs
View File

@ -86,3 +86,152 @@ impl Integer for $type {
i128: Signed,
isize: Signed
);
/// Creates a constant value ("cv") from an integer constant expression, with validity checked at
/// compile time.
///
/// This works for any type that implements [`FromConst`], with the target type inferred from
/// the context, or named explicitly with `cv!(value => Type)`. It can also be used in const
/// contexts.
///
/// # Pitfalls
///
/// Only the target type is inferred from the context. The expression passed to [`cv!`] is typed by
/// itself. So, unsuffixed integer literal expressions default to [`i32`]. For example
/// `cv!(2 << 31)` does not evaluate to the same value as `cv!(2i64 << 31)`, since `2 << 31` is
/// treated as an [`i32`] before being widened to [`i128`] internally. Literals and arithmetic
/// expressions that overflow [`i32`] are a compile error, it's just left shifts that require care.
///
/// # Examples
///
/// ```
/// use core::num::NonZero;
///
/// use kernel::{
/// num::Bounded,
/// ptr::Alignment, //
/// };
///
/// let v: NonZero<usize> = cv!(8);
/// assert_eq!(v.get(), 8);
///
/// // Any integer constant expression works, not only literals.
/// let m: NonZero<usize> = cv!(usize::MAX);
/// assert_eq!(m.get(), usize::MAX);
///
/// let b: Bounded<u32, 4> = cv!(15);
/// assert_eq!(b.get(), 15);
///
/// let a: Alignment = cv!(4096);
/// assert_eq!(a.as_usize(), 4096);
///
/// // Checked narrowing of integer constants, including in `const` items.
/// const SMALL: u8 = cv!(200u32);
/// assert_eq!(SMALL, 200);
///
/// const N: NonZero<u8> = cv!(5);
/// assert_eq!(N.get(), 5);
///
/// // The target type can be given explicitly.
/// let e = cv!(200u32 => u8);
/// assert_eq!(e, 200);
///
/// // With an explicit primitive target, the expression can use generic parameters.
/// const fn as_u64<const KEY: u16>() -> u64 {
/// cv!(KEY => u64)
/// }
/// assert_eq!(as_u64::<0x40>(), 0x40);
/// ```
#[macro_export]
#[doc(hidden)]
macro_rules! cv {
(@cast $v:expr => $t:ty) => {
const {
#[allow(unused_comparisons, unused_assignments, clippy::as_underscore)]
{
let v = $v;
let r = v as $t;
// Pin `back` to `v`'s type so `as _` casts back to the source type.
let mut back = v;
back = r as _;
::core::assert!(
back == v && (v < 0) == (r < 0),
"value does not fit into the target type"
);
r
}
}
};
// Using `FromConst<V>::VALUE` means const generic expressions can't be used, since it requires
// the `generic_const_exprs` feature. Provide a special path for each primitive type that allows
// const generic expressions.
($v:expr => u8) => { $crate::cv!(@cast $v => u8) };
($v:expr => u16) => { $crate::cv!(@cast $v => u16) };
($v:expr => u32) => { $crate::cv!(@cast $v => u32) };
($v:expr => u64) => { $crate::cv!(@cast $v => u64) };
($v:expr => u128) => { $crate::cv!(@cast $v => u128) };
($v:expr => usize) => { $crate::cv!(@cast $v => usize) };
($v:expr => i8) => { $crate::cv!(@cast $v => i8) };
($v:expr => i16) => { $crate::cv!(@cast $v => i16) };
($v:expr => i32) => { $crate::cv!(@cast $v => i32) };
($v:expr => i64) => { $crate::cv!(@cast $v => i64) };
($v:expr => i128) => { $crate::cv!(@cast $v => i128) };
($v:expr => isize) => { $crate::cv!(@cast $v => isize) };
($v:expr => $t:ty) => {
<$t as $crate::num::FromConst<{ $crate::cv!(@cast $v => i128) }>>::VALUE
};
($v:expr) => {
<_ as $crate::num::FromConst<{ $crate::cv!(@cast $v => i128) }>>::VALUE
};
}
#[doc(inline)]
pub use cv;
/// Types that can be created from an integer constant expression validated at compile time.
///
/// Implement this trait to make a type usable with [`cv!`]. Use the [`cv!`] macro, not this trait
/// directly, for creating values.
#[diagnostic::on_unimplemented(message = "`{Self}` cannot be converted from a constant")]
pub trait FromConst<const V: i128>: Sized {
/// The value that corresponds to the constant `V`.
///
/// Fails the build if `V` is not a valid value for [`Self`].
const VALUE: Self;
}
/// Implements [`FromConst`] for primitive integer types and their [`NonZero`](core::num::NonZero)
/// versions.
macro_rules! impl_from_const {
($($type:ty)*) => {
$(
impl<const V: i128> FromConst<V> for $type {
const VALUE: Self = {
// CAST: The macro is only used for types up to 64 bits wide, so `MIN` and `MAX`
// widen to `i128` losslessly.
assert!(
V >= <$type>::MIN as i128 && V <= <$type>::MAX as i128,
"constant cannot be represented by the target type"
);
// CAST: The assert above confirmed that `V` fits in `$type`.
V as $type
};
}
impl<const V: i128> FromConst<V> for core::num::NonZero<$type> {
const VALUE: Self = match core::num::NonZero::new(<$type as FromConst<V>>::VALUE) {
Some(value) => value,
None => panic!("constant cannot be zero"),
};
}
)*
};
}
impl_from_const!(
u8 u16 u32 u64 usize
i8 i16 i32 i64 isize
);

16
rust/kernel/num/bounded.rs
View File

@ -14,6 +14,7 @@
use kernel::{
num::{
FromConst,
Integer,
Unsigned, //
},
@ -272,6 +273,21 @@ pub const fn new<const VALUE: $type>() -> Self {
unsafe { Self::__new(VALUE) }
}
}
impl<const N: u32, const V: i128> FromConst<V> for Bounded<$type, N> {
const VALUE: Self = {
let value = <$type as FromConst<V>>::VALUE;
// Statically assert that `value` fits within the set number of bits.
assert!(
fits_within!(value, $type, N),
"constant cannot be represented within the given number of bits"
);
// SAFETY: The assert above confirmed that `value` can be represented within `N`
// bits.
unsafe { Self::__new(value) }
};
}
)*
};
}

9
rust/kernel/prelude.rs
View File

@ -106,14 +106,23 @@
Result, //
},
init::InPlaceInit,
num::cv,
pr_alert,
pr_alert_ratelimited,
pr_crit,
pr_crit_ratelimited,
pr_debug,
pr_debug_ratelimited,
pr_emerg,
pr_emerg_ratelimited,
pr_err,
pr_err_ratelimited,
pr_info,
pr_info_ratelimited,
pr_notice,
pr_notice_ratelimited,
pr_warn,
pr_warn_ratelimited,
str::CStrExt as _,
try_init,
try_pin_init,

12
rust/kernel/ptr.rs
View File

@ -11,7 +11,10 @@
};
use core::num::NonZero;
use crate::const_assert;
use crate::{
const_assert,
num::FromConst, //
};
/// Type representing an alignment, which is always a power of two.
///
@ -166,6 +169,13 @@ pub const fn mask(self) -> usize {
}
}
impl<const V: i128> FromConst<V> for Alignment {
const VALUE: Self = match Alignment::new_checked(<usize as FromConst<V>>::VALUE) {
Some(alignment) => alignment,
None => panic!("constant is not a power of two"),
};
}
/// Trait for items that can be aligned against an [`Alignment`].
pub trait Alignable: Sized {
/// Aligns `self` down to `alignment`.

201
rust/kernel/ratelimit.rs Normal file
View File

@ -0,0 +1,201 @@
// SPDX-License-Identifier: GPL-2.0
//! Rate limiting support.
//!
//! C header: [`include/linux/ratelimit.h`](srctree/include/linux/ratelimit.h)
use crate::{
bindings,
prelude::*,
types::Opaque, //
};
/// Defines a `static` containing a [`Ratelimit`].
#[macro_export]
macro_rules! ratelimit_state_init {
($name:ident, $interval:expr, $burst:expr $(,)?) => {
#[allow(non_upper_case_globals)]
static $name: $crate::ratelimit::Ratelimit = {
let name = $crate::c_str!(::core::stringify!($name));
let interval = $interval;
let burst = $burst;
// SAFETY: This will be stored in static memory.
unsafe { $crate::ratelimit::Ratelimit::new_static(name, interval, burst) }
};
};
}
pub use ratelimit_state_init;
/// Rate limiter state.
///
/// # Invariants
///
/// The `inner` field contains an initialized `struct ratelimit_state`.
#[pin_data(PinnedDrop)]
#[repr(transparent)]
pub struct Ratelimit {
#[pin]
inner: Opaque<bindings::ratelimit_state>,
}
// SAFETY: `Ratelimit` is safe to be sent to any task.
unsafe impl Send for Ratelimit {}
// SAFETY: `Ratelimit` is safe to be accessed concurrently as it is protected by an internal
// spinlock.
unsafe impl Sync for Ratelimit {}
impl Ratelimit {
/// Constructs a [`Ratelimit`] with the specified configuration.
///
/// If `interval` is zero, then no rate limit is applied.
#[inline]
pub fn new(interval: i32, burst: i32) -> impl PinInit<Self> {
// INVARIANT: This creates a `Ratelimit` containing an initialized `struct ratelimit_state`
pin_init!(Self {
inner <- Opaque::ffi_init(|slot: *mut bindings::ratelimit_state| {
// SAFETY: `slot` is a valid pointer to an uninitialized `struct ratelimit_state`.
// The memory is pinned so it remains valid until `ratelimit_state_exit` is called.
unsafe { bindings::ratelimit_state_init(slot, interval, burst) };
}),
})
}
/// Constructs a [`Ratelimit`] with the default configuration.
#[inline]
pub fn new_default() -> impl PinInit<Self> {
Ratelimit::new(Ratelimit::DEFAULT_INTERVAL, Ratelimit::DEFAULT_BURST)
}
/// Constructs a [`Ratelimit`] with the specified configuration.
///
/// The name will be used for the lockdep name of the internal spinlock. See [`Self::new`] for
/// the meaning of `interval` and `burst`.
///
/// # Safety
///
/// The resulting value must be stored in static memory.
#[inline]
pub const unsafe fn new_static(name: &'static CStr, interval: i32, burst: i32) -> Self {
// INVARIANT: This creates a `Ratelimit` containing an initialized `struct ratelimit_state`.
Self {
inner: Opaque::new(bindings::ratelimit_state {
lock: kernel::sync::lock::spinlock::raw_spin_lock_unlocked(name),
interval,
burst,
..pin_init::zeroed()
}),
}
}
/// The default interval used for rate limiting.
pub const DEFAULT_INTERVAL: i32 = bindings::DEFAULT_RATELIMIT_INTERVAL as i32;
/// The default burst size.
pub const DEFAULT_BURST: i32 = bindings::DEFAULT_RATELIMIT_BURST as i32;
/// Check if an action should be rate-limited.
///
/// Returns [`true`] if the action is allowed, and [`false`] if it should be suppressed.
#[inline]
pub fn ratelimit(&self, func: &CStr) -> bool {
// SAFETY: `self.inner.get()` is a valid pointer to a `struct ratelimit_state`.
// The lifetime of `func` ensures the pointer remains valid for the duration of the call.
// The C function `___ratelimit` handles its own internal locking, so it is safe to call
// concurrently.
unsafe { bindings::___ratelimit(self.inner.get(), func.as_char_ptr()) != 0 }
}
}
#[pinned_drop]
impl PinnedDrop for Ratelimit {
#[inline]
fn drop(self: Pin<&mut Self>) {
// SAFETY: By the type invariants, this struct contains an initialized `struct
// ratelimit_state`.
unsafe { bindings::ratelimit_state_exit(self.inner.get()) };
}
}
/// Helper macro to implement ratelimited printing.
#[macro_export]
#[doc(hidden)]
macro_rules! print_ratelimited {
($print_macro:ident, $($arg:tt)*) => {{
$crate::ratelimit::ratelimit_state_init!(
_rs,
$crate::ratelimit::Ratelimit::DEFAULT_INTERVAL,
$crate::ratelimit::Ratelimit::DEFAULT_BURST,
);
let loc = $crate::c_str!(::core::concat!(::core::file!(), ":", ::core::line!()));
if $crate::ratelimit::Ratelimit::ratelimit(&_rs, loc) {
$crate::$print_macro!($($arg)*);
}
}};
}
/// Prints an emergency-level message (level 0) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_emerg_ratelimited (
($($arg:tt)*) => (
$crate::print_ratelimited!(pr_emerg, $($arg)*)
)
);
/// Prints an alert-level message (level 1) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_alert_ratelimited (
($($arg:tt)*) => (
$crate::print_ratelimited!(pr_alert, $($arg)*)
)
);
/// Prints a critical-level message (level 2) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_crit_ratelimited (
($($arg:tt)*) => (
$crate::print_ratelimited!(pr_crit, $($arg)*)
)
);
/// Prints an error-level message (level 3) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_err_ratelimited (
($($arg:tt)*) => (
$crate::print_ratelimited!(pr_err, $($arg)*)
)
);
/// Prints a warning-level message (level 4) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_warn_ratelimited (
($($arg:tt)*) => (
$crate::print_ratelimited!(pr_warn, $($arg)*)
)
);
/// Prints a notice-level message (level 5) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_notice_ratelimited (
($($arg:tt)*) => (
$crate::print_ratelimited!(pr_notice, $($arg)*)
)
);
/// Prints an info-level message (level 6) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_info_ratelimited (
($($arg:tt)*) => (
$crate::print_ratelimited!(pr_info, $($arg)*)
)
);
/// Prints a debug-level message (level 7) if allowed by a rate limiter.
#[macro_export]
macro_rules! pr_debug_ratelimited (
($($arg:tt)*) => (
if cfg!(debug_assertions) {
$crate::print_ratelimited!(pr_debug, $($arg)*)
}
)
);

47
rust/kernel/sizes.rs
View File

@ -13,6 +13,11 @@
//! these constants as [`u64`] (or [`u32`]) rather than [`usize`], because
//! device address spaces are sized independently of the CPU pointer width.
//!
//! The trait is also implemented for [`Alignment`], providing each size as a
//! compile-time validated alignment.
//!
//! [`Alignment`]: crate::ptr::Alignment
//!
//! # Examples
//!
//! ```
@ -35,6 +40,26 @@
macro_rules! define_sizes {
($($type:ty),* $(,)?) => {
define_sizes!(@internal [$($type),*]
/// `0x0000_0001`.
SZ_1,
/// `0x0000_0002`.
SZ_2,
/// `0x0000_0004`.
SZ_4,
/// `0x0000_0008`.
SZ_8,
/// `0x0000_0010`.
SZ_16,
/// `0x0000_0020`.
SZ_32,
/// `0x0000_0040`.
SZ_64,
/// `0x0000_0080`.
SZ_128,
/// `0x0000_0100`.
SZ_256,
/// `0x0000_0200`.
SZ_512,
/// `0x0000_0400`.
SZ_1K,
/// `0x0000_0800`.
@ -85,6 +110,7 @@ macro_rules! define_sizes {
(@internal [$($type:ty),*] $($names_and_metas:tt)*) => {
define_sizes!(@consts_and_trait $($names_and_metas)*);
define_sizes!(@impls [$($type),*] $($names_and_metas)*);
define_sizes!(@impl_alignment $($names_and_metas)*);
};
(@consts_and_trait $($(#[$meta:meta])* $name:ident,)*) => {
@ -99,13 +125,22 @@ macro_rules! define_sizes {
/// choose the width that matches their hardware. All `SZ_*` values fit
/// in a [`u32`], so all implementations are lossless.
///
/// Also implemented for [`Alignment`], providing each size as a
/// compile-time validated alignment.
///
/// [`Alignment`]: crate::ptr::Alignment
///
/// # Examples
///
/// ```
/// use kernel::sizes::SizeConstants;
/// use kernel::{
/// ptr::Alignment,
/// sizes::SizeConstants, //
/// };
///
/// let gpu_heap = 14 * u64::SZ_1M;
/// let mmio_window = u32::SZ_16M;
/// let page_align = Alignment::SZ_4K;
/// ```
pub trait SizeConstants {
$(
@ -117,6 +152,16 @@ pub trait SizeConstants {
(@impls [] $($(#[$meta:meta])* $name:ident,)*) => {};
(@impl_alignment $($(#[$meta:meta])* $name:ident,)*) => {
impl SizeConstants for crate::ptr::Alignment {
$(
$(#[$meta])*
// A non-power-of-two constant will fail the build here if used.
const $name: Self = crate::ptr::Alignment::new::<{ self::$name }>();
)*
}
};
(@impls [$first:ty $(, $rest:ty)*] $($(#[$meta:meta])* $name:ident,)*) => {
impl SizeConstants for $first {
$(

2
rust/kernel/str.rs
View File

@ -411,7 +411,7 @@ fn as_ref(&self) -> &BStr {
/// const BAD: &CStr = c_str!("literal");
///
/// // `c_str!` is still needed for static non-literal C strings.
/// const GOOD: &CStr = c_str!(concat!(file!(), ":", line!(), ": My CStr!"));
/// const GOOD: &CStr = c_str!(concat!(file!(), ":", ::core::line!(), ": My CStr!"));
/// ```
#[macro_export]
macro_rules! c_str {

138
rust/kernel/sync.rs
View File

@ -5,10 +5,6 @@
//! This module contains the kernel APIs related to synchronisation that have been ported or
//! wrapped for usage by Rust code in the kernel.
use crate::prelude::*;
use crate::types::Opaque;
use pin_init;
mod arc;
pub mod aref;
pub mod atomic;
@ -16,6 +12,7 @@
pub mod completion;
mod condvar;
pub mod lock;
pub mod lockdep;
mod locked_by;
pub mod poll;
pub mod rcu;
@ -36,136 +33,11 @@
SpinLockIrq,
SpinLockIrqGuard, //
};
pub use lockdep::{
static_lock_class,
LockClassKey, //
};
pub use locked_by::LockedBy;
pub use refcount::Refcount;
pub use set_once::SetOnce;
pub use srcu::Srcu;
/// Represents a lockdep class.
///
/// Wraps the kernel's `struct lock_class_key`.
#[repr(transparent)]
#[pin_data(PinnedDrop)]
pub struct LockClassKey {
#[pin]
inner: Opaque<bindings::lock_class_key>,
}
// SAFETY: Unregistering a lock class key from a different thread than where it was registered is
// allowed.
unsafe impl Send for LockClassKey {}
// SAFETY: `bindings::lock_class_key` is designed to be used concurrently from multiple threads and
// provides its own synchronization.
unsafe impl Sync for LockClassKey {}
impl LockClassKey {
/// Initializes a statically allocated lock class key.
///
/// This is usually used indirectly through the [`static_lock_class!`] macro. See its
/// documentation for more information.
///
/// # Safety
///
/// * Before using the returned value, it must be pinned in a static memory location.
/// * The destructor must never run on the returned `LockClassKey`.
pub const unsafe fn new_static() -> Self {
LockClassKey {
inner: Opaque::uninit(),
}
}
/// Initializes a dynamically allocated lock class key.
///
/// In the common case of using a statically allocated lock class key, the
/// [`static_lock_class!`] macro should be used instead.
///
/// # Examples
///
/// ```
/// use kernel::alloc::KBox;
/// use kernel::types::ForeignOwnable;
/// use kernel::sync::{LockClassKey, SpinLock};
/// use pin_init::stack_pin_init;
///
/// let key = KBox::pin_init(LockClassKey::new_dynamic(), GFP_KERNEL)?;
/// let key_ptr = key.into_foreign();
///
/// {
/// stack_pin_init!(let num: SpinLock<u32> = SpinLock::new(
/// 0,
/// c"my_spinlock",
/// // SAFETY: `key_ptr` is returned by the above `into_foreign()`, whose
/// // `from_foreign()` has not yet been called.
/// unsafe { <Pin<KBox<LockClassKey>> as ForeignOwnable>::borrow(key_ptr) }
/// ));
/// }
///
/// // SAFETY: We dropped `num`, the only use of the key, so the result of the previous
/// // `borrow` has also been dropped. Thus, it's safe to use from_foreign.
/// unsafe { drop(<Pin<KBox<LockClassKey>> as ForeignOwnable>::from_foreign(key_ptr)) };
/// # Ok::<(), Error>(())
/// ```
pub fn new_dynamic() -> impl PinInit<Self> {
pin_init!(Self {
// SAFETY: lockdep_register_key expects an uninitialized block of memory
inner <- Opaque::ffi_init(|slot| unsafe { bindings::lockdep_register_key(slot) })
})
}
/// Returns a raw pointer to the inner C struct.
///
/// It is up to the caller to use the raw pointer correctly.
pub fn as_ptr(&self) -> *mut bindings::lock_class_key {
self.inner.get()
}
}
#[pinned_drop]
impl PinnedDrop for LockClassKey {
fn drop(self: Pin<&mut Self>) {
// SAFETY: `self.as_ptr()` was registered with lockdep and `self` is pinned, so the address
// hasn't changed. Thus, it's safe to pass it to unregister.
unsafe { bindings::lockdep_unregister_key(self.as_ptr()) }
}
}
/// Defines a new static lock class and returns a pointer to it.
///
/// # Examples
///
/// ```
/// use kernel::sync::{static_lock_class, Arc, SpinLock};
///
/// fn new_locked_int() -> Result<Arc<SpinLock<u32>>> {
/// Arc::pin_init(SpinLock::new(
/// 42,
/// c"new_locked_int",
/// static_lock_class!(),
/// ), GFP_KERNEL)
/// }
/// ```
#[macro_export]
macro_rules! static_lock_class {
() => {{
static CLASS: $crate::sync::LockClassKey =
// SAFETY: The returned `LockClassKey` is stored in static memory and we pin it. Drop
// never runs on a static global.
unsafe { $crate::sync::LockClassKey::new_static() };
$crate::prelude::Pin::static_ref(&CLASS)
}};
}
pub use static_lock_class;
/// Returns the given string, if one is provided, otherwise generates one based on the source code
/// location.
#[doc(hidden)]
#[macro_export]
macro_rules! optional_name {
() => {
$crate::c_str!(::core::concat!(::core::file!(), ":", ::core::line!()))
};
($name:literal) => {
$crate::c_str!($name)
};
}

28
rust/kernel/sync/lock/spinlock.rs
View File

@ -463,3 +463,31 @@ fn spinlock_irq_condvar() -> Result {
Ok(())
}
}
/// Helper for creating a raw unlocked `bindings::raw_spinlock_t`.
///
/// For use in statics containing raw spinlocks.
#[doc(alias("__SPIN_LOCK_UNLOCKED", "DEFINE_SPINLOCK"))]
pub(crate) const fn raw_spin_lock_unlocked(name: &'static CStr) -> bindings::raw_spinlock_t {
// Silence unused variable warnings.
#[cfg(not(CONFIG_DEBUG_LOCK_ALLOC))]
let _ = name;
bindings::raw_spinlock_t {
raw_lock: bindings::__ARCH_SPIN_LOCK_UNLOCKED,
#[cfg(CONFIG_DEBUG_SPINLOCK)]
magic: bindings::SPINLOCK_MAGIC,
#[cfg(CONFIG_DEBUG_SPINLOCK)]
owner_cpu: u32::MAX,
#[cfg(CONFIG_DEBUG_SPINLOCK)]
owner: usize::MAX as *mut c_void,
#[cfg(CONFIG_DEBUG_LOCK_ALLOC)]
dep_map: kernel::sync::lockdep::raw_lockdep_map(
name,
kernel::sync::lockdep::LD_WAIT_SPIN,
kernel::sync::lockdep::LD_WAIT_INV,
),
}
}

161
rust/kernel/sync/lockdep.rs Normal file
View File

@ -0,0 +1,161 @@
// SPDX-License-Identifier: GPL-2.0
//! Utilities related to lockdep.
//!
//! C headers: [`include/linux/lockdep.h`](srctree/include/linux/lockdep.h)
use crate::{
prelude::*,
types::Opaque, //
};
/// Represents a lockdep class.
///
/// Wraps the kernel's `struct lock_class_key`.
#[repr(transparent)]
#[pin_data(PinnedDrop)]
pub struct LockClassKey {
#[pin]
inner: Opaque<bindings::lock_class_key>,
}
// SAFETY: Unregistering a lock class key from a different thread than where it was registered is
// allowed.
unsafe impl Send for LockClassKey {}
// SAFETY: `bindings::lock_class_key` is designed to be used concurrently from multiple threads and
// provides its own synchronization.
unsafe impl Sync for LockClassKey {}
impl LockClassKey {
/// Initializes a statically allocated lock class key.
///
/// This is usually used indirectly through the [`static_lock_class!`] macro. See its
/// documentation for more information.
///
/// # Safety
///
/// * Before using the returned value, it must be pinned in a static memory location.
/// * The destructor must never run on the returned `LockClassKey`.
pub const unsafe fn new_static() -> Self {
LockClassKey {
inner: Opaque::uninit(),
}
}
/// Initializes a dynamically allocated lock class key.
///
/// In the common case of using a statically allocated lock class key, the
/// [`static_lock_class!`] macro should be used instead.
///
/// # Examples
///
/// ```
/// use kernel::alloc::KBox;
/// use kernel::types::ForeignOwnable;
/// use kernel::sync::{LockClassKey, SpinLock};
/// use pin_init::stack_pin_init;
///
/// let key = KBox::pin_init(LockClassKey::new_dynamic(), GFP_KERNEL)?;
/// let key_ptr = key.into_foreign();
///
/// {
/// stack_pin_init!(let num: SpinLock<u32> = SpinLock::new(
/// 0,
/// c"my_spinlock",
/// // SAFETY: `key_ptr` is returned by the above `into_foreign()`, whose
/// // `from_foreign()` has not yet been called.
/// unsafe { <Pin<KBox<LockClassKey>> as ForeignOwnable>::borrow(key_ptr) }
/// ));
/// }
///
/// // SAFETY: We dropped `num`, the only use of the key, so the result of the previous
/// // `borrow` has also been dropped. Thus, it's safe to use from_foreign.
/// unsafe { drop(<Pin<KBox<LockClassKey>> as ForeignOwnable>::from_foreign(key_ptr)) };
/// # Ok::<(), Error>(())
/// ```
pub fn new_dynamic() -> impl PinInit<Self> {
pin_init!(Self {
// SAFETY: lockdep_register_key expects an uninitialized block of memory
inner <- Opaque::ffi_init(|slot| unsafe { bindings::lockdep_register_key(slot) })
})
}
/// Returns a raw pointer to the inner C struct.
///
/// It is up to the caller to use the raw pointer correctly.
pub fn as_ptr(&self) -> *mut bindings::lock_class_key {
self.inner.get()
}
}
#[pinned_drop]
impl PinnedDrop for LockClassKey {
fn drop(self: Pin<&mut Self>) {
// SAFETY: `self.as_ptr()` was registered with lockdep and `self` is pinned, so the address
// hasn't changed. Thus, it's safe to pass it to unregister.
unsafe { bindings::lockdep_unregister_key(self.as_ptr()) }
}
}
/// Defines a new static lock class and returns a pointer to it.
///
/// # Examples
///
/// ```
/// use kernel::sync::{static_lock_class, Arc, SpinLock};
///
/// fn new_locked_int() -> Result<Arc<SpinLock<u32>>> {
/// Arc::pin_init(SpinLock::new(
/// 42,
/// c"new_locked_int",
/// static_lock_class!(),
/// ), GFP_KERNEL)
/// }
/// ```
#[macro_export]
macro_rules! static_lock_class {
() => {{
static CLASS: $crate::sync::LockClassKey =
// SAFETY: The returned `LockClassKey` is stored in static memory and we pin it. Drop
// never runs on a static global.
unsafe { $crate::sync::LockClassKey::new_static() };
$crate::prelude::Pin::static_ref(&CLASS)
}};
}
pub use static_lock_class;
/// Returns the given string, if one is provided, otherwise generates one based on the source code
/// location.
#[doc(hidden)]
#[macro_export]
macro_rules! optional_name {
() => {
$crate::c_str!(::core::concat!(::core::file!(), ":", ::core::line!()))
};
($name:literal) => {
$crate::c_str!($name)
};
}
/// Not checked, catch all.
pub const LD_WAIT_INV: u8 = bindings::lockdep_wait_type_LD_WAIT_INV as u8;
/// Spin loops, `raw_spinlock_t` etc
pub const LD_WAIT_SPIN: u8 = bindings::lockdep_wait_type_LD_WAIT_SPIN as u8;
/// Helper for declaring a raw `struct lockdep_map` for locks in statics.
///
/// It's up to the caller to use the returned `struct lockdep_map` correctly.
#[cfg(CONFIG_DEBUG_LOCK_ALLOC)]
pub(crate) const fn raw_lockdep_map(
name: &'static CStr,
wait_type_inner: u8,
wait_type_outer: u8,
) -> bindings::lockdep_map {
// SAFETY: All zeros is valid for this type.
let mut map: bindings::lockdep_map = unsafe { core::mem::zeroed() };
map.name = kernel::str::as_char_ptr_in_const_context(name);
map.wait_type_inner = wait_type_inner;
map.wait_type_outer = wait_type_outer;
map
}

173
rust/kernel/time.rs
View File

@ -39,6 +39,10 @@
/// The number of nanoseconds per second.
pub const NSEC_PER_SEC: i64 = bindings::NSEC_PER_SEC as i64;
/// The C side `MAX_JIFFY_OFFSET`, i.e. `((LONG_MAX >> 1) - 1)`. It is the upper
/// bound the kernel uses for a jiffies span, not a wait-forever value.
const MAX_JIFFY_OFFSET: isize = (isize::MAX >> 1) - 1;
/// The time unit of Linux kernel. One jiffy equals (1/HZ) second.
pub type Jiffies = crate::ffi::c_ulong;
@ -53,9 +57,9 @@ pub fn msecs_to_jiffies(msecs: Msecs) -> Jiffies {
unsafe { bindings::__msecs_to_jiffies(msecs) }
}
/// Trait for clock sources.
/// Trait for kernel clock identifiers.
///
/// Selection of the clock source depends on the use case. In some cases the usage of a
/// Selection of the clock depends on the use case. In some cases the usage of a
/// particular clock is mandatory, e.g. in network protocols, filesystems. In other
/// cases the user of the clock has to decide which clock is best suited for the
/// purpose. In most scenarios clock [`Monotonic`] is the best choice as it
@ -66,13 +70,13 @@ pub fn msecs_to_jiffies(msecs: Msecs) -> Jiffies {
/// Implementers must ensure that `ktime_get()` returns a value in the inclusive range
/// `0..=KTIME_MAX` (i.e., greater than or equal to 0 and less than or equal to
/// `KTIME_MAX`, where `KTIME_MAX` equals `i64::MAX`).
pub unsafe trait ClockSource {
/// The kernel clock ID associated with this clock source.
pub unsafe trait ClockId {
/// The kernel clock ID associated with this clock.
///
/// This constant corresponds to the C side `clockid_t` value.
const ID: bindings::clockid_t;
/// Get the current time from the clock source.
/// Get the current time from the clock.
///
/// The function must return a value in the range `0..=KTIME_MAX`.
fn ktime_get() -> bindings::ktime_t;
@ -93,7 +97,7 @@ pub unsafe trait ClockSource {
// SAFETY: The kernel's `ktime_get()` is guaranteed to return a value
// in `0..=KTIME_MAX`.
unsafe impl ClockSource for Monotonic {
unsafe impl ClockId for Monotonic {
const ID: bindings::clockid_t = bindings::CLOCK_MONOTONIC as bindings::clockid_t;
fn ktime_get() -> bindings::ktime_t {
@ -120,7 +124,7 @@ fn ktime_get() -> bindings::ktime_t {
// SAFETY: The kernel's `ktime_get_real()` is guaranteed to return a value
// in `0..=KTIME_MAX`.
unsafe impl ClockSource for RealTime {
unsafe impl ClockId for RealTime {
const ID: bindings::clockid_t = bindings::CLOCK_REALTIME as bindings::clockid_t;
fn ktime_get() -> bindings::ktime_t {
@ -140,7 +144,7 @@ fn ktime_get() -> bindings::ktime_t {
// SAFETY: The kernel's `ktime_get_boottime()` is guaranteed to return a value
// in `0..=KTIME_MAX`.
unsafe impl ClockSource for BootTime {
unsafe impl ClockId for BootTime {
const ID: bindings::clockid_t = bindings::CLOCK_BOOTTIME as bindings::clockid_t;
fn ktime_get() -> bindings::ktime_t {
@ -164,7 +168,7 @@ fn ktime_get() -> bindings::ktime_t {
// SAFETY: The kernel's `ktime_get_clocktai()` is guaranteed to return a value
// in `0..=KTIME_MAX`.
unsafe impl ClockSource for Tai {
unsafe impl ClockId for Tai {
const ID: bindings::clockid_t = bindings::CLOCK_TAI as bindings::clockid_t;
fn ktime_get() -> bindings::ktime_t {
@ -180,24 +184,24 @@ fn ktime_get() -> bindings::ktime_t {
/// The `inner` value is in the range from 0 to `KTIME_MAX`.
#[repr(transparent)]
#[derive(PartialEq, PartialOrd, Eq, Ord)]
pub struct Instant<C: ClockSource> {
pub struct Instant<C: ClockId> {
inner: bindings::ktime_t,
_c: PhantomData<C>,
}
impl<C: ClockSource> Clone for Instant<C> {
impl<C: ClockId> Clone for Instant<C> {
fn clone(&self) -> Self {
*self
}
}
impl<C: ClockSource> Copy for Instant<C> {}
impl<C: ClockId> Copy for Instant<C> {}
impl<C: ClockSource> Instant<C> {
impl<C: ClockId> Instant<C> {
/// Get the current time from the clock source.
#[inline]
pub fn now() -> Self {
// INVARIANT: The `ClockSource::ktime_get()` function returns a value in the range
// INVARIANT: The `ClockId::ktime_get()` function returns a value in the range
// from 0 to `KTIME_MAX`.
Self {
inner: C::ktime_get(),
@ -239,7 +243,7 @@ pub(crate) unsafe fn from_ktime(ktime: bindings::ktime_t) -> Self {
}
}
impl<C: ClockSource> ops::Sub for Instant<C> {
impl<C: ClockId> ops::Sub for Instant<C> {
type Output = Delta;
// By the type invariant, it never overflows.
@ -251,7 +255,7 @@ fn sub(self, other: Instant<C>) -> Delta {
}
}
impl<T: ClockSource> ops::Add<Delta> for Instant<T> {
impl<T: ClockId> ops::Add<Delta> for Instant<T> {
type Output = Self;
#[inline]
@ -271,7 +275,7 @@ fn add(self, rhs: Delta) -> Self::Output {
}
}
impl<T: ClockSource> ops::Sub<Delta> for Instant<T> {
impl<T: ClockId> ops::Sub<Delta> for Instant<T> {
type Output = Self;
#[inline]
@ -302,6 +306,9 @@ impl Sealed for super::Jiffy {}
pub trait TimeUnit: private::Sealed {
/// The underlying representation of the time unit.
type Repr: Copy + Clone + PartialEq + PartialOrd + Eq + Ord + core::fmt::Debug;
/// The zero value of [`Self::Repr`].
const ZERO: Self::Repr;
}
/// A time unit of nanoseconds.
@ -313,6 +320,7 @@ pub enum Nsec {}
impl TimeUnit for Nsec {
type Repr = i64;
const ZERO: Self::Repr = 0;
}
/// A time unit of jiffies.
@ -324,6 +332,7 @@ pub enum Jiffy {}
impl TimeUnit for Jiffy {
type Repr = isize;
const ZERO: Self::Repr = 0;
}
/// A span of time.
@ -337,6 +346,11 @@ pub struct Delta<U: TimeUnit = Nsec> {
value: U::Repr,
}
impl<U: TimeUnit> Delta<U> {
/// A span of time equal to zero.
pub const ZERO: Self = Self { value: U::ZERO };
}
impl Delta<Jiffy> {
/// Create a new [`Delta`] from a number of jiffies.
#[inline]
@ -349,6 +363,27 @@ pub const fn from_jiffies(jiffies: isize) -> Self {
pub const fn as_jiffies(self) -> isize {
self.value
}
/// Return the number of jiffies in the [`Delta`] as an unsigned value.
///
/// If the number of jiffies is negative, zero is returned.
///
/// # Examples
///
/// ```
/// use kernel::time::Delta;
///
/// // A negative span returns zero.
/// assert_eq!(Delta::from_jiffies(-1).as_jiffies_unsigned(), 0);
///
/// // A positive span returns its value.
/// assert_eq!(Delta::from_jiffies(1).as_jiffies_unsigned(), 1);
/// ```
#[inline]
pub fn as_jiffies_unsigned(self) -> usize {
// CAST: `isize::max()` makes the value non-negative, so the cast keeps it.
isize::max(self.value, 0) as usize
}
}
impl ops::Add for Delta {
@ -424,9 +459,6 @@ fn div(self, rhs: Self) -> Self::Output {
}
impl Delta {
/// A span of time equal to zero.
pub const ZERO: Self = Self { value: 0 };
/// Create a new [`Delta`] from a number of nanoseconds.
#[inline]
pub const fn from_nanos(nanos: i64) -> Self {
@ -554,6 +586,62 @@ pub fn as_millis_ceil(self) -> i64 {
}
}
/// Convert this span to a [`Delta<Jiffy>`] suitable for use as a timeout.
///
/// Unless the result saturates, the value is rounded up to the next whole
/// jiffy, so the resulting timeout is never shorter than `self`.
///
/// A negative span saturates at zero jiffies, i.e. an immediate timeout.
///
/// A span that does not fit saturates at the kernel's [`MAX_JIFFY_OFFSET`],
/// the upper bound for a jiffies span. That is a finite timeout, so a
/// saturated result can be shorter than the requested span. It is derived
/// from `long`, so only 32 bit can reach it, at about 12 days with `HZ=1000`.
///
/// # Examples
///
/// ```
/// use kernel::time::Delta;
///
/// // A negative span is an immediate timeout.
/// assert_eq!(Delta::from_millis(-1).to_jiffies_timeout().as_jiffies(), 0);
///
/// // A span shorter than a jiffy still waits, i.e. the timeout is never
/// // shorter than the span.
/// assert!(Delta::from_nanos(1).to_jiffies_timeout().as_jiffies() >= 1);
/// ```
///
/// [`MAX_JIFFY_OFFSET`]: srctree/include/linux/jiffies.h
#[inline]
pub fn to_jiffies_timeout(self) -> Delta<Jiffy> {
const HZ: u64 = bindings::HZ as u64;
// The quotient `(nsecs * HZ + NSEC_PER_SEC - 1) / NSEC_PER_SEC` has to fit in
// `u64`; `nsecs * HZ` does not. With `HZ <= NSEC_PER_SEC` the numerator is at
// most `(nsecs + 1) * NSEC_PER_SEC - 1`, so the quotient is at most `nsecs`.
crate::static_assert!(HZ <= NSEC_PER_SEC as u64);
// CAST: `i64::max()` makes the value non-negative, so the cast keeps it.
let nsecs = i64::max(self.as_nanos(), 0) as u64;
// SAFETY: `mul_u64_add_u64_div_u64()` must not be called with a zero divisor,
// and its result must fit in `u64`. `NSEC_PER_SEC` is a non-zero constant, and
// the assertion above bounds the quotient by `nsecs`.
let jiffies = unsafe {
bindings::mul_u64_add_u64_div_u64(
nsecs,
HZ,
(NSEC_PER_SEC - 1) as u64,
NSEC_PER_SEC as u64,
)
};
// CAST: `jiffies` is clamped to `MAX_JIFFY_OFFSET`, which is `<= isize::MAX`.
let jiffies = u64::min(jiffies, MAX_JIFFY_OFFSET as u64) as isize;
Delta::<Jiffy>::from_jiffies(jiffies)
}
/// Return `self % dividend` where `dividend` is in nanoseconds.
///
/// The kernel doesn't have any emulation for `s64 % s64` on 32 bit platforms, so this is
@ -580,3 +668,48 @@ pub fn rem_nanos(self, dividend: i32) -> Self {
}
}
}
#[cfg(CONFIG_RUST_TIME_KUNIT_TEST)]
#[macros::kunit_tests(rust_kernel_time)]
mod tests {
use super::*;
#[test]
fn to_jiffies_timeout_converts() {
const HZ: isize = bindings::HZ as isize;
// One second is exactly `HZ` jiffies, and the round-up must not add one.
assert_eq!(Delta::from_secs(1).to_jiffies_timeout().as_jiffies(), HZ);
// One nanosecond more has to round up to the next whole jiffy.
assert_eq!(
Delta::from_nanos(NSEC_PER_SEC + 1)
.to_jiffies_timeout()
.as_jiffies(),
HZ + 1
);
}
#[test]
fn to_jiffies_timeout_saturates() {
// The result never exceeds `MAX_JIFFY_OFFSET`. On 32 bit with `HZ=1000` this
// span is 2147483647 jiffies, so the clamp is what keeps it in range; on 64
// bit it fits and the check holds for every possible return value.
let clamped = Delta::from_millis(i64::from(i32::MAX)).to_jiffies_timeout();
assert!(clamped.as_jiffies() <= MAX_JIFFY_OFFSET);
// `MAX_JIFFY_OFFSET` is derived from `long`, so only 32 bit can reach it. On
// 64 bit `i64::MAX` nanoseconds is about 292 years, which is 9223372036855
// jiffies with `HZ=1000`, far below the limit.
#[cfg(not(CONFIG_64BIT))]
{
// An overlong span is clamped to `MAX_JIFFY_OFFSET`.
let overlong = Delta::from_nanos(i64::MAX).to_jiffies_timeout();
assert_eq!(overlong.as_jiffies(), MAX_JIFFY_OFFSET);
}
// A negative span is an immediate timeout, however long it is.
let negative = Delta::from_nanos(i64::MIN).to_jiffies_timeout();
assert_eq!(negative.as_jiffies(), 0);
}
}

17
rust/kernel/time/delay.rs
View File

@ -8,7 +8,10 @@
//! C header: [`include/linux/delay.h`](srctree/include/linux/delay.h).
use super::Delta;
use crate::prelude::*;
use crate::{
pr_warn_once,
prelude::*, //
};
/// Sleeps for a given duration at least.
///
@ -18,8 +21,7 @@
/// `delta` must be within `[0, i32::MAX]` microseconds;
/// otherwise, it is erroneous behavior. That is, it is considered a bug
/// to call this function with an out-of-range value, in which case the function
/// will sleep for at least the maximum value in the range and may warn
/// in the future.
/// will sleep for at least the maximum value in the range and warns once.
///
/// The behavior above differs from the C side [`fsleep()`] for which out-of-range
/// values mean "infinite timeout" instead.
@ -32,10 +34,13 @@ pub fn fsleep(delta: Delta) {
// overflow inside fsleep, which could lead to unintentional infinite sleep.
const MAX_DELTA: Delta = Delta::from_micros(i32::MAX as i64);
debug_assert!(delta.as_nanos() >= 0);
debug_assert!(delta <= MAX_DELTA);
let delta = if (Delta::ZERO..=MAX_DELTA).contains(&delta) {
delta
} else {
// TODO: Add WARN_ONCE() when it's supported.
pr_warn_once!("attempted to fsleep() with out of range delta\n");
MAX_DELTA
};
@ -54,7 +59,8 @@ pub fn fsleep(delta: Delta) {
///
/// `delta` must be within `[0, MAX_UDELAY_MS]` in milliseconds;
/// otherwise, it is erroneous behavior. That is, it is considered a bug to
/// call this function with an out-of-range value.
/// call this function with an out-of-range value, in which case the function
/// will delay for at least the maximum value in the range and warns once.
///
/// The behavior above differs from the C side [`udelay()`] for which out-of-range
/// values could lead to an overflow and unexpected behavior.
@ -69,6 +75,7 @@ pub fn udelay(delta: Delta) {
let delta = if (Delta::ZERO..=MAX_UDELAY_DELTA).contains(&delta) {
delta
} else {
pr_warn_once!("attempted to udelay() with out of range delta\n");
MAX_UDELAY_DELTA
};

93
rust/kernel/time/hrtimer.rs
View File

@ -402,15 +402,26 @@
//! ```
//!
//! [`Arc`]: kernel::sync::Arc
// Implementation details
//
// The reasoning for adopting a handle based approach:
// - If we explicitly drop the target of a timer callback in the timer callback, we
// may get a dangling reference.
// - If the callback owns the last reference to the target, target may be dropped
// in non-sleepable context when the callback is finished.
// - When dropping an object that is the target of an armed timer, we may drop
// fields accessed by the timer callback before we cancel the timer (drop order).
//
// By using a handle, we can make the handle own the callback target and avoid these problems.
use super::{ClockSource, Delta, Instant};
use super::{ClockId, Delta, Instant};
use crate::{prelude::*, types::Opaque};
use core::{marker::PhantomData, ptr::NonNull};
use pin_init::PinInit;
/// A type-alias to refer to the [`Instant<C>`] for a given `T` from [`HrTimer<T>`].
///
/// Where `C` is the [`ClockSource`] of the [`HrTimer`].
/// Where `C` is the [`ClockId`] of the [`HrTimer`].
pub type HrTimerInstant<T> = Instant<<<T as HasHrTimer<T>>::TimerMode as HrTimerMode>::Clock>;
/// A timer backed by a C `struct hrtimer`.
@ -855,7 +866,7 @@ pub trait HrTimerExpires {
fn as_nanos(&self) -> i64;
}
impl<C: ClockSource> HrTimerExpires for Instant<C> {
impl<C: ClockId> HrTimerExpires for Instant<C> {
#[inline]
fn as_nanos(&self) -> i64 {
Instant::<C>::as_nanos(self)
@ -870,22 +881,22 @@ fn as_nanos(&self) -> i64 {
}
mod private {
use crate::time::ClockSource;
use crate::time::ClockId;
pub trait Sealed {}
impl<C: ClockSource> Sealed for super::AbsoluteMode<C> {}
impl<C: ClockSource> Sealed for super::RelativeMode<C> {}
impl<C: ClockSource> Sealed for super::AbsolutePinnedMode<C> {}
impl<C: ClockSource> Sealed for super::RelativePinnedMode<C> {}
impl<C: ClockSource> Sealed for super::AbsoluteSoftMode<C> {}
impl<C: ClockSource> Sealed for super::RelativeSoftMode<C> {}
impl<C: ClockSource> Sealed for super::AbsolutePinnedSoftMode<C> {}
impl<C: ClockSource> Sealed for super::RelativePinnedSoftMode<C> {}
impl<C: ClockSource> Sealed for super::AbsoluteHardMode<C> {}
impl<C: ClockSource> Sealed for super::RelativeHardMode<C> {}
impl<C: ClockSource> Sealed for super::AbsolutePinnedHardMode<C> {}
impl<C: ClockSource> Sealed for super::RelativePinnedHardMode<C> {}
impl<C: ClockId> Sealed for super::AbsoluteMode<C> {}
impl<C: ClockId> Sealed for super::RelativeMode<C> {}
impl<C: ClockId> Sealed for super::AbsolutePinnedMode<C> {}
impl<C: ClockId> Sealed for super::RelativePinnedMode<C> {}
impl<C: ClockId> Sealed for super::AbsoluteSoftMode<C> {}
impl<C: ClockId> Sealed for super::RelativeSoftMode<C> {}
impl<C: ClockId> Sealed for super::AbsolutePinnedSoftMode<C> {}
impl<C: ClockId> Sealed for super::RelativePinnedSoftMode<C> {}
impl<C: ClockId> Sealed for super::AbsoluteHardMode<C> {}
impl<C: ClockId> Sealed for super::RelativeHardMode<C> {}
impl<C: ClockId> Sealed for super::AbsolutePinnedHardMode<C> {}
impl<C: ClockId> Sealed for super::RelativePinnedHardMode<C> {}
}
/// Operational mode of [`HrTimer`].
@ -894,16 +905,16 @@ pub trait HrTimerMode: private::Sealed {
const C_MODE: bindings::hrtimer_mode;
/// Type representing the clock source.
type Clock: ClockSource;
type Clock: ClockId;
/// Type representing the expiration specification (absolute or relative time).
type Expires: HrTimerExpires;
}
/// Timer that expires at a fixed point in time.
pub struct AbsoluteMode<C: ClockSource>(PhantomData<C>);
pub struct AbsoluteMode<C: ClockId>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for AbsoluteMode<C> {
impl<C: ClockId> HrTimerMode for AbsoluteMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_ABS;
type Clock = C;
@ -911,9 +922,9 @@ impl<C: ClockSource> HrTimerMode for AbsoluteMode<C> {
}
/// Timer that expires after a delay from now.
pub struct RelativeMode<C: ClockSource>(PhantomData<C>);
pub struct RelativeMode<C: ClockId>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for RelativeMode<C> {
impl<C: ClockId> HrTimerMode for RelativeMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_REL;
type Clock = C;
@ -921,8 +932,8 @@ impl<C: ClockSource> HrTimerMode for RelativeMode<C> {
}
/// Timer with absolute expiration time, pinned to its current CPU.
pub struct AbsolutePinnedMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for AbsolutePinnedMode<C> {
pub struct AbsolutePinnedMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for AbsolutePinnedMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_ABS_PINNED;
type Clock = C;
@ -930,8 +941,8 @@ impl<C: ClockSource> HrTimerMode for AbsolutePinnedMode<C> {
}
/// Timer with relative expiration time, pinned to its current CPU.
pub struct RelativePinnedMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for RelativePinnedMode<C> {
pub struct RelativePinnedMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for RelativePinnedMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_REL_PINNED;
type Clock = C;
@ -939,8 +950,8 @@ impl<C: ClockSource> HrTimerMode for RelativePinnedMode<C> {
}
/// Timer with absolute expiration, handled in soft irq context.
pub struct AbsoluteSoftMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for AbsoluteSoftMode<C> {
pub struct AbsoluteSoftMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for AbsoluteSoftMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_ABS_SOFT;
type Clock = C;
@ -948,8 +959,8 @@ impl<C: ClockSource> HrTimerMode for AbsoluteSoftMode<C> {
}
/// Timer with relative expiration, handled in soft irq context.
pub struct RelativeSoftMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for RelativeSoftMode<C> {
pub struct RelativeSoftMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for RelativeSoftMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_REL_SOFT;
type Clock = C;
@ -957,8 +968,8 @@ impl<C: ClockSource> HrTimerMode for RelativeSoftMode<C> {
}
/// Timer with absolute expiration, pinned to CPU and handled in soft irq context.
pub struct AbsolutePinnedSoftMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for AbsolutePinnedSoftMode<C> {
pub struct AbsolutePinnedSoftMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for AbsolutePinnedSoftMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_ABS_PINNED_SOFT;
type Clock = C;
@ -966,8 +977,8 @@ impl<C: ClockSource> HrTimerMode for AbsolutePinnedSoftMode<C> {
}
/// Timer with absolute expiration, pinned to CPU and handled in soft irq context.
pub struct RelativePinnedSoftMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for RelativePinnedSoftMode<C> {
pub struct RelativePinnedSoftMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for RelativePinnedSoftMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_REL_PINNED_SOFT;
type Clock = C;
@ -975,8 +986,8 @@ impl<C: ClockSource> HrTimerMode for RelativePinnedSoftMode<C> {
}
/// Timer with absolute expiration, handled in hard irq context.
pub struct AbsoluteHardMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for AbsoluteHardMode<C> {
pub struct AbsoluteHardMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for AbsoluteHardMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_ABS_HARD;
type Clock = C;
@ -984,8 +995,8 @@ impl<C: ClockSource> HrTimerMode for AbsoluteHardMode<C> {
}
/// Timer with relative expiration, handled in hard irq context.
pub struct RelativeHardMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for RelativeHardMode<C> {
pub struct RelativeHardMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for RelativeHardMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_REL_HARD;
type Clock = C;
@ -993,8 +1004,8 @@ impl<C: ClockSource> HrTimerMode for RelativeHardMode<C> {
}
/// Timer with absolute expiration, pinned to CPU and handled in hard irq context.
pub struct AbsolutePinnedHardMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for AbsolutePinnedHardMode<C> {
pub struct AbsolutePinnedHardMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for AbsolutePinnedHardMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_ABS_PINNED_HARD;
type Clock = C;
@ -1002,8 +1013,8 @@ impl<C: ClockSource> HrTimerMode for AbsolutePinnedHardMode<C> {
}
/// Timer with relative expiration, pinned to CPU and handled in hard irq context.
pub struct RelativePinnedHardMode<C: ClockSource>(PhantomData<C>);
impl<C: ClockSource> HrTimerMode for RelativePinnedHardMode<C> {
pub struct RelativePinnedHardMode<C: ClockId>(PhantomData<C>);
impl<C: ClockId> HrTimerMode for RelativePinnedHardMode<C> {
const C_MODE: bindings::hrtimer_mode = bindings::hrtimer_mode_HRTIMER_MODE_REL_PINNED_HARD;
type Clock = C;

16
rust/macros/helpers.rs
View File

@ -38,22 +38,6 @@ pub(crate) fn value(&self) -> String {
}
}
pub(crate) fn file() -> String {
#[cfg(not(CONFIG_RUSTC_HAS_SPAN_FILE))]
{
proc_macro::Span::call_site()
.source_file()
.path()
.to_string_lossy()
.into_owned()
}
#[cfg(CONFIG_RUSTC_HAS_SPAN_FILE)]
{
proc_macro::Span::call_site().file()
}
}
/// Obtain all `#[cfg]` attributes.
pub(crate) fn gather_cfg_attrs(attr: &[Attribute]) -> impl Iterator<Item = &Attribute> + '_ {
attr.iter().filter(|a| a.path().is_ident("cfg"))

5
rust/macros/kunit.rs
View File

@ -109,12 +109,11 @@ pub(crate) fn kunit_tests(test_suite: Ident, mut module: ItemMod) -> Result<Toke
// Before the test, override usual `assert!` and `assert_eq!` macros with ones that call
// KUnit instead.
let test_str = test.to_string();
let path = CString::new(crate::helpers::file()).expect("file path cannot contain NUL");
processed_items.push(parse_quote! {
#[allow(unused)]
macro_rules! assert {
($cond:expr $(,)?) => {{
kernel::kunit_assert!(#test_str, #path, 0, $cond);
kernel::kunit_assert!(#test_str, $cond);
}}
}
});
@ -122,7 +121,7 @@ macro_rules! assert {
#[allow(unused)]
macro_rules! assert_eq {
($left:expr, $right:expr $(,)?) => {{
kernel::kunit_assert_eq!(#test_str, #path, 0, $left, $right);
kernel::kunit_assert_eq!(#test_str, $left, $right);
}}
}
});

48
rust/macros/lib.rs
View File

@ -8,11 +8,6 @@
// Stable since Rust 1.87.0.
#![feature(extract_if)]
//
// Stable since Rust 1.88.0 under a different name, `proc_macro_span_file`,
// which was added in Rust 1.88.0. This is why `cfg_attr` is used here, i.e.
// to avoid depending on the full `proc_macro_span` on Rust >= 1.88.0.
#![cfg_attr(not(CONFIG_RUSTC_HAS_SPAN_FILE), feature(proc_macro_span))]
mod concat_idents;
mod export;
@ -129,7 +124,7 @@
/// # Supported argument types
/// - `type`: type which implements the [`Module`] trait (required).
/// - `name`: ASCII string literal of the name of the kernel module (required).
/// - `authors`: array of ASCII string literals of the authors of the kernel module.
/// - `authors`: array of string literals of the authors of the kernel module.
/// - `description`: string literal of the description of the kernel module.
/// - `license`: ASCII string literal of the license of the kernel module (required).
/// - `alias`: array of ASCII string literals of the alias names of the kernel module.
@ -156,25 +151,15 @@ pub fn module(input: TokenStream) -> TokenStream {
/// associated constant bool for each method in the trait that is set to true if
/// the implementer has overridden the associated method.
///
/// For a trait method to be optional, it must have a default implementation.
/// This is also the case for traits annotated with `#[vtable]`, but in this
/// case the default implementation will never be executed. The reason for this
/// is that the functions will be called through function pointers installed in
/// C side vtables. When an optional method is not implemented on a `#[vtable]`
/// trait, a `NULL` entry is installed in the vtable. Thus the default
/// implementation is never called. Since these traits are not designed to be
/// used on the Rust side, it should not be possible to call the default
/// implementation. This is done to ensure that we call the vtable methods
/// through the C vtable, and not through the Rust vtable. Therefore, the
/// default implementation should call `build_error!`, which prevents
/// calls to this function at compile time:
///
/// ```compile_fail
/// # // Intentionally missing `use`s to simplify `rusttest`.
/// build_error!(VTABLE_DEFAULT_ERROR)
/// ```
///
/// Note that you might need to import [`kernel::error::VTABLE_DEFAULT_ERROR`].
/// For a trait method to be optional for normal traits, it must have a default
/// implementation. However, for many users of `#[vtable]`, the functions will
/// be called through function pointers installed in C side vtables. When an
/// optional method is not implemented on a `#[vtable]` trait, a `NULL` entry is
/// installed in the vtable; thus the default implementation is never called. If
/// this is the case, `#[optional]` can be applied on the optional method.
/// Methods annotated as such do not need to be implemented nor need a default
/// implementation. Calling these methods on types that do not implement them
/// will fail to build using the `build_error!` mechanism.
///
/// This macro should not be used when all functions are required.
///
@ -187,7 +172,6 @@ pub fn module(input: TokenStream) -> TokenStream {
/// # Examples
///
/// ```
/// use kernel::error::VTABLE_DEFAULT_ERROR;
/// use kernel::prelude::*;
///
/// # struct LocalModule;
@ -204,13 +188,11 @@ pub fn module(input: TokenStream) -> TokenStream {
/// // Declares a `#[vtable]` trait
/// #[vtable]
/// pub trait Operations: Send + Sync + Sized {
/// fn foo(&self) -> Result<()> {
/// build_error!(VTABLE_DEFAULT_ERROR)
/// }
/// #[optional]
/// fn foo(&self) -> Result<()>;
///
/// fn bar(&self) -> Result<()> {
/// build_error!(VTABLE_DEFAULT_ERROR)
/// }
/// #[optional]
/// fn bar(&self) -> Result<()>;
/// }
///
/// struct Foo;
@ -228,8 +210,6 @@ pub fn module(input: TokenStream) -> TokenStream {
/// assert_eq!(<Foo as Operations>::HAS_BAR, false);
/// # }
/// ```
///
/// [`kernel::error::VTABLE_DEFAULT_ERROR`]: ../kernel/error/constant.VTABLE_DEFAULT_ERROR.html
#[proc_macro_attribute]
pub fn vtable(attr: TokenStream, input: TokenStream) -> TokenStream {
parse_macro_input!(attr as syn::parse::Nothing);

2
rust/macros/module.rs
View File

@ -369,7 +369,7 @@ pub(crate) struct ModuleInfo {
type_: Type,
license: AsciiLitStr,
name: AsciiLitStr,
authors: Option<Punctuated<AsciiLitStr, Token![,]>>,
authors: Option<Punctuated<LitStr, Token![,]>>,
description: Option<LitStr>,
alias: Option<Punctuated<AsciiLitStr, Token![,]>>,
firmware: Option<Punctuated<AsciiLitStr, Token![,]>>,

43
rust/macros/vtable.rs
View File

@ -9,7 +9,10 @@
Ident,
TokenStream, //
};
use quote::ToTokens;
use quote::{
quote,
ToTokens, //
};
use syn::{
parse_quote,
Error,
@ -46,7 +49,7 @@ fn handle_trait(mut item: ItemTrait) -> Result<ItemTrait> {
});
}
for item in &item.items {
for item in &mut item.items {
if let TraitItem::Fn(fn_item) = item {
let name = &fn_item.sig.ident;
let gen_const_name = Ident::new(
@ -54,14 +57,44 @@ fn handle_trait(mut item: ItemTrait) -> Result<ItemTrait> {
name.span(),
);
if fn_item
.attrs
.extract_if(.., |attr| attr.path().is_ident("optional"))
.count()
!= 0
{
if let Some(default) = &fn_item.default {
Err(Error::new_spanned(
default,
"`#[optional]` methods must not have a default implementation",
))?;
}
// Optional methods in Rust need a default implementation. Inject one that fails
// the build at compile time if not overridden.
fn_item.default = Some(parse_quote!({
::kernel::build_assert::build_error!(
"This function must not be called, see the `#[vtable]` documentation."
);
}));
// Ensure that the function is never code generated unless used.
fn_item.attrs.push(parse_quote!(#[inline]));
}
// We don't know on the implementation-site whether a method is required or provided
// so we have to generate a const for all methods.
// However, hide it for required methods as it will always be true.
let doc = if fn_item.default.is_some() {
let comment =
format!("Indicates if the `{name}` method is overridden by the implementor.");
quote!(#[doc = #comment])
} else {
quote!(#[doc(hidden)])
};
let cfg_attrs = crate::helpers::gather_cfg_attrs(&fn_item.attrs);
let comment =
format!("Indicates if the `{name}` method is overridden by the implementor.");
gen_items.push(parse_quote! {
#(#cfg_attrs)*
#[doc = #comment]
#doc
const #gen_const_name: bool = false;
});
}

60
rust/pin-init/examples/selfref.rs Normal file
View File

@ -0,0 +1,60 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
#![allow(clippy::disallowed_names)]
use pin_init::*;
#[pin_data]
struct SelfRef {
#[uses('_: invariant)]
not_cov: Box<dyn Fn(&'str str) -> bool + 'str>,
part: &'str str,
str: String,
mut_part: &'mut_str mut str,
#[borrowed(mut)]
mut_str: String,
}
fn use_self_ref() {
stack_pin_init!(let foo = pin_init!(SelfRef {
str: "hello world".to_owned(),
part: &str[..5],
mut_str: "hello world".to_owned(),
mut_part: &mut mut_str[..5],
not_cov: Box::new(move |s| s == str),
}));
// Access via projection.
println!("{}", foo.as_mut().project().part);
// Access via accessor.
println!("{}", foo.part());
// Access via `with_project`, gives mutable reference.
foo.as_mut().with_project(|proj| {
*proj.part = &proj.str[5..];
});
println!("{}", foo.part());
// Access fields that mutable borrow others are similar to those of shared borrow.
println!("{}", foo.as_mut().project().mut_part);
println!("{}", foo.mut_part());
foo.as_mut().with_project(|proj| {
proj.mut_part.make_ascii_uppercase();
});
// Access non-covariant type using `with_` accessor.
foo.with_not_cov(|not_cov| {
not_cov("");
});
// Access non-covariant type using `with_project`.
foo.as_mut().with_project(|proj| (proj.not_cov)(proj.str));
}
fn main() {
use_self_ref();
}

169
rust/pin-init/internal/src/diagnostics.rs
View File

@ -1,44 +1,149 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
use std::cell::RefCell;
use std::fmt::Display;
use std::marker::PhantomData;
use proc_macro2::TokenStream;
use quote::quote_spanned;
use proc_macro2::{Span, TokenStream};
use quote::{quote, quote_spanned};
use syn::{spanned::Spanned, Error};
pub(crate) struct DiagCtxt(TokenStream);
pub(crate) struct DiagCtxt(PhantomData<*mut ()>);
pub(crate) struct ErrorGuaranteed(());
impl DiagCtxt {
pub(crate) fn error(&mut self, span: impl Spanned, msg: impl Display) -> ErrorGuaranteed {
let error = Error::new(span.span(), msg);
self.0.extend(error.into_compile_error());
ErrorGuaranteed(())
}
struct DiagCtxtData {
diag: TokenStream,
}
pub(crate) fn warn(&mut self, span: impl Spanned, msg: impl Display) {
// Have the message start on a new line for visual clarity.
let msg = format!("\n{}", msg);
self.0.extend(quote_spanned!(span.span() =>
// Approximate using deprecated warning while `proc_macro_diagnostic` is unstable.
const _: () = {
#[deprecated = #msg]
const fn warn() {}
warn();
};
));
}
thread_local! {
static DIAGNOSTICS: RefCell<Option<DiagCtxtData>> = const { RefCell::new(None) };
}
pub(crate) fn with(
fun: impl FnOnce(&mut DiagCtxt) -> Result<TokenStream, ErrorGuaranteed>,
) -> TokenStream {
let mut dcx = Self(TokenStream::new());
match fun(&mut dcx) {
Ok(mut stream) => {
stream.extend(dcx.0);
stream
}
Err(ErrorGuaranteed(())) => dcx.0,
}
// Allows `syn::Error` to be emitted into the current diagnostic context with just `?`.
impl From<syn::Error> for ErrorGuaranteed {
fn from(error: syn::Error) -> Self {
DIAGNOSTICS.with_borrow_mut(|data| {
data.as_mut()
.unwrap()
.diag
.extend(error.into_compile_error());
});
Self(())
}
}
impl DiagCtxt {
pub(crate) fn error(&self, span: impl Spanned, msg: impl Display) -> ErrorGuaranteed {
Error::new(span.span(), msg).into()
}
pub(crate) fn warn(&self, span: impl Spanned, msg: impl Display) {
// Have the message start on a new line for visual clarity.
let msg = format!("\n{}", msg);
DIAGNOSTICS.with_borrow_mut(|data| {
data.as_mut()
.unwrap()
.diag
.extend(quote_spanned!(span.span() =>
// Approximate using deprecated warning while `proc_macro_diagnostic` is
// unstable.
const _: () = {
#[deprecated = #msg]
const fn warn() {}
warn();
};
))
});
}
/// Execute the provided function with the current diagnostic context.
pub(crate) fn current<R>(f: impl FnOnce(&DiagCtxt) -> R) -> R {
DIAGNOSTICS.with_borrow(|data| {
assert!(data.is_some(), "No active `DiagCtxt`");
});
f(&DiagCtxt(PhantomData))
}
fn with(
f: impl FnOnce(&mut DiagCtxt) -> Result<TokenStream, ErrorGuaranteed>,
merge_diag: impl FnOnce(TokenStream, TokenStream) -> TokenStream,
convert_diag: impl FnOnce(TokenStream) -> TokenStream,
) -> TokenStream {
DIAGNOSTICS.with_borrow_mut(|data| {
assert!(data.is_none(), "`DiagCtxt` cannot be nested");
*data = Some(DiagCtxtData {
diag: TokenStream::new(),
});
});
let mut dcx = DiagCtxt(PhantomData);
let result = match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| f(&mut dcx))) {
Ok(result) => result,
Err(payload) => {
// Robustness against panicking in macros.
//
// Ensure that any error messages are still emitted when this happens.
let message = if let Some(&s) = payload.downcast_ref::<&'static str>() {
s
} else if let Some(s) = payload.downcast_ref::<String>() {
s.as_str()
} else {
"Box<dyn Any>"
};
Err(dcx.error(
Span::mixed_site(),
format!("proc macro panicked: {message}"),
))
}
};
let data = DIAGNOSTICS.with_borrow_mut(|data| data.take().unwrap());
match result {
Ok(stream) => {
if data.diag.is_empty() {
stream
} else {
merge_diag(stream, data.diag)
}
}
Err(ErrorGuaranteed(())) => convert_diag(data.diag),
}
}
pub(crate) fn for_item(
f: impl FnOnce(&mut DiagCtxt) -> Result<TokenStream, ErrorGuaranteed>,
) -> TokenStream {
Self::with(
f,
|mut out, diag| {
out.extend(diag);
out
},
std::convert::identity,
)
}
pub(crate) fn for_expr(
f: impl FnOnce(&mut DiagCtxt) -> Result<TokenStream, ErrorGuaranteed>,
) -> TokenStream {
Self::with(
f,
|out, diag| {
// Diagnostics that we generate are always items.
// So for expressions create a block to place diagnostics in item position.
quote!({
#diag
#out
})
},
|diag| {
quote!({
#diag
})
},
)
}
}

683
rust/pin-init/internal/src/init.rs
View File

@ -1,26 +1,86 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
use proc_macro2::{Span, TokenStream};
use quote::{format_ident, quote};
use quote::{format_ident, quote, quote_spanned, ToTokens, TokenStreamExt};
use syn::{
braced,
braced, parenthesized,
parse::{End, Parse},
parse_quote,
punctuated::Punctuated,
punctuated::{Pair, Punctuated},
spanned::Spanned,
token, Attribute, Block, Expr, ExprCall, ExprPath, Ident, Path, Token, Type,
token, Attribute, Block, Expr, ExprCall, ExprPath, Ident, Index, LitInt, Member, Path, Token,
Type,
};
use crate::diagnostics::{DiagCtxt, ErrorGuaranteed};
use crate::{
diagnostics::{DiagCtxt, ErrorGuaranteed},
util::*,
};
pub(crate) struct Initializer {
pub(crate) struct Initializer<Kind = InitExprKind> {
attrs: Vec<InitializerAttribute>,
this: Option<This>,
kind: Kind,
error: Option<(Token![?], Type)>,
}
pub(crate) struct InitExprStruct {
path: Path,
brace_token: token::Brace,
fields: Punctuated<InitializerField, Token![,]>,
rest: Option<(Token![..], Expr)>,
error: Option<(Token![?], Type)>,
}
pub(crate) struct InitExprTuple {
path: Path,
paren_token: token::Paren,
fields: Punctuated<InitTupleField, Token![,]>,
}
pub(crate) enum InitExprKind {
Struct(InitExprStruct),
Tuple(InitExprTuple),
}
struct InitTupleField {
attrs: Vec<Attribute>,
value: Expr,
}
impl InitExprTuple {
fn normalize(self) -> InitExprStruct {
let InitExprTuple {
path,
paren_token,
fields,
} = self;
InitExprStruct {
path,
brace_token: token::Brace {
span: paren_token.span,
},
fields: fields
.into_pairs()
.enumerate()
.map(|(index, pair)| {
let (field, comma) = pair.into_tuple();
let span = field.value.span();
let field = InitializerField {
attrs: field.attrs,
kind: InitializerKind::Value {
member: Member::Unnamed(Index {
index: index.try_into().unwrap(),
span,
}),
value: Some((Token![:](span), field.value)),
},
};
Pair::new(field, comma)
})
.collect(),
rest: None,
}
}
}
struct This {
@ -36,12 +96,12 @@ struct InitializerField {
enum InitializerKind {
Value {
ident: Ident,
member: Member,
value: Option<(Token![:], Expr)>,
},
Init {
ident: Ident,
_left_arrow_token: Token![<-],
member: Member,
left_arrow_token: Token![<-],
value: Expr,
},
Code {
@ -52,9 +112,9 @@ enum InitializerKind {
}
impl InitializerKind {
fn ident(&self) -> Option<&Ident> {
fn member(&self) -> Option<&Member> {
match self {
Self::Value { ident, .. } | Self::Init { ident, .. } => Some(ident),
Self::Value { member, .. } | Self::Init { member, .. } => Some(member),
Self::Code { .. } => None,
}
}
@ -68,16 +128,101 @@ struct DefaultErrorAttribute {
ty: Box<Type>,
}
pub(crate) fn expand(
pub(crate) fn expand_with_cfg(
initializer: Initializer,
default_error: Option<&'static str>,
pinned: bool,
dcx: &mut DiagCtxt,
) -> Result<TokenStream, ErrorGuaranteed> {
let initializer = match initializer.kind {
InitExprKind::Tuple(expr) => {
let mut initializer = Initializer {
attrs: initializer.attrs,
this: initializer.this,
kind: expr,
error: initializer.error,
};
// Removing a tuple field renumbers every field after it, which cannot be expressed with
// a `cfg` attribute on the initializer of a single field. Therefore, resolve tuple
// field cfgs before continuing. Struct expression syntax uses explicit numbers, so
// there is no need to pre-expand them and we only need to emit their cfgs on generated
// code.
for (field_idx, field) in initializer.kind.fields.iter_mut().enumerate() {
let cfg = field.attrs.extract_cfg_attrs();
if cfg.is_empty() {
continue;
}
let true_initializer = initializer.to_token_stream();
initializer.kind.fields = initializer
.kind
.fields
.into_pairs()
.enumerate()
.filter(|&(index, _)| index != field_idx)
.map(|(_, pair)| pair)
.collect();
let false_initializer = &initializer;
let macro_name = if pinned {
quote!(::pin_init::pin_init)
} else {
quote!(::pin_init::init)
};
// Resolve one field at a time until we've got no more tuple field cfgs.
//
// This is linear time because macro invocations with false cfg will not be
// expanded.
return Ok(quote! {
{
// Use `{}` delimiter here so semicolon is not required, otherwise the
// expression becomes unit type.
#[cfg(all(#(#cfg,)*))]
#macro_name! { #true_initializer }
#[cfg(not(all(#(#cfg,)*)))]
#macro_name! { #false_initializer }
}
});
}
// No cfgs left, we can normalize the initializer to the struct kind.
Initializer {
attrs: initializer.attrs,
this: initializer.this,
kind: initializer.kind.normalize(),
error: initializer.error,
}
}
InitExprKind::Struct(expr) => Initializer {
attrs: initializer.attrs,
this: initializer.this,
kind: expr,
error: initializer.error,
},
};
expand(initializer, default_error, pinned, dcx)
}
fn expand(
Initializer {
attrs,
this,
path,
brace_token,
fields,
rest,
kind:
InitExprStruct {
path,
brace_token,
fields,
rest,
},
error,
}: Initializer,
}: Initializer<InitExprStruct>,
default_error: Option<&'static str>,
pinned: bool,
dcx: &mut DiagCtxt,
@ -96,30 +241,31 @@ pub(crate) fn expand(
} else if let Some(default_error) = default_error {
syn::parse_str(default_error).unwrap()
} else {
dcx.error(brace_token.span.close(), "expected `? <type>` after `}`");
dcx.error(
brace_token.span.close(),
"expected `? <type>` after initializer",
);
parse_quote!(::core::convert::Infallible)
}
},
|(_, err)| Box::new(err),
);
let slot = format_ident!("slot");
let (has_data_trait, get_data, init_from_closure) = if pinned {
let (get_pin_data, init_from_closure) = if pinned {
(
format_ident!("HasPinData"),
format_ident!("__pin_data"),
Some(
quote_spanned! { path.span().resolved_at(Span::mixed_site()) =>
let data = ::pin_init::__internal::HasPinData::__pin_data(data);
},
),
format_ident!("pin_init_from_closure"),
)
} else {
(
format_ident!("HasInitData"),
format_ident!("__init_data"),
format_ident!("init_from_closure"),
)
(None, format_ident!("init_from_closure"))
};
let init_kind = get_init_kind(rest, dcx);
let zeroable_check = match init_kind {
InitKind::Normal => quote!(),
InitKind::Zeroing => quote! {
InitKind::Zeroing => quote_spanned! { Span::mixed_site() =>
// The user specified `..Zeroable::zeroed()` at the end of the list of fields.
// Therefore we check if the struct implements `Zeroable` and then zero the memory.
// This allows us to also remove the check that all fields are present (since we
@ -128,45 +274,60 @@ fn assert_zeroable<T: ?::core::marker::Sized>(_: *mut T)
where T: ::pin_init::Zeroable
{}
// Ensure that the struct is indeed `Zeroable`.
assert_zeroable(#slot);
assert_zeroable(slot);
// SAFETY: The type implements `Zeroable` by the check above.
unsafe { ::core::ptr::write_bytes(#slot, 0, 1) };
unsafe { ::core::ptr::write_bytes(slot, 0, 1) };
},
};
let this = match this {
None => quote!(),
Some(This { ident, .. }) => quote! {
Some(This { ident, .. }) => quote_spanned! { Span::mixed_site() =>
// Create the `this` so it can be referenced by the user inside of the
// expressions creating the individual fields.
let #ident = unsafe { ::core::ptr::NonNull::new_unchecked(slot) };
},
};
// `mixed_site` ensures that the data is not accessible to the user-controlled code.
let data = Ident::new("__data", Span::mixed_site());
let init_fields = init_fields(&fields, pinned, &data, &slot);
let init_fields = make_field_init(&fields, pinned, false);
let drop_check = make_field_init(&fields, pinned, true);
let field_check = make_field_check(&fields, init_kind, &path);
Ok(quote! {{
Ok(quote_spanned! { Span::mixed_site() => {
// Get the data about fields from the supplied type.
// SAFETY: TODO
let #data = unsafe {
use ::pin_init::__internal::#has_data_trait;
let data = {
use ::pin_init::__internal::HasInitData;
// Can't use `<#path as #has_data_trait>::#get_data`, since the user is able to omit
// generics (which need to be present with that syntax).
#path::#get_data()
#path::__init_data()
};
// Ensure that `#data` really is of type `#data` and help with type inference:
let init = #data.__make_closure::<_, #error>(
move |slot| {
#get_pin_data
// Ensure that `data` really is of type `data` and help with type inference:
let init = data.__make_closure::<_, #error>(
move |slot, data_lt| {
#zeroable_check
#this
#init_fields
// Generate init twice, which is mostly identical except for lifetimes.
if true {
// In this path, we use the field lifetime from HRTB to prevent environment
// lifetime from entering the fields, and to ensure that the dependency of
// fields is consistent with the field drop of the struct.
#init_fields
} else {
// In this path, we use local lifetime, to make sure that if initialization
// fails, the destructor execution will not cause lifetime issues. This is
// separate as implied bounds between field lifetimes can be inconsistent with
// that of the drop.
#drop_check
}
#field_check
// SAFETY: we are the `init!` macro that is allowed to call this.
Ok(unsafe { ::pin_init::__internal::InitOk::new() })
}
);
let init = move |slot| -> ::core::result::Result<(), #error> {
init(slot).map(|__InitOk| ())
init(slot, data.__with_lt()).map(|__InitOk| ())
};
// SAFETY: TODO
unsafe { ::pin_init::#init_from_closure::<_, #error>(init) }
@ -213,14 +374,12 @@ fn get_init_kind(rest: Option<(Token![..], Expr)>, dcx: &mut DiagCtxt) -> InitKi
}
/// Generate the code that initializes the fields of the struct using the initializers in `field`.
fn init_fields(
fn make_field_init(
fields: &Punctuated<InitializerField, Token![,]>,
pinned: bool,
data: &Ident,
slot: &Ident,
dropck: bool,
) -> TokenStream {
let mut guards = vec![];
let mut guard_attrs = vec![];
let mut forget_guards = vec![];
let mut res = TokenStream::new();
for InitializerField { attrs, kind } in fields {
let cfgs = {
@ -229,9 +388,9 @@ fn init_fields(
cfgs
};
let ident = match kind {
InitializerKind::Value { ident, .. } => ident,
InitializerKind::Init { ident, .. } => ident,
let member = match kind {
InitializerKind::Value { member, .. } => member,
InitializerKind::Init { member, .. } => member,
InitializerKind::Code { block, .. } => {
let stmt = &block.stmts;
res.extend(quote! {
@ -243,78 +402,118 @@ fn init_fields(
continue;
}
};
let ident = member.as_ident();
let span = Span::mixed_site().located_at(ident.span());
let slot = if pinned {
quote! {
let data = if !dropck {
quote_spanned!(span => data_lt)
} else {
quote_spanned!(span => data.__with_lt())
};
quote_spanned! { span =>
// SAFETY:
// - `slot` is valid and properly aligned.
// - `make_field_check` checks that `&raw mut (*slot).#ident` is properly aligned.
// - `make_field_check` prevents `#ident` from being used twice, therefore
// `(*slot).#ident` is exclusively accessed and has not been initialized.
(unsafe { #data.#ident(#slot) })
// - `make_field_check` checks that `&raw mut (*slot).#member` is properly aligned.
// - `make_field_check` prevents `#member` from being used twice, therefore
// `(*slot).#member` is exclusively accessed and has not been initialized.
(unsafe { #data.#ident(slot) })
}
} else {
quote! {
quote_spanned! { span =>
// For `init!()` macro, everything is unpinned.
// SAFETY:
// - `&raw mut (*slot).#ident` is valid.
// - `make_field_check` checks that `&raw mut (*slot).#ident` is properly aligned.
// - `make_field_check` prevents `#ident` from being used twice, therefore
// `(*slot).#ident` is exclusively accessed and has not been initialized.
// - `&raw mut (*slot).#member` is valid.
// - `make_field_check` checks that `&raw mut (*slot).#member` is properly aligned.
// - `make_field_check` prevents `#member` from being used twice, therefore
// `(*slot).#member` is exclusively accessed and has not been initialized.
(unsafe {
::pin_init::__internal::Slot::<::pin_init::__internal::Unpinned, _>::new(
&raw mut (*#slot).#ident
&raw mut (*slot).#member
)
})
}
};
// `mixed_site` ensures that the guard is not accessible to the user-controlled code.
let guard = format_ident!("__{ident}_guard", span = Span::mixed_site());
let guard = format_ident!("__{ident}_guard", span = span);
let full_span = kind.span().resolved_at(Span::mixed_site());
let init = match kind {
InitializerKind::Value { ident, value } => {
let value = value
InitializerKind::Value { value, .. } => {
let (colon_span, value) = value
.as_ref()
.map(|(_, value)| quote!(#value))
.unwrap_or_else(|| quote!(#ident));
.map(|(colon_token, value)| (colon_token.span(), quote!(#value)))
.unwrap_or_else(|| (member.span(), quote!(#member)));
quote! {
// Use `:` as the span of the method name, so the type requirement appears to come
// from `:`.
let write =
format_ident!("write", span = colon_span.resolved_at(Span::mixed_site()));
quote_spanned! { full_span =>
#(#attrs)*
let mut #guard = #slot.write(#value);
let mut #guard = #slot.#write(#value);
}
}
InitializerKind::Init { value, .. } => {
quote! {
InitializerKind::Init {
value,
left_arrow_token,
..
} => {
// Use `<-` as the span of the method name, so the trait bound appears to come from
// `<-`.
let init = format_ident!(
"init",
span = left_arrow_token.span().resolved_at(Span::mixed_site())
);
quote_spanned! { full_span =>
#(#attrs)*
let mut #guard = #slot.init(#value)?;
let mut #guard = #slot.#init(#value)?;
}
}
InitializerKind::Code { .. } => unreachable!(),
};
// A tuple field has no name that could be bound here (the `_0` identifiers are considered
// implementation detail and not user-facing).
let let_binding_method = if !dropck {
format_ident!("let_binding", span = span)
} else {
format_ident!("let_binding_in_dropck", span = span)
};
let binding = match member {
Member::Named(ident) => quote_spanned! { span =>
#(#cfgs)*
// Allow `non_snake_case` since the same warning is going to be reported for the
// struct field.
#[allow(unused_variables, non_snake_case)]
// Include `mut` so that `Pin<&mut T>` bindings can be reborrowed via `.as_mut()`.
let mut #ident = #guard.#let_binding_method();
},
Member::Unnamed(_) => quote!(),
};
res.extend(quote! {
#init
#(#cfgs)*
// Allow `non_snake_case` since the same warning is going to be reported for the struct
// field.
#[allow(unused_variables, non_snake_case)]
let #ident = #guard.let_binding();
#binding
});
guards.push(guard);
guard_attrs.push(cfgs);
forget_guards.push(quote_spanned! { span =>
#(#cfgs)*
::core::mem::forget(#guard);
});
}
forget_guards.reverse();
quote! {
#res
// If execution reaches this point, all fields have been initialized. Therefore we can now
// dismiss the guards by forgetting them.
#(
#(#guard_attrs)*
::core::mem::forget(#guards);
)*
#(#forget_guards)*
}
}
@ -324,51 +523,67 @@ fn make_field_check(
init_kind: InitKind,
path: &Path,
) -> TokenStream {
let field_attrs: Vec<_> = fields
let align_checks: TokenStream = fields
.iter()
.filter_map(|f| f.kind.ident().map(|_| &f.attrs))
.filter_map(|f| {
let member = f.kind.member()?;
let span = member.span().resolved_at(Span::mixed_site());
let attrs = &f.attrs;
Some(quote_spanned! {span =>
// Create references to ensure that the initialized field is properly aligned.
// Unaligned fields will cause the compiler to emit E0793. We do not support
// unaligned fields since `Init::__init` requires an aligned pointer; the call to
// `ptr::write` for value-initialization case has the same requirement.
#(#attrs)*
let _ = &(*slot).#member;
})
})
.collect();
let fake_field_init: TokenStream = fields
.iter()
.filter_map(|f| {
let member = f.kind.member()?;
let span = member.span().resolved_at(Span::mixed_site());
let attrs = &f.attrs;
Some(quote_spanned! {span =>
#(#attrs)*
#member: loop {},
})
})
.collect();
let field_name: Vec<_> = fields.iter().filter_map(|f| f.kind.ident()).collect();
let zeroing_trailer = match init_kind {
InitKind::Normal => None,
InitKind::Zeroing => Some(quote! {
..::core::mem::zeroed()
}),
};
quote! {
let field_dup_checks = quote_spanned! { Span::mixed_site() =>
// If the zeroing trailer is not present, this checks that all fields have been
// mentioned exactly once. If the zeroing trailer is present, all missing fields will be
// zeroed, so this checks that all fields have been mentioned at most once. The use of
// struct initializer will still generate very natural error messages for any misuse.
::core::ptr::write(slot, #path {
#fake_field_init
#zeroing_trailer
})
};
quote_spanned! { Span::mixed_site() =>
#[allow(unreachable_code)]
// We use unreachable code to perform field checks. They're still checked by the compiler.
// SAFETY: this code is never executed.
let _ = || unsafe {
// Create references to ensure that the initialized field is properly aligned.
// Unaligned fields will cause the compiler to emit E0793. We do not support
// unaligned fields since `Init::__init` requires an aligned pointer; the call to
// `ptr::write` for value-initialization case has the same requirement.
#(
#(#field_attrs)*
let _ = &(*slot).#field_name;
)*
// If the zeroing trailer is not present, this checks that all fields have been
// mentioned exactly once. If the zeroing trailer is present, all missing fields will be
// zeroed, so this checks that all fields have been mentioned at most once. The use of
// struct initializer will still generate very natural error messages for any misuse.
::core::ptr::write(slot, #path {
#(
#(#field_attrs)*
#field_name: loop {},
)*
#zeroing_trailer
})
#align_checks
#field_dup_checks
};
}
}
impl Parse for Initializer {
fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
let attrs = input.call(Attribute::parse_outer)?;
let this = input.peek(Token![&]).then(|| input.parse()).transpose()?;
let path = input.parse()?;
impl InitExprStruct {
fn parse_with_path(path: Path, input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
let content;
let brace_token = braced!(content in input);
let mut fields = Punctuated::new();
@ -376,7 +591,8 @@ fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
let lh = content.lookahead1();
if lh.peek(End) || lh.peek(Token![..]) {
break;
} else if lh.peek(Ident) || lh.peek(Token![_]) || lh.peek(Token![#]) {
} else if lh.peek(Ident) || lh.peek(LitInt) || lh.peek(Token![_]) || lh.peek(Token![#])
{
fields.push_value(content.parse()?);
let lh = content.lookahead1();
if lh.peek(End) {
@ -394,6 +610,62 @@ fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
.peek(Token![..])
.then(|| Ok::<_, syn::Error>((content.parse()?, content.parse()?)))
.transpose()?;
Ok(Self {
path,
brace_token,
fields,
rest,
})
}
}
impl InitExprTuple {
fn parse_with_path(path: Path, input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
let content;
let paren_token = parenthesized!(content in input);
let mut fields = Punctuated::new();
while !content.is_empty() {
let attrs = content.call(Attribute::parse_outer)?;
if let Some(left_arrow_token) = content.parse::<Option<Token![<-]>>()? {
DiagCtxt::current(|dcx| {
dcx.error(
left_arrow_token,
"`<-` is not supported in tuple constructor syntax; name the fields by \
index instead, e.g. `Type { 0 <- initializer, 1: value }`",
)
});
}
fields.push_value(InitTupleField {
attrs,
value: content.parse()?,
});
if content.is_empty() {
break;
}
fields.push_punct(content.parse()?);
}
Ok(InitExprTuple {
path,
paren_token,
fields,
})
}
}
impl Parse for Initializer {
fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
let attrs = input.call(Attribute::parse_outer)?;
let this = input.peek(Token![&]).then(|| input.parse()).transpose()?;
let path = input.parse()?;
let kind = if input.peek(token::Brace) {
InitExprKind::Struct(InitExprStruct::parse_with_path(path, input)?)
} else if input.peek(token::Paren) {
InitExprKind::Tuple(InitExprTuple::parse_with_path(path, input)?)
} else {
return Err(input.error("expected curly braces or parentheses"));
};
let error = input
.peek(Token![?])
.then(|| Ok::<_, syn::Error>((input.parse()?, input.parse()?)))
@ -412,10 +684,7 @@ fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
Ok(Self {
attrs,
this,
path,
brace_token,
fields,
rest,
kind,
error,
})
}
@ -450,33 +719,167 @@ fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
impl Parse for InitializerKind {
fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
let lh = input.lookahead1();
if lh.peek(Token![_]) {
Ok(Self::Code {
let member = if lh.peek(Token![_]) {
return Ok(Self::Code {
_underscore_token: input.parse()?,
_colon_token: input.parse()?,
block: input.parse()?,
});
} else if lh.peek(Ident) || lh.peek(LitInt) {
input.parse::<Member>()?
} else {
return Err(lh.error());
};
let lh = input.lookahead1();
if lh.peek(Token![<-]) {
Ok(Self::Init {
member,
left_arrow_token: input.parse()?,
value: input.parse()?,
})
} else if lh.peek(Token![:]) {
Ok(Self::Value {
member,
value: Some((input.parse()?, input.parse()?)),
})
} else if matches!(member, Member::Named(_)) && (lh.peek(Token![,]) || lh.peek(End)) {
// Short-hand syntax, available for named fields only.
Ok(Self::Value {
member,
value: None,
})
} else if lh.peek(Ident) {
let ident = input.parse()?;
let lh = input.lookahead1();
if lh.peek(Token![<-]) {
Ok(Self::Init {
ident,
_left_arrow_token: input.parse()?,
value: input.parse()?,
})
} else if lh.peek(Token![:]) {
Ok(Self::Value {
ident,
value: Some((input.parse()?, input.parse()?)),
})
} else if lh.peek(Token![,]) || lh.peek(End) {
Ok(Self::Value { ident, value: None })
} else {
Err(lh.error())
}
} else {
Err(lh.error())
}
}
}
impl<Kind: ToTokens> ToTokens for Initializer<Kind> {
fn to_tokens(&self, tokens: &mut TokenStream) {
let Self {
attrs,
this,
kind,
error,
} = self;
tokens.append_all(attrs);
this.to_tokens(tokens);
kind.to_tokens(tokens);
if let Some((question, ty)) = error {
question.to_tokens(tokens);
ty.to_tokens(tokens);
}
}
}
impl ToTokens for InitExprKind {
fn to_tokens(&self, tokens: &mut TokenStream) {
match self {
Self::Struct(init) => init.to_tokens(tokens),
Self::Tuple(init) => init.to_tokens(tokens),
}
}
}
impl ToTokens for InitExprStruct {
fn to_tokens(&self, tokens: &mut TokenStream) {
let Self {
path,
brace_token,
fields,
rest,
} = self;
path.to_tokens(tokens);
brace_token.surround(tokens, |tokens| {
fields.to_tokens(tokens);
if let Some((dotdot, expr)) = rest {
dotdot.to_tokens(tokens);
expr.to_tokens(tokens);
}
});
}
}
impl ToTokens for InitExprTuple {
fn to_tokens(&self, tokens: &mut TokenStream) {
let Self {
path,
paren_token,
fields,
} = self;
path.to_tokens(tokens);
paren_token.surround(tokens, |tokens| fields.to_tokens(tokens));
}
}
impl ToTokens for InitTupleField {
fn to_tokens(&self, tokens: &mut TokenStream) {
let Self { attrs, value } = self;
tokens.append_all(attrs);
value.to_tokens(tokens);
}
}
impl ToTokens for InitializerAttribute {
fn to_tokens(&self, tokens: &mut TokenStream) {
match self {
Self::DefaultError(DefaultErrorAttribute { ty }) => {
quote!(#[default_error(#ty)]).to_tokens(tokens);
}
}
}
}
impl ToTokens for This {
fn to_tokens(&self, tokens: &mut TokenStream) {
let Self {
_and_token,
ident,
_in_token,
} = self;
_and_token.to_tokens(tokens);
ident.to_tokens(tokens);
_in_token.to_tokens(tokens);
}
}
impl ToTokens for InitializerField {
fn to_tokens(&self, tokens: &mut TokenStream) {
let Self { attrs, kind } = self;
tokens.append_all(attrs);
kind.to_tokens(tokens);
}
}
impl ToTokens for InitializerKind {
fn to_tokens(&self, tokens: &mut TokenStream) {
match self {
Self::Value { member, value } => {
member.to_tokens(tokens);
if let Some((colon, expr)) = value {
colon.to_tokens(tokens);
expr.to_tokens(tokens);
}
}
Self::Init {
member,
left_arrow_token,
value,
} => {
member.to_tokens(tokens);
left_arrow_token.to_tokens(tokens);
value.to_tokens(tokens);
}
Self::Code {
_underscore_token,
_colon_token,
block,
} => {
_underscore_token.to_tokens(tokens);
_colon_token.to_tokens(tokens);
block.to_tokens(tokens);
}
}
}
}

41
rust/pin-init/internal/src/lib.rs
View File

@ -10,7 +10,6 @@
#![allow(missing_docs)]
use proc_macro::TokenStream;
use syn::parse_macro_input;
use crate::diagnostics::DiagCtxt;
@ -18,42 +17,52 @@
mod init;
mod pin_data;
mod pinned_drop;
mod util;
mod zeroable;
#[proc_macro_attribute]
pub fn pin_data(args: TokenStream, input: TokenStream) -> TokenStream {
let args = parse_macro_input!(args);
let input = parse_macro_input!(input);
DiagCtxt::with(|dcx| pin_data::pin_data(args, input, dcx)).into()
DiagCtxt::for_item(|dcx| pin_data::expand_with_cfg(syn::parse(args)?, syn::parse(input)?, dcx))
.into()
}
#[proc_macro_attribute]
pub fn pinned_drop(args: TokenStream, input: TokenStream) -> TokenStream {
let args = parse_macro_input!(args);
let input = parse_macro_input!(input);
DiagCtxt::with(|dcx| pinned_drop::pinned_drop(args, input, dcx)).into()
DiagCtxt::for_item(|dcx| pinned_drop::pinned_drop(syn::parse(args)?, syn::parse(input)?, dcx))
.into()
}
#[proc_macro_derive(Zeroable)]
pub fn derive_zeroable(input: TokenStream) -> TokenStream {
let input = parse_macro_input!(input);
DiagCtxt::with(|dcx| zeroable::derive(input, dcx)).into()
DiagCtxt::for_item(|dcx| zeroable::derive(syn::parse(input)?, dcx)).into()
}
#[proc_macro_derive(MaybeZeroable)]
pub fn maybe_derive_zeroable(input: TokenStream) -> TokenStream {
let input = parse_macro_input!(input);
DiagCtxt::with(|dcx| zeroable::maybe_derive(input, dcx)).into()
DiagCtxt::for_item(|dcx| zeroable::maybe_derive(syn::parse(input)?, dcx)).into()
}
#[proc_macro]
pub fn init(input: TokenStream) -> TokenStream {
let input = parse_macro_input!(input);
DiagCtxt::with(|dcx| init::expand(input, Some("::core::convert::Infallible"), false, dcx))
.into()
DiagCtxt::for_expr(|dcx| {
init::expand_with_cfg(
syn::parse(input)?,
Some("::core::convert::Infallible"),
false,
dcx,
)
})
.into()
}
#[proc_macro]
pub fn pin_init(input: TokenStream) -> TokenStream {
let input = parse_macro_input!(input);
DiagCtxt::with(|dcx| init::expand(input, Some("::core::convert::Infallible"), true, dcx)).into()
DiagCtxt::for_expr(|dcx| {
init::expand_with_cfg(
syn::parse(input)?,
Some("::core::convert::Infallible"),
true,
dcx,
)
})
.into()
}

1859
rust/pin-init/internal/src/pin_data.rs
View File

File diff suppressed because it is too large Load Diff

500
rust/pin-init/internal/src/util.rs Normal file
View File

@ -0,0 +1,500 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
use std::collections::{BTreeMap, BTreeSet};
use proc_macro2::{Ident, TokenStream};
use quote::{format_ident, ToTokens};
use syn::{
parse_quote, visit::Visit, visit_mut::VisitMut, Attribute, BoundLifetimes, GenericParam,
Generics, Index, Lifetime, Member, Token, Type, TypePath,
};
use crate::DiagCtxt;
pub(crate) trait AttrListExt {
fn extract_cfg_attrs(&mut self) -> Vec<TokenStream>;
/// Extract attribute with identifier `path`.
///
/// Report error if the attribute appears multiple times.
fn extract_single_attr(&mut self, dcx: &DiagCtxt, path: &str) -> Option<Attribute>;
}
impl AttrListExt for Vec<Attribute> {
fn extract_cfg_attrs(&mut self) -> Vec<TokenStream> {
let cfg: Vec<_> = self
.iter()
.filter(|a| a.path().is_ident("cfg"))
.map(|a| {
a.parse_args::<TokenStream>()
.expect("parse as token stream cannot fail")
})
.collect();
if !cfg.is_empty() {
self.retain(|a| !a.path().is_ident("cfg"));
}
cfg
}
fn extract_single_attr(&mut self, dcx: &DiagCtxt, path: &str) -> Option<Attribute> {
// FIXME: Replace with `extract_if` when MSRV >= 1.85.
let attr_pos = self.iter().position(|attr| attr.path().is_ident(path))?;
let attr = self.remove(attr_pos);
self.retain(|attr| {
if !attr.path().is_ident(path) {
return true;
}
dcx.error(
attr,
format!("`#[{path}]` attribute specified more than once"),
);
false
});
Some(attr)
}
}
pub(crate) trait MemberExt {
/// Returns an identifier for the member.
///
/// Tuple fields have no name of their own, so they are named `_0`, `_1`, ... instead.
fn as_ident(&self) -> Ident;
/// Obtain a display name for the member in diagnostics.
fn display_name(&self) -> String;
}
impl MemberExt for Member {
fn as_ident(&self) -> Ident {
match self {
Member::Named(ident) => ident.clone(),
Member::Unnamed(Index { index, span }) => format_ident!("_{index}", span = *span),
}
}
fn display_name(&self) -> String {
match self {
Member::Named(ident) => format!("`{ident}`"),
Member::Unnamed(Index { index, .. }) => format!("index `{index}`"),
}
}
}
pub(crate) struct CombinedGenerics<'a>(pub(crate) Vec<&'a Generics>);
pub(crate) struct CombinedImplGenerics<'a>(&'a CombinedGenerics<'a>);
pub(crate) struct CombinedTypeGenerics<'a>(&'a CombinedGenerics<'a>);
pub(crate) struct CombinedWhereClauses<'a>(&'a CombinedGenerics<'a>);
impl CombinedGenerics<'_> {
pub(crate) fn split_for_impl(
&self,
) -> (
CombinedImplGenerics<'_>,
CombinedTypeGenerics<'_>,
CombinedWhereClauses<'_>,
) {
(
CombinedImplGenerics(self),
CombinedTypeGenerics(self),
CombinedWhereClauses(self),
)
}
}
impl ToTokens for CombinedGenerics<'_> {
fn to_tokens(&self, tokens: &mut TokenStream) {
// Most of the time we are only adding lifetimes, so we prefer to place
// the user provided generics last.
self.0
.last()
.and_then(|x| x.lt_token)
.unwrap_or_default()
.to_tokens(tokens);
let comma: Token![,] = Default::default();
// Output lifetimes first.
for generics in self.0.iter() {
for param in generics.params.pairs() {
if let GenericParam::Lifetime(lt) = param.value() {
lt.to_tokens(tokens);
param.punct().unwrap_or(&&comma).to_tokens(tokens);
}
}
}
for generics in self.0.iter() {
for param in generics.params.pairs() {
if let GenericParam::Lifetime(_) = param.value() {
continue;
};
param.value().to_tokens(tokens);
param.punct().unwrap_or(&&comma).to_tokens(tokens);
}
}
self.0
.last()
.and_then(|x| x.gt_token)
.unwrap_or_default()
.to_tokens(tokens);
}
}
impl ToTokens for CombinedImplGenerics<'_> {
fn to_tokens(&self, tokens: &mut TokenStream) {
self.0
.0
.last()
.and_then(|x| x.lt_token)
.unwrap_or_default()
.to_tokens(tokens);
let comma: Token![,] = Default::default();
// Output lifetimes first.
for generics in self.0 .0.iter() {
for param in generics.params.pairs() {
if let GenericParam::Lifetime(lt) = param.value() {
lt.to_tokens(tokens);
param.punct().unwrap_or(&&comma).to_tokens(tokens);
}
}
}
for generics in self.0 .0.iter() {
for param in generics.params.pairs() {
// Leave out defaults.
match param.value() {
GenericParam::Lifetime(_) => continue,
GenericParam::Type(param) => {
param.ident.to_tokens(tokens);
if !param.bounds.is_empty() {
param
.colon_token
.unwrap_or_else(Default::default)
.to_tokens(tokens);
param.bounds.to_tokens(tokens);
}
}
GenericParam::Const(param) => {
param.const_token.to_tokens(tokens);
param.ident.to_tokens(tokens);
param.colon_token.to_tokens(tokens);
param.ty.to_tokens(tokens);
}
}
param.punct().unwrap_or(&&comma).to_tokens(tokens);
}
}
self.0
.0
.last()
.and_then(|x| x.gt_token)
.unwrap_or_default()
.to_tokens(tokens);
}
}
impl ToTokens for CombinedTypeGenerics<'_> {
fn to_tokens(&self, tokens: &mut TokenStream) {
self.0
.0
.last()
.and_then(|x| x.lt_token)
.unwrap_or_default()
.to_tokens(tokens);
let comma: Token![,] = Default::default();
// Output lifetimes first.
for generics in self.0 .0.iter() {
for param in generics.params.pairs() {
if let GenericParam::Lifetime(lt) = param.value() {
// Leave out bounds
lt.lifetime.to_tokens(tokens);
param.punct().unwrap_or(&&comma).to_tokens(tokens);
}
}
}
for generics in self.0 .0.iter() {
for param in generics.params.pairs() {
// Leave out bounds and defaults.
match param.value() {
GenericParam::Lifetime(_) => continue,
GenericParam::Type(param) => {
param.ident.to_tokens(tokens);
}
GenericParam::Const(param) => {
param.ident.to_tokens(tokens);
}
}
param.punct().unwrap_or(&&comma).to_tokens(tokens);
}
}
self.0
.0
.last()
.and_then(|x| x.gt_token)
.unwrap_or_default()
.to_tokens(tokens);
}
}
impl ToTokens for CombinedWhereClauses<'_> {
fn to_tokens(&self, tokens: &mut TokenStream) {
self.0
.0
.iter()
.filter_map(|x| Some(x.where_clause.as_ref()?.where_token))
.next_back()
.unwrap_or_default()
.to_tokens(tokens);
let comma: Token![,] = Default::default();
for generics in self.0 .0.iter() {
let Some(where_clause) = &generics.where_clause else {
continue;
};
where_clause.predicates.to_tokens(tokens);
if !where_clause.predicates.empty_or_trailing() {
comma.to_tokens(tokens);
}
}
}
}
pub(crate) trait LifetimeExt {
/// Get a visitor that call the provided function for all unbound lifetimes.
fn visitor<'a>(f: impl FnMut(&'a Lifetime)) -> impl Visit<'a>;
/// Obtain a lifetime from a identifier.
///
/// The created lifetime has the same span.
fn from_ident(ident: &Ident) -> Self;
}
impl LifetimeExt for Lifetime {
fn visitor<'a>(f: impl FnMut(&'a Lifetime)) -> impl Visit<'a> {
LifetimeVisitor {
bound: BTreeSet::new(),
visit: f,
}
}
fn from_ident(ident: &Ident) -> Self {
Lifetime {
apostrophe: ident.span(),
ident: ident.clone(),
}
}
}
struct LifetimeVisitor<'a, F> {
bound: BTreeSet<&'a Lifetime>,
visit: F,
}
impl<'a, F> LifetimeVisitor<'a, F> {
fn with_bound_lifetimes(
&mut self,
bound: Option<&'a BoundLifetimes>,
f: impl FnOnce(&mut Self),
) {
// In case the type includes a lifetime binder, e.g. `dyn for<'a> Foo`,
// the lifetimes in the binder are bound and should not be visited.
let mut to_remove = Vec::new();
if let Some(bound) = bound {
for lt in &bound.lifetimes {
let GenericParam::Lifetime(lt) = lt else {
continue;
};
if !self.bound.contains(&&lt.lifetime) {
self.bound.insert(&lt.lifetime);
to_remove.push(&lt.lifetime);
}
}
}
f(self);
for lt in to_remove {
self.bound.remove(lt);
}
}
}
impl<'a, F: FnMut(&'a Lifetime)> Visit<'a> for LifetimeVisitor<'a, F> {
fn visit_lifetime(&mut self, lt: &'a Lifetime) {
if lt.ident == "static" {
return;
}
if !self.bound.contains(lt) {
(self.visit)(lt);
}
}
fn visit_trait_bound(&mut self, bound: &'a syn::TraitBound) {
self.with_bound_lifetimes(bound.lifetimes.as_ref(), |this| {
this.visit_path(&bound.path)
});
}
fn visit_type_bare_fn(&mut self, bare_fn: &'a syn::TypeBareFn) {
self.with_bound_lifetimes(bare_fn.lifetimes.as_ref(), |this| {
for input in bare_fn.inputs.iter() {
this.visit_bare_fn_arg(input);
}
this.visit_return_type(&bare_fn.output);
});
}
}
pub(crate) trait GenericParamExt {
fn maybe_type_params_visitor<'a>(f: impl FnMut(&'a Ident)) -> impl Visit<'a>;
}
impl GenericParamExt for GenericParam {
fn maybe_type_params_visitor<'a>(f: impl FnMut(&'a Ident)) -> impl Visit<'a> {
struct TypeParamVisitor<F>(F);
impl<'a, F> Visit<'a> for TypeParamVisitor<F>
where
F: FnMut(&'a Ident),
{
fn visit_type_path(&mut self, ty: &'a TypePath) {
if ty.qself.is_none()
&& ty.path.leading_colon.is_none()
&& ty.path.segments[0].arguments.is_none()
{
(self.0)(&ty.path.segments[0].ident);
}
syn::visit::visit_type_path(self, ty);
}
}
TypeParamVisitor(f)
}
}
pub(crate) trait TypeExt {
/// Check if the type includes macro invocations.
///
/// Proc-macros cannot expand macros and peek into them, so if macro is involved sometimes
/// special handling is required.
fn has_macro(&self) -> bool;
fn replace_lifetimes(&self, needle: &[&Lifetime], replacement: &[&Lifetime]) -> Type;
}
impl TypeExt for Type {
fn has_macro(&self) -> bool {
struct HasMacro(bool);
impl<'ast> Visit<'ast> for HasMacro {
fn visit_macro(&mut self, _: &'ast syn::Macro) {
self.0 = true;
}
}
let mut visitor = HasMacro(false);
visitor.visit_type(self);
visitor.0
}
fn replace_lifetimes(&self, needle: &[&Lifetime], replacement: &[&Lifetime]) -> Type {
if needle.is_empty() {
return self.clone();
}
// If the type has macro, we cannot peek into it. Use some different approach to replace
// the type using GAT.
if self.has_macro() {
return parse_quote!(
<
for<#(#needle,)*> fn(#(&#needle (),)*) -> #self
as
::pin_init::__internal::FnOutput<(#(&#replacement (),)*)>
>::Output
);
}
struct LifetimeReplacer<'a> {
to_replace: BTreeMap<&'a Lifetime, &'a Lifetime>,
}
impl<'a> LifetimeReplacer<'a> {
fn with_bound_lifetimes(
&mut self,
bound: Option<&BoundLifetimes>,
f: impl FnOnce(&mut Self),
) {
// In case the type includes a lifetime binder, e.g. `dyn for<'a> Foo`,
// temporarily remove them from to_replace if they're.
let mut removed = Vec::new();
if let Some(bound) = bound {
for lt in &bound.lifetimes {
let GenericParam::Lifetime(lt) = lt else {
continue;
};
if let Some(entry) = self.to_replace.remove_entry(&lt.lifetime) {
removed.push(entry);
}
}
}
f(self);
for (key, val) in removed {
self.to_replace.insert(key, val);
}
}
}
impl VisitMut for LifetimeReplacer<'_> {
fn visit_lifetime_mut(&mut self, lt: &mut syn::Lifetime) {
if let Some(&replacement) = self.to_replace.get(lt) {
*lt = replacement.clone();
}
}
fn visit_trait_bound_mut(&mut self, bound: &mut syn::TraitBound) {
self.with_bound_lifetimes(bound.lifetimes.as_ref(), |this| {
this.visit_path_mut(&mut bound.path)
});
}
fn visit_type_bare_fn_mut(&mut self, bare_fn: &mut syn::TypeBareFn) {
self.with_bound_lifetimes(bare_fn.lifetimes.as_ref(), |this| {
for input in bare_fn.inputs.iter_mut() {
this.visit_bare_fn_arg_mut(input);
}
this.visit_return_type_mut(&mut bare_fn.output);
});
}
}
let mut ret = self.clone();
LifetimeReplacer {
to_replace: needle
.iter()
.copied()
.zip(replacement.iter().copied())
.collect(),
}
.visit_type_mut(&mut ret);
ret
}
}

417
rust/pin-init/src/__internal.rs
View File

@ -5,6 +5,11 @@
//! These items must not be used outside of this crate and the pin-init-internal crate located at
//! `../internal`.
#![expect(clippy::new_without_default, reason = "private API")]
use core::marker::PhantomPinned;
use core::ops::Deref;
use super::*;
/// Zero-sized type used to mark a type as invariant.
@ -42,22 +47,6 @@ pub const fn new() -> Self {
}
}
/// Zero-sized type used to mark a lifetime as invariant.
///
/// This is a polyfill for the [unstable type] in the standard library of the same name.
///
/// [unstable type]: https://doc.rust-lang.org/nightly/std/marker/struct.PhantomInvariantLifetime.html
#[repr(transparent)]
#[derive(Clone, Copy, Default)]
pub struct PhantomInvariantLifetime<'a>(PhantomInvariant<&'a ()>);
impl PhantomInvariantLifetime<'_> {
#[inline(always)]
pub const fn new() -> Self {
Self(PhantomInvariant::new())
}
}
/// Token type to signify successful initialization.
///
/// Can only be constructed via the unsafe [`Self::new`] function. The initializer macros use this
@ -81,56 +70,55 @@ pub unsafe fn new() -> Self {
///
/// # Safety
///
/// Only the `init` module is allowed to use this trait.
/// `pin-init` relies on the correctness of the helper functions defined on `PinData`.
/// Thus, only the `#[pin_data]` can implement this trait.
#[diagnostic::on_unimplemented(
message = "`{Self}` cannot be used with `pin_init!` macro",
note = "did you forget to add `#[pin_data]` attribute to the struct?"
)]
pub unsafe trait HasPinData {
type PinData;
#[expect(clippy::missing_safety_doc)]
unsafe fn __pin_data() -> Self::PinData;
fn __pin_data(_: InitData<Self>) -> Self::PinData;
}
/// This trait is automatically implemented for every type. It aims to provide the same type
/// inference help as `HasPinData`.
/// This trait is automatically implemented for every type.
///
/// # Safety
///
/// Only the `init` module is allowed to use this trait.
pub unsafe trait HasInitData {
type InitData;
#[expect(clippy::missing_safety_doc)]
unsafe fn __init_data() -> Self::InitData;
/// It aims to provide type inference help; `PATH::__init_data()` would be able to retrieve an
/// instance of `InitData<PATH<Generics>>` without having to mention the generics explicitly.
pub trait HasInitData {
#[inline]
fn __init_data() -> InitData<Self> {
InitData(PhantomInvariant::new())
}
}
pub struct AllData<T: ?Sized>(PhantomInvariant<T>);
impl<T: ?Sized> HasInitData for T {}
impl<T: ?Sized> Clone for AllData<T> {
pub struct InitData<T: ?Sized>(PhantomInvariant<T>);
impl<T: ?Sized> Clone for InitData<T> {
#[inline]
fn clone(&self) -> Self {
*self
}
}
impl<T: ?Sized> Copy for AllData<T> {}
impl<T: ?Sized> Copy for InitData<T> {}
impl<T: ?Sized> AllData<T> {
impl<T: ?Sized> InitData<T> {
/// Type inference helper function.
#[inline(always)]
pub fn __make_closure<F, E>(self, f: F) -> F
where
F: FnOnce(*mut T) -> Result<InitOk, E>,
F: FnOnce(*mut T, Self) -> Result<InitOk, E>,
{
f
}
}
// SAFETY: TODO.
unsafe impl<T: ?Sized> HasInitData for T {
type InitData = AllData<T>;
#[inline]
unsafe fn __init_data() -> Self::InitData {
AllData(PhantomInvariant::new())
#[inline(always)]
pub fn __with_lt(self) -> Self {
self
}
}
@ -341,6 +329,12 @@ pub fn let_binding(&mut self) -> &mut T {
// SAFETY: Per type invariant.
unsafe { &mut *self.ptr }
}
/// Create a let binding for accessor use in dropck.
#[inline]
pub fn let_binding_in_dropck(&mut self) -> &mut T {
self.let_binding()
}
}
impl<T: ?Sized> DropGuard<Pinned, T> {
@ -351,6 +345,12 @@ pub fn let_binding(&mut self) -> Pin<&mut T> {
// pinned per type invariant.
unsafe { Pin::new_unchecked(&mut *self.ptr) }
}
/// Create a let binding for accessor use in dropck.
#[inline]
pub fn let_binding_in_dropck(&mut self) -> Pin<&mut T> {
self.let_binding()
}
}
impl<P, T: ?Sized> Drop for DropGuard<P, T> {
@ -361,6 +361,192 @@ fn drop(&mut self) {
}
}
pub struct Shared;
pub struct Mutable;
/// Represent an uninitialized field in a pinned struct that will be referenced by other fields.
///
/// # Invariants
///
/// - `ptr` is valid, properly aligned and points to uninitialized and exclusively accessed memory
/// and will live longer than `'a`.
/// - If `P` is `Pinned`, then `ptr` is structurally pinned.
pub struct SelfRefSlot<'a, M, P, T: ?Sized> {
ptr: *mut T,
_phantom: PhantomData<(M, P, &'a mut T)>,
}
impl<'a, M, P, T: ?Sized> SelfRefSlot<'a, M, P, T> {
/// # Safety
///
/// - `ptr` is valid, properly aligned and points to uninitialized and exclusively accessed
/// memory and will live longer than `'a`.
/// - If `P` is `Pinned`, then `ptr` is structurally pinned.
#[inline]
pub unsafe fn new(ptr: *mut T) -> Self {
// INVARIANT: Per safety requirement.
Self {
ptr,
_phantom: PhantomData,
}
}
/// Initialize the field by value.
#[inline]
pub fn write(self, value: T) -> SelfRefDropGuard<'a, M, P, T>
where
T: Sized,
{
// SAFETY: `self.ptr` is a valid and aligned pointer for write.
unsafe { self.ptr.write(value) }
// SAFETY:
// - `self.ptr` is valid, properly aligned and live longer than `'a` per type invariant.
// - `*self.ptr` is initialized above and the ownership is transferred to the guard.
// - If `P` is `Pinned`, `self.ptr` is pinned.
unsafe { SelfRefDropGuard::new(self.ptr) }
}
}
impl<'a, M, T: ?Sized> SelfRefSlot<'a, M, Unpinned, T> {
/// Initialize the field.
#[inline]
pub fn init<E>(self, init: impl Init<T, E>) -> Result<SelfRefDropGuard<'a, M, Unpinned, T>, E> {
// SAFETY:
// - `self.ptr` is valid and properly aligned.
// - when `Err` is returned, we also propagate the error without touching `ptr`;
// also `self` is consumed so it cannot be touched further.
unsafe { init.__init(self.ptr)? };
// SAFETY:
// - `self.ptr` is valid, properly aligned and live longer than `'a` per type invariant.
// - `*self.ptr` is initialized above and the ownership is transferred to the guard.
Ok(unsafe { SelfRefDropGuard::new(self.ptr) })
}
}
impl<'a, M, T: ?Sized> SelfRefSlot<'a, M, Pinned, T> {
/// Initialize the field.
#[inline]
pub fn init<E>(
self,
init: impl PinInit<T, E>,
) -> Result<SelfRefDropGuard<'a, M, Pinned, T>, E> {
// SAFETY:
// - `ptr` is valid
// - when `Err` is returned, we also propagate the error without touching `ptr`;
// also `self` is consumed so it cannot be touched further.
// - the drop guard will not hand out `&mut` (only `Pin<&mut T>`).
unsafe { init.__init(self.ptr)? };
// SAFETY:
// - `self.ptr` is valid, properly aligned and live longer than `'a` per type invariant.
// - `*self.ptr` is initialized above and the ownership is transferred to the guard.
Ok(unsafe { SelfRefDropGuard::new(self.ptr) })
}
}
/// When a value of this type is dropped, it drops a `T`.
///
/// Can be forgotten to prevent the drop.
///
/// # Invariants
///
/// - `ptr` is valid, properly aligned and live longer than `'a`.
/// - `*ptr` is initialized and owned by this guard.
/// - if `P` is `Pinned`, `ptr` is pinned.
pub struct SelfRefDropGuard<'a, M, P, T: ?Sized> {
ptr: *mut T,
phantom: PhantomData<(M, P, &'a mut T)>,
}
impl<'a, M, P, T: ?Sized> SelfRefDropGuard<'a, M, P, T> {
/// Creates a drop guard and transfer the ownership of the pointer content.
///
/// The ownership is only relinquished if the guard is forgotten via [`core::mem::forget`].
///
/// # Safety
///
/// - `ptr` is valid, properly aligned and live longer than `'a`.
/// - `*ptr` is initialized, and the ownership is transferred to this guard.
/// - if `P` is `Pinned`, `ptr` is pinned.
#[inline]
pub unsafe fn new(ptr: *mut T) -> Self {
// INVARIANT: By safety requirement.
Self {
ptr,
phantom: PhantomData,
}
}
}
impl<'a, T: ?Sized> SelfRefDropGuard<'a, Shared, Unpinned, T> {
/// Create a let binding for accessor use.
#[inline]
pub fn let_binding(&mut self) -> &'a T {
// SAFETY: Per type invariant.
unsafe { &*self.ptr }
}
/// Create a let binding for accessor use in dropck.
#[inline]
pub fn let_binding_in_dropck(&mut self) -> &T {
self.let_binding()
}
}
impl<'a, T: ?Sized> SelfRefDropGuard<'a, Shared, Pinned, T> {
/// Create a let binding for accessor use.
#[inline]
pub fn let_binding(&mut self) -> Pin<&'a T> {
// SAFETY: `self.ptr` is valid, properly aligned, live longer than `'a`, initialized,
// exclusively accessible and pinned per type invariant.
unsafe { Pin::new_unchecked(&*self.ptr) }
}
/// Create a let binding for accessor use in dropck.
#[inline]
pub fn let_binding_in_dropck(&mut self) -> Pin<&T> {
self.let_binding()
}
}
impl<'a, T: ?Sized> SelfRefDropGuard<'a, Mutable, Unpinned, T> {
/// Create a let binding for accessor use.
#[inline]
pub fn let_binding(&mut self) -> &'a mut T {
// SAFETY: Per type invariant.
unsafe { &mut *self.ptr }
}
/// Create a let binding for accessor use in dropck.
#[inline]
pub fn let_binding_in_dropck(&mut self) -> &mut T {
self.let_binding()
}
}
impl<'a, T: ?Sized> SelfRefDropGuard<'a, Mutable, Pinned, T> {
/// Create a let binding for accessor use.
#[inline]
pub fn let_binding(&mut self) -> Pin<&'a mut T> {
// SAFETY: `self.ptr` is valid, properly aligned, live longer than `'a`, initialized,
// exclusively accessible and pinned per type invariant.
unsafe { Pin::new_unchecked(&mut *self.ptr) }
}
#[inline]
pub fn let_binding_in_dropck(&mut self) -> Pin<&mut T> {
self.let_binding()
}
}
impl<M, P, T: ?Sized> Drop for SelfRefDropGuard<'_, M, P, T> {
#[inline]
fn drop(&mut self) {
// SAFETY: `self.ptr` is valid, properly aligned and `*self.ptr` is owned by this guard.
unsafe { ptr::drop_in_place(self.ptr) }
}
}
/// Token used by `PinnedDrop` to prevent calling the function without creating this unsafely
/// created struct. This is needed, because the `drop` function is safe, but should not be called
/// manually.
@ -407,3 +593,150 @@ unsafe fn __init(self, _slot: *mut T) -> Result<(), ()> {
Err(())
}
}
/// Polyfill of `FnOnce` trait to be able to reference output via associated type.
pub trait FnOutput<Args> {
type Output;
}
macro_rules! impl_fn_output {
() => {};
($ret:ident, $($arg:ident,)*) => {
impl<This, $ret, $($arg,)*> FnOutput<($($arg,)*)> for This
where
This: FnOnce($($arg,)*) -> $ret,
{
type Output = $ret;
}
impl_fn_output!($($arg,)*);
};
}
impl_fn_output!(A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U,);
/// Lifetime erasure facility.
///
/// Say we have `exists<'a, 'b> Foo<'a, 'b>` and we want to store it. There's no concrete
/// lifetimes we can use, so we want to erase the lifetime.
///
/// Such erasure can be encoded as
/// `Erased<for<'a> fn(&'a ()) -> for<'b> fn(&'b()) -> (Foo<'a, 'b>,)`.
///
/// This can be considered the stable version of Rust's `unsafe_binder` feature, without the
/// no-drop-glue requirement.
#[repr(transparent)]
#[allow(private_bounds)]
pub struct Erase<F: EraseLt>(F::Erased);
/// Helper trait to resolve the erased lifetime.
trait EraseLt {
type Erased;
}
impl<T> EraseLt for (T,) {
type Erased = T;
}
impl<T> EraseLt for T
where
T: for<'a> FnOutput<(&'a (),), Output: EraseLt>,
{
type Erased = <<T as FnOutput<(&'static (),)>>::Output as EraseLt>::Erased;
}
// The default `Send` and `Sync` are not sufficient, because one can use lifetime specialization to
// implement `Send` or `Sync` for a concrete instance of lifetime. Use HRTB to ensure that the type
// will only implement `Send` or `Sync` if it's implemented for *all* erased lifetimes.
// SAFETY: Trivial, no lifetime to erase.
unsafe impl<T: Send> Send for Erase<(T,)> {}
// SAFETY: If we erased a lifetime, then the type needs to be `Send` for across *all* that
// lifetimes.
unsafe impl<F: EraseLt> Send for Erase<F>
where
F: for<'a> FnOutput<(&'a (),), Output: EraseLt>,
for<'a> Erase<<F as FnOutput<(&'a (),)>>::Output>: Send,
{
}
// SAFETY: Trivial, no lifetime to erase.
unsafe impl<T: Sync> Sync for Erase<(T,)> {}
// SAFETY: If we erased a lifetime, then the type needs to be `Sync` for across *all* that
// lifetimes.
unsafe impl<F: EraseLt> Sync for Erase<F>
where
F: for<'a> FnOutput<(&'a (),), Output: EraseLt>,
for<'a> Erase<<F as FnOutput<(&'a (),)>>::Output>: Sync,
{
}
/// Wrapper for borrowed fields.
///
/// This should be switched to `UnsafePinned` when it is stable.
/// NOTE: This type needs to be covariant; Rust's 1.89+'s `UnsafePinned` is invariant.
#[repr(transparent)]
pub struct Borrowed<T: ?Sized, P>(PhantomInvariant<P>, PhantomPinned, T);
// Lifetimes not needed by drop glue are considered by Rust's drop check to be considered
// `#[may_dangle]`. In case for a self-referential struct, we may have fields which need lifetime of
// borrowed fields in their drop glue, so compiler's automatic check is insufficient.
//
// Code like this:
// ```
// #[pin_data]
// struct SelfRef<'a> {
// borrow: PrintOnDrop<&'owner str>,
// owner: &'a str,
// }
// ```
// may access `owner` during the drop, however Rust will determine that since `'a` only is used in
// `owner`, the `'a` lifetime may dangle during drop.
//
// This is undesirable for pin-init self references, because `&'a str` could be coerecd to
// `&'owner str` and this could further coerce if there're implied outlives, e.g.
// `&'earlier_field &'owner ()` would allow `&'owner str` to further coerce to `&'earlier_field`.
//
// Thus, if any self-referential field require field lifetime access in `Drop` impl, we would need
// to ensure that the all generic parameters visible by self-referential fields would strictly
// outlive the struct. And this can be done by a simple `Drop` impl that does nothing. Without a
// dropck eye patch, presence of `Drop` impl, albeit empty, tells the drop check that the strict
// outlive relation is needed.
impl<T: ?Sized, P> Drop for Borrowed<T, P> {
#[inline(always)]
fn drop(&mut self) {}
}
impl<T: ?Sized, P> Deref for Borrowed<T, P> {
type Target = T;
#[inline(always)]
fn deref(&self) -> &T {
&self.2
}
}
/// An alias of `PhantomData` but with a name to aid user in case of misuse.
pub struct NotVisible<T: ?Sized>(PhantomData<T>);
impl<T: ?Sized> NotVisible<T> {
#[inline(always)]
pub fn new() -> Self {
Self(PhantomData)
}
}
/// Marker type to capture outlive bounds coming from existential lifetimes.
pub struct ExistLt<'a>(PhantomData<&'a ()>);
// Dummy `Drop`, same reason as `Borrowed`.
impl Drop for ExistLt<'_> {
#[inline(always)]
fn drop(&mut self) {}
}
/// Type that allows additional `PhantomData` to be attached.
///
/// This allows changing variance of types without introducing additional fields.
#[repr(transparent)]
pub struct WithPhantom<T, P>(PhantomData<P>, T);

83
rust/pin-init/src/lib.rs
View File

@ -304,6 +304,9 @@
/// This macro enables the use of the [`pin_init!`] macro. When pin-initializing a `struct`,
/// then `#[pin]` directs the type of initializer that is required.
///
/// Tuple structs are supported as well. Their fields have no names, so the generated projection
/// is a tuple struct too and its fields are accessed by index.
///
/// If your `struct` implements `Drop`, then you need to add `PinnedDrop` as arguments to this
/// macro, and change your `Drop` implementation to `PinnedDrop` annotated with
/// `#[`[`macro@pinned_drop`]`]`, since dropping pinned values requires extra care.
@ -327,6 +330,26 @@
/// }
/// ```
///
/// The same as a tuple struct, projected by index:
///
/// ```
/// # #![feature(allocator_api)]
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
/// use core::pin::Pin;
/// use pin_init::pin_data;
///
/// enum Command {
/// /* ... */
/// }
///
/// #[pin_data]
/// struct DriverData(#[pin] CMutex<Vec<Command>>, Box<[u8; 1024 * 1024]>);
///
/// fn queue(data: Pin<&mut DriverData>) -> Pin<&mut CMutex<Vec<Command>>> {
/// data.project().0
/// }
/// ```
///
/// ```
/// # #![feature(allocator_api)]
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
@ -569,7 +592,7 @@ macro_rules! stack_try_pin_init {
};
}
/// Construct an in-place, fallible pinned initializer for `struct`s.
/// Construct an in-place, fallible pinned initializer for structs, including tuple structs.
///
/// The error type defaults to [`Infallible`]; if you need a different one, write `? Error` at the
/// end, after the struct initializer.
@ -603,6 +626,42 @@ macro_rules! stack_try_pin_init {
/// # Box::pin_init(demo()).unwrap();
/// ```
///
/// The fields of a tuple struct are addressed by their index:
///
/// ```rust
/// # use pin_init::*;
/// # use core::pin::Pin;
/// #[pin_data]
/// struct Pair(usize, Bar);
///
/// #[pin_data]
/// struct Bar {
/// x: u32,
/// }
///
/// # fn demo() -> impl PinInit<Pair> {
/// let initializer = pin_init!(Pair {
/// 0: 42,
/// 1 <- Bar { x: 64 },
/// });
/// # initializer }
/// # Box::pin_init(demo()).unwrap();
/// ```
///
/// A tuple struct whose fields are all set to a value can also be written like a call to its
/// constructor:
///
/// ```rust
/// # use pin_init::*;
/// #[pin_data]
/// struct Pair(usize, usize);
///
/// # fn demo() -> impl PinInit<Pair> {
/// let initializer = pin_init!(Pair(42, 64));
/// # initializer }
/// # Box::pin_init(demo()).unwrap();
/// ```
///
/// Arbitrary Rust expressions can be used to set the value of a variable.
///
/// The fields are initialized in the order that they appear in the initializer. So it is possible
@ -721,9 +780,13 @@ macro_rules! stack_try_pin_init {
///
/// # Syntax
///
/// As already mentioned in the examples above, inside of `pin_init!` a `struct` initializer with
/// the following modifications is expected:
/// As already mentioned in the examples above, inside of `pin_init!` a struct initializer with the
/// following modifications is expected:
/// - Fields that you want to initialize in-place have to use `<-` instead of `:`.
/// - Tuple struct fields are named by their index, as in `0: value` or `0 <- initializer`. They
/// are not exposed by a `let` binding, since they have no name to bind.
/// - A tuple struct can also be initialized with constructor syntax, as in `Type(value, value)`.
/// Since its arguments are not named, they cannot use `<-`; write them out by index instead.
/// - You can use `_: { /* run any user-code here */ },` anywhere where you can place fields in
/// order to run arbitrary code.
/// - In front of the initializer you can write `&this in` to have access to a [`NonNull<Self>`]
@ -762,7 +825,7 @@ macro_rules! stack_try_pin_init {
/// [`NonNull<Self>`]: core::ptr::NonNull
pub use pin_init_internal::pin_init;
/// Construct an in-place, fallible initializer for `struct`s.
/// Construct an in-place, fallible initializer for structs, including tuple structs.
///
/// This macro defaults the error to [`Infallible`]; if you need a different one, write `? Error`
/// at the end, after the struct initializer.
@ -857,8 +920,10 @@ macro_rules! assert_pinned {
($ty:ty, $field:ident, $field_ty:ty, inline) => {
// SAFETY: This code is unreachable.
let _ = move |ptr: *mut $ty| unsafe {
let data = <$ty as $crate::__internal::HasPinData>::__pin_data();
let data = <$ty as $crate::__internal::HasInitData>::__init_data();
let data = $crate::__internal::HasPinData::__pin_data(data);
_ = data
.__with_lt()
.$field(ptr)
.init($crate::__internal::AlwaysFail::<$field_ty>::new());
};
@ -902,6 +967,9 @@ macro_rules! assert_pinned {
#[cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
#[cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
#[must_use = "An initializer must be used in order to create its value."]
#[diagnostic::on_unimplemented(
message = "`{Self}` cannot be used to initialize `{T}` with error `{E}`"
)]
pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
/// Alias of [`PinInit::__init`].
///
@ -1036,6 +1104,11 @@ unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
#[cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
#[cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
#[must_use = "An initializer must be used in order to create its value."]
#[diagnostic::on_unimplemented(
message = "`{Self}` cannot be used to movably initialize `{T}` with error `{E}`",
note = "if your type implements `PinInit` but not `Init`, \
you might be forgetting a `#[pin]` annotation on fields"
)]
pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
/// First initializes the value using `self` then calls the function `f` with the initialized
/// value.

3
rust/proc-macro2/README.md
View File

@ -4,7 +4,8 @@ These source files come from the Rust `proc-macro2` crate, version
1.0.101 (released 2025-08-16), hosted in the
<https://github.com/dtolnay/proc-macro2> repository, licensed under
"Apache-2.0 OR MIT" and only modified to add the SPDX license
identifiers and to remove the `unicode-ident` dependency.
identifiers, to remove the `unicode-ident` dependency, and to build
with 1.85 with nightly features enabled.
For copyright details, please see:

9
rust/proc-macro2/probe/proc_macro_span.rs
View File

@ -10,7 +10,6 @@
use core::ops::{Range, RangeBounds};
use proc_macro::{Literal, Span};
use std::path::PathBuf;
pub fn byte_range(this: &Span) -> Range<usize> {
this.byte_range()
@ -32,14 +31,6 @@ pub fn column(this: &Span) -> usize {
this.column()
}
pub fn file(this: &Span) -> String {
this.file()
}
pub fn local_file(this: &Span) -> Option<PathBuf> {
this.local_file()
}
pub fn join(this: &Span, other: Span) -> Option<Span> {
this.join(other)
}

12
rust/zerocopy-derive/README.md
View File

@ -1,14 +1,14 @@
# `zerocopy-derive`
These source files come from the Rust `zerocopy-derive` crate, version v0.8.54
(released 2026-07-08), hosted in the <https://github.com/google/zerocopy>
These source files come from the Rust `zerocopy-derive` crate, version v0.8.62
(released 2026-10-08), hosted in the <https://github.com/google/zerocopy>
repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only
modified to tweak the SPDX license identifiers and to remove the generation of
non-ASCII identifiers.
For copyright details, please see:
https://github.com/google/zerocopy/blob/v0.8.54/README.md?plain=1
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT
https://github.com/google/zerocopy/blob/v0.8.62/zerocopy/README.md?plain=1
https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-BSD
https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-APACHE
https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-MIT

12
rust/zerocopy-derive/derive/from_bytes.rs
View File

@ -89,6 +89,12 @@ pub(crate) fn find_zero_variant(enm: &DataEnum) -> Result<usize, bool> {
Err(has_unknown_discriminants)
}
pub(crate) fn derive_from_zeros(ctx: &Ctx, top_level: Trait) -> Result<TokenStream, Error> {
if let Some(span) = ctx.invariant_span {
return ctx.error_or_skip(Error::new(
span,
"cannot derive `FromZeros` for a type with invariants",
));
}
let try_from_bytes = derive_try_from_bytes(ctx, top_level)?;
let from_zeros = match &ctx.ast.data {
Data::Struct(strct) => derive_from_zeros_struct(ctx, strct),
@ -98,6 +104,12 @@ pub(crate) fn derive_from_zeros(ctx: &Ctx, top_level: Trait) -> Result<TokenStre
Ok(IntoIterator::into_iter([try_from_bytes, from_zeros]).collect())
}
pub(crate) fn derive_from_bytes(ctx: &Ctx, top_level: Trait) -> Result<TokenStream, Error> {
if let Some(span) = ctx.invariant_span {
return ctx.error_or_skip(Error::new(
span,
"cannot derive `FromBytes` for a type with invariants",
));
}
let from_zeros = derive_from_zeros(ctx, top_level)?;
let from_bytes = match &ctx.ast.data {
Data::Struct(strct) => derive_from_bytes_struct(ctx, strct),

147
rust/zerocopy-derive/derive/into_bytes.rs
View File

@ -2,7 +2,7 @@
//
use proc_macro2::{Span, TokenStream};
use quote::quote;
use syn::{Data, DataEnum, DataStruct, DataUnion, Error, Type};
use syn::{parse_quote, Data, DataEnum, DataStruct, DataUnion, Error, Ident, Type, WherePredicate};
use crate::{
repr::{EnumRepr, StructUnionRepr},
@ -18,6 +18,74 @@ pub(crate) fn derive_into_bytes(ctx: &Ctx, _top_level: Trait) -> Result<TokenStr
Data::Union(unn) => derive_into_bytes_union(ctx, unn),
}
}
/// If every field is exactly `T`, `[T; _]`, or a final `[T]` for the same type
/// parameter `T`, returns the bounds required to prove this to rustc.
fn homogeneous_field_bounds(ctx: &Ctx, strct: &DataStruct) -> Option<Vec<WherePredicate>> {
fn strip_parens_and_groups(mut ty: &Type) -> &Type {
loop {
ty = match ty {
Type::Group(group) => &group.elem,
Type::Paren(paren) => &paren.elem,
ty => return ty,
};
}
}
fn type_is_parameter(ty: &Type, parameter: &Ident) -> bool {
match strip_parens_and_groups(ty) {
Type::Path(path) => path.qself.is_none() && path.path.is_ident(parameter),
_ => false,
}
}
let fields = strct.fields();
ctx.ast.generics.type_params().find_map(|parameter| {
let parameter = &parameter.ident;
let elements = fields
.iter()
.enumerate()
.map(|(index, (_, _, ty))| match strip_parens_and_groups(ty) {
Type::Array(array) => Some(&*array.elem),
Type::Slice(slice) if index + 1 == fields.len() => Some(&*slice.elem),
Type::Slice(_) => None,
ty => Some(ty),
})
.collect::<Option<Vec<_>>>()?;
if !elements.iter().all(|element| type_is_parameter(element, parameter)) {
return None;
}
let zerocopy_crate = &ctx.zerocopy_crate;
let mut bounds = fields
.iter()
.map(|(_, _, ty)| parse_quote!(#ty: #zerocopy_crate::IntoBytes))
.collect::<Vec<WherePredicate>>();
// `Ident` equality is only a syntactic prefilter above. In
// macro-generated input, two identifiers can both print as `T` and
// compare equal here while having different hygiene contexts. For
// example, a call-site `T` could resolve to a `u8` type alias while a
// def-site `T` resolves to this struct's type parameter. Instantiating
// that parameter with `u16` would make the fields actually be `u8` and
// `u16`; treating them as homogeneous could overlook padding and cause
// us to emit an unsound `IntoBytes` impl.
//
// Quote every element type from its original field AST so that its
// hygiene context is retained, then use `Identity` to have rustc verify
// that it resolves to `parameter`. Do not deduplicate these predicates:
// identically-printed types may have different hygiene contexts and
// must each be checked by rustc.
bounds.extend(elements.into_iter().map(|element| {
parse_quote! {
#element: #zerocopy_crate::util::macro_util::Identity<Type = #parameter>
}
}));
Some(bounds)
})
}
fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream, Error> {
let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
@ -25,8 +93,12 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
let is_c = repr.is_c();
let is_packed_1 = repr.is_packed_1();
let num_fields = strct.fields().len();
let mut homogeneous_bounds =
if is_c && !repr.is_align_gt_1() { homogeneous_field_bounds(ctx, strct) } else { None };
let (padding_check, require_unaligned_fields) = if is_transparent || is_packed_1 {
let (padding_check, require_unaligned_fields, explicit_field_bounds) = if is_transparent
|| is_packed_1
{
// No padding check needed.
// - repr(transparent): The layout and ABI of the whole struct is the
// same as its only non-ZST field (meaning there's no padding outside
@ -43,12 +115,12 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
// An important consequence of these rules is that a type with
// `#[repr(packed(1))]`` (or `#[repr(packed)]``) will have no
// inter-field padding.
(None, false)
(None, false, None)
} else if is_c && !repr.is_align_gt_1() && num_fields <= 1 {
// No padding check needed. A repr(C) struct with zero or one field has
// no padding unless #[repr(align)] explicitly adds padding, which we
// check for in this branch's condition.
(None, false)
(None, false, None)
} else if ctx.ast.generics.params.is_empty() {
// Is the last field a syntactic slice, i.e., `[SomeType]`.
let is_syntactic_dst =
@ -66,10 +138,31 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
// 2. The fields do not overlap.
// ...
if is_c && is_syntactic_dst {
(Some(PaddingCheck::ReprCStruct), false)
(Some(PaddingCheck::ReprCStruct), false, None)
} else {
(Some(PaddingCheck::Struct), false)
(Some(PaddingCheck::Struct), false, None)
}
} else if let Some(bounds) = homogeneous_bounds.take() {
// Let `a` be the alignment of `T` and `s` be its size. Rust guarantees
// that `s` is a multiple of `a`. `T`, `[T; N]`, and `[T]` all have
// alignment `a`, and their sizes are `s`, `N * s`, and `len * s`,
// respectively.
//
// Without a `packed` modifier, each field and the struct have
// alignment `a`. `align(1)` does not change that because every Rust
// type already has alignment at least 1. With `packed(P)`, each field
// and the struct instead have alignment `b = min(a, P)`. Rust
// alignments and valid values of `P` are powers of two, so `b` divides
// `a`; every field size is therefore also a multiple of `b`.
//
// Thus, after placing any field, the running offset is already aligned
// for the next field, and the final offset is already aligned for the
// struct. A repr(C) without an `align` modifier greater than 1
// therefore introduces neither inter-field nor trailing padding,
// including for a slice DST. The bounds returned by
// `homogeneous_field_bounds` require rustc to verify that the element
// type of every field is the same type `T`.
(None, false, Some(bounds))
} else if is_c && !repr.is_align_gt_1() {
// We can't use a padding check since there are generic type arguments.
// Instead, we require all field types to implement `Unaligned`. This
@ -79,7 +172,7 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
//
// FIXME(#10): Support type parameters for non-transparent, non-packed
// structs without requiring `Unaligned`.
(None, true)
(None, true, None)
} else {
return ctx.error_or_skip(Error::new(
Span::call_site(),
@ -87,7 +180,9 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
));
};
let field_bounds = if require_unaligned_fields {
let field_bounds = if let Some(bounds) = explicit_field_bounds {
FieldBounds::Explicit(bounds)
} else if require_unaligned_fields {
FieldBounds::All(&[TraitBound::Slf, TraitBound::Other(Trait::Unaligned)])
} else {
FieldBounds::ALL_SELF
@ -162,3 +257,39 @@ fn derive_into_bytes_union(ctx: &Ctx, unn: &DataUnion) -> Result<TokenStream, Er
.build();
Ok(quote!(#cfg_compile_error #impl_block))
}
#[cfg(test)]
mod tests {
use super::*;
fn derive(input: syn::DeriveInput) -> TokenStream {
let ctx = Ctx::try_from_derive_input(input).unwrap();
derive_into_bytes(&ctx, Trait::IntoBytes).unwrap()
}
#[test]
fn raw_repr_matches_ordinary_repr() {
let ordinary = derive(parse_quote! {
#[repr(C)]
#[repr(align(8))]
struct Response {
status: u8,
}
});
let raw = derive(parse_quote! {
#[repr(C)]
#[r#repr(r#align(8))]
struct Response {
status: u8,
}
});
let ordinary = ordinary.to_string();
let raw = raw.to_string();
// In particular, both expansions contain the padding check which
// prevents the resulting type from implementing `IntoBytes`.
assert!(ordinary.contains("PaddingFree"));
assert_eq!(ordinary, raw);
}
}

4
rust/zerocopy-derive/derive/known_layout.rs
View File

@ -81,7 +81,7 @@ fn raw_from_ptr_len(
#[inline(always)]
fn pointer_to_metadata(ptr: *mut Self) -> <Self as #zerocopy_crate::KnownLayout>::PointerMetadata {
<#trailing_field_ty>::pointer_to_metadata(ptr as *mut _)
<#trailing_field_ty as #zerocopy_crate::KnownLayout>::pointer_to_metadata(ptr as *mut _)
}
}
};
@ -144,6 +144,7 @@ fn pointer_to_metadata(ptr: *mut Self) -> <Self as #zerocopy_crate::KnownLayout>
// Define the collection of type-level field handles.
let field_defs = field_indices.iter().zip(fields).map(|(idx, (vis, _, _))| {
quote! {
#[allow(missing_debug_implementations)]
#vis struct #idx;
}
});
@ -200,6 +201,7 @@ unsafe impl #impl_generics #zerocopy_crate::util::macro_util::Field<#idx> for #i
// `#ty`, not `__ZerocopyKnownLayoutMaybeUninit` (see #2116).
#repr
#[doc(hidden)]
#[allow(missing_debug_implementations)]
#vis struct __ZerocopyKnownLayoutMaybeUninit<#params> (
#(#core::mem::MaybeUninit<
<#ident #ty_generics as

33
rust/zerocopy-derive/derive/mod.rs
View File

@ -3,6 +3,7 @@
pub mod from_bytes;
pub mod into_bytes;
pub mod known_layout;
pub mod project;
pub mod try_from_bytes;
pub mod unaligned;
@ -102,13 +103,6 @@ pub(crate) fn derive_split_at(ctx: &Ctx, _top_level: Trait) -> Result<TokenStrea
}
};
if repr.get_packed().is_some() {
return ctx.error_or_skip(Error::new(
Span::call_site(),
"must not have #[repr(packed)] attribute",
));
}
if !(repr.is_c() || repr.is_transparent()) {
return ctx.error_or_skip(Error::new(
Span::call_site(),
@ -124,13 +118,28 @@ pub(crate) fn derive_split_at(ctx: &Ctx, _top_level: Trait) -> Result<TokenStrea
};
let zerocopy_crate = &ctx.zerocopy_crate;
// SAFETY: `#ty`, per the above checks, is `repr(C)` or `repr(transparent)`
// and is not packed; its trailing field is guaranteed to be well-aligned
// for its type. By invariant on `FieldBounds::TRAILING_SELF`, the trailing
// slice of the trailing field is also well-aligned for its type.
let elem = quote!(<#trailing_field as #zerocopy_crate::SplitAt>::Elem);
let elem =
if repr.get_packed().is_some() { quote!(#zerocopy_crate::Unalign<#elem>) } else { elem };
// SAFETY: The checks above require `repr(C)` or `repr(transparent)`, and
// `FieldBounds::TRAILING_SELF` requires the trailing field to be `SplitAt`.
// Its `Elem` therefore has the size, bit validity, and `UnsafeCell`
// coverage of the actual trailing element. For an unpacked struct, the
// trailing field is aligned, so its `SplitAt` alignment guarantee applies.
// For a packed struct, wrapping its `Elem` in `Unalign` preserves those
// representation properties and gives alignment 1. Thus the trailing
// slice can be accessed as `[#elem]` even if packing misaligns its field.
// Packing does not change the field's internal layout [1], so its element
// size and byte offsets are preserved in either case.
//
// [1] Per https://doc.rust-lang.org/1.93.0/reference/type-layout.html#layout.repr.alignment.intro:
//
// `packed` may also alter the padding between fields (although it will
// not alter the padding inside of any field).
Ok(ImplBlockBuilder::new(ctx, &ctx.ast.data, Trait::SplitAt, FieldBounds::TRAILING_SELF)
.inner_extras(quote! {
type Elem = <#trailing_field as #zerocopy_crate::SplitAt>::Elem;
type Elem = #elem;
})
.build())
}

569
rust/zerocopy-derive/derive/project.rs Normal file
View File

@ -0,0 +1,569 @@
// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
//
// Copyright 2026 The Fuchsia Authors
//
// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
// This file may not be copied, modified, or distributed except according to
// those terms.
use proc_macro2::TokenStream;
use quote::quote;
use syn::{
parse_quote, spanned::Spanned as _, Data, DataEnum, DeriveInput, Error, Expr, Fields, Ident,
Index, Type,
};
use crate::{
repr::{EnumRepr, StructUnionRepr},
util::{
const_block, generate_tag_enum, Client, Ctx, DataExt, FieldBounds, FieldProjection,
ImplBlockBuilder, Trait,
},
};
pub(crate) fn tag_ident(variant: &Ident) -> Ident {
ident!(("___ZEROCOPY_TAG_{}", variant), variant.span())
}
pub(crate) fn variant_struct_ident(variant: &Ident) -> Ident {
ident!(("___ZerocopyVariantStruct_{}", variant), variant.span())
}
pub(crate) fn variants_union_field_ident(variant: &Ident) -> Ident {
ident!(("__field_{}", variant), variant.span())
}
/// Generates a constant for the tag associated with each variant of the enum.
/// When we match on the enum's tag, each arm matches one of these constants. We
/// have to use constants here because:
///
/// - The type that we're matching on is not the type of the tag, it's an
/// integer of the same size as the tag type and with the same bit patterns.
/// - We can't read the enum tag as an enum because the bytes may not represent
/// a valid variant.
/// - Patterns do not currently support const expressions, so we have to assign
/// these constants to names rather than use them inline in the `match`
/// statement.
pub(crate) fn generate_tag_consts(data: &DataEnum) -> TokenStream {
let tags = data.variants.iter().map(|variant| {
let variant_ident = &variant.ident;
let tag = tag_ident(variant_ident);
quote! {
// This casts the enum variant to its discriminant, and then
// converts the discriminant to the target integral type via a
// numeric cast [1].
//
// Because these are the same size, this is defined to be a no-op
// and therefore is a lossless conversion [2].
//
// [1] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#enum-cast:
//
// Casts an enum to its discriminant.
//
// [2] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#numeric-cast:
//
// Casting between two integers of the same size (e.g. i32 -> u32)
// is a no-op.
const #tag: ___ZerocopyTagPrimitive =
___ZerocopyTag::#variant_ident as ___ZerocopyTagPrimitive;
}
});
quote! {
#(#tags)*
}
}
#[derive(Copy, Clone, Eq, PartialEq)]
enum Validity {
Uninit,
Initialized,
Safe,
}
impl Validity {
fn as_type(self, ctx: &Ctx) -> Type {
let zerocopy_crate = &ctx.zerocopy_crate;
match self {
Validity::Uninit => parse_quote!(#zerocopy_crate::invariant::Uninit),
Validity::Initialized => {
parse_quote!(#zerocopy_crate::invariant::Initialized)
}
Validity::Safe => parse_quote!(#zerocopy_crate::invariant::Safe),
}
}
/// Computes the output validity of projecting a struct or union field, or
/// returns `None` if that projection must not be generated.
///
/// Struct fields preserve all three validity invariants. Union fields
/// preserve `Uninit` and `Initialized`. No projection is generated from a
/// `Safe` union.
fn output_validity_for_struct_or_union(self, data: &Data) -> Option<Self> {
match (data, self) {
(Data::Struct(_), validity) => Some(validity),
(Data::Union(_), validity @ (Validity::Uninit | Validity::Initialized)) => {
Some(validity)
}
(Data::Union(_), Validity::Safe) => None,
(Data::Enum(_), _) => unreachable!(),
}
}
}
/// Returns the projection variant ID for the struct or union described by
/// `ctx`.
///
/// This intentionally does not handle enums: an enum field's variant ID
/// identifies its source variant rather than the representation of its
/// containing type.
pub(crate) fn struct_union_variant_id(ctx: &Ctx) -> Box<Expr> {
let zerocopy_crate = &ctx.zerocopy_crate;
match &ctx.ast.data {
Data::Struct(_) => parse_quote!({ #zerocopy_crate::STRUCT_VARIANT_ID }),
Data::Union(_) => {
if StructUnionRepr::from_attrs(&ctx.ast.attrs).map(|repr| repr.is_c()).unwrap() {
parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID })
} else {
parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID })
}
}
Data::Enum(_) => unreachable!("struct_union_variant_id called for enum"),
}
}
/// Generates field projection implementations for a struct or union.
///
/// `repr(C)` unions use `REPR_C_UNION_VARIANT_ID`, so their projections
/// additionally implement `pointer::cast::Cast`. Other unions use
/// `UNION_VARIANT_ID`.
pub(crate) fn derive_projection_struct_union(
ctx: &Ctx,
data: &dyn DataExt,
client: Client,
) -> TokenStream {
let fields = ctx.ast.data.fields();
if fields.is_empty() {
return quote! {};
}
let field_tokens = fields.iter().map(|(vis, ident, _)| {
let ident = ident!(("__z{}", ident), ident.span());
quote! {
#vis enum #ident {}
}
});
let zerocopy_crate = &ctx.zerocopy_crate;
let variant_id = struct_union_variant_id(ctx);
let core = ctx.core_path();
let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag { client }, FieldBounds::None)
.inner_extras(quote! {
type Tag = ();
type ProjectToTag = #zerocopy_crate::pointer::cast::CastToUnit;
})
.build();
let has_fields = fields.iter().map(move |(_, ident, ty)| {
let field_token = ident!(("__z{}", ident), ident.span());
let projection = FieldProjection {
variant_id: variant_id.clone(),
field: parse_quote!(#field_token),
field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
};
let has_field_trait = Trait::HasField { client, projection: projection.clone() };
let has_field_path = has_field_trait.crate_path(ctx);
let has_field = ImplBlockBuilder::new(ctx, data, has_field_trait, FieldBounds::None)
.inner_extras(quote! {
type Type = #ty;
#[inline(always)]
fn project(
slf: #zerocopy_crate::pointer::PtrInner<'_, Self>,
) -> *mut <Self as #has_field_path>::Type {
let slf = slf.as_ptr();
// SAFETY: By invariant on `PtrInner`, `slf` is a non-null
// pointer whose referent is zero-sized or lives in a valid
// allocation. Since `#ident` is a struct or union field of
// `Self`, this projection preserves or shrinks the referent
// size, and so the resulting referent also fits in the same
// allocation.
unsafe { #core::ptr::addr_of_mut!((*slf).#ident) }
}
})
.build();
let derive_project_field = |input_validity: Validity| {
let output_validity =
match input_validity.output_validity_for_struct_or_union(&ctx.ast.data) {
Some(output_validity) => output_validity.as_type(ctx),
None => return TokenStream::new(),
};
let input_validity = input_validity.as_type(ctx);
let projection_preserves_alignment = StructUnionRepr::from_attrs(&ctx.ast.attrs)
.map(|repr| repr.get_packed().is_none())
.unwrap();
let output_alignment = if projection_preserves_alignment {
quote! { ___ZcAlignment }
} else {
quote! { #zerocopy_crate::invariant::Unaligned }
};
ImplBlockBuilder::new(
ctx,
data,
Trait::ProjectField {
client,
projection: projection.clone(),
invariants: parse_quote!((___ZcAliasing, ___ZcAlignment, #input_validity)),
},
FieldBounds::None,
)
.param_extras(vec![
parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Aliasing),
parse_quote!(___ZcAlignment: #zerocopy_crate::invariant::Alignment),
])
.inner_extras(quote! {
// SAFETY: Struct and union projections do not depend on the value of
// the referent, and are therefore infallible.
type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible;
// SAFETY: Projection preserves aliasing. It also preserves
// alignment unless the containing type is packed. `Validity`
// determines whether a projection is emitted and, if so,
// computes its output validity: struct fields preserve validity;
// union fields preserve `Uninit` and `Initialized`, and no
// projection is emitted for `Safe` union input.
type Invariants = (___ZcAliasing, #output_alignment, #output_validity);
})
.build()
};
let project_uninit = derive_project_field(Validity::Uninit);
let project_initialized = derive_project_field(Validity::Initialized);
let project_safe = derive_project_field(Validity::Safe);
quote! {
#has_field
#project_uninit
#project_initialized
#project_safe
}
});
const_block(field_tokens.into_iter().chain(Some(has_tag)).chain(has_fields).map(Some))
}
pub(crate) fn derive_enum(
ctx: &Ctx,
data: &DataEnum,
client: Client,
) -> Result<TokenStream, Error> {
// With no fields, there are no `HasField` or `ProjectField` impls to emit,
// and therefore no representation requirement to enforce.
if data.fields().is_empty() {
return Ok(TokenStream::new());
}
let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?;
let (outer_tag_type, inner_tag_type) = if repr.is_c() {
(quote! { ___ZerocopyTag }, quote! { () })
} else if repr.is_primitive() {
(quote! { () }, quote! { ___ZerocopyTag })
} else {
return Err(Error::new(
ctx.ast.span(),
"must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout",
));
};
let zerocopy_crate = &ctx.zerocopy_crate;
let core = &ctx.core_path();
let client_path = &client.crate_path(ctx);
let has_tag_trait = Trait::HasTag { client };
let has_tag_path = &has_tag_trait.crate_path(ctx);
let infallible =
quote! { #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible };
let tag_enum = generate_tag_enum(ctx, &repr, data);
let tag_consts = generate_tag_consts(data);
let generics = &ctx.ast.generics;
let (impl_generics, ref ty_generics, where_clause) = generics.split_for_impl();
// Synthesize a `repr(C)` struct for each fieldful variant. We must do this
// because Rust does not provide an `addr_of!` that works on enums.
let variant_structs = data.variants.iter().filter_map(move |variant| {
if matches!(variant.fields, Fields::Unit) {
return None;
}
let ident = variant_struct_ident(&variant.ident);
let field_types = variant.fields.iter().map(|field| &field.ty);
let enum_name = &ctx.ast.ident;
let variant_struct: DeriveInput = parse_quote! {
#[repr(C)]
struct #ident #impl_generics (
#core::mem::MaybeUninit<___ZerocopyInnerTag>,
#(#field_types,)*
#core::marker::PhantomData<#enum_name #ty_generics>,
) #where_clause;
};
let projections = derive_projection_struct_union(
&ctx.with_input(&variant_struct),
&variant_struct.data,
client,
);
Some(quote! {
#variant_struct
#projections
})
});
// Synthesize a `repr(C)` union encompassing all synthesized variants.
let variants_union = {
let fields = data.variants.iter().filter_map(|variant| {
if matches!(variant.fields, Fields::Unit) {
return None;
}
let field_name = variants_union_field_ident(&variant.ident);
let variant_struct = variant_struct_ident(&variant.ident);
Some(quote! {
#field_name: #core::mem::ManuallyDrop<#variant_struct #ty_generics>,
})
});
let variants_union: DeriveInput = parse_quote! {
#[repr(C)]
union ___ZerocopyVariants #generics {
#(#fields)*
// A fieldless enum produces no variant structs, but a union must
// have at least one field. This unit does not affect `repr(C)`
// layout.
__nonempty: (),
}
};
let projections = derive_projection_struct_union(
&ctx.with_input(&variants_union),
&variants_union.data,
client,
);
quote! {
#variants_union
#projections
}
};
// Synthesize a `repr(C)` struct that pairs the the tag with the synthesized
// variants union. In practice, the actual tag storage may be here, or
// inside the synthesized variant structs, depending on the `repr` of the
// enum.
let raw_enum: DeriveInput = parse_quote! {
#[repr(C)]
struct ___ZerocopyRawEnum #generics {
tag: ___ZerocopyOuterTag,
variants: ___ZerocopyVariants #ty_generics,
}
};
let raw_projections =
derive_projection_struct_union(&ctx.with_input(&raw_enum), &raw_enum.data, client);
let has_tag = ImplBlockBuilder::new(ctx, data, has_tag_trait, FieldBounds::None)
.inner_extras(quote! {
type Tag = ___ZerocopyTag;
type ProjectToTag = #zerocopy_crate::pointer::cast::CastSized;
})
.build();
let aliasing_param: &syn::GenericParam =
&parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Aliasing);
let reference_param: &syn::GenericParam =
&parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Reference);
let alignment_param: &syn::GenericParam =
&parse_quote!(___ZcAlignment: #zerocopy_crate::invariant::Alignment);
let has_fields = data.variants().into_iter().flat_map(|(variant, fields)| {
let variant_ident = &variant.unwrap().ident;
let variants_union_field = variants_union_field_ident(variant_ident);
let infallible = &infallible;
fields.into_iter().enumerate().map(move |(idx, (vis, ident, ty))| {
// Rust does not presently support explicit visibility modifiers on
// enum fields. Keep this assertion so that a future language
// change cannot silently invalidate the visibility invariant.
assert!(matches!(vis, syn::Visibility::Inherited));
let projection = FieldProjection {
variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }),
field: parse_quote!(()),
field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
};
// The generated variant struct's first field is its inner tag.
let variant_struct_field_index = Index::from(idx + 1);
let has_field_trait = Trait::HasField {
client,
projection: projection.clone(),
};
let has_field_path = has_field_trait.crate_path(ctx);
let has_field = ImplBlockBuilder::new(ctx, data, has_field_trait, FieldBounds::None)
.inner_extras(quote! {
type Type = #ty;
#[inline(always)]
fn project(
slf: #zerocopy_crate::pointer::PtrInner<'_, Self>,
) -> *mut <Self as #has_field_path>::Type {
use #zerocopy_crate::pointer::cast::{CastSized, Projection};
slf.project::<___ZerocopyRawEnum #ty_generics, CastSized>()
.project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(variants) }>>()
.project::<_, Projection<#client_path, _, { #zerocopy_crate::REPR_C_UNION_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variants_union_field) }>>()
.project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(value) }>>()
.project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variant_struct_field_index) }>>()
.as_ptr()
}
})
.build();
// SAFETY: `projection` is constructed from the current field and
// `variant_ident`, and that same `variant_ident` selects the
// variants union field through which `has_field` projects. Thus the
// `HasField` impl, `ProjectField` impls, and valid projection's tag
// check all identify the same enum field. `idx` is the current
// field's zero-based position, and the generated variant struct
// prepends its inner tag before reproducing the enum fields in
// source order, so `idx + 1` selects that same field. The assertion
// above enforces that `repr` is `repr(C)` or a primitive
// representation; `___ZerocopyRawEnum` and its nested types model
// exactly the representations specified by [1] and [2]. Those
// representations use `repr(C)` structs for variant fields, so field
// projection preserves alignment. Each `ProjectField` impl preserves
// the input validity. The `Uninit` and `Initialized` projections are
// infallible because those validity invariants do not depend on the
// enum's tag. The `Safe` projection first checks that the tag
// selects `variant_ident`; its `Reference` aliasing bound prevents
// mutation of the tag between that check and field projection. The
// corresponding `HasTag` impl projects to the raw representation's
// tag using `CastSized`.
//
// [1] Per https://doc.rust-lang.org/1.56.0/reference/type-layout.html#reprc-enums-with-fields:
//
// The representation of a `repr(C)` enum with fields is a
// `repr(C)` struct with two fields [...]: a `repr(C)` version of
// the enum with all fields removed ("the tag") [and] a `repr(C)`
// union of `repr(C)` structs for the fields of each variant [...].
//
// [2] Per https://doc.rust-lang.org/1.56.0/reference/type-layout.html#primitive-representation-of-enums-with-fields:
//
// The representation of a primitive representation enum is a
// `repr(C)` union of `repr(C)` structs for each variant with a
// field. The first field of each struct in the union is [...] the
// tag and the remaining fields are the fields of that variant.
let derive_enum_project_field = |validity| {
let (error, is_projectable, aliasing_bound) = if validity == Validity::Safe {
assert!(!matches!(validity, Validity::Uninit | Validity::Initialized));
(
quote! { () },
quote! {
#[inline(always)]
fn is_projectable(
tag: #zerocopy_crate::pointer::Ptr<
'_,
<Self as #has_tag_path>::Tag,
(
#zerocopy_crate::invariant::Shared,
___ZcAlignment,
#zerocopy_crate::invariant::Safe,
),
>,
) -> #core::result::Result<(), ()> {
let tag = tag.read::<#zerocopy_crate::BecauseImmutable>();
if tag == ___ZerocopyTag::#variant_ident {
#core::result::Result::Ok(())
} else {
#core::result::Result::Err(())
}
}
},
reference_param.clone(),
)
} else {
(infallible.clone(), quote! {}, aliasing_param.clone())
};
let validity = validity.as_type(ctx);
let invariants: Box<Type> =
parse_quote!((___ZcAliasing, ___ZcAlignment, #validity));
let project_field_trait = Trait::ProjectField {
client,
projection: projection.clone(),
invariants: invariants.clone(),
};
ImplBlockBuilder::new(ctx, data, project_field_trait, FieldBounds::None)
.param_extras(vec![aliasing_bound.clone(), alignment_param.clone()])
.inner_extras(quote! {
type Error = #error;
type Invariants = #invariants;
#is_projectable
})
.build()
};
let project_uninit = derive_enum_project_field(Validity::Uninit);
let project_initialized = derive_enum_project_field(Validity::Initialized);
let project_valid = derive_enum_project_field(Validity::Safe);
quote! {
#has_field
#project_uninit
#project_initialized
#project_valid
}
})
});
Ok(quote! {
#tag_enum
type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag<
{ #core::mem::size_of::<___ZerocopyTag>() },
>;
#tag_consts
type ___ZerocopyOuterTag = #outer_tag_type;
type ___ZerocopyInnerTag = #inner_tag_type;
#(#variant_structs)*
#variants_union
#raw_enum
#raw_projections
#has_tag
#(#has_fields)*
})
}
pub(crate) fn derive(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
match &ctx.ast.data {
Data::Struct(strct) => {
Ok(derive_projection_struct_union(ctx, strct, Client::ProjectDerive))
}
Data::Union(unn) => Ok(derive_projection_struct_union(ctx, unn, Client::ProjectDerive)),
Data::Enum(enm) => {
let projections = derive_enum(ctx, enm, Client::ProjectDerive)?;
if projections.is_empty() {
Ok(projections)
} else {
Ok(const_block([Some(projections)]))
}
}
}
}

881
rust/zerocopy-derive/derive/try_from_bytes.rs
View File

File diff suppressed because it is too large Load Diff

116
rust/zerocopy-derive/invariant.rs Normal file
View File

@ -0,0 +1,116 @@
// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
//
// Copyright 2026 The Fuchsia Authors
//
// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
// This file may not be copied, modified, or distributed except according to
// those terms.
use proc_macro2::{Ident, Span, TokenStream, TokenTree};
use quote::{quote, ToTokens as _};
use syn::{
punctuated::Punctuated, spanned::Spanned as _, Attribute, Data, DeriveInput, Error, Expr,
Field, MacroDelimiter, Meta, Token,
};
use crate::util::path_is_ident;
/// Parses field invariants in attribute order.
pub(crate) fn parse(attrs: &[Attribute]) -> Result<Vec<Expr>, Error> {
let mut invariants = Vec::new();
for attr in attrs.iter().filter(|attr| path_is_ident(attr.path(), "zerocopy")) {
let options = attr.parse_args_with(Punctuated::<Meta, Token![,]>::parse_terminated)?;
for option in options {
if !path_is_ident(option.path(), "invariant") {
return Err(Error::new_spanned(option, "expected `invariant(...)`"));
}
match option {
Meta::List(list) if matches!(list.delimiter, MacroDelimiter::Paren(_)) => {
invariants.push(list.parse_args::<Expr>()?)
}
other => return Err(Error::new_spanned(other, "expected `invariant(...)`")),
}
}
}
Ok(invariants)
}
/// Validates invariant syntax and placement, returning the first invariant's
/// span so derives which cannot honor invariants can reject them.
pub(crate) fn validate(ast: &DeriveInput) -> Result<Option<Span>, Error> {
let fields: Vec<&Field> = match &ast.data {
Data::Struct(data) => data.fields.iter().collect(),
Data::Enum(data) => {
for variant in &data.variants {
if let Some(invariant) = parse(&variant.attrs)?.first() {
return Err(Error::new(
invariant.span(),
"invariants are only supported on named fields",
));
}
}
data.variants.iter().flat_map(|variant| &variant.fields).collect()
}
Data::Union(data) => data.fields.named.iter().collect(),
};
let mut first = None;
for field in fields {
if let Some(invariant) = parse(&field.attrs)?.first() {
if field.ident.is_none() {
return Err(Error::new(
invariant.span(),
"invariants are only supported on named fields",
));
}
first.get_or_insert(invariant.span());
}
}
Ok(first)
}
/// Visits identifiers, including those inside macro arguments.
pub(crate) fn idents(tokens: TokenStream) -> Vec<Ident> {
tokens
.into_iter()
.flat_map(|token| match token {
TokenTree::Ident(ident) => vec![ident],
TokenTree::Group(group) => idents(group.stream()),
_ => Vec::new(),
})
.collect()
}
/// Binds semantic field names in every syntax context used by the expression.
/// Preserve the expression's tokens so its other names retain their hygiene.
pub(crate) fn bind_fields(fields: &[(&Field, Ident)], expression: &Expr) -> TokenStream {
let mut body = expression.to_token_stream();
let references = idents(body.clone());
for (field, binding) in fields {
let name = match &field.ident {
Some(name) => name,
None => continue,
};
let spelling = name.to_string();
let spelling = spelling.trim_start_matches("r#");
// Also support references produced by a macro invoked by the expression.
let mut call_site = name.clone();
call_site.set_span(Span::call_site());
for alias in [name.clone(), call_site].into_iter().chain(
references
.iter()
.filter(|ident| ident.to_string().trim_start_matches("r#") == spelling)
.cloned(),
) {
body = quote! {{
// Require a binding; reject names Rust would resolve as constant
// patterns. Item aliases could capture helper calls in macros.
#[allow(unused_variables, non_snake_case, clippy::redundant_pattern)]
let #alias @ _ = #binding;
#body
}};
}
}
body
}

118
rust/zerocopy-derive/lib.rs
View File

@ -28,6 +28,10 @@
#![allow(clippy::type_complexity)]
// Inlining format args isn't supported on our MSRV.
#![allow(clippy::uninlined_format_args)]
// `cargo-zerocopy` supplies this cfg for pinned-nightly tests. During UI tests,
// `testutil::UiTestRunner` explicitly supplies it to the host-built proc macro;
// ordinary `RUSTFLAGS` are not sufficient when Cargo receives `--target`.
#![cfg_attr(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, feature(proc_macro_def_site))]
#![deny(
rustdoc::bare_urls,
rustdoc::broken_intra_doc_links,
@ -46,6 +50,7 @@ macro_rules! ident {
}
mod derive;
mod invariant;
#[cfg(test)]
mod output_tests;
mod repr;
@ -79,7 +84,8 @@ macro_rules! ident {
/// are currently required to live at the crate root, and so the caller must
/// specify the name in order to avoid name collisions.
macro_rules! derive {
($trait:ident => $outer:ident => $inner:path) => {
($(#[$attr:meta])* $trait:ident => $outer:ident => $inner:path) => {
$(#[$attr])*
#[proc_macro_derive($trait, attributes(zerocopy))]
pub fn $outer(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
let ast = syn::parse_macro_input!(ts as DeriveInput);
@ -88,10 +94,13 @@ pub fn $outer(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
Err(e) => return e.into_compile_error().into(),
};
let ts = $inner(&ctx, Trait::$trait).into_ts();
// We wrap in `const_block` as a backstop in case any derive fails
// to wrap its output in `const_block` (and thus fails to annotate)
// with the full set of `#[allow(...)]` attributes).
let ts = const_block([Some(ts)]);
// Apply generated-code lint allowances as a backstop, except
// around caller-authored invariant expressions.
let ts = if matches!(Trait::$trait, Trait::TryFromBytes) {
ctx.const_block([Some(ts)])
} else {
const_block([Some(ts)])
};
#[cfg(test)]
crate::util::testutil::check_hygiene(ts.clone());
ts.into()
@ -120,6 +129,7 @@ fn into_ts(self) -> proc_macro2::TokenStream {
derive!(KnownLayout => derive_known_layout => crate::derive::known_layout::derive);
derive!(Immutable => derive_immutable => crate::derive::derive_immutable);
derive!(#[doc(hidden)] Project => derive_project => crate::derive::project::derive);
derive!(TryFromBytes => derive_try_from_bytes => crate::derive::try_from_bytes::derive_try_from_bytes);
derive!(FromZeros => derive_from_zeros => crate::derive::from_bytes::derive_from_zeros);
derive!(FromBytes => derive_from_bytes => crate::derive::from_bytes::derive_from_bytes);
@ -129,6 +139,89 @@ fn into_ts(self) -> proc_macro2::TokenStream {
derive!(ByteEq => derive_eq => crate::derive::derive_eq);
derive!(SplitAt => derive_split_at => crate::derive::derive_split_at);
/// Generates a struct whose two identically-printed field types resolve to
/// different types. This is used to test that derives preserve type identity
/// across macro hygiene contexts.
#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
#[doc(hidden)]
#[proc_macro]
pub fn __test_hygienically_mixed_into_bytes(
_input: proc_macro::TokenStream,
) -> proc_macro::TokenStream {
use proc_macro::{Group, Ident, Span, TokenStream, TokenTree};
fn rewrite(input: TokenStream) -> TokenStream {
input
.into_iter()
.map(|token| match token {
TokenTree::Ident(ident) if ident.to_string() == "CallT" => {
TokenTree::Ident(Ident::new("T", Span::call_site()))
}
TokenTree::Ident(ident) if ident.to_string() == "DefT" => {
TokenTree::Ident(Ident::new("T", Span::def_site()))
}
TokenTree::Group(group) => {
let mut rewritten = Group::new(group.delimiter(), rewrite(group.stream()));
rewritten.set_span(group.span());
TokenTree::Group(rewritten)
}
token => token,
})
.collect()
}
rewrite(
"#[derive(zerocopy_renamed::IntoBytes)] \
#[zerocopy(crate = \"zerocopy_renamed\")] \
#[repr(C)] \
struct IntoBytes15<DefT>(CallT, DefT);"
.parse()
.expect("test input must parse"),
)
}
/// Constructs an invariant whose field declaration and reference have different
/// syntax contexts, while preserving all other caller tokens.
#[doc(hidden)]
#[proc_macro]
pub fn __test_hygienically_mixed_invariant(
input: proc_macro::TokenStream,
) -> proc_macro::TokenStream {
// Cross-compilation does not necessarily pass the nightly test cfg to
// host proc macros. Omit the fixture and its test together in that case.
#[cfg(not(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS))]
{
let _ = input;
proc_macro::TokenStream::new()
}
#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
{
use proc_macro::{Group, Ident, Span, TokenStream, TokenTree};
fn rewrite(input: TokenStream) -> TokenStream {
input
.into_iter()
.map(|token| match token {
TokenTree::Ident(ident) if ident.to_string() == "DefField" => {
TokenTree::Ident(Ident::new("field", Span::def_site()))
}
TokenTree::Ident(ident) if ident.to_string() == "CallField" => {
TokenTree::Ident(Ident::new("field", Span::call_site()))
}
TokenTree::Group(group) => {
let mut rewritten = Group::new(group.delimiter(), rewrite(group.stream()));
rewritten.set_span(group.span());
TokenTree::Group(rewritten)
}
token => token,
})
.collect()
}
rewrite(input)
}
}
#[cfg_attr(not(zerocopy_unstable_linux), doc(hidden))]
#[proc_macro_derive(most_traits, attributes(zerocopy))]
pub fn most_traits(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
@ -153,11 +246,18 @@ pub fn most_traits(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
for (derive, t) in derives {
tokens.extend(derive(&ctx, t))
}
// Invariants prevent `FromBytes` from generating its usual supertrait
// impls, but still permit checked conversions through `TryFromBytes`.
if ctx.invariant_span.is_some() {
tokens.extend(crate::derive::try_from_bytes::derive_try_from_bytes(
&ctx,
Trait::TryFromBytes,
));
}
// We wrap in `const_block` as a backstop in case any derive fails
// to wrap its output in `const_block` (and thus fails to annotate)
// with the full set of `#[allow(...)]` attributes).
let ts = const_block([Some(tokens)]);
// Apply generated-code lint allowances as a backstop, except around
// caller-authored invariant expressions.
let ts = ctx.const_block([Some(tokens)]);
#[cfg(test)]
crate::util::testutil::check_hygiene(ts.clone());
ts.into()

46
rust/zerocopy-derive/repr.rs
View File

@ -20,6 +20,8 @@
MetaList,
};
use crate::util::{path_is_ident, to_ident_str};
/// The computed representation of a type.
///
/// This is the result of processing all `#[repr(...)]` attributes on a type, if
@ -524,11 +526,11 @@ fn from_attrs(
let mut reprs = Vec::new();
for attr in attrs {
// Ignore documentation attributes.
if attr.path().is_ident("doc") {
if path_is_ident(attr.path(), "doc") {
continue;
}
if let Meta::List(ref meta_list) = attr.meta {
if meta_list.path.is_ident("repr") {
if path_is_ident(&meta_list.path, "repr") {
let parsed: Punctuated<Meta, Comma> =
match meta_list.parse_args_with(Punctuated::parse_terminated) {
Ok(parsed) => parsed,
@ -564,6 +566,8 @@ fn from_meta(meta: &Meta) -> Result<RawRepr, UnrecognizedReprError> {
let ident = path.get_ident().ok_or(UnrecognizedReprError)?;
// Only returns `Ok` for non-zero power-of-two values.
// `NonZeroU32::is_power_of_two` is unsupported on our MSRV.
#[allow(clippy::needless_nonzero_get)]
let parse_nzu64 = |list: &MetaList| {
list.parse_args::<LitInt>()
.and_then(|int| int.base10_parse::<NonZeroU32>())
@ -578,7 +582,8 @@ fn from_meta(meta: &Meta) -> Result<RawRepr, UnrecognizedReprError> {
};
use RawRepr::*;
Ok(match (ident.to_string().as_str(), list) {
let ident = to_ident_str(ident);
Ok(match (ident.as_str(), list) {
("u8", None) => U8,
("u16", None) => U16,
("u32", None) => U32,
@ -743,7 +748,40 @@ macro_rules! test {
use PrimitiveRepr::*;
let nz = |n: u32| NonZeroU32::new(n).unwrap();
// Rust treats raw identifiers and their ordinary spellings as the
// same representation hints. Exercise every spelling recognized by
// `RawRepr::from_meta` so that adding a hint cannot accidentally
// reintroduce raw-identifier-sensitive parsing.
macro_rules! test_raw_repr {
($ordinary:meta, $raw:meta => $expected:expr) => {{
let ordinary: Meta = parse_quote!($ordinary);
let raw: Meta = parse_quote!($raw);
assert_eq!(RawRepr::from_meta(&ordinary).ok().unwrap(), $expected);
assert_eq!(RawRepr::from_meta(&raw).ok().unwrap(), $expected);
}};
}
test_raw_repr!(transparent, r#transparent => RawRepr::Transparent);
test_raw_repr!(C, r#C => RawRepr::C);
test_raw_repr!(Rust, r#Rust => RawRepr::Rust);
test_raw_repr!(u8, r#u8 => RawRepr::U8);
test_raw_repr!(u16, r#u16 => RawRepr::U16);
test_raw_repr!(u32, r#u32 => RawRepr::U32);
test_raw_repr!(u64, r#u64 => RawRepr::U64);
test_raw_repr!(u128, r#u128 => RawRepr::U128);
test_raw_repr!(usize, r#usize => RawRepr::Usize);
test_raw_repr!(i8, r#i8 => RawRepr::I8);
test_raw_repr!(i16, r#i16 => RawRepr::I16);
test_raw_repr!(i32, r#i32 => RawRepr::I32);
test_raw_repr!(i64, r#i64 => RawRepr::I64);
test_raw_repr!(i128, r#i128 => RawRepr::I128);
test_raw_repr!(isize, r#isize => RawRepr::Isize);
test_raw_repr!(align(2), r#align(2) => RawRepr::Align(nz(2)));
test_raw_repr!(packed, r#packed => RawRepr::Packed);
test_raw_repr!(packed(2), r#packed(2) => RawRepr::PackedN(nz(2)));
test!(#[repr(transparent)] => StructUnionRepr::Transparent(s()));
test!(#[r#repr(r#transparent)] => StructUnionRepr::Transparent(s()));
test!(#[repr()] => StructUnionRepr::Compound(Rust.into(), None));
test!(#[repr(packed)] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(1)).into())));
test!(#[repr(packed(2))] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(2)).into())));
@ -754,6 +792,8 @@ macro_rules! test {
test!(#[repr(C, packed(2))] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(2)).into())));
test!(#[repr(C, align(1))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(1)).into())));
test!(#[repr(C, align(2))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(2)).into())));
test!(#[r#repr(r#C, r#align(2))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(2)).into())));
test!(#[r#repr(r#C, r#packed(2))] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(2)).into())));
test!(#[repr(transparent)] => EnumRepr::Transparent(s()));
test!(#[repr()] => EnumRepr::Compound(Rust.into(), None));

414
rust/zerocopy-derive/util.rs
View File

@ -13,9 +13,9 @@
use proc_macro2::{Span, TokenStream};
use quote::{quote, quote_spanned, ToTokens};
use syn::{
parse_quote, spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, DeriveInput, Error,
Expr, ExprLit, Field, GenericParam, Ident, Index, Lit, LitStr, Meta, Path, Type, Variant,
Visibility, WherePredicate,
parse::ParseBuffer, parse_quote, spanned::Spanned as _, token::PathSep, Data, DataEnum,
DataStruct, DataUnion, DeriveInput, Error, Expr, ExprLit, Field, GenericParam, Ident, Index,
Lit, LitStr, Meta, Path, Type, Variant, Visibility, WherePredicate,
};
use crate::repr::{CompoundRepr, EnumRepr, PrimitiveRepr, Repr, Spanned};
@ -30,6 +30,59 @@ pub(crate) struct Ctx {
// The span of the last `#[zerocopy(on_error = ...)]` attribute, if any.
pub(crate) on_error_span: Option<proc_macro2::Span>,
/// The first field invariant on the source type, if any.
pub(crate) invariant_span: Option<Span>,
}
#[derive(Eq, PartialEq)]
enum CratePath {
External,
CrateRelative,
ModuleRelative,
}
fn validate_crate_path(path: &Path) -> Result<CratePath, ()> {
if path.segments.is_empty() {
return Err(());
}
enum ModuleRelative {
Yes,
No,
}
let first = path.segments[0].ident.to_string();
let (mut prev_segment, path_type) = match first.as_str() {
"Self" => return Err(()),
"crate" => {
if path.leading_colon.is_some() {
return Err(());
}
(ModuleRelative::No, CratePath::CrateRelative)
}
"self" | "super" => {
if path.leading_colon.is_some() {
return Err(());
}
(ModuleRelative::Yes, CratePath::ModuleRelative)
}
_ => (ModuleRelative::No, CratePath::External),
};
for seg in path.segments.iter().skip(1) {
let ident = seg.ident.to_string();
match ident.as_str() {
"Self" | "crate" | "self" => return Err(()),
"super" => match prev_segment {
ModuleRelative::Yes => {}
ModuleRelative::No => return Err(()),
},
_ => {
prev_segment = ModuleRelative::No;
}
}
}
Ok(path_type)
}
impl Ctx {
@ -42,24 +95,40 @@ pub(crate) fn try_from_derive_input(ast: DeriveInput) -> Result<Self, Error> {
for attr in &ast.attrs {
if let Meta::List(ref meta_list) = attr.meta {
if meta_list.path.is_ident("zerocopy") {
if path_is_ident(&meta_list.path, "zerocopy") {
attr.parse_nested_meta(|meta| {
if meta.path.is_ident("crate") {
let expr = meta.value().and_then(|value| value.parse());
if path_is_ident(&meta.path, "crate") {
let expr = meta.value().and_then(ParseBuffer::parse);
if let Ok(Expr::Lit(ExprLit { lit: Lit::Str(lit), .. })) = expr {
if let Ok(path_lit) = lit.parse::<Ident>() {
path = parse_quote!(::#path_lit);
return Ok(());
if let Ok(mut path_lit) = lit.parse_with(Path::parse_mod_style) {
if let Ok(crate_path) = validate_crate_path(&path_lit) {
// If not expressly relative, absolutize.
if path_lit.leading_colon.is_none() && crate_path == CratePath::External {
path_lit.leading_colon = Some(PathSep::default());
}
path = path_lit;
return Ok(());
}
return Err(Error::new(
lit.span(),
"`crate` attribute requires a valid module path",
));
}
return Err(Error::new(
lit.span(),
"`crate` attribute requires a path as the value",
));
}
return Err(Error::new(
Span::call_site(),
meta.path.span(),
"`crate` attribute requires a path as the value",
));
}
if meta.path.is_ident("on_error") {
if path_is_ident(&meta.path, "on_error") {
on_error_span = Some(meta.path.span());
let value = meta.value()?;
let s: LitStr = value.parse()?;
@ -86,7 +155,8 @@ pub(crate) fn try_from_derive_input(ast: DeriveInput) -> Result<Self, Error> {
}
}
Ok(Self { ast, zerocopy_crate: path, skip_on_error, on_error_span })
let invariant_span = crate::invariant::validate(&ast)?;
Ok(Self { ast, zerocopy_crate: path, skip_on_error, on_error_span, invariant_span })
}
pub(crate) fn with_input(&self, input: &DeriveInput) -> Self {
@ -95,6 +165,7 @@ pub(crate) fn with_input(&self, input: &DeriveInput) -> Self {
zerocopy_crate: self.zerocopy_crate.clone(),
skip_on_error: self.skip_on_error,
on_error_span: self.on_error_span,
invariant_span: self.invariant_span,
}
}
@ -108,15 +179,41 @@ pub(crate) fn core_path(&self) -> TokenStream {
quote!(#zerocopy_crate::util::macro_util::core_reexport)
}
/// Choose an implementation name absent from the caller's tokens. This is
/// best-effort; names introduced by nested macros are not considered.
/// Unlike local variables, type parameters are not hidden by mixed-site
/// hygiene.
pub(crate) fn fresh_ident(&self, name: &str) -> Ident {
let idents = crate::invariant::idents(self.ast.to_token_stream());
let mut name = name.to_owned();
while idents.iter().any(|ident| ident.to_string().trim_start_matches("r#") == name) {
name.push('_');
}
Ident::new(&name, Span::mixed_site())
}
/// Caller-authored invariant expressions must inherit the caller's lint
/// policy, rather than the blanket allowances for generated glue.
pub(crate) fn const_block(
&self,
items: impl IntoIterator<Item = Option<TokenStream>>,
) -> TokenStream {
if self.invariant_span.is_none() {
const_block(items)
} else {
let items = items.into_iter().flatten();
quote! { const _: () = { #(#items)* }; }
}
}
pub(crate) fn cfg_compile_error(&self) -> TokenStream {
// By checking both during the compilation of the proc macro *and* in
// the generated code, we ensure that `--cfg
// zerocopy_unstable_linux` need only be passed *either* when
// the generated code, we ensure that each cfg need only be passed when
// compiling this crate *or* when compiling the user's crate. The former
// is preferable, but in some situations (such as when cross-compiling
// using `cargo build --target`), it doesn't get propagated to this
// crate's build by default.
if cfg!(zerocopy_unstable_linux) {
let on_error = if cfg!(zerocopy_unstable_linux) {
quote!()
} else if let Some(span) = self.on_error_span {
let core = self.core_path();
@ -131,7 +228,24 @@ pub(crate) fn cfg_compile_error(&self) -> TokenStream {
}
} else {
quote!()
}
};
let invariant = if cfg!(zerocopy_unstable_ptr) {
quote!()
} else if let Some(span) = self.invariant_span {
let core = self.core_path();
let error_message =
"`invariant` is experimental; pass '--cfg zerocopy_unstable_ptr' to enable";
quote::quote_spanned! {span=>
#[allow(unused_attributes, unexpected_cfgs)]
const _: () = {
#[cfg(not(zerocopy_unstable_ptr))]
#core::compile_error!(#error_message);
};
}
} else {
quote!()
};
quote!(#on_error #invariant)
}
pub(crate) fn error_or_skip<E>(&self, error: E) -> Result<TokenStream, E> {
@ -151,7 +265,7 @@ pub(crate) trait DataExt {
/// FIXME: Extracting field names for enums doesn't really make sense. Types
/// makes sense because we don't care about where they live - we just care
/// about transitive ownership. But for field names, we'd only use them when
/// generating is_bit_valid, which cares about where they live.
/// generating is_safe, which cares about where they live.
fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)>;
fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)>;
@ -255,6 +369,18 @@ pub(crate) fn to_ident_str(t: &impl ToString) -> String {
}
}
/// Does `path` consist solely of the identifier `expected`?
///
/// Unlike [`Path::is_ident`], this treats a raw identifier and its ordinary
/// spelling as the same identifier. Rust applies that same normalization when
/// interpreting attribute names and arguments.
pub(crate) fn path_is_ident(path: &Path, expected: &str) -> bool {
match path.get_ident() {
Some(ident) => to_ident_str(ident) == expected,
None => false,
}
}
/// This enum describes what kind of padding check needs to be generated for the
/// associated impl.
pub(crate) enum PaddingCheck {
@ -298,22 +424,46 @@ pub(crate) fn validator_macro_context(&self) -> Option<&TokenStream> {
}
}
#[derive(Copy, Clone)]
pub(crate) enum Client {
ProjectDerive,
TryFromBytesDerive,
}
impl ToTokens for Client {
fn to_tokens(&self, tokens: &mut TokenStream) {
let s = match self {
Client::ProjectDerive => "ProjectDerive",
Client::TryFromBytesDerive => "TryFromBytesDerive",
};
let ident = Ident::new(s, Span::call_site());
tokens.extend(quote!(project_clients::#ident));
}
}
impl Client {
pub(crate) fn crate_path(&self, ctx: &Ctx) -> Path {
let zerocopy_crate = &ctx.zerocopy_crate;
parse_quote!(#zerocopy_crate::#self)
}
}
/// Bundles the trait parameters for `HasField` and `ProjectField`.
#[derive(Clone)]
pub(crate) struct FieldProjection {
pub(crate) variant_id: Box<Expr>,
pub(crate) field: Box<Type>,
pub(crate) field_id: Box<Expr>,
}
#[derive(Clone)]
pub(crate) enum Trait {
KnownLayout,
HasTag,
HasField {
variant_id: Box<Expr>,
field: Box<Type>,
field_id: Box<Expr>,
},
ProjectField {
variant_id: Box<Expr>,
field: Box<Type>,
field_id: Box<Expr>,
invariants: Box<Type>,
},
HasTag { client: Client },
HasField { client: Client, projection: FieldProjection },
ProjectField { client: Client, projection: FieldProjection, invariants: Box<Type> },
Immutable,
Project,
TryFromBytes,
FromZeros,
FromBytes,
@ -340,8 +490,9 @@ fn to_tokens(&self, tokens: &mut TokenStream) {
Trait::HasField { .. } => "HasField",
Trait::ProjectField { .. } => "ProjectField",
Trait::KnownLayout => "KnownLayout",
Trait::HasTag => "HasTag",
Trait::HasTag { .. } => "HasTag",
Trait::Immutable => "Immutable",
Trait::Project => "Project",
Trait::TryFromBytes => "TryFromBytes",
Trait::FromZeros => "FromZeros",
Trait::FromBytes => "FromBytes",
@ -354,15 +505,18 @@ fn to_tokens(&self, tokens: &mut TokenStream) {
};
let ident = Ident::new(s, Span::call_site());
let arguments: Option<syn::AngleBracketedGenericArguments> = match self {
Trait::HasField { variant_id, field, field_id } => {
Some(parse_quote!(<#field, #variant_id, #field_id>))
Trait::HasTag { client } => Some(parse_quote!(<#client>)),
Trait::HasField { client, projection } => {
let FieldProjection { variant_id, field, field_id } = projection;
Some(parse_quote!(<#client, #field, #variant_id, #field_id>))
}
Trait::ProjectField { variant_id, field, field_id, invariants } => {
Some(parse_quote!(<#field, #invariants, #variant_id, #field_id>))
Trait::ProjectField { client, projection, invariants } => {
let FieldProjection { variant_id, field, field_id } = projection;
Some(parse_quote!(<#client, #field, #invariants, #variant_id, #field_id>))
}
Trait::KnownLayout
| Trait::HasTag
| Trait::Immutable
| Trait::Project
| Trait::TryFromBytes
| Trait::FromZeros
| Trait::FromBytes
@ -383,6 +537,28 @@ pub(crate) fn crate_path(&self, ctx: &Ctx) -> Path {
let core = ctx.core_path();
match self {
Self::Sized => parse_quote!(#core::marker::#self),
Self::HasTag { client } => {
let client = client.crate_path(ctx);
parse_quote!(#zerocopy_crate::HasTag<#client>)
}
Self::HasField { client, projection } => {
let FieldProjection { variant_id, field, field_id } = projection;
let client = client.crate_path(ctx);
parse_quote!(#zerocopy_crate::HasField<#client, #field, #variant_id, #field_id>)
}
Self::ProjectField { client, projection, invariants } => {
let FieldProjection { variant_id, field, field_id } = projection;
let client = client.crate_path(ctx);
parse_quote!(
#zerocopy_crate::ProjectField<
#client,
#field,
#invariants,
#variant_id,
#field_id
>
)
}
_ => parse_quote!(#zerocopy_crate::#self),
}
}
@ -704,7 +880,12 @@ fn only_derive_is_allowed_to_implement_this_trait() {}
let outer_extras = self.outer_extras.filter(|e| !e.is_empty());
let cfg_compile_error = self.ctx.cfg_compile_error();
const_block([Some(cfg_compile_error), Some(impl_tokens), outer_extras])
let items = [Some(cfg_compile_error), Some(impl_tokens), outer_extras];
if matches!(self.trt, Trait::TryFromBytes) {
self.ctx.const_block(items)
} else {
const_block(items)
}
}
}
@ -730,8 +911,7 @@ fn then_some<T>(self, t: T) -> Option<T> {
}
}
pub(crate) fn const_block(items: impl IntoIterator<Item = Option<TokenStream>>) -> TokenStream {
let items = items.into_iter().flatten();
pub(crate) fn allow_generated_code() -> TokenStream {
quote! {
#[allow(
// FIXME(#553): Add a test that generates a warning when
@ -748,6 +928,14 @@ pub(crate) fn const_block(items: impl IntoIterator<Item = Option<TokenStream>>)
non_ascii_idents,
clippy::missing_inline_in_public_items,
)]
}
}
pub(crate) fn const_block(items: impl IntoIterator<Item = Option<TokenStream>>) -> TokenStream {
let items = items.into_iter().flatten();
let allow = allow_generated_code();
quote! {
#allow
#[deny(ambiguous_associated_items)]
// While there are not currently any warnings that this suppresses
// (that we're aware of), it's good future-proofing hygiene.
@ -778,7 +966,8 @@ pub(crate) fn generate_tag_enum(ctx: &Ctx, repr: &EnumRepr, data: &DataEnum) ->
quote! {
#repr
#[allow(dead_code)]
#[allow(dead_code, clippy::derive_partial_eq_without_eq)]
#[derive(Copy, Clone, PartialEq)]
pub enum ___ZerocopyTag {
#(#variants,)*
}
@ -824,18 +1013,40 @@ pub(crate) mod testutil {
pub(crate) fn check_hygiene(ts: TokenStream) {
struct AmbiguousItemVisitor;
fn panic_ambiguous_path(path: &impl quote::ToTokens) -> ! {
panic!(
"Found ambiguous path `{}` in generated output. \
All associated item access must be fully qualified (e.g., `<Self as Trait>::Item`) \
to prevent hygiene issues.",
quote::quote!(#path)
);
}
impl<'ast> Visit<'ast> for AmbiguousItemVisitor {
fn visit_expr_path(&mut self, i: &'ast syn::ExprPath) {
if let Some(qself) = &i.qself {
if qself.position == 0 {
panic_ambiguous_path(i);
}
}
visit::visit_expr_path(self, i);
}
fn visit_path(&mut self, i: &'ast syn::Path) {
if i.segments.len() > 1 && i.segments.first().unwrap().ident == "Self" {
panic!(
"Found ambiguous path `{}` in generated output. \
All associated item access must be fully qualified (e.g., `<Self as Trait>::Item`) \
to prevent hygiene issues.",
quote::quote!(#i)
);
panic_ambiguous_path(i);
}
visit::visit_path(self, i);
}
fn visit_type_path(&mut self, i: &'ast syn::TypePath) {
if let Some(qself) = &i.qself {
if qself.position == 0 {
panic_ambiguous_path(i);
}
}
visit::visit_type_path(self, i);
}
}
let file = syn::parse2::<syn::File>(ts).expect("failed to parse generated output as File");
@ -860,4 +1071,117 @@ fn foo() {
}
});
}
#[test]
#[should_panic(expected = "Found ambiguous path `< T > :: Ambiguous`")]
fn test_check_hygiene_type_relative_expr_failure() {
check_hygiene(quote::quote! {
fn foo<T>() {
let _ = <T>::Ambiguous;
}
});
}
#[test]
#[should_panic(expected = "Found ambiguous path `< T > :: Ambiguous`")]
fn test_check_hygiene_type_relative_type_failure() {
check_hygiene(quote::quote! {
fn foo<T>() {
let _: <T>::Ambiguous;
}
});
}
#[test]
fn test_validate_crate_path() {
use syn::parse_str;
let valid = [
"zerocopy",
"crate",
"crate::foo::bar",
"self",
"self::foo",
"self::super::foo",
"super",
"super::foo",
"super::super::foo",
"super::super::super",
"foo::bar::baz",
"::foo::bar",
];
for path_str in valid {
let path = parse_str::<syn::Path>(path_str).unwrap();
assert!(
super::validate_crate_path(&path).is_ok(),
"expected valid path for `{}`",
path_str
);
}
let invalid = [
"::crate::foo",
"::self::foo",
"::super::foo",
"Self",
"Self::foo",
"foo::Self::bar",
"foo::crate::bar",
"foo::super::bar",
"foo::self",
"super::foo::super",
"super::crate::foo",
"crate::super::foo",
"self::self::foo",
];
for path_str in invalid {
let path = parse_str::<syn::Path>(path_str).unwrap();
assert!(
super::validate_crate_path(&path).is_err(),
"expected invalid path for `{}`",
path_str
);
}
}
#[test]
fn test_path_is_ident() {
use syn::parse_str;
for (expected, ordinary, raw) in [
("doc", "doc", "r#doc"),
("repr", "repr", "r#repr"),
("zerocopy", "zerocopy", "r#zerocopy"),
("on_error", "on_error", "r#on_error"),
] {
let ordinary = parse_str::<syn::Path>(ordinary).unwrap();
let raw = parse_str::<syn::Path>(raw).unwrap();
assert!(super::path_is_ident(&ordinary, expected));
assert!(super::path_is_ident(&raw, expected));
}
// `r#crate` is forbidden by Rust's raw identifier grammar.
let crate_path = parse_str::<syn::Path>("crate").unwrap();
assert!(super::path_is_ident(&crate_path, "crate"));
for path in ["::zerocopy", "module::zerocopy"] {
let path = parse_str::<syn::Path>(path).unwrap();
assert!(!super::path_is_ident(&path, "zerocopy"));
}
}
#[test]
fn test_raw_zerocopy_attributes() {
let ast: syn::DeriveInput = syn::parse_quote! {
#[r#zerocopy(crate = "renamed", r#on_error = "skip")]
struct Foo;
};
let ctx = super::Ctx::try_from_derive_input(ast).unwrap();
let path = &ctx.zerocopy_crate;
assert_eq!(quote::quote!(#path).to_string(), ":: renamed");
assert!(ctx.skip_on_error);
}
}

12
rust/zerocopy/README.md
View File

@ -1,13 +1,13 @@
# `zerocopy`
These source files come from the Rust `zerocopy` crate, version v0.8.54
(released 2026-07-08), hosted in the <https://github.com/google/zerocopy>
These source files come from the Rust `zerocopy` crate, version v0.8.62
(released 2026-10-08), hosted in the <https://github.com/google/zerocopy>
repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only
modified to tweak the SPDX license identifiers.
For copyright details, please see:
https://github.com/google/zerocopy/blob/v0.8.54/README.md?plain=1
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT
https://github.com/google/zerocopy/blob/v0.8.62/zerocopy/README.md?plain=1
https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-BSD
https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-APACHE
https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-MIT

3
rust/zerocopy/benches/as_bytes_dynamic_size.x86-64
View File

@ -1,5 +1,4 @@
bench_as_bytes_dynamic_size:
mov rax, rdi
lea rdx, [2*rsi + 5]
and rdx, -2
lea rdx, [2*rsi + 4]
ret

20
rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca
View File

@ -1,11 +1,11 @@
Iterations: 100
Instructions: 400
Total Cycles: 137
Total uOps: 400
Instructions: 300
Total Cycles: 104
Total uOps: 300
Dispatch Width: 4
uOps Per Cycle: 2.92
IPC: 2.92
uOps Per Cycle: 2.88
IPC: 2.88
Block RThroughput: 1.0
@ -19,8 +19,7 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov rax, rdi
1 1 0.50 lea rdx, [2*rsi + 5]
1 1 0.33 and rdx, -2
1 1 0.50 lea rdx, [2*rsi + 4]
1 1 1.00 U ret
@ -37,11 +36,10 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 1.33 1.33 - 1.34 - -
- - 0.99 1.00 - 1.01 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.66 - 0.34 - - mov rax, rdi
- - 0.33 0.67 - - - - lea rdx, [2*rsi + 5]
- - 1.00 - - - - - and rdx, -2
- - 0.99 - - 0.01 - - mov rax, rdi
- - - 1.00 - - - - lea rdx, [2*rsi + 4]
- - - - - 1.00 - - ret

4
rust/zerocopy/benches/extend_vec_zeroed.x86-64
View File

@ -41,8 +41,8 @@ bench_extend_vec_zeroed:
.LBB6_6:
mov r13, rsi
lea rcx, [rax + rax]
cmp r15, rcx
cmova rcx, r15
cmp rcx, r15
cmovbe rcx, r15
cmp rcx, 5
mov r14d, 4
cmovae r14, rcx

8
rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca
View File

@ -55,8 +55,8 @@ Instruction Info:
1 1 1.00 U ret
1 1 0.33 mov r13, rsi
1 1 0.50 lea rcx, [rax + rax]
1 1 0.33 cmp r15, rcx
3 3 1.00 cmova rcx, r15
1 1 0.33 cmp rcx, r15
3 3 1.00 cmovbe rcx, r15
1 1 0.33 cmp rcx, 5
1 1 0.33 mov r14d, 4
2 2 0.67 cmovae r14, rcx
@ -128,8 +128,8 @@ Resource pressure by instruction:
- - - - - 1.00 - - ret
- - 1.00 - - - - - mov r13, rsi
- - 0.01 0.99 - - - - lea rcx, [rax + rax]
- - 0.99 0.01 - - - - cmp r15, rcx
- - 2.00 0.01 - 0.99 - - cmova rcx, r15
- - 0.99 0.01 - - - - cmp rcx, r15
- - 2.00 0.01 - 0.99 - - cmovbe rcx, r15
- - 0.01 0.99 - - - - cmp rcx, 5
- - 0.01 0.99 - - - - mov r14d, 4
- - 1.00 0.01 - 0.99 - - cmovae r14, rcx

4
rust/zerocopy/benches/insert_vec_zeroed.x86-64
View File

@ -54,8 +54,8 @@ bench_insert_vec_zeroed:
mov r15, rsi
mov rbp, rdx
lea rcx, [rax + rax]
cmp r12, rcx
cmova rcx, r12
cmp rcx, r12
cmovbe rcx, r12
cmp rcx, 5
mov r14d, 4
cmovae r14, rcx

8
rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca
View File

@ -68,8 +68,8 @@ Instruction Info:
1 1 0.33 mov r15, rsi
1 1 0.33 mov rbp, rdx
1 1 0.50 lea rcx, [rax + rax]
1 1 0.33 cmp r12, rcx
3 3 1.00 cmova rcx, r12
1 1 0.33 cmp rcx, r12
3 3 1.00 cmovbe rcx, r12
1 1 0.33 cmp rcx, 5
1 1 0.33 mov r14d, 4
2 2 0.67 cmovae r14, rcx
@ -159,8 +159,8 @@ Resource pressure by instruction:
- - 0.49 0.51 - - - - mov r15, rsi
- - 0.51 0.49 - - - - mov rbp, rdx
- - 0.49 0.51 - - - - lea rcx, [rax + rax]
- - 0.49 0.50 - 0.01 - - cmp r12, rcx
- - 1.04 0.50 - 1.46 - - cmova rcx, r12
- - 0.49 0.50 - 0.01 - - cmp rcx, r12
- - 1.04 0.50 - 1.46 - - cmovbe rcx, r12
- - 0.49 0.49 - 0.02 - - cmp rcx, 5
- - 0.50 - - 0.50 - - mov r14d, 4
- - 0.50 0.51 - 0.99 - - cmovae r14, rcx

3
rust/zerocopy/benches/new_vec_zeroed.x86-64
View File

@ -29,8 +29,7 @@ bench_new_vec_zeroed:
mov qword ptr [rax + 16], rsi
ret
.LBB5_5:
movabs rcx, -9223372036854775808
mov qword ptr [rax], rcx
mov qword ptr [rax], -1
ret
.LBB5_2:
mov ecx, 2

46
rust/zerocopy/benches/new_vec_zeroed.x86-64.mca
View File

@ -1,11 +1,11 @@
Iterations: 100
Instructions: 3700
Instructions: 3600
Total Cycles: 3486
Total uOps: 4600
Total uOps: 4500
Dispatch Width: 4
uOps Per Cycle: 1.32
IPC: 1.06
uOps Per Cycle: 1.29
IPC: 1.03
Block RThroughput: 12.0
@ -47,8 +47,7 @@ Instruction Info:
1 1 1.00 * mov qword ptr [rax + 8], rcx
1 1 1.00 * mov qword ptr [rax + 16], rsi
1 1 1.00 U ret
1 1 0.33 movabs rcx, -9223372036854775808
1 1 1.00 * mov qword ptr [rax], rcx
1 1 1.00 * mov qword ptr [rax], -1
1 1 1.00 U ret
1 1 0.33 mov ecx, 2
1 1 1.00 * mov qword ptr [rax], rsi
@ -70,31 +69,31 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 6.99 6.99 12.00 10.02 8.00 9.00
- - 5.99 6.01 12.00 11.00 8.00 9.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - 0.01 0.98 - 0.01 - - mov rax, rdi
- - 0.98 0.02 - - - - movabs rcx, 1537228672809129301
- - 0.02 0.98 - - - - cmp rsi, rcx
- - 0.97 0.02 - 0.01 - - mov rax, rdi
- - 0.02 0.98 - - - - movabs rcx, 1537228672809129301
- - 0.97 0.03 - - - - cmp rsi, rcx
- - - - - 1.00 - - ja .LBB5_5
- - 0.98 - - 0.02 - - test rsi, rsi
- - 0.03 0.96 - 0.01 - - test rsi, rsi
- - - - - 1.00 - - je .LBB5_2
- - - - 1.00 - - 1.00 push r15
- - - - 1.00 - 1.00 - push r14
- - - - 1.00 - - 1.00 push rbx
- - - 1.00 - - - - lea rcx, [rsi + rsi]
- - - 1.00 - - - - lea rbx, [rcx + 2*rcx]
- - 1.00 - - - - - mov r14, rax
- - 1.00 - - - - - mov r15, rsi
- - 0.99 0.01 - - - - lea rcx, [rsi + rsi]
- - 0.99 0.01 - - - - lea rbx, [rcx + 2*rcx]
- - 0.01 0.99 - - - - mov r14, rax
- - 0.01 0.01 - 0.98 - - mov r15, rsi
- - - - 1.00 1.00 2.00 - call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
- - - 0.01 - 0.99 - - mov esi, 2
- - 0.01 0.99 - - - - mov rdi, rbx
- - - 0.98 - 0.02 - - mov esi, 2
- - 0.96 0.02 - 0.02 - - mov rdi, rbx
- - - - 1.00 1.00 - 2.00 call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
- - 0.01 - - 0.99 - - mov rsi, r15
- - 0.99 0.01 - - - - mov rcx, rax
- - - 0.99 - 0.01 - - mov rax, r14
- - 0.99 0.01 - - - - test rcx, rcx
- - 0.02 0.96 - 0.02 - - mov rsi, r15
- - 0.02 0.02 - 0.96 - - mov rcx, rax
- - 0.96 0.02 - 0.02 - - mov rax, r14
- - 0.02 0.02 - 0.96 - - test rcx, rcx
- - - - - - - 1.00 pop rbx
- - - - - - 1.00 - pop r14
- - - - - - - 1.00 pop r15
@ -103,10 +102,9 @@ Resource pressure by instruction:
- - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx
- - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi
- - - - - 1.00 - - ret
- - 0.01 0.99 - - - - movabs rcx, -9223372036854775808
- - - - 1.00 - - 1.00 mov qword ptr [rax], rcx
- - - - 1.00 - - 1.00 mov qword ptr [rax], -1
- - - - - 1.00 - - ret
- - 0.99 0.01 - - - - mov ecx, 2
- - 0.02 0.98 - - - - mov ecx, 2
- - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
- - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx
- - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi

35
rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64
View File

@ -1,22 +1,21 @@
bench_ref_from_bytes_dynamic_padding:
test dil, 3
jne .LBB5_3
movabs rax, 9223372036854775804
and rax, rsi
cmp rax, 9
jb .LBB5_3
add rax, -9
movabs rcx, -6148914691236517205
mul rcx
shr rdx
lea rax, [rdx + 2*rdx]
or rax, 3
add rax, 9
cmp rsi, rax
je .LBB5_4
.LBB5_3:
cmp rsi, 12
setae al
mov ecx, esi
or ecx, edi
test cl, 3
sete cl
and cl, al
cmp cl, 1
je .LBB5_2
xor edi, edi
mov rdx, rsi
.LBB5_4:
mov rax, rdi
ret
.LBB5_2:
add rsi, -9
movabs rcx, -6148914691236517205
mov rax, rsi
mul rcx
shr rdx
mov rax, rdi
ret

70
rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca
View File

@ -1,11 +1,11 @@
Iterations: 100
Instructions: 1900
Total Cycles: 645
Total Cycles: 641
Total uOps: 2000
Dispatch Width: 4
uOps Per Cycle: 3.10
IPC: 2.95
uOps Per Cycle: 3.12
IPC: 2.96
Block RThroughput: 5.0
@ -18,23 +18,23 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 test dil, 3
1 1 1.00 jne .LBB5_3
1 1 0.33 movabs rax, 9223372036854775804
1 1 0.33 and rax, rsi
1 1 0.33 cmp rax, 9
1 1 1.00 jb .LBB5_3
1 1 0.33 add rax, -9
1 1 0.33 cmp rsi, 12
1 1 0.50 setae al
1 1 0.33 mov ecx, esi
1 1 0.33 or ecx, edi
1 1 0.33 test cl, 3
1 1 0.50 sete cl
1 1 0.33 and cl, al
1 1 0.33 cmp cl, 1
1 1 1.00 je .LBB5_2
1 0 0.25 xor edi, edi
1 1 0.33 mov rax, rdi
1 1 1.00 U ret
1 1 0.33 add rsi, -9
1 1 0.33 movabs rcx, -6148914691236517205
1 1 0.33 mov rax, rsi
2 4 1.00 mul rcx
1 1 0.50 shr rdx
1 1 0.50 lea rax, [rdx + 2*rdx]
1 1 0.33 or rax, 3
1 1 0.33 add rax, 9
1 1 0.33 cmp rsi, rax
1 1 1.00 je .LBB5_4
1 0 0.25 xor edi, edi
1 1 0.33 mov rdx, rsi
1 1 0.33 mov rax, rdi
1 1 1.00 U ret
@ -52,26 +52,26 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 6.32 6.33 - 6.35 - -
- - 6.33 6.33 - 6.34 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - 0.64 0.35 - 0.01 - - test dil, 3
- - - - - 1.00 - - jne .LBB5_3
- - 0.34 0.65 - 0.01 - - movabs rax, 9223372036854775804
- - 0.35 0.65 - - - - and rax, rsi
- - 0.33 0.34 - 0.33 - - cmp rax, 9
- - - - - 1.00 - - jb .LBB5_3
- - 0.35 - - 0.65 - - add rax, -9
- - 0.97 0.01 - 0.02 - - movabs rcx, -6148914691236517205
- - 1.00 1.00 - - - - mul rcx
- - 0.99 - - 0.01 - - shr rdx
- - 0.33 0.67 - - - - lea rax, [rdx + 2*rdx]
- - 0.34 0.66 - - - - or rax, 3
- - 0.33 0.66 - 0.01 - - add rax, 9
- - 0.01 0.99 - - - - cmp rsi, rax
- - - - - 1.00 - - je .LBB5_4
- - 0.99 - - 0.01 - - cmp rsi, 12
- - 1.00 - - - - - setae al
- - - 0.34 - 0.66 - - mov ecx, esi
- - - 0.99 - 0.01 - - or ecx, edi
- - - 1.00 - - - - test cl, 3
- - 0.67 - - 0.33 - - sete cl
- - - 1.00 - - - - and cl, al
- - - 1.00 - - - - cmp cl, 1
- - - - - 1.00 - - je .LBB5_2
- - - - - - - - xor edi, edi
- - 0.32 0.01 - 0.67 - - mov rdx, rsi
- - 0.02 0.34 - 0.64 - - mov rax, rdi
- - 0.66 - - 0.34 - - mov rax, rdi
- - - - - 1.00 - - ret
- - 0.35 0.01 - 0.64 - - add rsi, -9
- - 0.65 0.33 - 0.02 - - movabs rcx, -6148914691236517205
- - 0.01 0.33 - 0.66 - - mov rax, rsi
- - 1.00 1.00 - - - - mul rcx
- - 1.00 - - - - - shr rdx
- - - 0.33 - 0.67 - - mov rax, rdi
- - - - - 1.00 - - ret

21
rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64
View File

@ -1,20 +1,11 @@
bench_ref_from_bytes_dynamic_size:
mov rdx, rsi
cmp rsi, 4
setb al
or al, dil
test al, 1
je .LBB5_2
mov ecx, edx
or ecx, edi
xor eax, eax
ret
.LBB5_2:
lea rcx, [rdx - 4]
mov rsi, rcx
and rsi, -2
add rsi, 4
shr rcx
xor eax, eax
cmp rdx, rsi
cmove rdx, rcx
sub rdx, 4
cmovb rdi, rax
shr rdx
test cl, 1
cmove rax, rdi
ret

58
rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca
View File

@ -1,12 +1,12 @@
Iterations: 100
Instructions: 1800
Total Cycles: 704
Total uOps: 2000
Instructions: 1000
Total Cycles: 373
Total uOps: 1200
Dispatch Width: 4
uOps Per Cycle: 2.84
IPC: 2.56
Block RThroughput: 5.0
uOps Per Cycle: 3.22
IPC: 2.68
Block RThroughput: 3.0
Instruction Info:
@ -19,21 +19,13 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov rdx, rsi
1 1 0.33 cmp rsi, 4
1 1 0.50 setb al
1 1 0.33 or al, dil
1 1 0.33 test al, 1
1 1 1.00 je .LBB5_2
1 1 0.33 mov ecx, edx
1 1 0.33 or ecx, edi
1 0 0.25 xor eax, eax
1 1 1.00 U ret
1 1 0.50 lea rcx, [rdx - 4]
1 1 0.33 mov rsi, rcx
1 1 0.33 and rsi, -2
1 1 0.33 add rsi, 4
1 1 0.50 shr rcx
1 0 0.25 xor eax, eax
1 1 0.33 cmp rdx, rsi
2 2 0.67 cmove rdx, rcx
1 1 0.33 sub rdx, 4
2 2 0.67 cmovb rdi, rax
1 1 0.50 shr rdx
1 1 0.33 test cl, 1
2 2 0.67 cmove rax, rdi
1 1 1.00 U ret
@ -51,25 +43,17 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 5.97 5.98 - 6.05 - -
- - 3.66 3.67 - 3.67 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - 0.97 0.01 - 0.02 - - mov rdx, rsi
- - 0.01 0.02 - 0.97 - - cmp rsi, 4
- - 0.03 - - 0.97 - - setb al
- - 0.01 0.02 - 0.97 - - or al, dil
- - - 0.98 - 0.02 - - test al, 1
- - - - - 1.00 - - je .LBB5_2
- - 0.32 0.01 - 0.67 - - mov rdx, rsi
- - 0.33 0.35 - 0.32 - - mov ecx, edx
- - 0.01 0.34 - 0.65 - - or ecx, edi
- - - - - - - - xor eax, eax
- - - - - 1.00 - - ret
- - 0.98 0.02 - - - - lea rcx, [rdx - 4]
- - 0.01 0.99 - - - - mov rsi, rcx
- - - 0.98 - 0.02 - - and rsi, -2
- - 0.98 0.01 - 0.01 - - add rsi, 4
- - 0.99 - - 0.01 - - shr rcx
- - - - - - - - xor eax, eax
- - 0.02 0.97 - 0.01 - - cmp rdx, rsi
- - 0.99 0.99 - 0.02 - - cmove rdx, rcx
- - 0.98 0.99 - 0.03 - - cmove rax, rdi
- - 0.67 0.33 - - - - sub rdx, 4
- - 0.99 0.98 - 0.03 - - cmovb rdi, rax
- - 0.03 - - 0.97 - - shr rdx
- - 0.32 0.67 - 0.01 - - test cl, 1
- - 0.99 0.99 - 0.02 - - cmove rax, rdi
- - - - - 1.00 - - ret

7
rust/zerocopy/benches/ref_from_bytes_static_size.x86-64
View File

@ -1,8 +1,7 @@
bench_ref_from_bytes_static_size:
mov ecx, edi
and ecx, 1
xor rsi, 6
xor eax, eax
or rsi, rcx
test dil, 1
cmovne rdi, rax
cmp rsi, 6
cmove rax, rdi
ret

26
rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca
View File

@ -1,11 +1,11 @@
Iterations: 100
Instructions: 700
Total Cycles: 240
Instructions: 600
Total Cycles: 305
Total uOps: 800
Dispatch Width: 4
uOps Per Cycle: 3.33
IPC: 2.92
uOps Per Cycle: 2.62
IPC: 1.97
Block RThroughput: 2.0
@ -18,11 +18,10 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov ecx, edi
1 1 0.33 and ecx, 1
1 1 0.33 xor rsi, 6
1 0 0.25 xor eax, eax
1 1 0.33 or rsi, rcx
1 1 0.33 test dil, 1
2 2 0.67 cmovne rdi, rax
1 1 0.33 cmp rsi, 6
2 2 0.67 cmove rax, rdi
1 1 1.00 U ret
@ -40,14 +39,13 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 2.33 2.33 - 2.34 - -
- - 1.95 2.28 - 2.77 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - 0.01 0.98 - 0.01 - - mov ecx, edi
- - 0.02 0.66 - 0.32 - - and ecx, 1
- - 0.33 0.66 - 0.01 - - xor rsi, 6
- - - - - - - - xor eax, eax
- - 0.98 0.02 - - - - or rsi, rcx
- - 0.99 0.01 - 1.00 - - cmove rax, rdi
- - 0.05 0.06 - 0.89 - - test dil, 1
- - 0.95 0.94 - 0.11 - - cmovne rdi, rax
- - - 0.34 - 0.66 - - cmp rsi, 6
- - 0.95 0.94 - 0.11 - - cmove rax, rdi
- - - - - 1.00 - - ret

26
rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64
View File

@ -1,19 +1,19 @@
bench_ref_from_prefix_dynamic_padding:
xor edx, edx
mov eax, 0
test dil, 3
je .LBB5_1
sete al
cmp rsi, 12
setae cl
test al, cl
jne .LBB5_2
xor edx, edx
test dil, 3
sete dl
xor edi, edi
mov rax, rdi
ret
.LBB5_1:
movabs rax, 9223372036854775804
and rsi, rax
cmp rsi, 9
jae .LBB5_3
mov edx, 1
xor eax, eax
ret
.LBB5_3:
add rsi, -9
.LBB5_2:
or rsi, 3
add rsi, -12
movabs rcx, -6148914691236517205
mov rax, rsi
mul rcx

64
rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca
View File

@ -1,12 +1,12 @@
Iterations: 100
Instructions: 1900
Total Cycles: 608
Total uOps: 2000
Instructions: 2000
Total Cycles: 640
Total uOps: 2100
Dispatch Width: 4
uOps Per Cycle: 3.29
uOps Per Cycle: 3.28
IPC: 3.13
Block RThroughput: 5.0
Block RThroughput: 5.3
Instruction Info:
@ -18,19 +18,20 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
1 0 0.25 xor edx, edx
1 1 0.33 mov eax, 0
1 1 0.33 test dil, 3
1 1 1.00 je .LBB5_1
1 1 0.50 sete al
1 1 0.33 cmp rsi, 12
1 1 0.50 setae cl
1 1 0.33 test al, cl
1 1 1.00 jne .LBB5_2
1 0 0.25 xor edx, edx
1 1 0.33 test dil, 3
1 1 0.50 sete dl
1 0 0.25 xor edi, edi
1 1 0.33 mov rax, rdi
1 1 1.00 U ret
1 1 0.33 movabs rax, 9223372036854775804
1 1 0.33 and rsi, rax
1 1 0.33 cmp rsi, 9
1 1 1.00 jae .LBB5_3
1 1 0.33 mov edx, 1
1 0 0.25 xor eax, eax
1 1 1.00 U ret
1 1 0.33 add rsi, -9
1 1 0.33 or rsi, 3
1 1 0.33 add rsi, -12
1 1 0.33 movabs rcx, -6148914691236517205
1 1 0.33 mov rax, rsi
2 4 1.00 mul rcx
@ -52,26 +53,27 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 6.00 6.00 - 6.00 - -
- - 6.33 6.33 - 6.34 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - 0.65 0.34 - 0.01 - - test dil, 3
- - 0.99 - - 0.01 - - sete al
- - 0.01 0.33 - 0.66 - - cmp rsi, 12
- - 0.34 - - 0.66 - - setae cl
- - 1.00 - - - - - test al, cl
- - - - - 1.00 - - jne .LBB5_2
- - - - - - - - xor edx, edx
- - 0.01 0.98 - 0.01 - - mov eax, 0
- - 0.98 0.01 - 0.01 - - test dil, 3
- - - - - 1.00 - - je .LBB5_1
- - - 0.99 - 0.01 - - test dil, 3
- - - - - 1.00 - - sete dl
- - - - - - - - xor edi, edi
- - - 1.00 - - - - mov rax, rdi
- - - - - 1.00 - - ret
- - 0.01 0.99 - - - - movabs rax, 9223372036854775804
- - - 1.00 - - - - and rsi, rax
- - - 1.00 - - - - cmp rsi, 9
- - - - - 1.00 - - jae .LBB5_3
- - 1.00 - - - - - mov edx, 1
- - - - - - - - xor eax, eax
- - - - - 1.00 - - ret
- - 0.02 0.02 - 0.96 - - add rsi, -9
- - 0.99 0.01 - - - - movabs rcx, -6148914691236517205
- - 0.01 0.99 - - - - mov rax, rsi
- - 0.01 0.99 - - - - or rsi, 3
- - 0.33 0.67 - - - - add rsi, -12
- - 0.66 0.34 - - - - movabs rcx, -6148914691236517205
- - 0.34 0.33 - 0.33 - - mov rax, rsi
- - 1.00 1.00 - - - - mul rcx
- - 1.00 - - - - - shr rdx
- - 0.98 - - 0.02 - - mov rax, rdi
- - - 0.34 - 0.66 - - mov rax, rdi
- - - - - 1.00 - - ret

21
rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64
View File

@ -1,17 +1,10 @@
bench_ref_from_prefix_dynamic_size:
xor edx, edx
mov eax, 0
test dil, 1
jne .LBB5_4
cmp rsi, 4
jae .LBB5_3
mov edx, 1
xor eax, eax
ret
.LBB5_3:
add rsi, -4
shr rsi
mov rdx, rsi
mov rax, rdi
.LBB5_4:
xor eax, eax
sub rdx, 4
mov rcx, rdi
cmovb rcx, rax
shr rdx
test dil, 1
cmove rax, rcx
ret

52
rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca
View File

@ -1,12 +1,12 @@
Iterations: 100
Instructions: 1400
Total Cycles: 405
Total uOps: 1400
Instructions: 900
Total Cycles: 339
Total uOps: 1100
Dispatch Width: 4
uOps Per Cycle: 3.46
IPC: 3.46
Block RThroughput: 4.0
uOps Per Cycle: 3.24
IPC: 2.65
Block RThroughput: 2.8
Instruction Info:
@ -18,19 +18,14 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
1 0 0.25 xor edx, edx
1 1 0.33 mov eax, 0
1 1 0.33 test dil, 1
1 1 1.00 jne .LBB5_4
1 1 0.33 cmp rsi, 4
1 1 1.00 jae .LBB5_3
1 1 0.33 mov edx, 1
1 0 0.25 xor eax, eax
1 1 1.00 U ret
1 1 0.33 add rsi, -4
1 1 0.50 shr rsi
1 1 0.33 mov rdx, rsi
1 1 0.33 mov rax, rdi
1 0 0.25 xor eax, eax
1 1 0.33 sub rdx, 4
1 1 0.33 mov rcx, rdi
2 2 0.67 cmovb rcx, rax
1 1 0.50 shr rdx
1 1 0.33 test dil, 1
2 2 0.67 cmove rax, rcx
1 1 1.00 U ret
@ -47,21 +42,16 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 3.99 3.99 - 4.02 - -
- - 3.33 3.33 - 3.34 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - - - - - xor edx, edx
- - 0.01 0.98 - 0.01 - - mov eax, 0
- - 0.98 0.02 - - - - test dil, 1
- - - - - 1.00 - - jne .LBB5_4
- - 0.02 0.98 - - - - cmp rsi, 4
- - - - - 1.00 - - jae .LBB5_3
- - 0.98 0.01 - 0.01 - - mov edx, 1
- - 0.33 0.66 - 0.01 - - mov rdx, rsi
- - - - - - - - xor eax, eax
- - - - - 1.00 - - ret
- - 0.01 0.99 - - - - add rsi, -4
- - 1.00 - - - - - shr rsi
- - - 1.00 - - - - mov rdx, rsi
- - 0.99 0.01 - - - - mov rax, rdi
- - 0.01 - - 0.99 - - sub rdx, 4
- - 0.66 0.34 - - - - mov rcx, rdi
- - 1.00 1.00 - - - - cmovb rcx, rax
- - 0.33 - - 0.67 - - shr rdx
- - - 0.33 - 0.67 - - test dil, 1
- - 1.00 1.00 - - - - cmove rax, rcx
- - - - - 1.00 - - ret

29
rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64
View File

@ -1,23 +1,22 @@
bench_ref_from_suffix_dynamic_padding:
lea eax, [rsi + rdi]
test al, 3
jne .LBB5_1
movabs rax, 9223372036854775804
and rax, rsi
cmp rax, 9
jae .LBB5_3
.LBB5_1:
xor eax, eax
setne al
cmp rsi, 12
setb cl
or cl, al
je .LBB5_2
xor edi, edi
mov rax, rdi
ret
.LBB5_3:
add rax, -9
.LBB5_2:
mov rax, rsi
or rax, 3
add rax, -12
movabs rcx, -6148914691236517205
mul rcx
shr rdx
lea rax, [rdx + 2*rdx]
sub rsi, rax
or rax, -4
add rsi, rdi
add rax, rsi
add rax, -8
and esi, 3
add rdi, rsi
mov rax, rdi
ret

68
rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca
View File

@ -1,11 +1,11 @@
Iterations: 100
Instructions: 2000
Total Cycles: 682
Total Cycles: 673
Total uOps: 2100
Dispatch Width: 4
uOps Per Cycle: 3.08
IPC: 2.93
uOps Per Cycle: 3.12
IPC: 2.97
Block RThroughput: 5.3
@ -20,23 +20,23 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.50 lea eax, [rsi + rdi]
1 1 0.33 test al, 3
1 1 1.00 jne .LBB5_1
1 1 0.33 movabs rax, 9223372036854775804
1 1 0.33 and rax, rsi
1 1 0.33 cmp rax, 9
1 1 1.00 jae .LBB5_3
1 0 0.25 xor eax, eax
1 1 0.50 setne al
1 1 0.33 cmp rsi, 12
1 1 0.50 setb cl
1 1 0.33 or cl, al
1 1 1.00 je .LBB5_2
1 0 0.25 xor edi, edi
1 1 0.33 mov rax, rdi
1 1 1.00 U ret
1 1 0.33 add rax, -9
1 1 0.33 mov rax, rsi
1 1 0.33 or rax, 3
1 1 0.33 add rax, -12
1 1 0.33 movabs rcx, -6148914691236517205
2 4 1.00 mul rcx
1 1 0.50 shr rdx
1 1 0.50 lea rax, [rdx + 2*rdx]
1 1 0.33 sub rsi, rax
1 1 0.33 or rax, -4
1 1 0.33 add rsi, rdi
1 1 0.33 add rax, rsi
1 1 0.33 add rax, -8
1 1 0.33 and esi, 3
1 1 0.33 add rdi, rsi
1 1 0.33 mov rax, rdi
1 1 1.00 U ret
@ -53,27 +53,27 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 6.65 6.67 - 6.68 - -
- - 6.67 6.66 - 6.67 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - 0.90 0.10 - - - - lea eax, [rsi + rdi]
- - 0.93 - - 0.07 - - test al, 3
- - - - - 1.00 - - jne .LBB5_1
- - 0.51 0.47 - 0.02 - - movabs rax, 9223372036854775804
- - - - - 1.00 - - and rax, rsi
- - - 0.09 - 0.91 - - cmp rax, 9
- - - - - 1.00 - - jae .LBB5_3
- - - - - - - - xor eax, eax
- - 0.35 0.65 - - - - lea eax, [rsi + rdi]
- - 0.01 0.99 - - - - test al, 3
- - 1.00 - - - - - setne al
- - 0.97 - - 0.03 - - cmp rsi, 12
- - 0.66 - - 0.34 - - setb cl
- - - 1.00 - - - - or cl, al
- - - - - 1.00 - - je .LBB5_2
- - - - - - - - xor edi, edi
- - 0.01 0.01 - 0.98 - - mov rax, rdi
- - - - - 1.00 - - ret
- - 0.43 0.47 - 0.10 - - add rax, -9
- - 0.42 0.39 - 0.19 - - movabs rcx, -6148914691236517205
- - 0.01 0.99 - - - - mov rax, rsi
- - - 0.34 - 0.66 - - or rax, 3
- - - 1.00 - - - - add rax, -12
- - 0.66 0.31 - 0.03 - - movabs rcx, -6148914691236517205
- - 1.00 1.00 - - - - mul rcx
- - 0.69 - - 0.31 - - shr rdx
- - 0.54 0.46 - - - - lea rax, [rdx + 2*rdx]
- - 0.07 0.91 - 0.02 - - sub rsi, rax
- - 0.91 0.05 - 0.04 - - or rax, -4
- - 0.08 0.90 - 0.02 - - add rsi, rdi
- - 0.09 0.91 - - - - add rax, rsi
- - 0.08 0.92 - - - - add rax, -8
- - 1.00 - - - - - shr rdx
- - 0.67 0.01 - 0.32 - - and esi, 3
- - 0.32 0.01 - 0.67 - - add rdi, rsi
- - 0.01 0.35 - 0.64 - - mov rax, rdi
- - - - - 1.00 - - ret

36
rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64
View File

@ -1,27 +1,21 @@
bench_ref_from_suffix_with_elems_dynamic_padding:
movabs rax, 3074457345618258598
cmp rdx, rax
ja .LBB5_1
lea r8d, [rsi + rdi]
xor ecx, ecx
mov eax, 0
test r8b, 3
je .LBB5_3
mov rdx, rcx
ret
movabs rcx, 3074457345618258598
cmp rdx, rcx
ja .LBB5_3
mov rax, rdi
lea rcx, [rdx + 2*rdx]
or rcx, 3
add rcx, 9
add edi, esi
test dil, 3
setne dil
sub rsi, rcx
setb cl
or cl, dil
je .LBB5_4
.LBB5_3:
lea rax, [rdx + 2*rdx]
or rax, 3
add rax, 9
sub rsi, rax
jae .LBB5_4
.LBB5_1:
xor eax, eax
mov edx, 1
ret
.LBB5_4:
add rdi, rsi
mov rcx, rdx
mov rax, rdi
mov rdx, rcx
add rax, rsi
ret

86
rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
View File

@ -1,12 +1,12 @@
Iterations: 100
Instructions: 2300
Total Cycles: 706
Total uOps: 2300
Instructions: 1800
Total Cycles: 572
Total uOps: 1800
Dispatch Width: 4
uOps Per Cycle: 3.26
IPC: 3.26
Block RThroughput: 6.0
uOps Per Cycle: 3.15
IPC: 3.15
Block RThroughput: 4.5
Instruction Info:
@ -18,28 +18,23 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 movabs rax, 3074457345618258598
1 1 0.33 cmp rdx, rax
1 1 1.00 ja .LBB5_1
1 1 0.50 lea r8d, [rsi + rdi]
1 0 0.25 xor ecx, ecx
1 1 0.33 mov eax, 0
1 1 0.33 test r8b, 3
1 1 1.00 je .LBB5_3
1 1 0.33 mov rdx, rcx
1 1 1.00 U ret
1 1 0.50 lea rax, [rdx + 2*rdx]
1 1 0.33 or rax, 3
1 1 0.33 add rax, 9
1 1 0.33 sub rsi, rax
1 1 1.00 jae .LBB5_4
1 0 0.25 xor eax, eax
1 1 0.33 mov edx, 1
1 1 1.00 U ret
1 1 0.33 add rdi, rsi
1 1 0.33 mov rcx, rdx
1 1 0.33 movabs rcx, 3074457345618258598
1 1 0.33 cmp rdx, rcx
1 1 1.00 ja .LBB5_3
1 1 0.33 mov rax, rdi
1 1 0.33 mov rdx, rcx
1 1 0.50 lea rcx, [rdx + 2*rdx]
1 1 0.33 or rcx, 3
1 1 0.33 add rcx, 9
1 1 0.33 add edi, esi
1 1 0.33 test dil, 3
1 1 0.50 setne dil
1 1 0.33 sub rsi, rcx
1 1 0.50 setb cl
1 1 0.33 or cl, dil
1 1 1.00 je .LBB5_4
1 0 0.25 xor eax, eax
1 1 1.00 U ret
1 1 0.33 add rax, rsi
1 1 1.00 U ret
@ -56,30 +51,25 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 6.99 7.00 - 7.01 - -
- - 5.66 5.66 - 5.68 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.99 - 0.01 - - movabs rax, 3074457345618258598
- - 0.01 0.50 - 0.49 - - cmp rdx, rax
- - - - - 1.00 - - ja .LBB5_1
- - - 1.00 - - - - lea r8d, [rsi + rdi]
- - - - - - - - xor ecx, ecx
- - 0.50 0.49 - 0.01 - - mov eax, 0
- - 0.49 0.51 - - - - test r8b, 3
- - - - - 1.00 - - je .LBB5_3
- - 0.51 0.49 - - - - mov rdx, rcx
- - - - - 1.00 - - ret
- - 0.50 0.50 - - - - lea rax, [rdx + 2*rdx]
- - 1.00 - - - - - or rax, 3
- - 1.00 - - - - - add rax, 9
- - 0.99 0.01 - - - - sub rsi, rax
- - - - - 1.00 - - jae .LBB5_4
- - 0.67 - - 0.33 - - movabs rcx, 3074457345618258598
- - 0.34 - - 0.66 - - cmp rdx, rcx
- - - - - 1.00 - - ja .LBB5_3
- - 0.33 0.01 - 0.66 - - mov rax, rdi
- - 0.33 0.67 - - - - lea rcx, [rdx + 2*rdx]
- - 0.01 0.99 - - - - or rcx, 3
- - 0.01 0.99 - - - - add rcx, 9
- - 0.99 - - 0.01 - - add edi, esi
- - 0.99 0.01 - - - - test dil, 3
- - 0.99 - - 0.01 - - setne dil
- - - 1.00 - - - - sub rsi, rcx
- - 1.00 - - - - - setb cl
- - - 0.99 - 0.01 - - or cl, dil
- - - - - 1.00 - - je .LBB5_4
- - - - - - - - xor eax, eax
- - - 1.00 - - - - mov edx, 1
- - - - - 1.00 - - ret
- - 1.00 - - - - - add rdi, rsi
- - - 1.00 - - - - mov rcx, rdx
- - 0.99 0.01 - - - - mov rax, rdi
- - - 0.50 - 0.50 - - mov rdx, rcx
- - - 1.00 - - - - add rax, rsi
- - - - - 1.00 - - ret

31
rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64
View File

@ -1,23 +1,18 @@
bench_ref_from_suffix_with_elems_dynamic_size:
movabs rax, 4611686018427387901
cmp rdx, rax
ja .LBB5_1
lea r8d, [rsi + rdi]
xor ecx, ecx
mov eax, 0
test r8b, 1
jne .LBB5_5
lea rax, [2*rdx + 4]
sub rsi, rax
jae .LBB5_4
.LBB5_1:
movabs rcx, 4611686018427387901
cmp rdx, rcx
ja .LBB5_3
mov rax, rdi
lea rcx, [2*rdx + 4]
add edi, esi
sub rsi, rcx
setb cl
or cl, dil
test cl, 1
je .LBB5_4
.LBB5_3:
xor eax, eax
mov edx, 1
ret
.LBB5_4:
add rdi, rsi
mov rcx, rdx
mov rax, rdi
.LBB5_5:
mov rdx, rcx
add rax, rsi
ret

72
rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca
View File

@ -1,12 +1,12 @@
Iterations: 100
Instructions: 1900
Total Cycles: 571
Total uOps: 1900
Instructions: 1500
Total Cycles: 472
Total uOps: 1500
Dispatch Width: 4
uOps Per Cycle: 3.33
IPC: 3.33
Block RThroughput: 5.0
uOps Per Cycle: 3.18
IPC: 3.18
Block RThroughput: 4.0
Instruction Info:
@ -18,24 +18,20 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 movabs rax, 4611686018427387901
1 1 0.33 cmp rdx, rax
1 1 1.00 ja .LBB5_1
1 1 0.50 lea r8d, [rsi + rdi]
1 0 0.25 xor ecx, ecx
1 1 0.33 mov eax, 0
1 1 0.33 test r8b, 1
1 1 1.00 jne .LBB5_5
1 1 0.50 lea rax, [2*rdx + 4]
1 1 0.33 sub rsi, rax
1 1 1.00 jae .LBB5_4
1 0 0.25 xor eax, eax
1 1 0.33 mov edx, 1
1 1 1.00 U ret
1 1 0.33 add rdi, rsi
1 1 0.33 mov rcx, rdx
1 1 0.33 movabs rcx, 4611686018427387901
1 1 0.33 cmp rdx, rcx
1 1 1.00 ja .LBB5_3
1 1 0.33 mov rax, rdi
1 1 0.33 mov rdx, rcx
1 1 0.50 lea rcx, [2*rdx + 4]
1 1 0.33 add edi, esi
1 1 0.33 sub rsi, rcx
1 1 0.50 setb cl
1 1 0.33 or cl, dil
1 1 0.33 test cl, 1
1 1 1.00 je .LBB5_4
1 0 0.25 xor eax, eax
1 1 1.00 U ret
1 1 0.33 add rax, rsi
1 1 1.00 U ret
@ -52,26 +48,22 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - 5.66 5.66 - 5.68 - -
- - 4.66 4.66 - 4.68 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - 0.66 0.33 - 0.01 - - movabs rax, 4611686018427387901
- - 0.01 0.99 - - - - cmp rdx, rax
- - - - - 1.00 - - ja .LBB5_1
- - 0.99 0.01 - - - - lea r8d, [rsi + rdi]
- - - - - - - - xor ecx, ecx
- - 0.33 0.33 - 0.34 - - mov eax, 0
- - 0.33 0.34 - 0.33 - - test r8b, 1
- - - - - 1.00 - - jne .LBB5_5
- - 0.34 0.66 - - - - lea rax, [2*rdx + 4]
- - - 1.00 - - - - sub rsi, rax
- - - - - 1.00 - - jae .LBB5_4
- - 0.99 - - 0.01 - - movabs rcx, 4611686018427387901
- - 1.00 - - - - - cmp rdx, rcx
- - - - - 1.00 - - ja .LBB5_3
- - 0.63 0.02 - 0.35 - - mov rax, rdi
- - 0.35 0.65 - - - - lea rcx, [2*rdx + 4]
- - 0.65 0.03 - 0.32 - - add edi, esi
- - 0.03 0.97 - - - - sub rsi, rcx
- - 1.00 - - - - - setb cl
- - - 1.00 - - - - or cl, dil
- - 0.01 0.99 - - - - test cl, 1
- - - - - 1.00 - - je .LBB5_4
- - - - - - - - xor eax, eax
- - 1.00 - - - - - mov edx, 1
- - - - - 1.00 - - ret
- - - 1.00 - - - - add rdi, rsi
- - 1.00 - - - - - mov rcx, rdx
- - 0.32 0.68 - - - - mov rax, rdi
- - 0.68 0.32 - - - - mov rdx, rcx
- - - 1.00 - - - - add rax, rsi
- - - - - 1.00 - - ret

19
rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs Normal file
View File

@ -0,0 +1,19 @@
// Copyright 2026 The Fuchsia Authors
//
// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
// This file may not be copied, modified, or distributed except according to
// those terms.
use zerocopy::Ref;
#[path = "formats/coco_dynamic_padding.rs"]
mod format;
#[unsafe(no_mangle)]
fn bench_ref_type_bytes_dynamic_padding<'a>(
source: &'a Ref<&[u8], format::LocoPacket>,
) -> &'a [u8] {
Ref::bytes(source)
}

4
rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64 Normal file
View File

@ -0,0 +1,4 @@
bench_ref_type_bytes_dynamic_padding:
mov rax, qword ptr [rdi]
mov rdx, qword ptr [rdi + 8]
ret

Some files were not shown because too many files have changed in this diff Show More