Enforce local fence for IB_WR_REG_MR to avoid spurious
FASTREG_VALID_MKEY async events during heavy invalidation
and registration activity.
Commit 69e8e429bc ("RDMA/irdma: Enforce local fence for LOCAL_INV WRs")
was very similar, but was not sufficient to prevent all occurrences
of these async events.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260901160014.2026285-1-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
These did not seem worth sending as a dedicated rc PR during the last week
of the cycle.
* ko-rdma/for-rc:
RDMA/ipoib: Drain RCU callbacks during module teardown
RDMA/mlx5: Drain RCU callbacks during module teardown
RDMA/core: Wait for RCU callbacks before unloading ib_core
RDMA/irdma: Prevent overflows in memory contiguity checks
RDMA/siw: publish QP after initialization
RDMA/hns: Fix potential integer overflow in mhop hem cleanup
RDMA/core: Fix memory leak in __ib_create_cq() on invalid cqe
RDMA/mana_ib: initialize err for empty send WR lists
RDMA/erdma: initialize ret for empty receive WR lists
RDMA/irdma: Prevent user-triggered null deref on QP create
RDMA/irdma: Prevent rereg_mr for non-mem regions
RDMA/cma: Fix hardware address comparison length in netevent callback
RDMa/mlx5: Avoid frame overflow warning
IB/mad: Drop unmatched RMPP responses before reassembly
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Now that ib_umem_check_rereg entirely blocks rereg for dmabuf
umems, the extra logic can be removed from irdma.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260728141501.1425737-2-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The irdma driver has been audited to confirm that:
1. Methods which do not accept udata input perform an explicit
check for no (or zero value) input.
2. Methods which do accept input perform the correct validation
to ensure that additional udata beyond the kernel's current
ABI definition is zero, and to enforce the required minimum
length.
3. Methods which do not return udata responses use the proper
helper.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-7-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The irdma driver maintains backward compatibility with the
legacy i40iw userspace provider by checking the length of
the user response buffer in irdma_create_qp.
Previously, the check relied on udata->outlen < sizeof(uresp).
That is technically okay since there have only ever been two
sizes for the resp struct (legacy and current). However, it
would be a problem if the resp struct is ever expanded in
the future because it would end up triggering the legacy
fallback path for non-legacy irdma providers that just haven't
moved over to the newer expanded struct yet.
Fix this by explicitly checking for the exact legacy resp size.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-6-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Replace the manual udata input copy and validation during
QP creation with the robust helper.
The irdma driver is backwards compatible with the legacy
i40iw userspace provider. The current create_qp ABI contains
two 8 byte fields. The legacy i40iw ABI was the same but
also contained two additional fields which were never actually
used. Furthermore, the i40iw userspace provider never explicitly
zero-initialized those extra fields, so there is a chance that
existing binaries are passing non-zero garbage values down
to the kernel.
Previously, the irdma driver only copied out the first 16
bytes and did not have any check for the rest of the buffer
being zero, so that additional garbage didn't matter.
By switching to ib_copy_validate_udata_in(), we will now be
checking to ensure that data beyond the kernel's definition
of the request is all zero.
In order to avoid breaking legacy binaries, we therefore need
to increase the request structure size to cover those garbage
fields.
- Legacy binaries will continue to pass down a 32 byte request,
with the driver copying the entire 32 bytes out but ignoring
the second 16 bytes, just as before.
- Newer binaries will pass down the normal 16 byte request. The
ib_copy_validate_udata_in() call will allow this to succeed
because we use user_compl_ctx as our minimum length (16 bytes).
- If the request is ever extended, the new fields would be
added after the "don't use" fields and would work as per
the normal uAPI mechanism.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-5-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Replace the use of ib_copy_from_udata() with
ib_copy_validate_udata_in() where applicable.
For each modified call site, the last argument of
ib_copy_validate_udata_in() was determined by taking
the last member of the ABI struct as per its original
definition (i.e., when it was first committed).
Some methods like irdma_create_cq required special care
because the last member of the current ABI def is beyond
that of the legacy i40iw's ABI def which we need to
remain compatible with. In some other cases like modify_qp,
the legacy i40iw provider never provided any udata at all
so the validation is only performed if inlen > 0.
irdma_create_qp is more challenging because the legacy ABI
was actually larger but the additional fields were never used,
and even worse, never initialized in the provider. This will
be handled in a followup commit.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-4-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Methods that may accept udata input but do not provide a
udata response should use the ib_respond_empty_udata()
helper to ensure that user response buffers are cleared.
Since the ib_respond_empty_udata() call itself can fail if
the user intentionally provides a bogus output buffer, it is
called at the beginning of the method to fail early before
mutating any state that would be difficult to unwind.
Additionally, add missing bounds validation for udata->outlen
in irdma_create_srq() to ensure it is large enough to hold the
response struct as per its original (and so far, only) definition.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-3-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Several methods do not accept udata input and do not provide
a udata response. Use the ib_no_udata_io helper to check that
the input buffers are empty and to zero fill any user response
buffers. For methods that do provide a response, enforce the
input buffer is empty using ib_is_udata_in_empty.
The irdma rdma-core provider as well as the legacy i40iw
provider were both checked to ensure they never passed any
udata to these ops.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-2-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Previously, the user QP creation path would only attempt to
populate iwqp->iwpbl if the user-provided req.user_wqe_bufs
field was non-zero. The problem is that iwqp->iwpbl is
unconditionally dereferenced later on in irdma_setup_virt_qp.
While there was a check for iwqp->iwpbl != NULL, this check
would only occur if req.user_wqe_bufs was non-zero. The end
result is that a user could send a zero user_wqe_bufs value
and trigger a null ptr deref.
Fix this by unconditionally calling irdma_get_pbl and bailing
if it fails, similar to the CQ and SRQ paths.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The query_device callbacks that neither accept driver-specific input nor
return a driver-specific response open-code the empty udata handling as
ib_is_udata_in_empty() on entry and ib_respond_empty_udata() on exit.
ib_no_udata_io() already combines both steps, so replace the entry check
with it and simply return 0 on success.
Unlike the create and destroy flows, query_device owns no uobject or HW
resource - the extended path fills a stack ib_device_attr that the core
discards on error - so clearing the empty response buffer on entry rather
than on exit is a mechanical change with no functional difference.
Link: https://patch.msgid.link/20260714-convert-to-noio-udata-v1-1-f1f6b6c7c988@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
The core always hands the driver's query_device() callback a zeroed
ib_device_attr. There are only two callers of the op and both clear the
structure before invoking it: setup_device() memsets &device->attrs, and
ib_uverbs_ex_query_device() passes an on-stack structure initialized to {}.
The open-coded memset(props, 0, sizeof(*props)) at the top of the driver
callbacks is therefore redundant. Remove it from all drivers.
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
irdma_check_mem_contiguous() and irdma_check_mr_contiguous() verify that
PBL entries describe physically contiguous memory ranges.
Both functions calculate byte offsets using 32-bit operands. For example,
with 4 KiB pages, pg_size * pg_idx overflows 32-bit arithmetic when
pg_idx reaches 1048576. In the level-2 check, PBLE_PER_PAGE is 512, so
i * pg_size * PBLE_PER_PAGE overflows when i reaches 2048.
These values are reachable in the driver. For MRs, palloc->total_cnt
comes from iwmr->page_cnt, which is calculated by
ib_umem_num_dma_blocks(). The MR size is limited by IRDMA_MAX_MR_SIZE,
so a 4 GiB MR with 4 KiB pages can reach page_cnt of 1048576. PBLE
resources do not exclude this value either: for gen3, the limit is based
on avail_sds * MAX_PBLE_PER_SD, and MAX_PBLE_PER_SD is 0x40000, so 4 SDs
are enough for 1048576 PBLEs.
Cast one operand to u64 before the multiplications so that the offset
calculations are performed in 64-bit arithmetic.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Aleksandrova Alyona <aga@itb.spb.ru>
Link: https://patch.msgid.link/20260624144846.61242-1-aga@itb.spb.ru
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The irdma driver currently issues an unconditional PF reset whenever the
HMC Error interrupt (PFINT_OICR bit 26) fires:
if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) {
ibdev_err(&iwdev->ibdev, "HMC Error\n");
iwdev->rf->reset = true;
}
request_reset() issues an IIDC_PFR to ice. In practice a single HMC_ERR
can trigger cascading PF resets, IOMMU faults during teardown, and
teardown of every RDMA connection on the device.
i40e handles the identically-named interrupt by reading
PFHMC_ERRORINFO and PFHMC_ERRORDATA and logging them without touching
device state; see commit 9c010ee0ea ("i40e: Suppress HMC error to
Interrupt message level") which removed the reset as "not necessary".
This patch mirrors that handling on irdma.
With this change, repeated HMC_ERR no longer produces a reset storm and
RDMA traffic on the device continues uninterrupted.
Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com>
Link: https://patch.msgid.link/20260619050044.1807044-1-seyeong.kim@canonical.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Prevent userspace from deregistering the MRs that back QP/CQ/SRQ rings
by bumping the MR's refcount upon association.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260618201458.875740-5-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
User QP/CQ/SRQ rings are registered with the normal reg_mr
mechanism prior to creating the actual QP/CQ/SRQ object. In
order to prevent userspace from deregistering these special MRs
while the child object still exists, a refcount will be used.
This commit adds the refcount and logic to reject a dereg_mr
with active references. Subsequent commits will add logic to
bump this refcount when the user QP/CQ/SRQ objects are created.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260618201458.875740-3-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Simplify/dedup the irdma_del_memlist logic in preparation for
the QP/CQ/SRQ ring MR refcounting change that will follow in
a subsequent commit.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260618201458.875740-2-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Previously, the user QP creation path would only attempt to
populate iwqp->iwpbl if the user-provided req.user_wqe_bufs
field was non-zero. The problem is that iwqp->iwpbl is
unconditionally dereferenced later on in irdma_setup_virt_qp.
While there was a check for iwqp->iwpbl != NULL, this check
would only occur if req.user_wqe_bufs was non-zero. The end
result is that a user could send a zero user_wqe_bufs value
and trigger a null ptr deref.
Fix this by unconditionally calling irdma_get_pbl and bailing
if it fails, similar to the CQ and SRQ paths.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Link: https://patch.msgid.link/r/20260617164013.280790-1-jmoroni@google.com
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Reviewed-by: David Hu <xuehaohu@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
When a QP/CQ/SRQ is created, a two step process is used
where the buffer is allocated in userspace and explicitly
registered with the normal reg_mr mechanism prior to creating
the actual QP/CQ/SRQ object.
These special registrations are indicated via an ABI field
so the driver knows that they do not have a valid mkey and
to skip the actual CQP command submission.
Since these are real MR objects from the core's perspective,
it is possible for a user application to invoke rereg_mr on them
and cause a real CQP op to be emitted with the zero-initialized
mkey value of 0.
Fix this by preventing rereg_mr on these special regions.
Fixes: 5ac388db27 ("RDMA/irdma: Add support to re-register a memory region")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Reviewed-by: David Hu <xuehaohu@google.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Many AI driven bug fixes, and several big driver API cleanups
- Driver bug fixes and minor cleanups in mlx5, hns, rxe, efa, siw, rtrs,
mana, irdma, mlx4. Commonly error path flows, integer arithmetic
overflows on unsafe data, out of bounds access, and use after free
issues under races.
- Second half of the new udata API for drivers focusing on uAPI response
- bnxt_re supports more options for QP creation that will allow a dv
path in rdma-core
- Untangle the module dependencies so drivers don't link to ib_uverbs.ko
as was originall intended
- Provide a new way to handle umems with a consistent simplified uAPI and
update several drivers to use it. This brings dmabuf support to more
places and more drivers
- Support for mlx5 rate limit and packet pacing for UD and UC
- A batch of fixes for the new shared FRMR pools infrastructure
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRRRCHOFoQz/8F5bUaFwuHvBreFYQUCajLevAAKCRCFwuHvBreF
YVFuAP0X6gxaON+6cRGoaHp+KjxNj3zbuRj9VccTOy1OjerUdwD+LsbK/Pa9xrcO
ByA3LVvbw4BoQw9qcQpTu2klFuYV5g8=
=qnpN
-----END PGP SIGNATURE-----
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma
Pull rdma updates from Jason Gunthorpe:
"Many AI driven bug fixes, and several big driver API cleanups
- Driver bug fixes and minor cleanups in mlx5, hns, rxe, efa, siw,
rtrs, mana, irdma, mlx4. Commonly error path flows, integer
arithmetic overflows on unsafe data, out of bounds access, and use
after free issues under races.
- Second half of the new udata API for drivers focusing on uAPI
response
- bnxt_re supports more options for QP creation that will allow a dv
path in rdma-core
- Untangle the module dependencies so drivers don't link to
ib_uverbs.ko as was originall intended
- Provide a new way to handle umems with a consistent simplified uAPI
and update several drivers to use it. This brings dmabuf support to
more places and more drivers
- Support for mlx5 rate limit and packet pacing for UD and UC
- A batch of fixes for the new shared FRMR pools infrastructure"
* tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma: (148 commits)
RDMA/irdma: Replace waitqueue and flag with completion
RDMA/hns: Fix memory leak of bonding resources
RDMA/rtrs-srv: Bound RDMA-Write length to chunk size in rdma_write_sg
docs: infiniband: correct name of option to enable the ib_uverbs module
RDMA/bnxt_re: Reject GET_TOGGLE_MEM when toggle page was not allocated
RDMA/bnxt_re: Fail DBR related page allocation UAPIs if the feature is disabled
RDMA/bnxt_re: Avoid repeated requests to allocate WC pages
RDMA/bnxt_re: Proper rollback if the ioremap fails
RDMA/bnxt_re: Add a max slot check for SQ
RDMA/bnxt_re: Avoid displaying the kernel pointer
RDMA/bnxt_re: Free CQ toggle page after firmware teardown
RDMA/bnxt_re: Free SRQ toggle page after firmware teardown
RDMA/bnxt_re: Initialize dpi variable to zero
ABI: sysfs-class-infiniband: minor cleanup
RDMA/mlx5: Release the HW‑provided UAR index rather than the SW one
RDMA/mlx5: Fix undefined shift of user RQ WQE size
RDMA/mlx5: Remove raw RSS QP restrack tracking
RDMA/mlx5: Remove DCT restrack tracking
RDMA/mlx5: Drop FRMR pool handle on UMR revoke failure
RDMA/core: Add ib_frmr_pool_drop for unrecoverable handles
...
The driver previously used a waitqueue along with an explicit
request_done flag, but without proper barriers around request_done.
An earlier patch by Gui-Dong Han <hanguidong02@gmail.com> attempted
to fix this by adding the missing memory barriers. Rather than
adding the barriers, this patch replaces the waitqueue+flag with
a completion, which is designed for this exact purpose.
Fixes: 44d9e52977 ("RDMA/irdma: Implement device initialization definitions")
Fixes: 915cc7ac0f ("RDMA/irdma: Add miscellaneous utility definitions")
Link: https://patch.msgid.link/r/20260616155601.1081448-1-jmoroni@google.com
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
If IB_MR_REREG_ACCESS changes from RO to RW then the umem has to be
re-evaluated to ensure it is properly pinned as RW. Since the umem is
hidden inside each driver's mr struct add a ib_umem_check_rereg() function
that each driver has to call before processing IB_MR_REREG_ACCESS.
mlx4 has to retain its duplicate ib_access_writable check because it
implements IB_MR_REREG_ACCESS | IB_MR_REREG_TRANS by changing both items
in place sequentially while the MR is live, so it will continue to not
support this combination.
Cc: stable@vger.kernel.org
Fixes: b40656aa7d ("RDMA/umem: remove FOLL_FORCE usage")
Link: https://patch.msgid.link/r/0-v1-06fb1a2d6cf5+107-rereg_access_jgg@nvidia.com
Reported-by: Philip Tsukerman <philiptsukerman@gmail.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Initialize iwmr->access during initial user mem registration so
that it contains a valid value during a subsequent rereg_mr.
Otherwise, a rereg_mr that doesn't set IB_MR_REREG_ACCESS (for
example, one that only changes the PD) ends up clearing the
access flags in HW since iwmr->access is zero-initialized, which
is not intended.
Fixes: 5ac388db27 ("RDMA/irdma: Add support to re-register a memory region")
Link: https://patch.msgid.link/r/20260604154104.4035581-1-jmoroni@google.com
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Sashiko pointed out an unrelated bug during a previous patch:
https://sashiko.dev/#/patchset/20260512183852.614045-1-jmoroni%40google.com
This change fixes the bug by eliminating the cqmr->split field which
was not being set properly and instead just checks the CQ resize
feature flag directly.
The cqmr->split field essentially tracks whether IRDMA_FEATURE_CQ_RESIZE
is set, but it was not being set until CQ creation time, which is _after_
CQ memory registration (the only other place where it is referenced).
As a result, it would always be false during MR registration and would
therefore cause irdma_handle_q_mem to populate cqmr->shadow even for GEN_2
HW and beyond:
cqmr->shadow = (dma_addr_t)arr[req->cq_pages];
The issue is that for GEN_2 and beyond, req->cq_pages may be exactly equal
to iwmr->page_cnt and therefore equal to the size of arr, which would cause
an OOB read by one.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Link: https://patch.msgid.link/r/20260602214423.1315105-2-jmoroni@google.com
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
The driver has the following invariants:
1. legacy_mode is only allowed on GEN_1 hardware (enforced
in irdma_alloc_ucontext).
2. GEN_1 hardware does not set IRDMA_FEATURE_CQ_RESIZE or
IRDMA_FEATURE_RTS_AE. These feature flags are only set
for GEN_2 and GEN_3 hardware.
Therefore, legacy_mode is always false if IRDMA_FEATURE_CQ_RESIZE
or IRDMA_FEATURE_RTS_AE is set, so remove the redundant checks.
Link: https://patch.msgid.link/r/20260602214423.1315105-1-jmoroni@google.com
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
This op hasn't followed the normal pattern of passing NULL for udata when
invoked by the kernel. Instead the kernel caller creates a dummy ib_udata
on the stack and passes that in. It does not seem to currently be a bug,
but this flow should be modernized to use the new API flow and in the
process accept NULL as well.
Only mlx4 uses an input request structure, have every other driver call
ib_is_udata_in_empty() to enforce the lack of request structs.
Use ib_respond_empty_udata() in every driver that does not use a response
struct.
Ensure a check for NULL udata before calling ib_respond_udata() in
bnxt_re, efa, and mlx5.
Make mlx4 safe to be called with NULL.
Link: https://patch.msgid.link/r/2-v1-922fa8e828ba+f7-ib_udata_stack_jgg@nvidia.com
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
When we generate completion for SQ the opcode while being properly read
from ring buffer is ignored when written back to completion. Seems
to be a simple typo.
Link: https://patch.msgid.link/r/ahjB87k54bYdFbft@grain
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
The new umem getter family being introduced in follow-up patches
need a fitting name for the central all-source helper that resolves
attributes, legacy fillers and a UHW VA fallback.
Rename the existing VA-pinning helper ib_umem_get() to ib_umem_get_va()
so the name is freed up. The new name is consistent with names of rest
of the helpers that are about to be introduced.
Link: https://patch.msgid.link/r/20260529134312.2836341-2-jiri@resnulli.us
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
The irdma_copy_user_pgaddrs function loops through all of the umem DMA
blocks to populate the PBLEs and will stop when either the last DMA
block is reached or palloc->total_cnt is reached. The issue is that
the logic for checking palloc->total_cnt would only work for non-zero
values.
When irdma_setup_pbles is called with lvl==0, it
calls irdma_copy_user_pgaddrs with palloc->total_cnt==0, which means
the only way to break out of the loop is to reach the last umem DMA
block, which means it could end up going beyond the fixed size of 4
iwmr->pgaddrmem array that is used in the lvl==0 case.
In the case of QP/CQ/SRQ rings, the value of lvl is determined by a
separate input (for example, req.cq_pages in the case of a CQ). So,
we must perform explicit checking to ensure we don't overflow the
pgaddrmem array if the user provides a umem that consists of more
blocks than their provided req.cq_pages.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Link: https://patch.msgid.link/r/20260512183852.614045-1-jmoroni@google.com
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
irdma has a comp_mask field that was never checked for validity, check
it.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Go treewide and consolidate all existing patterns using:
* offsetofend() and variations
* ib_is_udata_cleared()
* ib_copy_from_udata()
into a direct call to the new ib_copy_validate_udata_in().
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Instead of checking whether the number of CQEs is negative or zero, fix the
.resize_user_cq() declaration to use unsigned int. This better reflects the
expected value range. The sanity check is then handled correctly in ib_uvbers.
Link: https://patch.msgid.link/20260319-resize_cq-cqe-v1-1-b78c6efc1def@nvidia.com
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
GEN4 hardware is similar to GEN3 and requires only a few special cases.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
For GEN3 and higher, FW requires scratch buffers for bookkeeping
during cleanup, specifically during QP and MR destroy ops.
Signed-off-by: Jay Bhat <jay.bhat@intel.com>
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
An issue was exposed where OS can pass in U32_MAX for SQ/RQ/SRQ size.
This can cause integer overflow and truncation of SQ/RQ/SRQ depth
returning a success when it should have failed.
Harden the functions to do all depth calculations and boundary
checking in u64 sizes.
Fixes: 563e1feb5f ("RDMA/irdma: Add SRQ support")
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When rdma_connect() fails due to an invalid arp index, user space rdma core
reports ENOMEM which is confusing. Modify irdma_make_cm_node() to return the
correct error code.
Fixes: 146b9756f1 ("RDMA/irdma: Add connection manager")
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Resolve deadlock that occurs when user executes netdev reset while RDMA
applications (e.g., rping) are active. The netdev reset causes ice
driver to remove irdma auxiliary driver, triggering device_delete and
subsequent client removal. During client removal, uverbs_client waits
for QP reference count to reach zero while cma_client holds the final
reference, creating circular dependency and indefinite wait in iWARP
mode. Skip QP reference count wait during device reset to prevent
deadlock.
Fixes: c8f304d75f ("RDMA/irdma: Prevent QP use after free")
Signed-off-by: Anil Samal <anil.samal@intel.com>
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
During reset, irdma_modify_qp() to error should be called to disconnect
the QP. Without this fix, if not preceded by irdma_modify_qp() to error, the
API call irdma_destroy_qp() gets stuck waiting for the QP refcount to go
to zero, because the cm_node associated with this QP isn't disconnected.
Fixes: 915cc7ac0f ("RDMA/irdma: Add miscellaneous utility definitions")
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The cm_node is available and the usage of cm_node and event->cm_node
seems arbitrary. Clean up unnecessary dereference of event->cm_node.
Fixes: 146b9756f1 ("RDMA/irdma: Add connection manager")
Signed-off-by: Ivan Barrera <ivan.d.barrera@intel.com>
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Remove a NOP wait_event() in irdma_modify_qp_roce() which is relevant
for iWARP and likely a copy and paste artifact for RoCEv2. The wait event
is for sending a reset on a TCP connection, after the reset has been
requested in irdma_modify_qp(), which occurs only in iWarp mode.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
In irdma_modify_qp() update ibqp state to error if the irdma QP is already
in error state, otherwise the ibqp state which is visible to the consumer
app remains stale.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
In irdma_create_qp, if ib_copy_to_udata fails, it will call
irdma_destroy_qp to clean up which will attempt to wait on
the free_qp completion, which is not initialized yet. Fix this
by initializing the completion before the ib_copy_to_udata call.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Use the new API to support importing pinned dmabufs from exporters
that require revocation, such as VFIO. The revoke semantic is
achieved by issuing a HW invalidation command but not freeing
the key. This prevents further accesses to the region (they will
result in an invalid key AE), but also keeps the key reserved
until the region is actually deregistered (i.e., ibv_dereg_mr)
so that a new MR registration cannot acquire the same key.
Tested with lockdep+kasan and a memfd backed dmabuf.
The rereg_mr path is explicitly blocked in libibverbs for dmabuf MRs
(more specifically, any MR not of type IBV_MR_TYPE_MR), so the rereg_mr
path for dmabufs was tested with a modified libibverbs.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260305170826.3803155-6-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
This series adds support for Transaction Layer Packet (TLP) emulation
response gateway regions, enabling userspace device emulation software
to write TLP responses directly to lower layers without kernel driver
involvement.
Currently, the mlx5 driver exposes VirtIO emulation access regions via
the MLX5_IB_METHOD_VAR_OBJ_ALLOC ioctl. This series extends that
ioctl to also support allocating TLP response gateway channels for
PCI device emulation use cases.
Signed-off-by: Leon Romanovsky <leon@kernel.org>
If IB_MR_REREG_TRANS is set during rereg_user_mr, the
umem will be released and a new one will be allocated
in irdma_rereg_mr_trans. If any step of irdma_rereg_mr_trans
fails after the new umem is allocated, it releases the umem,
but does not set iwmr->region to NULL. The problem is that
this failure is propagated to the user, who will then call
ibv_dereg_mr (as they should). Then, the dereg_mr path will
see a non-NULL umem and attempt to call ib_umem_release again.
Fix this by setting iwmr->region to NULL after ib_umem_release.
Fixed: 5ac388db27 ("RDMA/irdma: Add support to re-register a memory region")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260227152743.1183388-1-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Commits bf4afc53b7 ("Convert 'alloc_obj' family to use the new default
GFP_KERNEL argument") and 69050f8d6d ("treewide: Replace kmalloc with
kmalloc_obj for non-scalar types") updated various k[z|m|c]alloc calls to their
k[z|m]alloc_obj counterparts.
This commit finalizes that transition within the RDMA subsystem.
Link: https://patch.msgid.link/20260226-complete-alloc-conversion-v1-1-ebf1df1c2518@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
