Each CEQ interrupt handler only schedules eqc->tasklet. The tasklet calls
erdma_ceq_completion_handler(), which reads the DMA-coherent EQ ring
through get_next_valid_eqe() and updates eq->dbrec through notify_eq().
erdma_ceqs_uninit() frees each CEQ IRQ and then destroys its EQ.
free_irq() prevents another hard IRQ and waits for an in-flight handler,
but it does not drain a tasklet that the handler already scheduled. The
tasklet can therefore access eq->qbuf or eq->dbrec after
erdma_eq_destroy() frees them.
Clearing ceq_cb->ready does not synchronize with a tasklet that already
passed the check at the start of erdma_ceq_completion_handler().
Kill the tasklet after free_irq(), when no handler can schedule it again,
and before erdma_ceq_uninit_one() releases the EQ buffers.
Fixes: f2a0a630b9 ("RDMA/erdma: Add event queue implementation")
Co-developed-by: Ijae Kim <ae878000@gmail.com>
Signed-off-by: Ijae Kim <ae878000@gmail.com>
Signed-off-by: Myeonghun Pak <mhun512@gmail.com>
Link: https://patch.msgid.link/20260721082545.47395-1-mhun512@gmail.com
Acked-by: Cheng Xu <chengyou@linux.alibaba.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
srpt_handle_cmd() maps the initiator-supplied srp_cmd->task_attr into
cmd->sam_task_attr, but then hands a hardcoded TCM_SIMPLE_TAG to
target_init_cmd().
Pass the already mapped cmd->sam_task_attr instead, so target core sees the
attribute the initiator requested.
Fixes: 9474b04313 ("ib_srpt: Convert I/O path to target_submit_cmd + drop legacy ioctx->kref")
Link: https://patch.msgid.link/20260721-b4-scsi-ordering-violation-due-to-hardc-v1-1-07205aab71bb@nvidia.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
RDMA_NLDEV_ATTR_STAT_HWCOUNTERS is a nested attribute whose children are
consumed directly with nla_get_u32(). The top-level policy validates only
the container, so it does not establish the fixed shape of each child.
Require every child payload to be exactly one u32 before reading it.
Fixes: 3c3c1f1416 ("RDMA/nldev: Allow optional-counter status configuration through RDMA netlink")
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Link: https://patch.msgid.link/20260720114918.70323-1-pengpeng@iscas.ac.cn
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Enable the robust udata contract by setting uverbs_robust_udata and
adding proper input validation and output handling to all verbs that
accept struct ib_udata.
For verbs with no driver request or response struct, add
ib_no_udata_io(). For create_ah, which already responds with
ionic_ah_resp, add the missing input validation via
ib_is_udata_in_empty().
Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
Link: https://patch.msgid.link/20260717092345.2533564-1-abhijit.gangurde@amd.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Enable the uverbs robust udata interface in mana_ib by setting
uverbs_robust_udata and converting the driver to the new udata
handling model.
Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
mlx5r_umr_alloc_xlt() allocates physically contiguous scratch buffers
that are DMA mapped only in the DMA_TO_DEVICE direction. kmalloc()
provides the required contiguity and alignment for these sizes while
preserving the existing GFP allocation policy.
The emergency translation buffer has the same requirements. Convert all
of these UMR buffers to kmalloc() and release them with kfree(), which no
longer requires the caller to supply the allocation order.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-4-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
usnic_uiom_get_pages() uses a page-sized array of struct page pointers
as temporary storage for pin_user_pages(). Nothing requires the array to
come directly from the page allocator.
Use kmalloc() for the array and kfree() after the pinning loop.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-3-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
mlx4_alloc_priv_pages() allocates a zeroed, page-sized buffer for a
DMA-to-device page list. kmalloc() provides the required physical
contiguity, and a PAGE_SIZE allocation retains the alignment needed to
keep the list within one page.
Use kzalloc() for the buffer and kfree() on the error and teardown paths.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-2-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
c4iw_reg_user_mr() allocates a page-sized temporary array of DMA
addresses while programming a PBL. The array has no page-specific
requirements, so allocate it with kmalloc() and release it with kfree().
This avoids the casts required by the page allocator and lets the free
operation derive the allocation size from the object.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-1-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
The irdma driver has been audited to confirm that:
1. Methods which do not accept udata input perform an explicit
check for no (or zero value) input.
2. Methods which do accept input perform the correct validation
to ensure that additional udata beyond the kernel's current
ABI definition is zero, and to enforce the required minimum
length.
3. Methods which do not return udata responses use the proper
helper.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-7-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The irdma driver maintains backward compatibility with the
legacy i40iw userspace provider by checking the length of
the user response buffer in irdma_create_qp.
Previously, the check relied on udata->outlen < sizeof(uresp).
That is technically okay since there have only ever been two
sizes for the resp struct (legacy and current). However, it
would be a problem if the resp struct is ever expanded in
the future because it would end up triggering the legacy
fallback path for non-legacy irdma providers that just haven't
moved over to the newer expanded struct yet.
Fix this by explicitly checking for the exact legacy resp size.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-6-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Replace the manual udata input copy and validation during
QP creation with the robust helper.
The irdma driver is backwards compatible with the legacy
i40iw userspace provider. The current create_qp ABI contains
two 8 byte fields. The legacy i40iw ABI was the same but
also contained two additional fields which were never actually
used. Furthermore, the i40iw userspace provider never explicitly
zero-initialized those extra fields, so there is a chance that
existing binaries are passing non-zero garbage values down
to the kernel.
Previously, the irdma driver only copied out the first 16
bytes and did not have any check for the rest of the buffer
being zero, so that additional garbage didn't matter.
By switching to ib_copy_validate_udata_in(), we will now be
checking to ensure that data beyond the kernel's definition
of the request is all zero.
In order to avoid breaking legacy binaries, we therefore need
to increase the request structure size to cover those garbage
fields.
- Legacy binaries will continue to pass down a 32 byte request,
with the driver copying the entire 32 bytes out but ignoring
the second 16 bytes, just as before.
- Newer binaries will pass down the normal 16 byte request. The
ib_copy_validate_udata_in() call will allow this to succeed
because we use user_compl_ctx as our minimum length (16 bytes).
- If the request is ever extended, the new fields would be
added after the "don't use" fields and would work as per
the normal uAPI mechanism.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-5-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Replace the use of ib_copy_from_udata() with
ib_copy_validate_udata_in() where applicable.
For each modified call site, the last argument of
ib_copy_validate_udata_in() was determined by taking
the last member of the ABI struct as per its original
definition (i.e., when it was first committed).
Some methods like irdma_create_cq required special care
because the last member of the current ABI def is beyond
that of the legacy i40iw's ABI def which we need to
remain compatible with. In some other cases like modify_qp,
the legacy i40iw provider never provided any udata at all
so the validation is only performed if inlen > 0.
irdma_create_qp is more challenging because the legacy ABI
was actually larger but the additional fields were never used,
and even worse, never initialized in the provider. This will
be handled in a followup commit.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-4-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Methods that may accept udata input but do not provide a
udata response should use the ib_respond_empty_udata()
helper to ensure that user response buffers are cleared.
Since the ib_respond_empty_udata() call itself can fail if
the user intentionally provides a bogus output buffer, it is
called at the beginning of the method to fail early before
mutating any state that would be difficult to unwind.
Additionally, add missing bounds validation for udata->outlen
in irdma_create_srq() to ensure it is large enough to hold the
response struct as per its original (and so far, only) definition.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-3-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Several methods do not accept udata input and do not provide
a udata response. Use the ib_no_udata_io helper to check that
the input buffers are empty and to zero fill any user response
buffers. For methods that do provide a response, enforce the
input buffer is empty using ib_is_udata_in_empty.
The irdma rdma-core provider as well as the legacy i40iw
provider were both checked to ensure they never passed any
udata to these ops.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-2-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Previously, the user QP creation path would only attempt to
populate iwqp->iwpbl if the user-provided req.user_wqe_bufs
field was non-zero. The problem is that iwqp->iwpbl is
unconditionally dereferenced later on in irdma_setup_virt_qp.
While there was a check for iwqp->iwpbl != NULL, this check
would only occur if req.user_wqe_bufs was non-zero. The end
result is that a user could send a zero user_wqe_bufs value
and trigger a null ptr deref.
Fix this by unconditionally calling irdma_get_pbl and bailing
if it fails, similar to the CQ and SRQ paths.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When srpt_alloc_rw_ctxs() fails partway through a multi-buffer indirect
descriptor, the unwind path destroys RDMA contexts but leaves stale
n_rw_ctx and n_rdma values (and a dangling rw_ctxs pointer). Later
sq_wr_avail accounting in srpt_queue_response() or srpt_write_pending()
can then subtract the wrong number of send queue credits.
Reset the counters and clear rw_ctxs after freeing the heap
allocation before returning an error.
Fixes: b99f8e4d7b ("IB/srpt: convert to the generic RDMA READ/WRITE API")
Signed-off-by: TanZheng <tanzheng@kylinos.cn>
Link: https://patch.msgid.link/20260715101550.45345-1-kensanya@163.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The destroy callbacks currently zero the udata output after tearing down
driver resources. If the userspace access fails, uverbs preserves the
uobject and allows the destroy callback to run again, even though the
driver resource has already been freed.
Call ib_no_udata_io() before teardown so udata failures are detected
while the resource is still intact, then return success after teardown
completes.
As part of this change, move ib_respond_empty_udata() to the start of
the create and modify flows. While this is not strictly required for
general create flows, as the core layer unwinds uobjects on failure, it
is necessary for create AH. In _rdma_create_ah(), the HW object is
otherwise leaked.
Fixes: bed686d8dc ("RDMA/bnxt_re: Use ib_respond_empty_udata()")
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Link: https://patch.msgid.link/20260714-fix-destroy-no-udata-v2-1-734fdcf667d5@kernel.org
Acked-by: Selvin Xavier <selvin.xavier@broadcom.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
mlx4_ib_multiplex_cm_handler() allocates an id_map_entry for CM
transactions, but the entry is normally released only on DREQ or REJ
flows.
In the duplicate REP handling scenario, cm_dup_rep_handler() may be
invoked when the remote side receives a REP for which no matching
cm_id_priv exists. In such cases the CM handshake never reaches RTU, and
the sender side may never receive either DREQ or REJ cleanup events.
As a result, the allocated id_map_entry remains indefinitely, resulting in
a stale mapping leak.
Fix this by arming an RTU-abandon cleanup timeout when the id_map_entry is
allocated. The timeout uses the mlx4 CM workqueue and the existing
schedule_delayed() path, so later DREQ/REJ cleanup can shorten the pending
timeout with mod_delayed_work().
Track whether a pending cleanup timeout is still waiting for RTU. RTU
cancels only that initial timeout; if DREQ/REJ has already converted it to
normal teardown cleanup, a late or duplicate RTU does not cancel the
teardown timer. If the RTU timeout callback has already started, leave the
entry on the timeout path and make the RTU packet lose that race.
Hold id_map_lock while looking up the entry, canceling the RTU timeout,
scheduling teardown cleanup, and copying the id values needed by the CM
handlers. The delayed-work callback rechecks scheduled_delete under the
same lock before removing and freeing the entry, avoiding use-after-free
when RTU races with timeout execution.
Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
Link: https://patch.msgid.link/20260715080738.1357072-1-praveen.kannoju@oracle.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The query_device callbacks that neither accept driver-specific input nor
return a driver-specific response open-code the empty udata handling as
ib_is_udata_in_empty() on entry and ib_respond_empty_udata() on exit.
ib_no_udata_io() already combines both steps, so replace the entry check
with it and simply return 0 on success.
Unlike the create and destroy flows, query_device owns no uobject or HW
resource - the extended path fills a stack ib_device_attr that the core
discards on error - so clearing the empty response buffer on entry rather
than on exit is a mechanical change with no functional difference.
Link: https://patch.msgid.link/20260714-convert-to-noio-udata-v1-1-f1f6b6c7c988@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
For a user QP, qp->sq.queue is a ring the application writes directly,
so rxe_post_send() takes the is_user branch and only schedules send_task
without validating the WQE. rxe_requester() consumes it in place via
req_next_wqe() and calls copy_data(), which indexes
&wqe->dma.sge[cur_sge] with the attacker-controlled num_sge/cur_sge.
Only the kernel path bounds num_sge (validate_send_wr()); the user WQE
is never checked, so a local unprivileged user can post a WQE with an
out-of-range cur_sge or oversized num_sge and force an out-of-bounds
read of the per-WQE sge array in copy_data() (vmalloc OOB read, local
DoS).
Bound num_sge to qp->sq.max_sge in rxe_requester() before use, the way
get_srq_wqe() already guards SRQ entries, and bound cur_sge only when
the WQE carries payload (dma.resid): copy_data() returns early on a
zero-length copy before touching dma->sge[], so a zero-payload WQE --
the only kind a max_sge == 0 QP can post -- stays valid.
Reproduced under KASAN; the vmalloc-out-of-bounds in copy_data() is gone.
Fixes: 8700e3e7c4 ("Soft RoCE driver")
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Signed-off-by: Ibrahim Hashimov <security@auditcode.ai>
Link: https://patch.msgid.link/20260712122149.78142-1-security@auditcode.ai
Assisted-by: AuditCode-AI:2026.07
Signed-off-by: Leon Romanovsky <leon@kernel.org>
rxe_qp_from_attr() handles IB_QP_MAX_DEST_RD_ATOMIC outside the
IB_QP_STATE path, so it holds no state_lock and runs while the responder
task rxe_receiver() (recv_task on rxe_wq) is live. A modify_qp() setting
only that attribute calls free_rd_atomic_resources() then
alloc_rd_atomic_resources(), swapping qp->resp.resources[] while
rxe_prepare_res()/find_resource() walk it; free_rd_atomic_resources()
also leaves the cached pointer qp->resp.res dangling. A local
unprivileged user can race the free/realloc into a use-after-free in
rxe_receiver() (local DoS).
Drain recv_task around the swap with rxe_disable_task()/rxe_enable_task(),
as rxe_qp_reset() already does when tearing this array down, re-enabling
only after alloc_rd_atomic_resources() succeeds so the responder never
resumes against a NULL qp->resp.resources on the ENOMEM path. Also clear
qp->resp.res in free_rd_atomic_resources(), like the rxe_resp.c
completion paths.
Reproduced under KASAN; the slab-use-after-free in rxe_receiver() is gone.
Fixes: 8700e3e7c4 ("Soft RoCE driver")
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Signed-off-by: Ibrahim Hashimov <security@auditcode.ai>
Link: https://patch.msgid.link/20260712121720.78001-1-security@auditcode.ai
Assisted-by: AuditCode-AI:2026.07
Signed-off-by: Leon Romanovsky <leon@kernel.org>
set_txreq_header_ahg() ignores the return value of sdma_txinit_ahg().
If sdma_txinit_ahg() fails, it returns before initializing tx->txreq.
However, set_txreq_header_ahg() ignores the error and returns the AHG
change count, causing the caller to continue processing the request as
though initialization had succeeded.
Propagate sdma_txinit_ahg() failures to the caller and abort request
processing when initialization fails.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Fixes: e3304b7cc4 ("IB/hfi1: Optimize cachelines for user SDMA request structure")
Signed-off-by: Danila Chernetsov <listdansp@mail.ru>
Link: https://patch.msgid.link/20260708162252.936634-1-listdansp@mail.ru
Signed-off-by: Leon Romanovsky <leon@kernel.org>
'struct dma_buf_attach_ops' are not modified in this driver.
Constifying these structures moves some data to a read-only section, so
increases overall security, especially when the structure holds some
function pointers.
On a x86_64, with allmodconfig, as an example:
Before:
======
text data bss dec hex filename
10300 1216 0 11516 2cfc drivers/infiniband/core/umem_dmabuf.o
After:
=====
text data bss dec hex filename
10428 1088 0 11516 2cfc drivers/infiniband/core/umem_dmabuf.o
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Link: https://patch.msgid.link/3ca4ace543a02ccfdcce1ba568895c994aad7abb.1784018825.git.christophe.jaillet@wanadoo.fr
Signed-off-by: Leon Romanovsky <leon@kernel.org>
'struct ib_frmr_pool_ops' and 'struct dma_buf_attach_ops' are not modified
in this driver.
Constifying these structures moves some data to a read-only section, so
increases overall security, especially when the structure holds some
function pointers.
While at it, change a '1' into a 'true' into the mlx5_ib_dmabuf_attach_ops
structure. The 'allow_peer2peer' field is a bool and other usages of
'struct dma_buf_attach_ops' prefer using true/false.
On a x86_64, with allmodconfig, as an example:
Before:
======
text data bss dec hex filename
77631 10392 320 88343 15917 drivers/infiniband/hw/mlx5/mr.o
After:
=====
text data bss dec hex filename
77759 10264 320 88343 15917 drivers/infiniband/hw/mlx5/mr.o
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Link: https://patch.msgid.link/22f2263c04cc94e242cee712e6e6d82b86ac353d.1784017128.git.christophe.jaillet@wanadoo.fr
Signed-off-by: Leon Romanovsky <leon@kernel.org>
rxe advertises ODP but not IB_ODP_SUPPORT_IMPLICIT. The reg_user_mr path
still contained a dead branch that checked the implicit capability and
could never succeed.
Return -EOPNOTSUPP for the implicit ODP address range up front so the
intent is obvious and the unreachable code is gone.
Signed-off-by: xiongweimin <xiongweimin@kylinos.cn>
Cc: linux-rdma@vger.kernel.org
Cc: Jason Gunthorpe <jgg@nvidia.com>
Link: https://patch.msgid.link/20260713010439.331054-1-15927021679@163.com
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
get_map_page() allocates bitmap pages using get_zeroed_page().
The bitmaps can be allocated with kmalloc() as there's nothing special
about them to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of get_zeroed_page() with kzalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260713-b4-rdma-v2-5-65d2a1a5180c@kernel.org
Signed-off-by: Leon Romanovsky <leon@kernel.org>
mthca_array is essentially a sparse array of pointers and there is no
need to allocate its memory using page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of get_zeroed_page() with kzalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260713-b4-rdma-v2-4-65d2a1a5180c@kernel.org
Signed-off-by: Leon Romanovsky <leon@kernel.org>
mthca_reg_user_mr() allocates an array of DMA addresses during memory
registration.
This buffer can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of __get_free_page() with kmalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260713-b4-rdma-v2-3-65d2a1a5180c@kernel.org
Signed-off-by: Leon Romanovsky <leon@kernel.org>
mlx5_ib_mr_wqe_pfault_handler() allocates a scratch buffer for
parsing work queue entries during page fault handling.
This buffer can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of __get_free_page() with kmalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260713-b4-rdma-v2-2-65d2a1a5180c@kernel.org
Signed-off-by: Leon Romanovsky <leon@kernel.org>
ib_umem_get() allocates an array of pointers to struct page for
pin_user_pages_fast() calls during memory registration.
This array can be allocated with kmalloc() as there's nothing special
about it to go directly to the page allocator.
kmalloc() provides a better API that does not require ugly casts and
kfree() does not need to know the size of the freed object.
Performance difference between kmalloc() and __get_free_pages() is not
measurable as both allocators take an object/page from a per-CPU list for
fast path allocations.
For the slow path the performance is anyway determined by the amount of
reclaim involved rather than by what allocator is used.
Replace use of __get_free_page() with kmalloc() and free_page() with
kfree().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Link: https://patch.msgid.link/20260713-b4-rdma-v2-1-65d2a1a5180c@kernel.org
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The core always hands the driver's query_device() callback a zeroed
ib_device_attr. There are only two callers of the op and both clear the
structure before invoking it: setup_device() memsets &device->attrs, and
ib_uverbs_ex_query_device() passes an on-stack structure initialized to {}.
The open-coded memset(props, 0, sizeof(*props)) at the top of the driver
callbacks is therefore redundant. Remove it from all drivers.
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Update device interface adding one more bit from reserved to enable
>4GB page sizes that can be supported on 0xefa4 devices.
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
Link: https://patch.msgid.link/20260712134413.19226-2-mrgolin@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
init_one() defers handling errors from hfi1_init() and
hfi1_register_ib_device() to a combined block. This allows IB registration
to run after device initialization has failed. Cleanup then depends on two
unrelated error values.
The late probe failure path also differs from the common teardown in
remove_one(), even though both release the same initialized hardware and
driver resources. Maintaining separate sequences obscures ownership and
allows the paths to drift whenever initialization changes.
Unwind at each failing stage and keep late probe teardown ordered like
remove_one(), so partial initialization releases exactly the resources it
owns and normal removal remains the reference cleanup flow.
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-13-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Commit ed6f653fe4 ("staging/rdma/hfi1: Fix debugfs access race") moved
debugfs creation after device initialization and IB registration so users
cannot access the files before the driver is ready. However, init_one()
still creates them before character device creation and SDMA startup
finish.
Move hfi1_dbg_ibdev_init() to the end of the successful probe path,
matching hfi1_dbg_ibdev_exit() as the first action in remove_one().
Fixes: ed6f653fe4 ("staging/rdma/hfi1: Fix debugfs access race")
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-12-b9e9641268a5@nvidia.com
Reviewed-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
init_one() creates the character device before checking whether generic or
IB initialization failed, only to remove it immediately while unwinding.
Moreover, user_add() already calls user_remove() when device creation
fails.
Move hfi1_device_create() after the initialization failure path,
immediately before starting SDMA. The failure path then has no character
device to remove, and hfi1_device_create() continues to unwind its own
failures.
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-11-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
hfi1 does not queue work on ib_wq. QSFP and link work run on the per-port
link_wq, while the remaining device work uses hfi1_wq or dedicated queues.
The probe failure path destroys both per-port workqueues, and normal device
removal flushes them in shutdown_device() before destroying them.
Remove the flushes of the core-owned global workqueue. Waiting for
unrelated core or other device work is not part of hfi1 teardown.
Fixes: 71d47008ca ("IB/hfi1: Create workqueue for link events")
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-10-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
create_workqueues() is called only from init_one(), immediately after
hfi1_alloc_devdata() returns a zero-initialized hfi1_devdata. As a
result, the per-port hfi1_wq and link_wq pointers are always NULL on
entry, and nothing modifies them between allocation and this call.
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-9-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
After the preceding changes, module parameter validation and common PCI
setup no longer need hfi1_devdata. init_one() nevertheless allocates it
first, so failures in those early steps return without releasing it.
Move the allocation after hfi1_pcie_init() and return directly when no
resources are held. Use dev_err() and pci_info() for diagnostics emitted
before allocation so they retain the adapter BDF. Once PCI setup succeeds,
unwind allocation failures through hfi1_pcie_cleanup() to disable the
device and release its regions.
Some PCI error delivery paths are not serialized against probe. Keep their
diagnostics based on pci_dev and skip resume while driver data is absent,
preventing recovery from dereferencing a missing hfi1_devdata.
Fixes: 57f97e9662 ("IB/hfi1: Get the hfi1_devdata structure as early as possible")
Reported-by: Dawei Feng <dawei.feng@seu.edu.cn>
Closes: https://lore.kernel.org/all/20260627060159.2543686-1-dawei.feng@seu.edu.cn/
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-8-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
hfi1_init_dd() allocates the shared AIP/VNIC RX support before returning.
If hfi1_init() or hfi1_register_ib_device() later fails, init_one() tears
down the device data without calling hfi1_free_rx(). This leaks netdev_rx
and its dummy netdev.
Free the RX support after IB unregistration and before postinit_cleanup(),
as done on normal device removal.
Fixes: 4730f4a6c6 ("IB/hfi1: Activate the dummy netdev")
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-7-b9e9641268a5@nvidia.com
Reviewed-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
create_workqueues() only needs fields set up by hfi1_alloc_devdata().
Call it before hfi1_init_dd() so a workqueue allocation failure happens
before chip resources are initialized.
To keep the reordered error paths safe, make init_one() own hfi1_devdata.
hfi1_init_dd() unwinds its partial setup but leaves the allocation for the
caller to free. If device initialization fails, destroy the workqueues
before freeing the device data.
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-6-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
hfi1_validate_rcvhdrcnt() only needs hfi1_devdata to identify the adapter
in error messages. Requiring the full device data prevents module parameter
validation from running before hfi1_devdata is allocated.
Pass pci_dev instead and use dev_err(), allowing validation to move earlier
without losing the PCI BDF needed on multi-device systems. Use %u for the
unsigned count while changing the messages.
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-5-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
hfi1_pcie_init() only needs hfi1_devdata to reach the PCI device. This
unnecessary dependency prevents common PCI setup from running before
hfi1_devdata is allocated.
Pass pci_dev directly and report failures with dev_err(), preserving the
device BDF needed to identify the failing adapter on multi-device systems.
Use %pe while changing the messages so errno values are decoded.
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-4-b9e9641268a5@nvidia.com
Reviewed-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
The PCI core calls init_one() only after pci_match_device() has selected
an ID. For normal probing, hfi1_pci_tbl already restricts matches to the
two supported Intel device IDs. Dynamic IDs and driver_override are
explicit requests to attempt binding, so the probe should not second-guess
the PCI core's decision.
Remove the redundant check.
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-3-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
hfi1_free_devdata() assumes that the device was inserted into the unit
table and unconditionally erases dd->unit. If xa_alloc_irq() fails, the
zero-initialized unit remains zero, so full cleanup can remove an
unrelated device from index 0.
Release only the rdmavt allocation and return immediately while the unit
table has not acquired the device.
Fixes: 03b92789e5 ("hfi1: Convert hfi1_unit_table to XArray")
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-2-b9e9641268a5@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
rvt_alloc_device() deallocates the IB device when its port array cannot
be allocated but then returns the pointer to the released allocation.
Callers treat any non-NULL value as valid and dereference it, resulting
in a use-after-free.
Return NULL immediately after deallocation so callers can propagate the
allocation failure.
Fixes: ff6acd6951 ("IB/rdmavt: Add device structure allocation")
Link: https://patch.msgid.link/20260708-clean-init-one-hfi1-v1-1-b9e9641268a5@nvidia.com
Reviewed-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
The mlx4 CM paravirtualization layer rewrites a VF's local
communication ID to a PF-visible ID when CM MADs are sent from the VF.
For messages that start or advance a connection from the VF side, such
as REQ, REP, MRA and SIDR_REQ, mlx4_ib_multiplex_cm_handler() allocates
an id_map_entry when no existing mapping is found.
A REJ is different because it is a terminal response to an already known
exchange. It should either find an existing id_map_entry, rewrite the
local communication ID, and schedule that entry for deletion, or it
should pass through unchanged when no mapping exists.
Some REJ messages, such as rejects for an inbound REQ before an MRA or
REP was sent, do not have an id_map_entry because their local_comm_id is
zero. Timeout REJ messages are handled in the initial lookup branch, but
a lookup miss there must not fall through to id_map_alloc(); such a miss
means there is no existing mapping to translate or delete for the REJ.
Commit 227a0e142e ("IB/mlx4: Add support for REJ due to timeout")
added the timeout REJ case to the initial branch so an outgoing timeout
REJ could reuse the id_map_entry that was created when the VF's REQ was
multiplexed. Reusing that entry is the useful part: it rewrites the
timeout REJ local_comm_id to the same PF-visible ID that was sent in the
REQ. If the lookup misses, allocating a new id_map_entry does not help
because the peer has never seen that new PF-visible ID, and REJ is not
starting a new exchange.
Keep timeout REJ handling in the initial lookup branch, but return before
allocation if no mapping is found. Handle the other REJ cases with the
same lookup-only behavior. When a mapping is found, translate the local
communication ID and schedule delayed deletion, as is already done for
DREQ and for received REJ in the demux path. When no mapping is found,
keep the existing pass-through behavior.
Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
Link: https://patch.msgid.link/20260615171759.557425-1-praveen.kannoju@oracle.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
