github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-09-14 16:10:02 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

RDMA v7.3 merge window pull

Browse Source 
Quite alot of buf fixes again:
 
 - Assorted locking, bounds-checking, cleanup, and error-path fixes across
   UCMA/CMA, bng_re, bnxt_re, cxgb4, EFA, ERDMA, HFI1, HNS, ionic, iRDMA,
   mlx4/mlx5, RXE, SIW, SRP/SRPT, and iSER target.
 
 - netlink report for max # of supported resources
 
 - get_zeroed_page()/etc removal
 
 - Robust udata for ionic
 
 - Allow unique RDMA device names per network namespace
 
 - Completion counters and v2 admit queue support for EFA
 
 - UC QP support for MANA
 
 - Completion timestamps for ionic
 
 - Harden uverbs data validation and resource lifetime handling, fixing several core use-after-free conditions.
 
 - bnxt_re toggle-page ownership and lifetime bug fixes
 
 - dmabuf SRQ support for mlx5
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQRRRCHOFoQz/8F5bUaFwuHvBreFYQUCaoeO2wAKCRCFwuHvBreF
 YcyJAQCn8JeqTuayLchCARJX+9fazTaJES9zj41i3M8a7BfxCAEA2b266g0S660m
 7eQ761NiIb1iklSl9rmt8vM22EMcngM=
 =qcwp
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma

Pull RDMA updates from Jason Gunthorpe:
 "About the normal size, still a lot of AI bug fixes and so on, but some
  interesting new functionality too:

   - Assorted locking, bounds-checking, cleanup, and error-path fixes
     across UCMA/CMA, bng_re, bnxt_re, cxgb4, EFA, ERDMA, HFI1, HNS,
     ionic, iRDMA, mlx4/mlx5, RXE, SIW, SRP/SRPT, and iSER target.

   - netlink report for max # of supported resources

   - get_zeroed_page()/etc removal

   - Robust udata for ionic

   - Allow unique RDMA device names per network namespace

   - Completion counters and v2 admit queue support for EFA

   - UC QP support for MANA

   - Completion timestamps for ionic

   - Harden uverbs data validation and resource lifetime handling,
     fixing several core use-after-free conditions.

   - bnxt_re toggle-page ownership and lifetime bug fixes

   - dmabuf SRQ support for mlx5"

* tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma: (160 commits)
  RDMA/ucma: Allow path records to exactly fit the output buffer
  RDMA/uverbs: Guard legacy bundles without method_elm
  RDMA/efa: Add support for 128B admin v2 SQ entry
  RDMA/efa: Generalize the admin SQ
  RDMA/efa: Decouple admin command payload from admin header
  RDMA/rxe: Fix OOB in free_rd_atomic_resources()
  RDMA/cma: Fix WARNING in res_to_rt
  RDMA/cxgb4: Free debugfs on registration failure
  RDMA/cxgb4: Cancel reg_work before freeing device on remove
  RDMA/ucma: Lock the handler in ucma_set_ib_path()
  RDMA/ucma: Lock the handler in ucma_write_cm_event()
  RDMA/erdma: restrict the driver to little-endian systems
  RDMA/ionic: Embed counter driver data in rdma_counter allocation
  RDMA/ionic: Cap eq_count to the eth driver's interrupt vector budget
  RDMA/siw: Fix use-after-free in siw_accept()
  IB/isert: post the full-feature receive buffers after session registration
  IB/isert: delay the final Login Response until the session is registered
  RDMA/srp: fix heap information leak on a truncated SRP_CRED_REQ
  RDMA/erdma: Hold QP references for AE and CM processing
  RDMA/erdma: Hold CQ references when processing EQ events
  ...
This commit is contained in:
Linus Torvalds 2026-08-21 11:48:54 -07:00
commit 21bd0802cd
176 changed files with 4587 additions and 1594 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Documentation/ABI/testing/configfs-rdma_cm
View File

@ -12,6 +12,10 @@ Description: Interface is used to configure RDMA-cable HCAs in respect to
for this HCA has to be created:
mkdir -p /config/rdma_cm/<hca>
Note: configfs has no network namespace support, so this
interface cannot represent two devices that share a name in
different network namespaces (possible in exclusive netns mode).
What: /config/rdma_cm/<hca>/ports/<port-num>/default_roce_mode
Date: November 29, 2015

8
Documentation/admin-guide/cgroup-v1/rdma.rst
View File

@ -91,6 +91,13 @@ Following resources can be accounted by rdma controller.
hca_object Maximum number of HCA Objects
========== =============================
RDMA devices from all network namespaces are listed. Each line starts with
the device name. If more than one device has the same name, ``index=N``
follows the name, where ``N`` is the system-wide RDMA device index, unique
among registered devices. When configuring a limit, the index is optional
for a globally unique name and required for a duplicate name. A write without
the required index fails with ``-ENOTUNIQ``.
2. Usage Examples
=================
@ -98,6 +105,7 @@ Following resources can be accounted by rdma controller.
echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max
echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max
echo "rxe0 index=5 hca_handle=2" > /sys/fs/cgroup/rdma/3/rdma.max
(b) Query resource limit::

15
Documentation/admin-guide/cgroup-v2.rst
View File

@ -2809,6 +2809,11 @@ RDMA
The "rdma" controller regulates the distribution and accounting of
RDMA resources.
RDMA devices from all network namespaces are listed. Each line starts with
the device name. If more than one device has the same name, ``index=N``
follows the name, where ``N`` is the system-wide RDMA device index, unique
among registered devices.
RDMA Interface Files
~~~~~~~~~~~~~~~~~~~~
@ -2817,7 +2822,11 @@ RDMA Interface Files
except root that describes current configured resource limit
for a RDMA/IB device.
Lines are keyed by device name and are not ordered.
Lines are keyed by device name and are not ordered. A write may
include ``index=N`` after the device name. The index is optional
when the name is globally unique. If multiple devices have that
name, the index is required and a write without it fails with
``-ENOTUNIQ``.
Each line contains space separated resource name and its configured
limit that can be distributed.
@ -2833,6 +2842,10 @@ RDMA Interface Files
mlx4_0 hca_handle=2 hca_object=2000
ocrdma1 hca_handle=3 hca_object=max
For devices with duplicate names, select the device by index::
echo "rxe0 index=5 hca_handle=2" > rdma.max
rdma.current
A read-only file that describes current resource usage.
It exists for all the cgroup except root.

2
Documentation/infiniband/user_mad.rst
View File

@ -62,7 +62,7 @@ Receiving MADs
struct ib_user_mad *mad;
mad = malloc(sizeof *mad + 256);
ret = read(fd, mad, sizeof *mad + 256);
if (ret == -ENOSPC)) {
if (ret == -ENOSPC) {
length = mad.length;
free(mad);
mad = malloc(sizeof *mad + length);

1
drivers/infiniband/core/Makefile
View File

@ -38,6 +38,7 @@ ib_umad-y := user_mad.o
ib_uverbs-y := uverbs_main.o uverbs_cmd.o uverbs_marshall.o \
uverbs_std_types.o uverbs_ioctl.o \
uverbs_std_types_cq.o \
uverbs_std_types_comp_cntr.o \
uverbs_std_types_dmabuf.o \
uverbs_std_types_dmah.o \
uverbs_std_types_flow_action.o uverbs_std_types_dm.o \

2
drivers/infiniband/core/addr.c
View File

@ -769,7 +769,7 @@ void rdma_addr_cancel(struct rdma_dev_addr *addr)
/*
* sync canceling the work after removing it from the req_list
* guarentees no work is running and none will be started.
* guarantees no work is running and none will be started.
*/
cancel_delayed_work_sync(&found->work);
kfree(found);

1
drivers/infiniband/core/cgroup.c
View File

@ -17,6 +17,7 @@
void ib_device_register_rdmacg(struct ib_device *device)
{
device->cg_device.name = device->name;
device->cg_device.index = device->index;
rdmacg_register_device(&device->cg_device);
}

8
drivers/infiniband/core/cma.c
View File

@ -2104,7 +2104,7 @@ static void destroy_id_handler_unlock(struct rdma_id_private *id_priv)
/*
* Setting the state to destroyed under the handler mutex provides a
* fence against calling handler callbacks. If this is invoked due to
* the failure of a handler callback then it guarentees that no future
* the failure of a handler callback then it guarantees that no future
* handlers will be called.
*/
lockdep_assert_held(&id_priv->handler_mutex);
@ -3528,10 +3528,12 @@ static void addr_handler(int status, struct sockaddr *src_addr,
memcpy(addr, src_addr, rdma_addr_size(src_addr));
if (!status && !id_priv->cma_dev) {
status = cma_acquire_dev_by_src_ip(id_priv);
if (status)
if (status) {
pr_debug_ratelimited("RDMA CM: ADDR_ERROR: failed to acquire device. status %d\n",
status);
rdma_restrack_add(&id_priv->res);
} else {
rdma_restrack_add(&id_priv->res);
}
} else if (status) {
pr_debug_ratelimited("RDMA CM: ADDR_ERROR: failed to resolve IP. status %d\n", status);
}

4
drivers/infiniband/core/cma_configfs.c
View File

@ -65,6 +65,10 @@ static struct cma_dev_port_group *to_dev_port_group(struct config_item *item)
return container_of(group, struct cma_dev_port_group, group);
}
/*
* configfs is not net namespace aware, so a name shared by devices in
* different namespaces resolves to the first match here.
*/
static bool filter_by_name(struct ib_device *ib_dev, void *cookie)
{
return !strcmp(dev_name(&ib_dev->dev), cookie);

3
drivers/infiniband/core/core_priv.h
View File

@ -356,7 +356,8 @@ void ib_port_unregister_client_groups(struct ib_device *ibdev, u32 port_num,
const struct attribute_group **groups);
int ib_device_set_netns_put(struct sk_buff *skb,
struct ib_device *dev, u32 ns_fd);
struct ib_device *dev, u32 ns_fd, const char *name,
struct netlink_ext_ack *extack);
int rdma_nl_net_init(struct rdma_dev_net *rnet);
void rdma_nl_net_exit(struct rdma_dev_net *rnet);

5
drivers/infiniband/core/counters.c
View File

@ -234,7 +234,6 @@ static void rdma_counter_free(struct rdma_counter *counter)
mutex_unlock(&port_counter->lock);
rdma_restrack_del(&counter->res);
rdma_free_hw_stats_struct(counter->stats);
kfree(counter);
}
@ -329,6 +328,7 @@ static void counter_release(struct kref *kref)
counter = container_of(kref, struct rdma_counter, kref);
counter_history_stat_update(counter);
rdma_restrack_del(&counter->res);
counter->device->ops.counter_dealloc(counter);
rdma_counter_free(counter);
}
@ -490,7 +490,8 @@ static struct rdma_counter *rdma_get_counter_by_id(struct ib_device *dev,
return NULL;
counter = container_of(res, struct rdma_counter, res);
kref_get(&counter->kref);
if (!kref_get_unless_zero(&counter->kref))
counter = NULL;
rdma_restrack_put(res);
return counter;

5
drivers/infiniband/core/cq.c
View File

@ -327,6 +327,7 @@ void ib_free_cq(struct ib_cq *cq)
if (WARN_ON_ONCE(cq->cqe_used))
return;
rdma_restrack_del(&cq->res);
if (cq->device->ops.pre_destroy_cq) {
ret = cq->device->ops.pre_destroy_cq(cq);
WARN_ONCE(ret, "Disable of kernel CQ shouldn't fail");
@ -353,7 +354,6 @@ void ib_free_cq(struct ib_cq *cq)
else
ret = cq->device->ops.destroy_cq(cq, NULL);
WARN_ONCE(ret, "Destroy of kernel CQ shouldn't fail");
rdma_restrack_del(&cq->res);
kfree(cq->wc);
kfree(cq);
}
@ -393,8 +393,7 @@ static int ib_alloc_cqs(struct ib_device *dev, unsigned int nr_cqes,
* a reasonable batch size so that we can share CQs between
* multiple users instead of allocating a larger number of CQs.
*/
nr_cqes = min_t(unsigned int, dev->attrs.max_cqe,
max(nr_cqes, IB_MAX_SHARED_CQ_SZ));
nr_cqes = min(dev->attrs.max_cqe, max(nr_cqes, IB_MAX_SHARED_CQ_SZ));
nr_cqs = min_t(unsigned int, dev->num_comp_vectors, num_online_cpus());
for (i = 0; i < nr_cqs; i++) {
cq = ib_alloc_cq(dev, NULL, nr_cqes, i, poll_ctx);

255
drivers/infiniband/core/device.c
View File

@ -129,7 +129,7 @@ static DECLARE_RWSEM(rdma_nets_rwsem);
bool ib_devices_shared_netns = true;
module_param_named(netns_mode, ib_devices_shared_netns, bool, 0444);
MODULE_PARM_DESC(netns_mode,
"Share device among net namespaces; default=1 (shared)");
"Share device among net namespaces; default=1 (shared). In exclusive mode device names are unique per net namespace");
/**
* rdma_dev_access_netns() - Return whether an rdma device can be accessed
* from a specified net namespace or not.
@ -268,7 +268,8 @@ static struct notifier_block ibdev_lsm_nb = {
};
static int rdma_dev_change_netns(struct ib_device *device, struct net *cur_net,
struct net *net);
struct net *net, const char *requested_name,
const char *fallback_pattern);
/* Pointer to the RCU head at the start of the ib_port_data array */
struct ib_port_data_rcu {
@ -351,13 +352,15 @@ void ib_device_put(struct ib_device *device)
}
EXPORT_SYMBOL(ib_device_put);
static struct ib_device *__ib_device_get_by_name(const char *name)
static struct ib_device *__ib_device_get_by_name(const char *name,
const struct net *net)
{
struct ib_device *device;
unsigned long index;
xa_for_each (&devices, index, device)
if (!strcmp(name, dev_name(&device->dev)))
if (rdma_dev_access_netns(device, net) &&
!strcmp(name, dev_name(&device->dev)))
return device;
return NULL;
@ -395,7 +398,7 @@ int ib_device_rename(struct ib_device *ibdev, const char *name)
return 0;
}
if (__ib_device_get_by_name(name)) {
if (__ib_device_get_by_name(name, rdma_dev_net(ibdev))) {
up_write(&devices_rwsem);
return -EEXIST;
}
@ -435,7 +438,13 @@ int ib_device_set_dim(struct ib_device *ibdev, u8 use_dim)
return 0;
}
static int alloc_name(struct ib_device *ibdev, const char *name)
/*
* Pick a free index for the '%d' style @name pattern within net namespace
* @net. Returns the index on success or a negative errno. The caller builds
* the final unique device name from the returned index.
*/
static int __alloc_name_id(struct net *net, const char *name,
const struct ib_device *skip)
{
struct ib_device *device;
unsigned long index;
@ -448,6 +457,8 @@ static int alloc_name(struct ib_device *ibdev, const char *name)
xa_for_each (&devices, index, device) {
char buf[IB_DEVICE_NAME_MAX];
if (device == skip || !rdma_dev_access_netns(device, net))
continue;
if (sscanf(dev_name(&device->dev), name, &i) != 1)
continue;
if (i < 0 || i >= INT_MAX)
@ -462,15 +473,27 @@ static int alloc_name(struct ib_device *ibdev, const char *name)
}
rc = ida_alloc(&inuse, GFP_KERNEL);
if (rc < 0)
goto out;
rc = dev_set_name(&ibdev->dev, name, rc);
out:
ida_destroy(&inuse);
return rc;
}
static int alloc_name_id(struct net *net, const char *name)
{
return __alloc_name_id(net, name, NULL);
}
static int alloc_name(struct ib_device *ibdev, const char *name)
{
int id;
id = alloc_name_id(rdma_dev_net(ibdev), name);
if (id < 0)
return id;
return dev_set_name(&ibdev->dev, name, id);
}
static void ib_device_release(struct device *device)
{
struct ib_device *dev = container_of(device, struct ib_device, dev);
@ -1151,8 +1174,17 @@ static void rdma_dev_exit_net(struct net *net)
/*
* If the real device is in the NS then move it back to init.
* Provide a fallback pattern so a name conflict in init_net
* cannot make the teardown move fail.
*/
rdma_dev_change_netns(dev, net, &init_net);
if (net_eq(net, read_pnet(&dev->coredev.rdma_net))) {
ret = rdma_dev_change_netns(dev, net, &init_net,
NULL, "ibdev%d");
if (ret && ret != -ENODEV)
WARN(1,
"Failed to move RDMA device %s to init_net on netns exit: %d\n",
dev_name(&dev->dev), ret);
}
put_device(&dev->dev);
down_read(&devices_rwsem);
@ -1206,7 +1238,8 @@ static __net_init int rdma_dev_init_net(struct net *net)
}
/*
* Assign the unique string device name and the unique device index. This is
* Assign the unique string device name and the unique device index. The device
* name is unique within the net namespace the device is assigned to. This is
* undone by ib_dealloc_device.
*/
static int assign_name(struct ib_device *device, const char *name)
@ -1223,7 +1256,8 @@ static int assign_name(struct ib_device *device, const char *name)
if (ret)
goto out;
if (__ib_device_get_by_name(dev_name(&device->dev))) {
if (__ib_device_get_by_name(dev_name(&device->dev),
rdma_dev_net(device))) {
ret = -ENFILE;
goto out;
}
@ -1388,8 +1422,9 @@ static void ib_device_notify_register(struct ib_device *device)
/**
* ib_register_device - Register an IB device with IB core
* @device: Device to register
* @name: unique string device name. This may include a '%' which will
* cause a unique index to be added to the passed device name.
* @name: device name, unique within the device's net namespace. This may
* include a '%' which will cause a unique index to be added to the
* passed device name.
* @dma_device: pointer to a DMA-capable device. If %NULL, then the IB
* device will be used. In this case the caller should fully
* setup the ibdev for DMA. This usually means using dma_virt_ops.
@ -1670,14 +1705,83 @@ void ib_unregister_device_queued(struct ib_device *ib_dev)
}
EXPORT_SYMBOL(ib_unregister_device_queued);
static bool rdma_dev_name_in_netns(struct ib_device *skip, struct net *net,
const char *name)
{
struct ib_device *device;
unsigned long index;
lockdep_assert_held_write(&devices_rwsem);
xa_for_each(&devices, index, device)
if (device != skip &&
rdma_dev_access_netns(device, net) &&
!strcmp(name, dev_name(&device->dev)))
return true;
return false;
}
/*
* Choose the name @device should use in net namespace @net. @requested_name
* is used as a literal device name when set. Otherwise keep the current name
* when it is free, or use a trusted '%d' @fallback_pattern for teardown. The
* caller must hold the write side of devices_rwsem.
*/
static int rdma_dev_pick_netns_name(struct ib_device *device, struct net *net,
const char *requested_name,
const char *fallback_pattern,
char *buf, size_t buf_len,
const char **new_name)
{
int id;
lockdep_assert_held_write(&devices_rwsem);
if (requested_name) {
if (!rdma_dev_name_in_netns(device, net, requested_name)) {
*new_name = requested_name;
return 0;
}
return -EEXIST;
}
if (!rdma_dev_name_in_netns(device, net, dev_name(&device->dev))) {
*new_name = dev_name(&device->dev);
return 0;
}
if (!fallback_pattern)
return -EEXIST;
snprintf(buf, buf_len, "ibdev%u", device->index);
if (!rdma_dev_name_in_netns(device, net, buf)) {
*new_name = buf;
return 0;
}
id = __alloc_name_id(net, fallback_pattern, device);
if (id < 0)
return id;
snprintf(buf, buf_len, fallback_pattern, id);
*new_name = buf;
return 0;
}
/*
* The caller must pass in a device that has the kref held and the refcount
* released. If the device is in cur_net and still registered then it is moved
* into net.
*
* Naming rules are handled by rdma_dev_pick_netns_name().
*/
static int rdma_dev_change_netns(struct ib_device *device, struct net *cur_net,
struct net *net)
struct net *net, const char *requested_name,
const char *fallback_pattern)
{
char buf[IB_DEVICE_NAME_MAX];
const char *new_name;
int ret2 = -EINVAL;
int ret;
@ -1694,31 +1798,66 @@ static int rdma_dev_change_netns(struct ib_device *device, struct net *cur_net,
goto out;
}
if (!fallback_pattern) {
/*
* Reject a predictable name conflict before tearing anything
* down, so a doomed user move does not disable a live device.
*/
down_write(&devices_rwsem);
ret = rdma_dev_pick_netns_name(device, net, requested_name,
fallback_pattern, buf,
sizeof(buf), &new_name);
up_write(&devices_rwsem);
if (ret)
goto out;
}
kobject_uevent(&device->dev.kobj, KOBJ_REMOVE);
disable_device(device);
/*
* At this point no one can be using the device, so it is safe to
* change the namespace.
* Recompute the destination name under the write side of devices_rwsem
* now that the device is disabled, closing races with a concurrent
* registration or rename, then publish the new namespace at the sysfs
* level.
*/
write_pnet(&device->coredev.rdma_net, net);
down_read(&devices_rwsem);
/*
* Currently rdma devices are system wide unique. So the device name
* is guaranteed free in the new namespace. Publish the new namespace
* at the sysfs level.
*/
ret = device_rename(&device->dev, dev_name(&device->dev));
up_read(&devices_rwsem);
down_write(&devices_rwsem);
ret = rdma_dev_pick_netns_name(device, net, requested_name,
fallback_pattern, buf, sizeof(buf),
&new_name);
if (ret) {
dev_warn(&device->dev,
"%s: Couldn't rename device after namespace change\n",
__func__);
/* Try and put things back and re-enable the device */
write_pnet(&device->coredev.rdma_net, cur_net);
if (fallback_pattern) {
WARN(1,
"%s: failed to pick device name during namespace teardown: %d\n",
__func__, ret);
write_pnet(&device->coredev.rdma_net, net);
ret = 0;
}
goto rename_done;
}
write_pnet(&device->coredev.rdma_net, net);
ret = device_rename(&device->dev, new_name);
if (ret) {
if (fallback_pattern) {
WARN(1,
"%s: failed to rename device during namespace teardown: %d\n",
__func__, ret);
ret = 0;
} else {
dev_warn(&device->dev,
"%s: Couldn't rename device after namespace change\n",
__func__);
/* Try and put things back and re-enable the device */
write_pnet(&device->coredev.rdma_net, cur_net);
}
} else {
strscpy(device->name, dev_name(&device->dev),
IB_DEVICE_NAME_MAX);
}
rename_done:
up_write(&devices_rwsem);
ret2 = enable_device_and_get(device);
if (ret2) {
/*
@ -1740,36 +1879,72 @@ static int rdma_dev_change_netns(struct ib_device *device, struct net *cur_net,
}
int ib_device_set_netns_put(struct sk_buff *skb,
struct ib_device *dev, u32 ns_fd)
struct ib_device *dev, u32 ns_fd, const char *name,
struct netlink_ext_ack *extack)
{
struct net *net;
int ret;
net = get_net_ns_by_fd(ns_fd);
if (IS_ERR(net)) {
NL_SET_ERR_MSG(extack, "Invalid target net namespace fd");
ret = PTR_ERR(net);
goto net_err;
}
if (!netlink_ns_capable(skb, net->user_ns, CAP_NET_ADMIN)) {
NL_SET_ERR_MSG(extack,
"Missing CAP_NET_ADMIN in the target net namespace");
ret = -EPERM;
goto ns_err;
}
/*
* Moving a device to the namespace it already lives in is a no-op; a
* supplied name still renames it in place.
*/
if (net_eq(net, read_pnet(&dev->coredev.rdma_net))) {
ret = name ? ib_device_rename(dev, name) : 0;
if (ret == -EEXIST)
NL_SET_ERR_MSG(extack,
"Device name already exists in the target net namespace");
else if (ret == -EINVAL && name)
NL_SET_ERR_MSG(extack,
"Unable to use requested device name in the target net namespace");
goto ns_err;
}
/*
* All the ib_clients, including uverbs, are reset when the namespace is
* changed and this cannot be blocked waiting for userspace to do
* something, so disassociation is mandatory.
*/
if (!dev->ops.disassociate_ucontext || ib_devices_shared_netns) {
if (ib_devices_shared_netns) {
NL_SET_ERR_MSG(extack,
"Cannot change net namespace of RDMA device in shared netns mode");
ret = -EOPNOTSUPP;
goto ns_err;
}
if (!dev->ops.disassociate_ucontext) {
NL_SET_ERR_MSG(extack,
"Device does not support namespace changes (no disassociate support)");
ret = -EOPNOTSUPP;
goto ns_err;
}
get_device(&dev->dev);
ib_device_put(dev);
ret = rdma_dev_change_netns(dev, current->nsproxy->net_ns, net);
ret = rdma_dev_change_netns(dev, current->nsproxy->net_ns, net, name,
NULL);
put_device(&dev->dev);
if (ret == -EEXIST)
NL_SET_ERR_MSG(extack,
"Device name already exists in the target net namespace");
else if (ret == -EINVAL && name)
NL_SET_ERR_MSG(extack,
"Unable to use requested device name in the target net namespace");
put_net(net);
return ret;
@ -2742,6 +2917,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, create_ah);
SET_DEVICE_OP(dev_ops, create_counters);
SET_DEVICE_OP(dev_ops, create_cq);
SET_DEVICE_OP(dev_ops, create_comp_cntr);
SET_DEVICE_OP(dev_ops, create_user_cq);
SET_DEVICE_OP(dev_ops, create_flow);
SET_DEVICE_OP(dev_ops, create_qp);
@ -2762,6 +2938,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, destroy_ah);
SET_DEVICE_OP(dev_ops, destroy_counters);
SET_DEVICE_OP(dev_ops, destroy_cq);
SET_DEVICE_OP(dev_ops, destroy_comp_cntr);
SET_DEVICE_OP(dev_ops, destroy_flow);
SET_DEVICE_OP(dev_ops, destroy_flow_action);
SET_DEVICE_OP(dev_ops, destroy_qp);
@ -2813,6 +2990,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, modify_hw_stat);
SET_DEVICE_OP(dev_ops, modify_port);
SET_DEVICE_OP(dev_ops, modify_qp);
SET_DEVICE_OP(dev_ops, qp_attach_comp_cntr);
SET_DEVICE_OP(dev_ops, modify_srq);
SET_DEVICE_OP(dev_ops, modify_wq);
SET_DEVICE_OP(dev_ops, peek_cq);
@ -2826,6 +3004,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, post_srq_recv);
SET_DEVICE_OP(dev_ops, process_mad);
SET_DEVICE_OP(dev_ops, query_ah);
SET_DEVICE_OP(dev_ops, query_comp_cntr_caps);
SET_DEVICE_OP(dev_ops, query_device);
SET_DEVICE_OP(dev_ops, query_gid);
SET_DEVICE_OP(dev_ops, query_pkey);
@ -2836,12 +3015,14 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, query_ucontext);
SET_DEVICE_OP(dev_ops, rdma_netdev_get_params);
SET_DEVICE_OP(dev_ops, read_counters);
SET_DEVICE_OP(dev_ops, read_comp_cntr);
SET_DEVICE_OP(dev_ops, reg_dm_mr);
SET_DEVICE_OP(dev_ops, reg_user_mr);
SET_DEVICE_OP(dev_ops, reg_user_mr_dmabuf);
SET_DEVICE_OP(dev_ops, req_notify_cq);
SET_DEVICE_OP(dev_ops, rereg_user_mr);
SET_DEVICE_OP(dev_ops, resize_user_cq);
SET_DEVICE_OP(dev_ops, modify_comp_cntr);
SET_DEVICE_OP(dev_ops, set_vf_guid);
SET_DEVICE_OP(dev_ops, set_vf_link_state);
SET_DEVICE_OP(dev_ops, ufile_hw_cleanup);
@ -2850,6 +3031,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_OBJ_SIZE(dev_ops, ib_ah);
SET_OBJ_SIZE(dev_ops, ib_counters);
SET_OBJ_SIZE(dev_ops, ib_cq);
SET_OBJ_SIZE(dev_ops, ib_comp_cntr);
SET_OBJ_SIZE(dev_ops, ib_dmah);
SET_OBJ_SIZE(dev_ops, ib_mw);
SET_OBJ_SIZE(dev_ops, ib_pd);
@ -3150,6 +3332,7 @@ static void __exit ib_core_cleanup(void)
/* Make sure that any pending umem accounting work is done. */
destroy_workqueue(ib_wq);
destroy_workqueue(ib_unreg_wq);
rcu_barrier();
WARN_ON(!xa_empty(&clients));
WARN_ON(!xa_empty(&devices));
}

3
drivers/infiniband/core/ib_core_uverbs.c
View File

@ -424,6 +424,9 @@ static uverbs_api_ioctl_handler_fn uverbs_get_handler_fn(struct ib_udata *udata)
lockdep_assert_held(&bundle->ufile->device->disassociate_srcu);
if (!bundle->method_elm)
return NULL;
return srcu_dereference(bundle->method_elm->handler,
&bundle->ufile->device->disassociate_srcu);
}

2
drivers/infiniband/core/iwpm_msg.c
View File

@ -268,7 +268,7 @@ int iwpm_add_and_query_mapping(struct iwpm_sa_data *pm_msg, u8 nl_client)
if (ret)
goto query_mapping_error;
/* If flags are required and we're not V4, then return a quite error */
/* If flags are required and we're not V4, then return a quiet error */
if (pm_msg->flags && iwpm_ulib_version == IWPM_UABI_VERSION_MIN) {
ret = -EINVAL;
goto query_mapping_error_nowarn;

69
drivers/infiniband/core/nldev.c
View File

@ -188,6 +188,7 @@ static const struct nla_policy nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
[RDMA_NLDEV_ATTR_FRMR_POOLS_AGING_PERIOD] = { .type = NLA_U32 },
[RDMA_NLDEV_ATTR_FRMR_POOL_PINNED_HANDLES] = { .type = NLA_U32 },
[RDMA_NLDEV_ATTR_FRMR_POOL_KEY_KERNEL_VENDOR_KEY] = { .type = NLA_U64 },
[RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_MAX] = { .type = NLA_U64 },
};
static int put_driver_name_print_type(struct sk_buff *msg, const char *name,
@ -413,7 +414,7 @@ static int fill_port_info(struct sk_buff *msg,
}
static int fill_res_info_entry(struct sk_buff *msg,
const char *name, u64 curr)
const char *name, u64 curr, u64 max)
{
struct nlattr *entry_attr;
@ -427,6 +428,9 @@ static int fill_res_info_entry(struct sk_buff *msg,
if (nla_put_u64_64bit(msg, RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_CURR, curr,
RDMA_NLDEV_ATTR_PAD))
goto err;
if (max && nla_put_u64_64bit(msg, RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY_MAX,
max, RDMA_NLDEV_ATTR_PAD))
goto err;
nla_nest_end(msg, entry_attr);
return 0;
@ -447,10 +451,13 @@ static int fill_res_info(struct sk_buff *msg, struct ib_device *device,
[RDMA_RESTRACK_MR] = "mr",
[RDMA_RESTRACK_CTX] = "ctx",
[RDMA_RESTRACK_SRQ] = "srq",
[RDMA_RESTRACK_COMP_CNTR] = "comp_cntr",
};
struct ib_comp_cntr_caps comp_cntr_caps = {};
struct nlattr *table_attr;
int ret, i, curr;
u64 curr, max;
int ret, i;
if (fill_nldev_handle(msg, device))
return -EMSGSIZE;
@ -459,11 +466,36 @@ static int fill_res_info(struct sk_buff *msg, struct ib_device *device,
if (!table_attr)
return -EMSGSIZE;
if (device->ops.query_comp_cntr_caps)
device->ops.query_comp_cntr_caps(device, &comp_cntr_caps, NULL);
for (i = 0; i < RDMA_RESTRACK_MAX; i++) {
if (!names[i])
continue;
curr = rdma_restrack_count(device, i, show_details);
ret = fill_res_info_entry(msg, names[i], curr);
switch (i) {
case RDMA_RESTRACK_QP:
max = device->attrs.max_qp;
break;
case RDMA_RESTRACK_CQ:
max = device->attrs.max_cq;
break;
case RDMA_RESTRACK_MR:
max = device->attrs.max_mr;
break;
case RDMA_RESTRACK_PD:
max = device->attrs.max_pd;
break;
case RDMA_RESTRACK_SRQ:
max = device->attrs.max_srq;
break;
case RDMA_RESTRACK_COMP_CNTR:
max = comp_cntr_caps.max_counters;
break;
default:
max = 0;
}
ret = fill_res_info_entry(msg, names[i], curr, max);
if (ret)
goto err;
}
@ -1154,6 +1186,24 @@ static int nldev_set_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
if (!device)
return -EINVAL;
if (tb[RDMA_NLDEV_NET_NS_FD]) {
char name[IB_DEVICE_NAME_MAX] = {};
u32 ns_fd;
if (tb[RDMA_NLDEV_ATTR_DEV_NAME]) {
nla_strscpy(name, tb[RDMA_NLDEV_ATTR_DEV_NAME],
IB_DEVICE_NAME_MAX);
if (strlen(name) == 0) {
err = -EINVAL;
goto done;
}
}
ns_fd = nla_get_u32(tb[RDMA_NLDEV_NET_NS_FD]);
err = ib_device_set_netns_put(skb, device, ns_fd,
name[0] ? name : NULL, extack);
goto put_done;
}
if (tb[RDMA_NLDEV_ATTR_DEV_NAME]) {
char name[IB_DEVICE_NAME_MAX] = {};
@ -1167,14 +1217,6 @@ static int nldev_set_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
goto done;
}
if (tb[RDMA_NLDEV_NET_NS_FD]) {
u32 ns_fd;
ns_fd = nla_get_u32(tb[RDMA_NLDEV_NET_NS_FD]);
err = ib_device_set_netns_put(skb, device, ns_fd);
goto put_done;
}
if (tb[RDMA_NLDEV_ATTR_DEV_DIM]) {
u8 use_dim;
@ -2133,6 +2175,11 @@ static int nldev_stat_set_counter_dynamic_doit(struct nlattr *tb[],
nla_for_each_nested(entry_attr, tb[RDMA_NLDEV_ATTR_STAT_HWCOUNTERS],
rem) {
if (nla_len(entry_attr) != sizeof(u32)) {
ret = -EINVAL;
goto out;
}
index = nla_get_u32(entry_attr);
if ((index >= stats->num_counters) ||
!(stats->descs[index].flags & IB_STAT_FLAG_OPTIONAL)) {

1
drivers/infiniband/core/rdma_core.h
View File

@ -159,6 +159,7 @@ void uverbs_user_mmap_disassociate(struct ib_uverbs_file *ufile);
extern const struct uapi_definition uverbs_def_obj_async_fd[];
extern const struct uapi_definition uverbs_def_obj_counters[];
extern const struct uapi_definition uverbs_def_obj_comp_cntr[];
extern const struct uapi_definition uverbs_def_obj_cq[];
extern const struct uapi_definition uverbs_def_obj_device[];
extern const struct uapi_definition uverbs_def_obj_dm[];

169
drivers/infiniband/core/restrack.c
View File

@ -61,7 +61,7 @@ void rdma_restrack_clean(struct ib_device *dev)
* @type: actual type of object to operate
* @show_details: count driver specific objects
*/
int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type,
u32 rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type,
bool show_details)
{
struct rdma_restrack_root *rt = &dev->res[type];
@ -104,6 +104,8 @@ static struct ib_device *res_to_dev(struct rdma_restrack_entry *res)
return container_of(res, struct ib_srq, res)->device;
case RDMA_RESTRACK_DMAH:
return container_of(res, struct ib_dmah, res)->device;
case RDMA_RESTRACK_COMP_CNTR:
return container_of(res, struct ib_comp_cntr, res)->device;
default:
WARN_ONCE(true, "Wrong resource tracking type %u\n", res->type);
return NULL;
@ -129,6 +131,46 @@ static void rdma_restrack_attach_task(struct rdma_restrack_entry *res,
res->user = true;
}
static struct rdma_restrack_root *res_to_rt(struct rdma_restrack_entry *res)
{
struct ib_device *dev = res_to_dev(res);
if (WARN_ON(!dev))
return NULL;
return &dev->res[res->type];
}
static void restrack_drain_res(struct rdma_restrack_root *rt,
struct rdma_restrack_entry *res)
{
if (rt) {
struct rdma_restrack_entry *old;
old = xa_cmpxchg(&rt->xa, res->id, res, XA_ZERO_ENTRY,
GFP_KERNEL);
WARN_ON(old != res);
}
rdma_restrack_put(res);
wait_for_completion(&res->comp);
}
static void restrack_restore_res(struct rdma_restrack_root *rt,
struct rdma_restrack_entry *res)
{
reinit_completion(&res->comp);
kref_init(&res->kref);
if (rt) {
struct rdma_restrack_entry *old;
old = xa_cmpxchg(&rt->xa, res->id, XA_ZERO_ENTRY, res,
GFP_KERNEL);
WARN_ON(old);
}
}
/**
* rdma_restrack_set_name() - set the task for this resource
* @res: resource entry
@ -177,22 +219,23 @@ void rdma_restrack_new(struct rdma_restrack_entry *res,
EXPORT_SYMBOL(rdma_restrack_new);
/**
* rdma_restrack_add() - add object to the resource tracking database
* rdma_restrack_add() - add object to the resource tracking database.
* If this resource reuses an ID of a resource that was already destroyed
* after calling rdma_restrack_begin() but didn't yet call
* rdma_restrack_commit_del() it can result in an untracked QP.
* @res: resource entry
*/
void rdma_restrack_add(struct rdma_restrack_entry *res)
{
struct ib_device *dev = res_to_dev(res);
struct rdma_restrack_root *rt;
int ret = 0;
if (!dev)
return;
if (res->no_track)
goto out;
rt = &dev->res[res->type];
rt = res_to_rt(res);
if (!rt)
return;
if (res->type == RDMA_RESTRACK_QP) {
/* Special case to ensure that LQPN points to right QP */
@ -229,6 +272,28 @@ void rdma_restrack_add(struct rdma_restrack_entry *res)
}
EXPORT_SYMBOL(rdma_restrack_add);
/**
* rdma_restrack_abort_del() - re-add object to the resource tracking database
* it can only be used after rdma_restrack_begin_del().
* @res: resource entry
*/
void rdma_restrack_abort_del(struct rdma_restrack_entry *res)
{
struct rdma_restrack_root *rt = NULL;
if (!res->valid)
return;
if (!res->no_track) {
rt = res_to_rt(res);
if (!rt)
return;
}
restrack_restore_res(rt, res);
}
EXPORT_SYMBOL(rdma_restrack_abort_del);
int __must_check rdma_restrack_get(struct rdma_restrack_entry *res)
{
return kref_get_unless_zero(&res->kref);
@ -265,7 +330,7 @@ static void restrack_release(struct kref *kref)
struct rdma_restrack_entry *res;
res = container_of(kref, struct rdma_restrack_entry, kref);
if (res->task) {
if (res->task && !res->valid) {
put_task_struct(res->task);
res->task = NULL;
}
@ -291,37 +356,20 @@ EXPORT_SYMBOL(rdma_restrack_put);
*/
void rdma_restrack_sync(struct rdma_restrack_entry *res)
{
struct rdma_restrack_entry *old;
struct rdma_restrack_root *rt;
struct task_struct *task;
struct ib_device *dev;
if (!res->valid || res->no_track)
return;
dev = res_to_dev(res);
if (WARN_ON(!dev))
rt = res_to_rt(res);
if (!rt)
return;
rt = &dev->res[res->type];
if (WARN_ON(xa_get_mark(&rt->xa, res->id, RESTRACK_DD)))
return;
old = xa_cmpxchg(&rt->xa, res->id, res, XA_ZERO_ENTRY, GFP_KERNEL);
if (WARN_ON(old != res))
return;
task = res->task;
if (task)
get_task_struct(task);
rdma_restrack_put(res);
wait_for_completion(&res->comp);
reinit_completion(&res->comp);
if (task)
res->task = task;
kref_init(&res->kref);
xa_cmpxchg(&rt->xa, res->id, XA_ZERO_ENTRY, res, GFP_KERNEL);
restrack_drain_res(rt, res);
restrack_restore_res(rt, res);
}
EXPORT_SYMBOL(rdma_restrack_sync);
@ -333,7 +381,6 @@ void rdma_restrack_del(struct rdma_restrack_entry *res)
{
struct rdma_restrack_entry *old;
struct rdma_restrack_root *rt;
struct ib_device *dev;
if (!res->valid) {
if (res->task) {
@ -346,12 +393,10 @@ void rdma_restrack_del(struct rdma_restrack_entry *res)
if (res->no_track)
goto out;
dev = res_to_dev(res);
if (WARN_ON(!dev))
rt = res_to_rt(res);
if (!rt)
return;
rt = &dev->res[res->type];
old = xa_erase(&rt->xa, res->id);
WARN_ON(old != res);
@ -359,5 +404,61 @@ void rdma_restrack_del(struct rdma_restrack_entry *res)
res->valid = false;
rdma_restrack_put(res);
wait_for_completion(&res->comp);
if (res->task) {
put_task_struct(res->task);
res->task = NULL;
}
}
EXPORT_SYMBOL(rdma_restrack_del);
/**
* rdma_restrack_begin_del() - invalidate the object from the resource tracking
* database but preserve its index in the array.
* Since this preserves the index in the array until rdma_restrack_commit_del()
* is called, if rdma_restrack_add() is called in between with an old QP ID it
* can result in an untracked QP.
* @res: resource entry
*/
void rdma_restrack_begin_del(struct rdma_restrack_entry *res)
{
struct rdma_restrack_root *rt = NULL;
if (!res->valid)
return;
if (!res->no_track) {
rt = res_to_rt(res);
if (!rt)
return;
}
restrack_drain_res(rt, res);
}
EXPORT_SYMBOL(rdma_restrack_begin_del);
/**
* rdma_restrack_commit_del() - delete object from the resource tracking
* database and free the task.
* @res: resource entry
*/
void rdma_restrack_commit_del(struct rdma_restrack_entry *res)
{
struct rdma_restrack_root *rt;
if (!res->valid || res->no_track)
goto out;
rt = res_to_rt(res);
if (!rt)
return;
xa_erase(&rt->xa, res->id);
out:
res->valid = false;
if (res->task) {
put_task_struct(res->task);
res->task = NULL;
}
}
EXPORT_SYMBOL(rdma_restrack_commit_del);

3
drivers/infiniband/core/restrack.h
View File

@ -26,8 +26,11 @@ struct rdma_restrack_root {
int rdma_restrack_init(struct ib_device *dev);
void rdma_restrack_clean(struct ib_device *dev);
void rdma_restrack_add(struct rdma_restrack_entry *res);
void rdma_restrack_abort_del(struct rdma_restrack_entry *res);
void rdma_restrack_del(struct rdma_restrack_entry *res);
void rdma_restrack_sync(struct rdma_restrack_entry *res);
void rdma_restrack_begin_del(struct rdma_restrack_entry *res);
void rdma_restrack_commit_del(struct rdma_restrack_entry *res);
void rdma_restrack_new(struct rdma_restrack_entry *res,
enum rdma_restrack_type type);
void rdma_restrack_set_name(struct rdma_restrack_entry *res,

6
drivers/infiniband/core/security.c
View File

@ -700,6 +700,12 @@ int ib_mad_agent_security_setup(struct ib_mad_agent *agent,
if (qp_type != IB_QPT_SMI)
return 0;
/*
* SELinux labels an endport by (device name, port) from a global
* policy. If devices in different net namespaces share a name, they get
* the same label; distinguishing them would need net namespace support
* in the policy language and tooling.
*/
spin_lock(&mad_agent_list_lock);
ret = security_ib_endport_manage_subnet(agent->security,
dev_name(&agent->device->dev),

16
drivers/infiniband/core/ucma.c
View File

@ -951,7 +951,7 @@ static ssize_t ucma_query_path(struct ucma_context *ctx,
resp->num_paths = ctx->cm_id->route.num_pri_alt_paths;
for (i = 0, out_len -= sizeof(*resp);
i < resp->num_paths && out_len > sizeof(struct ib_path_rec_data);
i < resp->num_paths && out_len >= sizeof(struct ib_path_rec_data);
i++, out_len -= sizeof(struct ib_path_rec_data)) {
struct sa_path_rec *rec = &ctx->cm_id->route.path_rec[i];
@ -1404,7 +1404,10 @@ static int ucma_set_ib_path(struct ucma_context *ctx,
memset(&event, 0, sizeof event);
event.event = RDMA_CM_EVENT_ROUTE_RESOLVED;
return ucma_event_handler(ctx->cm_id, &event);
rdma_lock_handler(ctx->cm_id);
ret = ucma_event_handler(ctx->cm_id, &event);
rdma_unlock_handler(ctx->cm_id);
return ret;
}
static int ucma_set_option_ib(struct ucma_context *ctx, int optname,
@ -1776,6 +1779,13 @@ static ssize_t ucma_write_cm_event(struct ucma_file *file,
goto out;
}
rdma_lock_handler(ctx->cm_id);
if (!ctx->uid) {
kfree(uevent);
ret = -EINVAL;
goto err_unlock;
}
uevent->ctx = ctx;
uevent->resp.uid = ctx->uid;
uevent->resp.id = ctx->id;
@ -1789,6 +1799,8 @@ static ssize_t ucma_write_cm_event(struct ucma_file *file,
mutex_unlock(&ctx->file->mut);
wake_up_interruptible(&ctx->file->poll_wait);
err_unlock:
rdma_unlock_handler(ctx->cm_id);
out:
ucma_put_ctx(ctx);
return ret;

8
drivers/infiniband/core/umem.c
View File

@ -209,7 +209,8 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
mmgrab(mm);
page_list = (struct page **) __get_free_page(GFP_KERNEL);
/* TODO: switch to "fast and as large as possible" allocation helper */
page_list = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!page_list) {
ret = -ENOMEM;
goto umem_kfree;
@ -269,7 +270,7 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
__ib_umem_release(device, umem, 0);
atomic64_sub(ib_umem_num_pages(umem), &mm->pinned_vm);
out:
free_page((unsigned long) page_list);
kfree(page_list);
umem_kfree:
if (ret) {
mmdrop(umem->owning_mm);
@ -676,6 +677,9 @@ int ib_umem_check_rereg(struct ib_umem *umem, int flags, int new_access_flags)
if (!umem)
return 0;
if (umem->is_dmabuf)
return -EOPNOTSUPP;
if ((flags & IB_MR_REREG_ACCESS) && !(flags & IB_MR_REREG_TRANS))
if (ib_access_writable(new_access_flags) && !umem->writable)
return -EACCES;

4
drivers/infiniband/core/umem_dmabuf.c
View File

@ -181,7 +181,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
}
EXPORT_SYMBOL(ib_umem_dmabuf_get);
static struct dma_buf_attach_ops ib_umem_dmabuf_attach_pinned_ops = {
static const struct dma_buf_attach_ops ib_umem_dmabuf_attach_pinned_ops = {
.allow_peer2peer = true,
};
@ -205,7 +205,7 @@ static void ib_umem_dmabuf_revoke_locked(struct dma_buf_attachment *attach)
umem_dmabuf->revoked = 1;
}
static struct dma_buf_attach_ops ib_umem_dmabuf_attach_pinned_revocable_ops = {
static const struct dma_buf_attach_ops ib_umem_dmabuf_attach_pinned_revocable_ops = {
.allow_peer2peer = true,
.invalidate_mappings = ib_umem_dmabuf_revoke_locked,
};

1
drivers/infiniband/core/uverbs_main.c
View File

@ -556,6 +556,7 @@ static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf,
bundle.ufile = file;
bundle.context = NULL; /* only valid if bundle has uobject */
bundle.uobject = NULL;
bundle.method_elm = NULL;
if (!method_elm->is_ex) {
size_t in_len = hdr.in_words * 4 - sizeof(hdr);
size_t out_len = hdr.out_words * 4;

172
drivers/infiniband/core/uverbs_std_types_comp_cntr.c Normal file
View File

@ -0,0 +1,172 @@
// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
/*
* Copyright Amazon.com, Inc. or its affiliates. All rights reserved.
*/
#include <rdma/uverbs_std_types.h>
#include "rdma_core.h"
#include "uverbs.h"
#include "restrack.h"
static int uverbs_free_comp_cntr(struct ib_uobject *uobject, enum rdma_remove_reason why,
struct uverbs_attr_bundle *attrs)
{
struct ib_comp_cntr *cc = uobject->object;
int ret;
if (atomic_read(&cc->usecnt))
return -EBUSY;
rdma_restrack_begin_del(&cc->res);
ret = cc->device->ops.destroy_comp_cntr(cc);
if (ret) {
rdma_restrack_abort_del(&cc->res);
return ret;
}
rdma_restrack_commit_del(&cc->res);
kfree(cc);
return 0;
}
static int UVERBS_HANDLER(UVERBS_METHOD_COMP_CNTR_CREATE)(struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *uobj = uverbs_attr_get_uobject(attrs,
UVERBS_ATTR_CREATE_COMP_CNTR_HANDLE);
struct ib_device *ib_dev = attrs->context->device;
struct ib_comp_cntr *cc;
int ret;
if (!ib_dev->ops.create_comp_cntr ||
!ib_dev->ops.destroy_comp_cntr ||
!ib_dev->ops.qp_attach_comp_cntr)
return -EOPNOTSUPP;
cc = rdma_zalloc_drv_obj(ib_dev, ib_comp_cntr);
if (!cc)
return -ENOMEM;
cc->device = ib_dev;
cc->uobject = uobj;
rdma_restrack_new(&cc->res, RDMA_RESTRACK_COMP_CNTR);
rdma_restrack_set_name(&cc->res, NULL);
ret = ib_dev->ops.create_comp_cntr(cc, attrs);
if (ret)
goto err_free;
uobj->object = cc;
rdma_restrack_add(&cc->res);
uverbs_finalize_uobj_create(attrs, UVERBS_ATTR_CREATE_COMP_CNTR_HANDLE);
return 0;
err_free:
rdma_restrack_put(&cc->res);
kfree(cc);
return ret;
}
static int UVERBS_HANDLER(UVERBS_METHOD_COMP_CNTR_MODIFY)(struct uverbs_attr_bundle *attrs)
{
struct ib_comp_cntr *cc = uverbs_attr_get_obj(attrs, UVERBS_ATTR_MODIFY_COMP_CNTR_HANDLE);
enum ib_comp_cntr_modify_op op;
enum ib_comp_cntr_entry entry;
u64 value;
int ret;
if (!cc->device->ops.modify_comp_cntr)
return -EOPNOTSUPP;
ret = uverbs_get_const(&entry, attrs, UVERBS_ATTR_MODIFY_COMP_CNTR_ENTRY);
if (ret)
return ret;
ret = uverbs_get_const(&op, attrs, UVERBS_ATTR_MODIFY_COMP_CNTR_OP);
if (ret)
return ret;
ret = uverbs_copy_from(&value, attrs, UVERBS_ATTR_MODIFY_COMP_CNTR_VALUE);
if (ret)
return ret;
return cc->device->ops.modify_comp_cntr(cc, entry, op, value);
}
static int UVERBS_HANDLER(UVERBS_METHOD_COMP_CNTR_READ)(struct uverbs_attr_bundle *attrs)
{
struct ib_comp_cntr *cc = uverbs_attr_get_obj(attrs, UVERBS_ATTR_READ_COMP_CNTR_HANDLE);
enum ib_comp_cntr_entry entry;
u64 value = 0;
int ret;
if (!cc->device->ops.read_comp_cntr)
return -EOPNOTSUPP;
ret = uverbs_get_const(&entry, attrs, UVERBS_ATTR_READ_COMP_CNTR_ENTRY);
if (ret)
return ret;
ret = cc->device->ops.read_comp_cntr(cc, entry, &value);
if (ret)
return ret;
return uverbs_copy_to(attrs, UVERBS_ATTR_READ_COMP_CNTR_RESP_VALUE, &value, sizeof(value));
}
DECLARE_UVERBS_NAMED_METHOD(
UVERBS_METHOD_COMP_CNTR_CREATE,
UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_COMP_CNTR_HANDLE,
UVERBS_OBJECT_COMP_CNTR,
UVERBS_ACCESS_NEW,
UA_MANDATORY));
DECLARE_UVERBS_NAMED_METHOD_DESTROY(
UVERBS_METHOD_COMP_CNTR_DESTROY,
UVERBS_ATTR_IDR(UVERBS_ATTR_DESTROY_COMP_CNTR_HANDLE,
UVERBS_OBJECT_COMP_CNTR,
UVERBS_ACCESS_DESTROY,
UA_MANDATORY));
DECLARE_UVERBS_NAMED_METHOD(
UVERBS_METHOD_COMP_CNTR_MODIFY,
UVERBS_ATTR_IDR(UVERBS_ATTR_MODIFY_COMP_CNTR_HANDLE,
UVERBS_OBJECT_COMP_CNTR,
UVERBS_ACCESS_WRITE,
UA_MANDATORY),
UVERBS_ATTR_CONST_IN(UVERBS_ATTR_MODIFY_COMP_CNTR_ENTRY,
enum ib_uverbs_comp_cntr_entry,
UA_MANDATORY),
UVERBS_ATTR_CONST_IN(UVERBS_ATTR_MODIFY_COMP_CNTR_OP,
enum ib_uverbs_comp_cntr_modify_op,
UA_MANDATORY),
UVERBS_ATTR_PTR_IN(UVERBS_ATTR_MODIFY_COMP_CNTR_VALUE,
UVERBS_ATTR_TYPE(u64),
UA_MANDATORY));
DECLARE_UVERBS_NAMED_METHOD(
UVERBS_METHOD_COMP_CNTR_READ,
UVERBS_ATTR_IDR(UVERBS_ATTR_READ_COMP_CNTR_HANDLE,
UVERBS_OBJECT_COMP_CNTR,
UVERBS_ACCESS_READ,
UA_MANDATORY),
UVERBS_ATTR_CONST_IN(UVERBS_ATTR_READ_COMP_CNTR_ENTRY,
enum ib_uverbs_comp_cntr_entry,
UA_MANDATORY),
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_READ_COMP_CNTR_RESP_VALUE,
UVERBS_ATTR_TYPE(u64),
UA_MANDATORY));
DECLARE_UVERBS_NAMED_OBJECT(
UVERBS_OBJECT_COMP_CNTR,
UVERBS_TYPE_ALLOC_IDR(uverbs_free_comp_cntr),
&UVERBS_METHOD(UVERBS_METHOD_COMP_CNTR_CREATE),
&UVERBS_METHOD(UVERBS_METHOD_COMP_CNTR_DESTROY),
&UVERBS_METHOD(UVERBS_METHOD_COMP_CNTR_MODIFY),
&UVERBS_METHOD(UVERBS_METHOD_COMP_CNTR_READ));
const struct uapi_definition uverbs_def_obj_comp_cntr[] = {
UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_COMP_CNTR,
UAPI_DEF_OBJ_NEEDS_FN(destroy_comp_cntr)),
{}
};

51
drivers/infiniband/core/uverbs_std_types_device.c
View File

@ -472,6 +472,42 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_GID_ENTRY)(
return ret;
}
static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_COMP_CNTR_CAPS)(
struct uverbs_attr_bundle *attrs)
{
struct ib_comp_cntr_caps caps = {};
struct ib_ucontext *ucontext;
struct ib_device *ib_dev;
int ret;
ucontext = ib_uverbs_get_ucontext(attrs);
if (IS_ERR(ucontext))
return PTR_ERR(ucontext);
ib_dev = ucontext->device;
if (!ib_dev->ops.query_comp_cntr_caps)
return -EOPNOTSUPP;
ret = ib_dev->ops.query_comp_cntr_caps(ib_dev, &caps, attrs);
if (ret)
return ret;
ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_COMP_CNTR_CAPS_MAX_COUNTERS,
&caps.max_counters, sizeof(caps.max_counters));
if (IS_UVERBS_COPY_ERR(ret))
return ret;
ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_COMP_CNTR_CAPS_MAX_VALUE,
&caps.max_value, sizeof(caps.max_value));
if (IS_UVERBS_COPY_ERR(ret))
return ret;
ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_COMP_CNTR_CAPS_SUPPORTED_QP_ATTACH_OPS,
&caps.supported_qp_attach_ops,
sizeof(caps.supported_qp_attach_ops));
return IS_UVERBS_COPY_ERR(ret) ? ret : 0;
}
DECLARE_UVERBS_NAMED_METHOD(
UVERBS_METHOD_GET_CONTEXT,
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_GET_CONTEXT_NUM_COMP_VECTORS,
@ -542,6 +578,18 @@ DECLARE_UVERBS_NAMED_METHOD(
netdev_ifindex),
UA_MANDATORY));
DECLARE_UVERBS_NAMED_METHOD(
UVERBS_METHOD_QUERY_COMP_CNTR_CAPS,
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_COMP_CNTR_CAPS_MAX_COUNTERS,
UVERBS_ATTR_TYPE(u32),
UA_OPTIONAL),
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_COMP_CNTR_CAPS_MAX_VALUE,
UVERBS_ATTR_TYPE(u64),
UA_OPTIONAL),
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_COMP_CNTR_CAPS_SUPPORTED_QP_ATTACH_OPS,
UVERBS_ATTR_TYPE(u32),
UA_OPTIONAL));
DECLARE_UVERBS_GLOBAL_METHODS(UVERBS_OBJECT_DEVICE,
&UVERBS_METHOD(UVERBS_METHOD_GET_CONTEXT),
&UVERBS_METHOD(UVERBS_METHOD_INVOKE_WRITE),
@ -550,7 +598,8 @@ DECLARE_UVERBS_GLOBAL_METHODS(UVERBS_OBJECT_DEVICE,
&UVERBS_METHOD(UVERBS_METHOD_QUERY_PORT_SPEED),
&UVERBS_METHOD(UVERBS_METHOD_QUERY_CONTEXT),
&UVERBS_METHOD(UVERBS_METHOD_QUERY_GID_TABLE),
&UVERBS_METHOD(UVERBS_METHOD_QUERY_GID_ENTRY));
&UVERBS_METHOD(UVERBS_METHOD_QUERY_GID_ENTRY),
&UVERBS_METHOD(UVERBS_METHOD_QUERY_COMP_CNTR_CAPS));
const struct uapi_definition uverbs_def_obj_device[] = {
UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_DEVICE),

7
drivers/infiniband/core/uverbs_std_types_dmah.c
View File

@ -18,11 +18,14 @@ static int uverbs_free_dmah(struct ib_uobject *uobject,
if (atomic_read(&dmah->usecnt))
return -EBUSY;
rdma_restrack_begin_del(&dmah->res);
ret = dmah->device->ops.dealloc_dmah(dmah, attrs);
if (ret)
if (ret) {
rdma_restrack_abort_del(&dmah->res);
return ret;
}
rdma_restrack_del(&dmah->res);
rdma_restrack_commit_del(&dmah->res);
kfree(dmah);
return 0;
}

6
drivers/infiniband/core/uverbs_std_types_mr.c
View File

@ -364,7 +364,8 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
dmah, attrs);
else
mr = pd->device->ops.reg_user_mr(pd, addr, length, iova,
access_flags, dmah, NULL);
access_flags, dmah,
&attrs->driver_udata);
if (IS_ERR(mr))
return PTR_ERR(mr);
@ -527,7 +528,8 @@ DECLARE_UVERBS_NAMED_METHOD(
UA_MANDATORY),
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_REG_MR_RESP_RKEY,
UVERBS_ATTR_TYPE(u32),
UA_MANDATORY));
UA_MANDATORY),
UVERBS_ATTR_UHW());
DECLARE_UVERBS_NAMED_METHOD_DESTROY(
UVERBS_METHOD_MR_DESTROY,

67
drivers/infiniband/core/uverbs_std_types_qp.c
View File

@ -372,11 +372,76 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_TYPE(struct ib_uverbs_destroy_qp_resp),
UA_MANDATORY));
static int UVERBS_HANDLER(UVERBS_METHOD_QP_ATTACH_COMP_CNTR)(
struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *qp_uobj = uverbs_attr_get_uobject(
attrs, UVERBS_ATTR_QP_ATTACH_COMP_CNTR_HANDLE);
struct ib_comp_cntr *cc = uverbs_attr_get_obj(
attrs, UVERBS_ATTR_QP_ATTACH_COMP_CNTR_CNTR_HANDLE);
struct ib_qp_attach_comp_cntr_attr attr = {};
struct ib_qp *qp = qp_uobj->object;
int ret;
if (!cc->device->ops.qp_attach_comp_cntr)
return -EOPNOTSUPP;
if (qp->real_qp != qp)
return -EINVAL;
ret = uverbs_get_flags32(&attr.op_mask, attrs,
UVERBS_ATTR_QP_ATTACH_COMP_CNTR_OP_MASK,
IB_UVERBS_QP_ATTACH_COMP_CNTR_OP_SEND |
IB_UVERBS_QP_ATTACH_COMP_CNTR_OP_RECV |
IB_UVERBS_QP_ATTACH_COMP_CNTR_OP_RDMA_READ |
IB_UVERBS_QP_ATTACH_COMP_CNTR_OP_REMOTE_RDMA_READ |
IB_UVERBS_QP_ATTACH_COMP_CNTR_OP_RDMA_WRITE |
IB_UVERBS_QP_ATTACH_COMP_CNTR_OP_REMOTE_RDMA_WRITE);
if (ret)
return ret;
if (!attr.op_mask)
return -EINVAL;
if (attr.op_mask & qp->comp_cntr_op_mask)
return -EBUSY;
ret = xa_err(xa_store(&qp->comp_cntrs, attr.op_mask, cc, GFP_KERNEL));
if (ret)
return ret;
ret = qp->device->ops.qp_attach_comp_cntr(qp, cc, &attr);
if (ret) {
xa_erase(&qp->comp_cntrs, attr.op_mask);
return ret;
}
atomic_inc(&cc->usecnt);
qp->comp_cntr_op_mask |= attr.op_mask;
return 0;
}
DECLARE_UVERBS_NAMED_METHOD(
UVERBS_METHOD_QP_ATTACH_COMP_CNTR,
UVERBS_ATTR_IDR(UVERBS_ATTR_QP_ATTACH_COMP_CNTR_HANDLE,
UVERBS_OBJECT_QP,
UVERBS_ACCESS_WRITE,
UA_MANDATORY),
UVERBS_ATTR_IDR(UVERBS_ATTR_QP_ATTACH_COMP_CNTR_CNTR_HANDLE,
UVERBS_OBJECT_COMP_CNTR,
UVERBS_ACCESS_READ,
UA_MANDATORY),
UVERBS_ATTR_FLAGS_IN(UVERBS_ATTR_QP_ATTACH_COMP_CNTR_OP_MASK,
enum ib_uverbs_qp_attach_comp_cntr_op,
UA_MANDATORY));
DECLARE_UVERBS_NAMED_OBJECT(
UVERBS_OBJECT_QP,
UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_uqp_object), uverbs_free_qp),
&UVERBS_METHOD(UVERBS_METHOD_QP_CREATE),
&UVERBS_METHOD(UVERBS_METHOD_QP_DESTROY));
&UVERBS_METHOD(UVERBS_METHOD_QP_DESTROY),
&UVERBS_METHOD(UVERBS_METHOD_QP_ATTACH_COMP_CNTR));
const struct uapi_definition uverbs_def_obj_qp[] = {
UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_QP,

2
drivers/infiniband/core/uverbs_std_types_srq.c
View File

@ -192,6 +192,8 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_CREATE_SRQ_RESP_SRQ_NUM,
UVERBS_ATTR_TYPE(u32),
UA_OPTIONAL),
UVERBS_ATTR_UMEM(UVERBS_ATTR_CREATE_SRQ_BUF_UMEM,
UA_OPTIONAL),
UVERBS_ATTR_UHW());
static int UVERBS_HANDLER(UVERBS_METHOD_SRQ_DESTROY)(

1
drivers/infiniband/core/uverbs_uapi.c
View File

@ -628,6 +628,7 @@ void uverbs_destroy_api(struct uverbs_api *uapi)
static const struct uapi_definition uverbs_core_api[] = {
UAPI_DEF_CHAIN(uverbs_def_obj_async_fd),
UAPI_DEF_CHAIN(uverbs_def_obj_counters),
UAPI_DEF_CHAIN(uverbs_def_obj_comp_cntr),
UAPI_DEF_CHAIN(uverbs_def_obj_cq),
UAPI_DEF_CHAIN(uverbs_def_obj_device),
UAPI_DEF_CHAIN(uverbs_def_obj_dm),

40
drivers/infiniband/core/verbs.c
View File

@ -392,6 +392,7 @@ int ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata)
{
int ret;
rdma_restrack_begin_del(&pd->res);
if (pd->__internal_mr) {
ret = pd->device->ops.dereg_mr(pd->__internal_mr, NULL);
WARN_ON(ret);
@ -399,10 +400,12 @@ int ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata)
}
ret = pd->device->ops.dealloc_pd(pd, udata);
if (ret)
if (ret) {
rdma_restrack_abort_del(&pd->res);
return ret;
}
rdma_restrack_del(&pd->res);
rdma_restrack_commit_del(&pd->res);
kfree(pd);
return ret;
}
@ -1140,16 +1143,20 @@ int ib_destroy_srq_user(struct ib_srq *srq, struct ib_udata *udata)
if (atomic_read(&srq->usecnt))
return -EBUSY;
rdma_restrack_begin_del(&srq->res);
ret = srq->device->ops.destroy_srq(srq, udata);
if (ret)
if (ret) {
rdma_restrack_abort_del(&srq->res);
return ret;
}
atomic_dec(&srq->pd->usecnt);
if (srq->srq_type == IB_SRQT_XRC && srq->ext.xrc.xrcd)
atomic_dec(&srq->ext.xrc.xrcd->usecnt);
if (ib_srq_has_cq(srq->srq_type))
atomic_dec(&srq->ext.cq->usecnt);
rdma_restrack_del(&srq->res);
rdma_restrack_commit_del(&srq->res);
kfree(srq);
return ret;
@ -1293,6 +1300,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,
qp->qp_context = attr->qp_context;
spin_lock_init(&qp->mr_lock);
xa_init(&qp->comp_cntrs);
INIT_LIST_HEAD(&qp->rdma_mrs);
INIT_LIST_HEAD(&qp->sig_mrs);
init_completion(&qp->srq_completion);
@ -1327,6 +1335,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,
qp, uattrs ? uverbs_get_cleared_udata(uattrs) : NULL);
err_create:
rdma_restrack_put(&qp->res);
xa_destroy(&qp->comp_cntrs);
kfree(qp);
return ERR_PTR(ret);
@ -2144,6 +2153,8 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
const struct ib_gid_attr *alt_path_sgid_attr = qp->alt_path_sgid_attr;
const struct ib_gid_attr *av_sgid_attr = qp->av_sgid_attr;
struct ib_qp_security *sec;
struct ib_comp_cntr *cc;
unsigned long index;
int ret;
WARN_ON_ONCE(qp->mrs_used > 0);
@ -2154,6 +2165,8 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
if (qp->real_qp != qp)
return __ib_destroy_shared_qp(qp);
rdma_restrack_begin_del(&qp->res);
sec = qp->qp_sec;
if (sec)
ib_destroy_qp_security_begin(sec);
@ -2166,6 +2179,7 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
if (ret) {
if (sec)
ib_destroy_qp_security_abort(sec);
rdma_restrack_abort_del(&qp->res);
return ret;
}
@ -2174,11 +2188,15 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
if (av_sgid_attr)
rdma_put_gid_attr(av_sgid_attr);
xa_for_each(&qp->comp_cntrs, index, cc)
atomic_dec(&cc->usecnt);
xa_destroy(&qp->comp_cntrs);
ib_qp_usecnt_dec(qp);
if (sec)
ib_destroy_qp_security_end(sec);
rdma_restrack_del(&qp->res);
rdma_restrack_commit_del(&qp->res);
kfree(qp);
return ret;
}
@ -2244,11 +2262,15 @@ int ib_destroy_cq_user(struct ib_cq *cq, struct ib_udata *udata)
if (atomic_read(&cq->usecnt))
return -EBUSY;
ret = cq->device->ops.destroy_cq(cq, udata);
if (ret)
return ret;
rdma_restrack_begin_del(&cq->res);
rdma_restrack_del(&cq->res);
ret = cq->device->ops.destroy_cq(cq, udata);
if (ret) {
rdma_restrack_abort_del(&cq->res);
return ret;
}
rdma_restrack_commit_del(&cq->res);
kfree(cq);
return ret;
}

10
drivers/infiniband/hw/bng_re/bng_fw.c
View File

@ -401,14 +401,15 @@ static int __wait_for_resp(struct bng_re_rcfw *rcfw, u16 cookie)
{
struct bng_re_cmdq_ctx *cmdq;
struct bng_re_crsqe *crsqe;
unsigned long time_left;
cmdq = &rcfw->cmdq;
crsqe = &rcfw->crsqe_tbl[cookie];
do {
wait_event_timeout(cmdq->waitq,
!crsqe->is_in_used,
secs_to_jiffies(rcfw->max_timeout));
time_left = wait_event_timeout(cmdq->waitq,
!crsqe->is_in_used,
secs_to_jiffies(rcfw->max_timeout));
if (!crsqe->is_in_used)
return 0;
@ -417,6 +418,9 @@ static int __wait_for_resp(struct bng_re_rcfw *rcfw, u16 cookie)
if (!crsqe->is_in_used)
return 0;
if (!time_left)
return -ENODEV;
} while (true);
};

6
drivers/infiniband/hw/bnxt_re/bnxt_re.h
View File

@ -41,7 +41,6 @@
#define __BNXT_RE_H__
#include <rdma/uverbs_ioctl.h>
#include "hw_counters.h"
#include <linux/hashtable.h>
#define ROCE_DRV_MODULE_NAME "bnxt_re"
#define BNXT_RE_DESC "Broadcom NetXtreme-C/E RoCE Driver"
@ -158,9 +157,6 @@ struct bnxt_re_nq_record {
struct mutex load_lock;
};
#define MAX_CQ_HASH_BITS (16)
#define MAX_SRQ_HASH_BITS (16)
static inline bool bnxt_re_chip_gen_p7(u16 chip_num)
{
return (chip_num == CHIP_NUM_58818 ||
@ -215,8 +211,6 @@ struct bnxt_re_dev {
struct bnxt_re_pacing pacing;
struct work_struct dbq_fifo_check_work;
struct delayed_work dbq_pacing_work;
DECLARE_HASHTABLE(cq_hash, MAX_CQ_HASH_BITS);
DECLARE_HASHTABLE(srq_hash, MAX_SRQ_HASH_BITS);
struct dentry *dbg_root;
struct dentry *qp_debugfs;
unsigned long event_bitmap;

31
drivers/infiniband/hw/bnxt_re/debugfs.c
View File

@ -308,21 +308,12 @@ static ssize_t bnxt_re_cc_config_set(struct file *filp, const char __user *buffe
struct bnxt_re_dev *rdev = dbg_cc_param->rdev;
u32 offset = dbg_cc_param->offset;
u8 cc_gen = dbg_cc_param->cc_gen;
char buf[16];
u32 val;
int rc;
if (count >= sizeof(buf))
return -EINVAL;
if (copy_from_user(buf, buffer, count))
return -EFAULT;
buf[count] = '\0';
if (kstrtou32(buf, 0, &val))
return -EINVAL;
rc = bnxt_re_configure_cc(rdev, cc_gen, offset, val);
rc = kstrtou32_from_user(buffer, count, 0, &val);
if (!rc)
rc = bnxt_re_configure_cc(rdev, cc_gen, offset, val);
return rc ? rc : count;
}
@ -374,20 +365,12 @@ static ssize_t cq_coal_cfg_write(struct file *file,
struct seq_file *s = file->private_data;
struct bnxt_re_cq_coal_param *param = s->private;
struct bnxt_re_dev *rdev = param->rdev;
int offset = param->offset;
char lbuf[16] = { };
int ret, offset = param->offset;
u32 val;
if (count > sizeof(lbuf))
return -EINVAL;
if (copy_from_user(lbuf, buf, count))
return -EFAULT;
lbuf[sizeof(lbuf) - 1] = '\0';
if (kstrtou32(lbuf, 0, &val))
return -EINVAL;
ret = kstrtou32_from_user(buf, count, 0, &val);
if (ret)
return ret;
switch (offset) {
case BNXT_RE_COAL_CQ_BUF_MAXTIME:

191
drivers/infiniband/hw/bnxt_re/ib_verbs.c
View File

@ -193,7 +193,6 @@ int bnxt_re_query_device(struct ib_device *ibdev,
if (rc)
return rc;
memset(ib_attr, 0, sizeof(*ib_attr));
memcpy(&ib_attr->fw_ver, dev_attr->fw_ver,
min(sizeof(dev_attr->fw_ver),
sizeof(ib_attr->fw_ver)));
@ -695,7 +694,7 @@ int bnxt_re_dealloc_pd(struct ib_pd *ib_pd, struct ib_udata *udata)
struct bnxt_re_dev *rdev = pd->rdev;
int ret;
ret = ib_is_udata_in_empty(udata);
ret = ib_no_udata_io(udata);
if (ret)
return ret;
@ -712,7 +711,7 @@ int bnxt_re_dealloc_pd(struct ib_pd *ib_pd, struct ib_udata *udata)
&pd->qplib_pd))
atomic_dec(&rdev->stats.res.pd_count);
}
return ib_respond_empty_udata(udata);
return 0;
}
int bnxt_re_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
@ -844,7 +843,7 @@ int bnxt_re_create_ah(struct ib_ah *ib_ah, struct rdma_ah_init_attr *init_attr,
u8 nw_type;
int rc;
rc = ib_is_udata_in_empty(udata);
rc = ib_no_udata_io(udata);
if (rc)
return rc;
@ -901,7 +900,7 @@ int bnxt_re_create_ah(struct ib_ah *ib_ah, struct rdma_ah_init_attr *init_attr,
if (active_ahs > rdev->stats.res.ah_watermark)
rdev->stats.res.ah_watermark = active_ahs;
return ib_respond_empty_udata(udata);
return 0;
}
int bnxt_re_query_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr)
@ -1015,7 +1014,7 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
unsigned int flags;
int rc;
rc = ib_is_udata_in_empty(udata);
rc = ib_no_udata_io(udata);
if (rc)
return rc;
@ -1064,7 +1063,7 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
if (scq_nq != rcq_nq)
bnxt_re_synchronize_nq(rcq_nq);
return ib_respond_empty_udata(udata);
return 0;
}
static u8 __from_ib_qp_type(enum ib_qp_type type)
@ -2148,18 +2147,33 @@ int bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata)
struct bnxt_qplib_srq *qplib_srq = &srq->qplib_srq;
int ret;
ret = ib_is_udata_in_empty(udata);
ret = ib_no_udata_io(udata);
if (ret)
return ret;
if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT)
hash_del(&srq->hash_entry);
if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) {
struct bnxt_re_ucontext *uctx =
rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx);
/*
* Untrack the SRQ before releasing its hardware ID below, so a
* concurrent create that gets the same ID reused by firmware
* cannot have its fresh XArray entry erased by this destroy.
*/
if (uctx)
xa_erase(&uctx->srq_xa, srq->qplib_srq.id);
}
bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq);
if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT)
free_page((unsigned long)srq->uctx_srq_page);
if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) {
struct bnxt_re_ucontext *uctx =
rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx);
if (uctx)
rdma_user_mmap_entry_remove(&srq->toggle_entry->rdma_entry);
}
ib_umem_release(srq->umem);
atomic_dec(&rdev->stats.res.srq_count);
return ib_respond_empty_udata(udata);
return 0;
}
static int bnxt_re_init_user_srq(struct bnxt_re_dev *rdev,
@ -2263,20 +2277,28 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
resp.srqid = srq->qplib_srq.id;
if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) {
hash_add(rdev->srq_hash, &srq->hash_entry, srq->qplib_srq.id);
srq->uctx_srq_page = (void *)get_zeroed_page(GFP_KERNEL);
if (!srq->uctx_srq_page) {
rc = -ENOMEM;
goto fail;
goto fail_destroy_srq;
}
srq->toggle_entry = bnxt_re_mmap_entry_insert(uctx, (u64)srq->uctx_srq_page,
BNXT_RE_MMAP_TOGGLE_PAGE,
NULL);
if (!srq->toggle_entry) {
rc = -ENOMEM;
goto fail_free_srq_page;
}
if (xa_is_err(xa_store(&uctx->srq_xa, srq->qplib_srq.id,
ib_srq->uobject, GFP_KERNEL))) {
rc = -ENOMEM;
goto fail_remove_toggle_entry;
}
resp.comp_mask |= BNXT_RE_SRQ_TOGGLE_PAGE_SUPPORT;
}
rc = ib_respond_udata(udata, resp);
if (rc) {
bnxt_qplib_destroy_srq(&rdev->qplib_res,
&srq->qplib_srq);
goto fail;
}
if (rc)
goto fail_respond;
}
active_srqs = atomic_inc_return(&rdev->stats.res.srq_count);
if (active_srqs > rdev->stats.res.srq_watermark)
@ -2285,6 +2307,20 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
return 0;
fail_respond:
if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) {
xa_erase(&uctx->srq_xa, srq->qplib_srq.id);
goto fail_remove_toggle_entry;
}
bnxt_qplib_destroy_srq(&rdev->qplib_res, &srq->qplib_srq);
goto fail;
fail_remove_toggle_entry:
rdma_user_mmap_entry_remove(&srq->toggle_entry->rdma_entry);
goto fail_destroy_srq;
fail_free_srq_page:
free_page((unsigned long)srq->uctx_srq_page);
fail_destroy_srq:
bnxt_qplib_destroy_srq(&rdev->qplib_res, &srq->qplib_srq);
fail:
ib_umem_release(srq->umem);
exit:
@ -2297,34 +2333,25 @@ int bnxt_re_modify_srq(struct ib_srq *ib_srq, struct ib_srq_attr *srq_attr,
{
struct bnxt_re_srq *srq = container_of(ib_srq, struct bnxt_re_srq,
ib_srq);
struct bnxt_re_dev *rdev = srq->rdev;
int ret;
ret = ib_is_udata_in_empty(udata);
ret = ib_no_udata_io(udata);
if (ret)
return ret;
switch (srq_attr_mask) {
case IB_SRQ_MAX_WR:
/* SRQ resize is not supported */
if (srq_attr_mask != IB_SRQ_LIMIT)
return -EINVAL;
case IB_SRQ_LIMIT:
/* Change the SRQ threshold */
if (srq_attr->srq_limit > srq->qplib_srq.max_wqe)
return -EINVAL;
srq->qplib_srq.threshold = srq_attr->srq_limit;
bnxt_qplib_srq_arm_db(&srq->qplib_srq.dbinfo, srq->qplib_srq.threshold);
/* On success, update the shadow */
srq->srq_limit = srq_attr->srq_limit;
/* No need to Build and send response back to udata */
return ib_respond_empty_udata(udata);
default:
ibdev_err(&rdev->ibdev,
"Unsupported srq_attr_mask 0x%x", srq_attr_mask);
if (srq_attr->srq_limit > srq->qplib_srq.max_wqe)
return -EINVAL;
}
srq->qplib_srq.threshold = srq_attr->srq_limit;
bnxt_qplib_srq_arm_db(&srq->qplib_srq.dbinfo, srq->qplib_srq.threshold);
/* On success, update the shadow */
srq->srq_limit = srq_attr->srq_limit;
/* No need to Build and send response back to udata */
return 0;
}
int bnxt_re_query_srq(struct ib_srq *ib_srq, struct ib_srq_attr *srq_attr)
@ -2437,7 +2464,7 @@ int bnxt_re_modify_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr,
unsigned int flags;
u8 nw_type;
rc = ib_is_udata_in_empty(udata);
rc = ib_no_udata_io(udata);
if (rc)
return rc;
@ -2689,7 +2716,7 @@ int bnxt_re_modify_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr,
if (rc)
return rc;
}
return ib_respond_empty_udata(udata);
return 0;
}
int bnxt_re_query_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr,
@ -3471,22 +3498,37 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
nq = cq->qplib_cq.nq;
cctx = rdev->chip_ctx;
ret = ib_is_udata_in_empty(udata);
ret = ib_no_udata_io(udata);
if (ret)
return ret;
if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
hash_del(&cq->hash_entry);
if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT) {
struct bnxt_re_ucontext *uctx =
rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx);
/*
* Untrack the CQ before releasing its hardware ID below, so a
* concurrent create that gets the same ID reused by firmware
* cannot have its fresh XArray entry erased by this destroy.
*/
if (uctx)
xa_erase(&uctx->cq_xa, cq->qplib_cq.id);
}
bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
free_page((unsigned long)cq->uctx_cq_page);
if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT) {
struct bnxt_re_ucontext *uctx =
rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx);
if (uctx)
rdma_user_mmap_entry_remove(&cq->toggle_entry->rdma_entry);
}
bnxt_re_put_nq(rdev, nq);
atomic_dec(&rdev->stats.res.cq_count);
kfree(cq->cql);
ib_umem_release(cq->umem);
return ib_respond_empty_udata(udata);
return 0;
}
int bnxt_re_create_user_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
@ -3554,14 +3596,22 @@ int bnxt_re_create_user_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *att
spin_lock_init(&cq->cq_lock);
if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT) {
hash_add(rdev->cq_hash, &cq->hash_entry, cq->qplib_cq.id);
/* Allocate a page */
cq->uctx_cq_page = (void *)get_zeroed_page(GFP_KERNEL);
if (!cq->uctx_cq_page) {
rc = -ENOMEM;
goto destroy_cq;
}
cq->toggle_entry = bnxt_re_mmap_entry_insert(uctx, (u64)cq->uctx_cq_page,
BNXT_RE_MMAP_TOGGLE_PAGE, NULL);
if (!cq->toggle_entry) {
rc = -ENOMEM;
goto free_cq_page;
}
if (xa_is_err(xa_store(&uctx->cq_xa, cq->qplib_cq.id,
ibcq->uobject, GFP_KERNEL))) {
rc = -ENOMEM;
goto remove_toggle_entry;
}
resp.comp_mask |= BNXT_RE_CQ_TOGGLE_PAGE_SUPPORT;
}
resp.cqid = cq->qplib_cq.id;
@ -3574,6 +3624,13 @@ int bnxt_re_create_user_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *att
return 0;
free_mem:
if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
xa_erase(&uctx->cq_xa, cq->qplib_cq.id);
remove_toggle_entry:
if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
rdma_user_mmap_entry_remove(&cq->toggle_entry->rdma_entry);
goto destroy_cq;
free_cq_page:
free_page((unsigned long)cq->uctx_cq_page);
destroy_cq:
bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
@ -3688,6 +3745,10 @@ int bnxt_re_resize_cq(struct ib_cq *ibcq, unsigned int cqe,
if (rc)
goto fail;
rc = ib_respond_empty_udata(udata);
if (rc)
goto fail;
cq->resize_umem = ib_umem_get_va(&rdev->ibdev, req.cq_va,
entries * sizeof(struct cq_base),
IB_ACCESS_LOCAL_WRITE);
@ -3717,7 +3778,7 @@ int bnxt_re_resize_cq(struct ib_cq *ibcq, unsigned int cqe,
cq->ib_cq.cqe = cq->resize_cqe;
atomic_inc(&rdev->stats.res.resize_count);
return ib_respond_empty_udata(udata);
return 0;
fail:
if (cq->resize_umem) {
@ -4449,7 +4510,7 @@ int bnxt_re_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
struct bnxt_re_dev *rdev = mr->rdev;
int rc;
rc = ib_is_udata_in_empty(udata);
rc = ib_no_udata_io(udata);
if (rc)
return rc;
@ -4472,7 +4533,7 @@ int bnxt_re_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
atomic_dec(&rdev->stats.res.mr_count);
if (rc)
return rc;
return ib_respond_empty_udata(udata);
return 0;
}
static int bnxt_re_set_page(struct ib_mr *ib_mr, u64 addr)
@ -4795,6 +4856,8 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata)
goto cfail;
}
uctx->shpage_mmap = &entry->rdma_entry;
xa_init(&uctx->cq_xa);
xa_init(&uctx->srq_xa);
if (rdev->pacing.dbr_pacing)
resp.comp_mask |= BNXT_RE_UCNTX_CMASK_DBR_PACING_ENABLED;
@ -4804,6 +4867,8 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata)
if (_is_modify_qp_rate_limit_supported(dev_attr->dev_cap_flags2))
resp.comp_mask |= BNXT_RE_UCNTX_CMASK_QP_RATE_LIMIT_ENABLED;
resp.comp_mask |= BNXT_RE_UCNTX_CMASK_TOGGLE_MEM_UOBJ_SUPPORT;
if (udata->inlen) {
rc = ib_copy_validate_udata_in_cm(
udata, ureq, comp_mask,
@ -4847,6 +4912,8 @@ void bnxt_re_dealloc_ucontext(struct ib_ucontext *ib_uctx)
uctx->shpage_mmap = NULL;
if (uctx->shpg)
free_page((unsigned long)uctx->shpg);
xa_destroy(&uctx->cq_xa);
xa_destroy(&uctx->srq_xa);
if (uctx->dpi.dbr) {
/* Free DPI only if this is the first PD allocated by the
@ -4991,11 +5058,13 @@ int bnxt_re_mmap(struct ib_ucontext *ib_uctx, struct vm_area_struct *vma)
case BNXT_RE_MMAP_DBR_PAGE:
case BNXT_RE_MMAP_TOGGLE_PAGE:
/* Driver doesn't expect write access for user space */
if (vma->vm_flags & VM_WRITE)
if (vma->vm_flags & VM_WRITE) {
ret = -EFAULT;
else
} else {
vm_flags_clear(vma, VM_MAYWRITE);
ret = vm_insert_page(vma, vma->vm_start,
virt_to_page((void *)bnxt_entry->mem_offset));
}
break;
default:
ret = -EINVAL;
@ -5013,6 +5082,16 @@ void bnxt_re_mmap_free(struct rdma_user_mmap_entry *rdma_entry)
bnxt_entry = container_of(rdma_entry, struct bnxt_re_user_mmap_entry,
rdma_entry);
/*
* For toggle pages the kernel VA was stored directly in mem_offset
* at creation time (bnxt_re_create_user_cq / bnxt_re_create_srq).
* Free it here — this is the only place it is freed, ensuring the
* page outlives every concurrent bnxt_re_mmap() call that may have
* incremented the entry's reference count.
*/
if (bnxt_entry->mmap_flag == BNXT_RE_MMAP_TOGGLE_PAGE)
free_page((unsigned long)bnxt_entry->mem_offset);
if (bnxt_entry->dpi_valid)
bnxt_qplib_free_uc_dpi(&bnxt_entry->uctx->rdev->qplib_res,
&bnxt_entry->dpi);

8
drivers/infiniband/hw/bnxt_re/ib_verbs.h
View File

@ -70,6 +70,8 @@ struct bnxt_re_ah {
struct bnxt_qplib_ah qplib_ah;
};
struct bnxt_re_user_mmap_entry;
struct bnxt_re_srq {
struct ib_srq ib_srq;
struct bnxt_re_dev *rdev;
@ -78,7 +80,7 @@ struct bnxt_re_srq {
struct ib_umem *umem;
spinlock_t lock; /* protect srq */
void *uctx_srq_page;
struct hlist_node hash_entry;
struct bnxt_re_user_mmap_entry *toggle_entry;
};
struct bnxt_re_qp {
@ -113,7 +115,7 @@ struct bnxt_re_cq {
struct ib_umem *resize_umem;
int resize_cqe;
void *uctx_cq_page;
struct hlist_node hash_entry;
struct bnxt_re_user_mmap_entry *toggle_entry;
};
struct bnxt_re_mr {
@ -147,6 +149,8 @@ struct bnxt_re_ucontext {
void *shpg;
spinlock_t sh_lock; /* protect shpg */
struct rdma_user_mmap_entry *shpage_mmap;
struct xarray cq_xa; /* cqid → ib_uobject, per-context toggle page lookup */
struct xarray srq_xa; /* srqid → ib_uobject, per-context toggle page lookup */
u64 cmask;
};

4
drivers/infiniband/hw/bnxt_re/main.c
View File

@ -2337,10 +2337,6 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type)
if (!(rdev->qplib_res.en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT))
bnxt_re_vf_res_config(rdev);
}
hash_init(rdev->cq_hash);
if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT)
hash_init(rdev->srq_hash);
bnxt_re_debugfs_add_pdev(rdev);
bnxt_re_init_dcb_wq(rdev);

181
drivers/infiniband/hw/bnxt_re/uapi.c
View File

@ -22,31 +22,6 @@
#include "bnxt_re.h"
#include "ib_verbs.h"
static struct bnxt_re_cq *bnxt_re_search_for_cq(struct bnxt_re_dev *rdev, u32 cq_id)
{
struct bnxt_re_cq *cq = NULL, *tmp_cq;
hash_for_each_possible(rdev->cq_hash, tmp_cq, hash_entry, cq_id) {
if (tmp_cq->qplib_cq.id == cq_id) {
cq = tmp_cq;
break;
}
}
return cq;
}
static struct bnxt_re_srq *bnxt_re_search_for_srq(struct bnxt_re_dev *rdev, u32 srq_id)
{
struct bnxt_re_srq *srq = NULL, *tmp_srq;
hash_for_each_possible(rdev->srq_hash, tmp_srq, hash_entry, srq_id) {
if (tmp_srq->qplib_srq.id == srq_id) {
srq = tmp_srq;
break;
}
}
return srq;
}
static int UVERBS_HANDLER(BNXT_RE_METHOD_NOTIFY_DRV)(struct uverbs_attr_bundle *attrs)
{
@ -238,21 +213,23 @@ DECLARE_UVERBS_GLOBAL_METHODS(BNXT_RE_OBJECT_NOTIFY_DRV,
&UVERBS_METHOD(BNXT_RE_METHOD_NOTIFY_DRV));
/* Toggle MEM */
struct bnxt_re_toggle_mem {
struct bnxt_re_user_mmap_entry *toggle_entry;
u64 mmap_offset;
};
static int UVERBS_HANDLER(BNXT_RE_METHOD_GET_TOGGLE_MEM)(struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *uobj = uverbs_attr_get_uobject(attrs, BNXT_RE_TOGGLE_MEM_HANDLE);
enum bnxt_re_mmap_flag mmap_flag = BNXT_RE_MMAP_TOGGLE_PAGE;
struct bnxt_re_user_mmap_entry *toggle_entry = NULL;
enum bnxt_re_get_toggle_mem_type res_type;
struct bnxt_re_user_mmap_entry *entry;
struct bnxt_re_toggle_mem *tmem;
struct ib_uobject *res_uobj;
struct bnxt_re_ucontext *uctx;
struct ib_ucontext *ib_uctx;
struct bnxt_re_dev *rdev;
struct bnxt_re_srq *srq;
u32 length = PAGE_SIZE;
struct bnxt_re_cq *cq;
u64 mem_offset;
u64 mmap_offset = 0;
u32 offset = 0;
u64 addr = 0;
u32 res_id;
int err;
@ -260,48 +237,115 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_GET_TOGGLE_MEM)(struct uverbs_attr_bund
if (IS_ERR(ib_uctx))
return PTR_ERR(ib_uctx);
uctx = container_of(ib_uctx, struct bnxt_re_ucontext, ib_uctx);
/* New path: updated libbnxt_re passes the CQ or SRQ uverbs handle */
if (uverbs_attr_is_valid(attrs, BNXT_RE_TOGGLE_MEM_CQ_HANDLE)) {
struct bnxt_re_cq *cq;
res_uobj = uverbs_attr_get_uobject(attrs,
BNXT_RE_TOGGLE_MEM_CQ_HANDLE);
if (IS_ERR(res_uobj))
return PTR_ERR(res_uobj);
cq = container_of(res_uobj->object, struct bnxt_re_cq, ib_cq);
if (!cq->toggle_entry)
return -EOPNOTSUPP;
mmap_offset = rdma_user_mmap_get_offset(&cq->toggle_entry->rdma_entry);
if (!mmap_offset)
return -EOPNOTSUPP;
kref_get(&cq->toggle_entry->rdma_entry.ref);
toggle_entry = cq->toggle_entry;
goto alloc_tmem;
} else if (uverbs_attr_is_valid(attrs, BNXT_RE_TOGGLE_MEM_SRQ_HANDLE)) {
struct bnxt_re_srq *srq;
res_uobj = uverbs_attr_get_uobject(attrs,
BNXT_RE_TOGGLE_MEM_SRQ_HANDLE);
if (IS_ERR(res_uobj))
return PTR_ERR(res_uobj);
srq = container_of(res_uobj->object, struct bnxt_re_srq, ib_srq);
if (!srq->toggle_entry)
return -EOPNOTSUPP;
mmap_offset = rdma_user_mmap_get_offset(&srq->toggle_entry->rdma_entry);
if (!mmap_offset)
return -EOPNOTSUPP;
kref_get(&srq->toggle_entry->rdma_entry.ref);
toggle_entry = srq->toggle_entry;
goto alloc_tmem;
}
err = uverbs_get_const(&res_type, attrs, BNXT_RE_TOGGLE_MEM_TYPE);
if (err)
return err;
uctx = container_of(ib_uctx, struct bnxt_re_ucontext, ib_uctx);
rdev = uctx->rdev;
err = uverbs_copy_from(&res_id, attrs, BNXT_RE_TOGGLE_MEM_RES_ID);
if (err)
return err;
switch (res_type) {
case BNXT_RE_CQ_TOGGLE_MEM:
cq = bnxt_re_search_for_cq(rdev, res_id);
if (!cq)
return -EINVAL;
/*
* Legacy path: old libbnxt_re sends TYPE + RES_ID.
* Hold xa_lock across xa_load + kref_get so that a concurrent
* bnxt_re_destroy_cq/srq cannot call __xa_erase and remove the
* toggle_entry between our load and our reference on it.
*
* bnxt_re_create_cq/srq() publishes the uobject into cq_xa/srq_xa
* before returning to the uverbs core, but the core only sets
* uobject->object once the create callback has returned success.
* A lookup that races with an in-progress create can therefore
* find a uobject whose ->object is still NULL; skip it instead of
* feeding NULL to container_of().
*/
if (res_type == BNXT_RE_CQ_TOGGLE_MEM) {
struct bnxt_re_cq *cq;
addr = (u64)cq->uctx_cq_page;
if (!addr)
return -EOPNOTSUPP;
break;
case BNXT_RE_SRQ_TOGGLE_MEM:
srq = bnxt_re_search_for_srq(rdev, res_id);
if (!srq)
return -EINVAL;
xa_lock(&uctx->cq_xa);
res_uobj = xa_load(&uctx->cq_xa, res_id);
if (res_uobj && res_uobj->object) {
cq = container_of(res_uobj->object, struct bnxt_re_cq, ib_cq);
if (cq->toggle_entry)
mmap_offset =
rdma_user_mmap_get_offset(&cq->toggle_entry->rdma_entry);
if (mmap_offset) {
kref_get(&cq->toggle_entry->rdma_entry.ref);
toggle_entry = cq->toggle_entry;
}
}
xa_unlock(&uctx->cq_xa);
} else if (res_type == BNXT_RE_SRQ_TOGGLE_MEM) {
struct bnxt_re_srq *srq;
addr = (u64)srq->uctx_srq_page;
if (!addr)
return -EOPNOTSUPP;
break;
default:
xa_lock(&uctx->srq_xa);
res_uobj = xa_load(&uctx->srq_xa, res_id);
if (res_uobj && res_uobj->object) {
srq = container_of(res_uobj->object, struct bnxt_re_srq, ib_srq);
if (srq->toggle_entry)
mmap_offset =
rdma_user_mmap_get_offset(&srq->toggle_entry->rdma_entry);
if (mmap_offset) {
kref_get(&srq->toggle_entry->rdma_entry.ref);
toggle_entry = srq->toggle_entry;
}
}
xa_unlock(&uctx->srq_xa);
} else {
return -EOPNOTSUPP;
}
entry = bnxt_re_mmap_entry_insert(uctx, addr, mmap_flag, &mem_offset);
if (!entry)
return -ENOMEM;
if (!mmap_offset)
return -EOPNOTSUPP;
uobj->object = entry;
alloc_tmem:
tmem = kzalloc_obj(*tmem);
if (!tmem) {
rdma_user_mmap_entry_put(&toggle_entry->rdma_entry);
return -ENOMEM;
}
tmem->toggle_entry = toggle_entry;
tmem->mmap_offset = mmap_offset;
uobj->object = tmem;
uverbs_finalize_uobj_create(attrs, BNXT_RE_TOGGLE_MEM_HANDLE);
err = uverbs_copy_to(attrs, BNXT_RE_TOGGLE_MEM_MMAP_PAGE,
&mem_offset, sizeof(mem_offset));
&mmap_offset, sizeof(mmap_offset));
if (err)
return err;
@ -322,9 +366,10 @@ static int get_toggle_mem_obj_cleanup(struct ib_uobject *uobject,
enum rdma_remove_reason why,
struct uverbs_attr_bundle *attrs)
{
struct bnxt_re_user_mmap_entry *entry = uobject->object;
struct bnxt_re_toggle_mem *tmem = uobject->object;
rdma_user_mmap_entry_remove(&entry->rdma_entry);
rdma_user_mmap_entry_put(&tmem->toggle_entry->rdma_entry);
kfree(tmem);
return 0;
}
@ -335,10 +380,10 @@ DECLARE_UVERBS_NAMED_METHOD(BNXT_RE_METHOD_GET_TOGGLE_MEM,
UA_MANDATORY),
UVERBS_ATTR_CONST_IN(BNXT_RE_TOGGLE_MEM_TYPE,
enum bnxt_re_get_toggle_mem_type,
UA_MANDATORY),
UA_OPTIONAL),
UVERBS_ATTR_PTR_IN(BNXT_RE_TOGGLE_MEM_RES_ID,
UVERBS_ATTR_TYPE(u32),
UA_MANDATORY),
UA_OPTIONAL),
UVERBS_ATTR_PTR_OUT(BNXT_RE_TOGGLE_MEM_MMAP_PAGE,
UVERBS_ATTR_TYPE(u64),
UA_MANDATORY),
@ -347,7 +392,15 @@ DECLARE_UVERBS_NAMED_METHOD(BNXT_RE_METHOD_GET_TOGGLE_MEM,
UA_MANDATORY),
UVERBS_ATTR_PTR_OUT(BNXT_RE_TOGGLE_MEM_MMAP_LENGTH,
UVERBS_ATTR_TYPE(u32),
UA_MANDATORY));
UA_MANDATORY),
UVERBS_ATTR_IDR(BNXT_RE_TOGGLE_MEM_CQ_HANDLE,
UVERBS_OBJECT_CQ,
UVERBS_ACCESS_READ,
UA_OPTIONAL),
UVERBS_ATTR_IDR(BNXT_RE_TOGGLE_MEM_SRQ_HANDLE,
UVERBS_OBJECT_SRQ,
UVERBS_ACCESS_READ,
UA_OPTIONAL));
DECLARE_UVERBS_NAMED_METHOD_DESTROY(BNXT_RE_METHOD_RELEASE_TOGGLE_MEM,
UVERBS_ATTR_IDR(BNXT_RE_RELEASE_TOGGLE_MEM_HANDLE,

8
drivers/infiniband/hw/cxgb4/device.c
View File

@ -933,6 +933,7 @@ static void c4iw_rdev_close(struct c4iw_rdev *rdev)
void c4iw_dealloc(struct uld_ctx *ctx)
{
debugfs_remove_recursive(ctx->dev->debugfs_root);
c4iw_rdev_close(&ctx->dev->rdev);
WARN_ON(!xa_empty(&ctx->dev->cqs));
WARN_ON(!xa_empty(&ctx->dev->qps));
@ -951,7 +952,12 @@ void c4iw_dealloc(struct uld_ctx *ctx)
static void c4iw_remove(struct uld_ctx *ctx)
{
pr_debug("c4iw_dev %p\n", ctx->dev);
debugfs_remove_recursive(ctx->dev->debugfs_root);
/* c4iw_register_device() may still be using ctx->dev. */
cancel_work_sync(&ctx->reg_work);
if (!ctx->dev)
return;
c4iw_unregister_device(ctx->dev);
c4iw_dealloc(ctx);
}

64
drivers/infiniband/hw/cxgb4/mem.c
View File

@ -74,11 +74,8 @@ static int _c4iw_write_mem_dma_aligned(struct c4iw_rdev *rdev, u32 addr,
c4iw_init_wr_wait(wr_waitp);
wr_len = roundup(sizeof(*req) + sizeof(*sgl), 16);
if (!skb) {
if (!skb)
skb = alloc_skb(wr_len, GFP_KERNEL | __GFP_NOFAIL);
if (!skb)
return -ENOMEM;
}
set_wr_txq(skb, CPL_PRIORITY_CONTROL, 0);
req = __skb_put_zero(skb, wr_len);
@ -134,11 +131,8 @@ static int _c4iw_write_mem_inline(struct c4iw_rdev *rdev, u32 addr, u32 len,
roundup(copy_len, T4_ULPTX_MIN_IO),
16);
if (!skb) {
if (!skb)
skb = alloc_skb(wr_len, GFP_KERNEL | __GFP_NOFAIL);
if (!skb)
return -ENOMEM;
}
set_wr_txq(skb, CPL_PRIORITY_CONTROL, 0);
req = __skb_put_zero(skb, wr_len);
@ -199,7 +193,8 @@ static int _c4iw_write_mem_dma(struct c4iw_rdev *rdev, u32 addr, u32 len,
daddr = dma_map_single(&rdev->lldi.pdev->dev, data, len, DMA_TO_DEVICE);
if (dma_mapping_error(&rdev->lldi.pdev->dev, daddr))
return -1;
return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
wr_waitp);
save = daddr;
while (remain > inline_threshold) {
@ -235,30 +230,12 @@ static int write_adapter_mem(struct c4iw_rdev *rdev, u32 addr, u32 len,
void *data, struct sk_buff *skb,
struct c4iw_wr_wait *wr_waitp)
{
int ret;
if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl) {
ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl ||
len <= inline_threshold)
return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
wr_waitp);
goto out;
}
if (len <= inline_threshold) {
ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
wr_waitp);
goto out;
}
ret = _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
if (ret) {
pr_warn_ratelimited("%s: dma map failure (non fatal)\n",
pci_name(rdev->lldi.pdev));
ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
wr_waitp);
}
out:
return ret;
return _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
}
/*
@ -277,14 +254,19 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
int err;
struct fw_ri_tpte *tpt;
u32 stag_idx;
bool stag_idx_allocated = false;
static atomic_t key;
if (c4iw_fatal_error(rdev))
if (c4iw_fatal_error(rdev)) {
kfree_skb(skb);
return -EIO;
}
tpt = kmalloc_obj(*tpt);
if (!tpt)
if (!tpt) {
kfree_skb(skb);
return -ENOMEM;
}
stag_state = stag_state > 0;
stag_idx = (*stag) >> 8;
@ -296,9 +278,11 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
rdev->stats.stag.fail++;
mutex_unlock(&rdev->stats.lock);
kfree(tpt);
kfree_skb(skb);
return -ENOMEM;
}
mutex_lock(&rdev->stats.lock);
stag_idx_allocated = true;
rdev->stats.stag.cur += 32;
if (rdev->stats.stag.cur > rdev->stats.stag.max)
rdev->stats.stag.max = rdev->stats.stag.cur;
@ -333,7 +317,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
(rdev->lldi.vr->stag.start >> 5),
sizeof(*tpt), tpt, skb, wr_waitp);
if (reset_tpt_entry) {
if (reset_tpt_entry || (err && stag_idx_allocated)) {
c4iw_put_resource(&rdev->resource.tpt_table, stag_idx);
mutex_lock(&rdev->stats.lock);
rdev->stats.stag.cur -= 32;
@ -469,8 +453,10 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
FW_RI_STAG_NSMR, mhp->attr.perms,
mhp->attr.mw_bind_enable, 0, 0, ~0ULL, 0, 0, 0,
NULL, mhp->wr_waitp);
if (ret)
goto err_free_skb;
if (ret) {
kfree_skb(mhp->dereg_skb);
goto err_free_wr_wait;
}
ret = finish_mem_reg(mhp, stag);
if (ret)
@ -479,8 +465,6 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
err_dereg_mem:
dereg_mem(&rhp->rdev, mhp->attr.stag, mhp->attr.pbl_size,
mhp->attr.pbl_addr, mhp->dereg_skb, mhp->wr_waitp);
err_free_skb:
kfree_skb(mhp->dereg_skb);
err_free_wr_wait:
c4iw_put_wr_wait(mhp->wr_waitp);
err_free_mhp:
@ -541,7 +525,7 @@ struct ib_mr *c4iw_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
if (err)
goto err_umem_release;
pages = (__be64 *) __get_free_page(GFP_KERNEL);
pages = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!pages) {
err = -ENOMEM;
goto err_pbl_free;
@ -568,7 +552,7 @@ struct ib_mr *c4iw_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
mhp->wr_waitp);
pbl_done:
free_page((unsigned long) pages);
kfree(pages);
if (err)
goto err_pbl_free;

4
drivers/infiniband/hw/cxgb4/provider.c
View File

@ -263,7 +263,7 @@ static int c4iw_query_device(struct ib_device *ibdev, struct ib_device_attr *pro
pr_debug("ibdev %p\n", ibdev);
err = ib_is_udata_in_empty(uhw);
err = ib_no_udata_io(uhw);
if (err)
return err;
@ -300,7 +300,7 @@ static int c4iw_query_device(struct ib_device *ibdev, struct ib_device_attr *pro
props->max_fast_reg_page_list_len =
t4_max_fr_depth(dev->rdev.lldi.ulptx_memwrite_dsgl && use_dsgl);
return ib_respond_empty_udata(uhw);
return 0;
}
static int c4iw_query_port(struct ib_device *ibdev, u32 port,

4
drivers/infiniband/hw/efa/Makefile
View File

@ -1,9 +1,9 @@
# SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
# Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
# Copyright 2018-2026 Amazon.com, Inc. or its affiliates. All rights reserved.
#
# Makefile for Amazon Elastic Fabric Adapter (EFA) device driver.
#
obj-$(CONFIG_INFINIBAND_EFA) += efa.o
efa-y := efa_com_cmd.o efa_com.o efa_main.o efa_verbs.o
efa-y := efa_com_cmd.o efa_ah_cache.o efa_com.o efa_main.o efa_verbs.o

20
drivers/infiniband/hw/efa/efa.h
View File

@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
/*
* Copyright 2018-2025 Amazon.com, Inc. or its affiliates. All rights reserved.
* Copyright 2018-2026 Amazon.com, Inc. or its affiliates. All rights reserved.
*/
#ifndef _EFA_H_
@ -110,6 +110,14 @@ struct efa_cq {
struct ib_umem *umem;
};
struct efa_comp_cntr {
struct ib_comp_cntr ibcc;
struct ib_umem *comp_umem;
struct ib_umem *err_umem;
u32 comp_handle;
u32 err_handle;
};
struct efa_qp {
struct ib_qp ibqp;
dma_addr_t rq_dma_addr;
@ -164,6 +172,16 @@ int efa_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
int efa_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata);
int efa_create_user_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
struct uverbs_attr_bundle *attrs);
int efa_query_comp_cntr_caps(struct ib_device *ibdev,
struct ib_comp_cntr_caps *caps,
struct uverbs_attr_bundle *attrs);
int efa_create_comp_cntr(struct ib_comp_cntr *ibcc,
struct uverbs_attr_bundle *attrs);
int efa_destroy_comp_cntr(struct ib_comp_cntr *ibcc);
int efa_modify_comp_cntr(struct ib_comp_cntr *ibcc, enum ib_comp_cntr_entry entry,
enum ib_comp_cntr_modify_op op, u64 value);
int efa_qp_attach_comp_cntr(struct ib_qp *ibqp, struct ib_comp_cntr *ibcc,
struct ib_qp_attach_comp_cntr_attr *attr);
struct ib_mr *efa_reg_mr(struct ib_pd *ibpd, u64 start, u64 length,
u64 virt_addr, int access_flags,
struct ib_dmah *dmah,

296
drivers/infiniband/hw/efa/efa_admin_cmds_defs.h
View File

@ -28,7 +28,13 @@ enum efa_admin_aq_opcode {
EFA_ADMIN_CREATE_EQ = 18,
EFA_ADMIN_DESTROY_EQ = 19,
EFA_ADMIN_ALLOC_MR = 20,
EFA_ADMIN_MAX_OPCODE = 20,
EFA_ADMIN_SERVICE = 21,
EFA_ADMIN_CREATE_EVENT_COUNTER = 25,
EFA_ADMIN_DESTROY_EVENT_COUNTER = 26,
EFA_ADMIN_ATTACH_EVENT_COUNTER = 27,
EFA_ADMIN_MODIFY_EVENT_COUNTER = 28,
EFA_ADMIN_DETACH_EVENT_COUNTER = 29,
EFA_ADMIN_MAX_OPCODE = 29,
};
enum efa_admin_aq_feature_id {
@ -96,9 +102,6 @@ struct efa_admin_qp_alloc_size {
};
struct efa_admin_create_qp_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* Protection Domain associated with this QP */
u16 pd;
@ -113,7 +116,10 @@ struct efa_admin_create_qp_cmd {
* 2 : unsolicited_write_recv - If set, work requests
* will not be consumed for incoming RDMA write with
* immediate
* 7:3 : reserved - MBZ
* 3 : sq_64_bit_req_id - If set, requests posted on
* SQ will use 64-bit ids. The corresponding CQ must
* also have 64-bit ids enabled.
* 7:4 : reserved - MBZ
*/
u8 flags;
@ -158,7 +164,7 @@ struct efa_admin_create_qp_cmd {
/* MBZ */
u32 reserved2;
};
} __packed;
struct efa_admin_create_qp_resp {
/* Common Admin Queue completion descriptor */
@ -199,9 +205,6 @@ struct efa_admin_create_qp_resp {
};
struct efa_admin_modify_qp_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/*
* Mask indicating which fields should be updated
* 0 : qp_state
@ -237,7 +240,7 @@ struct efa_admin_modify_qp_cmd {
/* MBZ */
u16 reserved2;
};
} __packed;
struct efa_admin_modify_qp_resp {
/* Common Admin Queue completion descriptor */
@ -245,12 +248,9 @@ struct efa_admin_modify_qp_resp {
};
struct efa_admin_query_qp_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* QP handle returned by create_qp command */
u32 qp_handle;
};
} __packed;
struct efa_admin_query_qp_resp {
/* Common Admin Queue completion descriptor */
@ -276,12 +276,9 @@ struct efa_admin_query_qp_resp {
};
struct efa_admin_destroy_qp_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* QP handle returned by create_qp command */
u32 qp_handle;
};
} __packed;
struct efa_admin_destroy_qp_resp {
/* Common Admin Queue completion descriptor */
@ -293,9 +290,6 @@ struct efa_admin_destroy_qp_resp {
* once for the same destination
*/
struct efa_admin_create_ah_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* Destination address in network byte order */
u8 dest_addr[16];
@ -304,7 +298,7 @@ struct efa_admin_create_ah_cmd {
/* MBZ */
u16 reserved;
};
} __packed;
struct efa_admin_create_ah_resp {
/* Common Admin Queue completion descriptor */
@ -318,15 +312,12 @@ struct efa_admin_create_ah_resp {
};
struct efa_admin_destroy_ah_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* Target interface address handle (opaque) */
u16 ah;
/* PD number */
u16 pd;
};
} __packed;
struct efa_admin_destroy_ah_resp {
/* Common Admin Queue completion descriptor */
@ -340,9 +331,6 @@ struct efa_admin_destroy_ah_resp {
* on users working with very large datasets (i.e. full GPU memory mapping).
*/
struct efa_admin_reg_mr_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* Protection Domain */
u16 pd;
@ -367,10 +355,10 @@ struct efa_admin_reg_mr_cmd {
/*
* flags and page size
* 4:0 : phys_page_size_shift - page size is (1 <<
* 5:0 : phys_page_size_shift - page size is (1 <<
* phys_page_size_shift). Page size is used for
* building the Virtual to Physical address mapping
* 6:5 : reserved - MBZ
* 6 : reserved - MBZ
* 7 : mem_addr_phy_mode_en - Enable bit for physical
* memory registration (no translation), can be used
* only by privileged clients. If set, PBL must
@ -404,7 +392,7 @@ struct efa_admin_reg_mr_cmd {
* the region.
*/
u64 iova;
};
} __packed;
struct efa_admin_reg_mr_resp {
/* Common Admin Queue completion descriptor */
@ -450,12 +438,9 @@ struct efa_admin_reg_mr_resp {
};
struct efa_admin_dereg_mr_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* L_Key, memory region's l_key */
u32 l_key;
};
} __packed;
struct efa_admin_dereg_mr_resp {
/* Common Admin Queue completion descriptor */
@ -467,9 +452,6 @@ struct efa_admin_dereg_mr_resp {
* Addresses in kernel verbs semantics, ready for fast registration use.
*/
struct efa_admin_alloc_mr_cmd {
/* Common Admin Queue descriptor */
struct efa_admin_aq_common_desc aq_common_desc;
/* Protection Domain */
u16 pd;
@ -478,7 +460,7 @@ struct efa_admin_alloc_mr_cmd {
/* Maximum number of pages this MR supports. */
u32 max_pages;
};
} __packed;
struct efa_admin_alloc_mr_resp {
/* Common Admin Queue completion descriptor */
@ -498,8 +480,6 @@ struct efa_admin_alloc_mr_resp {
};
struct efa_admin_create_cq_cmd {
struct efa_admin_aq_common_desc aq_common_desc;
/*
* 4:0 : reserved5 - MBZ
* 5 : interrupt_mode_enabled - if set, cq operates
@ -517,7 +497,9 @@ struct efa_admin_create_cq_cmd {
* 5 : set_src_addr - If set, source address will be
* filled on RX completions from unknown senders.
* Requires 8 words CQ entry size.
* 7:6 : reserved7 - MBZ
* 6 : sq_comp_64_bit_req_id - If set, send
* completions will use 64-bit work request ids
* 7 : reserved7 - MBZ
*/
u8 cq_caps_2;
@ -550,7 +532,7 @@ struct efa_admin_create_cq_cmd {
/* UAR number */
u16 uar;
};
} __packed;
struct efa_admin_create_cq_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -571,13 +553,11 @@ struct efa_admin_create_cq_resp {
};
struct efa_admin_destroy_cq_cmd {
struct efa_admin_aq_common_desc aq_common_desc;
u16 cq_idx;
/* MBZ */
u16 reserved1;
};
} __packed;
struct efa_admin_destroy_cq_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -588,14 +568,7 @@ struct efa_admin_destroy_cq_resp {
* buffer pointed by AQ entry
*/
struct efa_admin_aq_get_stats_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
union {
/* command specific inline data */
u32 inline_data_w1[3];
struct efa_admin_ctrl_buff_info control_buffer;
} u;
struct efa_admin_ctrl_buff_info control_buffer;
/* stats type as defined in enum efa_admin_get_stats_type */
u8 type;
@ -604,7 +577,7 @@ struct efa_admin_aq_get_stats_cmd {
u8 scope;
u16 scope_modifier;
};
} __packed;
struct efa_admin_basic_stats {
u64 tx_bytes;
@ -722,7 +695,12 @@ struct efa_admin_feature_device_attr_desc {
* on TX queues
* 4 : unsolicited_write_recv - If set, unsolicited
* write with imm. receive is supported
* 31:5 : reserved - MBZ
* 5 : event_counters - If set, event counters are
* supported
* 9:6 : reserved1 - MBZ
* 10 : sq_64_bit_req_id - If set, SQ can use 64-bit
* work request ids
* 31:11 : reserved2 - MBZ
*/
u32 device_caps;
@ -811,6 +789,34 @@ struct efa_admin_feature_queue_attr_desc_1 {
struct efa_admin_feature_queue_attr_desc_2 {
/* Maximum size of data that can be sent inline in a Send WQE */
u16 inline_buf_size_ex;
/* MBZ */
u8 reserved[6];
/*
* Supported counter QP events
* 0 : send_comp
* 1 : send_comp_err
* 2 : recv_comp
* 3 : recv_comp_err
* 4 : read_comp
* 5 : read_comp_err
* 6 : write_comp
* 7 : write_comp_err
* 8 : remote_read_comp
* 9 : remote_write_comp
* 31:10 : reserved - MBZ
*/
u32 supported_event_counter_qp_events;
/* Maximum number of counters */
u32 max_event_counters;
/*
* Maximum counter value, counter wraps around to 0 after reaching
* this value
*/
u64 event_counter_max_val;
};
struct efa_admin_event_queue_attr_desc {
@ -859,14 +865,12 @@ struct efa_admin_hw_hints {
};
struct efa_admin_get_feature_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
struct efa_admin_ctrl_buff_info control_buffer;
struct efa_admin_get_set_feature_common_desc feature_common;
u32 raw[11];
};
} __packed;
struct efa_admin_get_feature_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -891,8 +895,6 @@ struct efa_admin_get_feature_resp {
};
struct efa_admin_set_feature_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
struct efa_admin_ctrl_buff_info control_buffer;
struct efa_admin_get_set_feature_common_desc feature_common;
@ -903,7 +905,7 @@ struct efa_admin_set_feature_cmd {
/* AENQ configuration */
struct efa_admin_feature_aenq_desc aenq;
} u;
};
} __packed;
struct efa_admin_set_feature_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -913,10 +915,6 @@ struct efa_admin_set_feature_resp {
} u;
};
struct efa_admin_alloc_pd_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
};
struct efa_admin_alloc_pd_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -928,23 +926,17 @@ struct efa_admin_alloc_pd_resp {
};
struct efa_admin_dealloc_pd_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
/* PD number */
u16 pd;
/* MBZ */
u16 reserved;
};
} __packed;
struct efa_admin_dealloc_pd_resp {
struct efa_admin_acq_common_desc acq_common_desc;
};
struct efa_admin_alloc_uar_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
};
struct efa_admin_alloc_uar_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -956,22 +948,18 @@ struct efa_admin_alloc_uar_resp {
};
struct efa_admin_dealloc_uar_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
/* UAR number */
u16 uar;
/* MBZ */
u16 reserved;
};
} __packed;
struct efa_admin_dealloc_uar_resp {
struct efa_admin_acq_common_desc acq_common_desc;
};
struct efa_admin_create_eq_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
/* Size of the EQ in entries, must be power of 2 */
u16 depth;
@ -997,7 +985,7 @@ struct efa_admin_create_eq_cmd {
/* MBZ */
u32 reserved;
};
} __packed;
struct efa_admin_create_eq_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -1010,14 +998,12 @@ struct efa_admin_create_eq_resp {
};
struct efa_admin_destroy_eq_cmd {
struct efa_admin_aq_common_desc aq_common_descriptor;
/* EQ number */
u16 eqn;
/* MBZ */
u16 reserved;
};
} __packed;
struct efa_admin_destroy_eq_resp {
struct efa_admin_acq_common_desc acq_common_desc;
@ -1089,10 +1075,123 @@ struct efa_admin_host_info {
u32 flags;
};
struct efa_admin_service_cmd {
u8 buffer[60];
} __packed;
struct efa_admin_service_resp {
struct efa_admin_acq_common_desc acq_common_desc;
u8 buffer[56];
};
/* Create Counter command */
struct efa_admin_create_event_counter_cmd {
/* UAR number */
u16 uar;
/* MBZ */
u16 reserved;
/* Counter physical address */
u64 paddr;
} __packed;
struct efa_admin_create_event_counter_resp {
struct efa_admin_acq_common_desc acq_common_desc;
/* Counter handle */
u32 cntr_handle;
/* MBZ */
u32 reserved;
};
struct efa_admin_destroy_event_counter_cmd {
/* Counter handle */
u32 cntr_handle;
} __packed;
struct efa_admin_destroy_event_counter_resp {
struct efa_admin_acq_common_desc acq_common_desc;
};
enum efa_admin_event_counter_attach_type {
EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS = 0,
};
struct efa_admin_event_counter_attach_qp_events {
/* QP handle */
u32 qp_handle;
/*
* Bitmask of counter QP events
* 0 : send_comp
* 1 : send_comp_err
* 2 : recv_comp
* 3 : recv_comp_err
* 4 : read_comp
* 5 : read_comp_err
* 6 : write_comp
* 7 : write_comp_err
* 8 : remote_read_comp
* 9 : remote_write_comp
* 31:10 : reserved - MBZ
*/
u32 events;
};
struct efa_admin_attach_detach_event_counter_cmd {
/* Counter handle */
u32 cntr_handle;
/* efa_admin_event_counter_attach_type */
u8 attach_type;
/* MBZ */
u8 reserved[3];
union {
struct efa_admin_event_counter_attach_qp_events qp_events;
} u;
} __packed;
struct efa_admin_attach_detach_event_counter_resp {
struct efa_admin_acq_common_desc acq_common_desc;
};
/* Counter modify operations */
enum efa_admin_event_counter_modify_ops {
/* Set counter value */
EFA_ADMIN_EVENT_COUNTER_MODIFY_SET = 0,
/* Add to counter value */
EFA_ADMIN_EVENT_COUNTER_MODIFY_ADD = 1,
};
struct efa_admin_modify_event_counter_cmd {
/* Counter handle */
u32 cntr_handle;
/* Counter operation type (efa_admin_event_counter_modify_ops) */
u8 operation;
/* MBZ */
u8 reserved[7];
/* Value for SET or ADD */
u64 value;
} __packed;
struct efa_admin_modify_event_counter_resp {
struct efa_admin_acq_common_desc acq_common_desc;
};
/* create_qp_cmd */
#define EFA_ADMIN_CREATE_QP_CMD_SQ_VIRT_MASK BIT(0)
#define EFA_ADMIN_CREATE_QP_CMD_RQ_VIRT_MASK BIT(1)
#define EFA_ADMIN_CREATE_QP_CMD_UNSOLICITED_WRITE_RECV_MASK BIT(2)
#define EFA_ADMIN_CREATE_QP_CMD_SQ_64_BIT_REQ_ID_SHIFT 3
#define EFA_ADMIN_CREATE_QP_CMD_SQ_64_BIT_REQ_ID_MASK BIT(3)
/* modify_qp_cmd */
#define EFA_ADMIN_MODIFY_QP_CMD_QP_STATE_MASK BIT(0)
@ -1103,7 +1202,7 @@ struct efa_admin_host_info {
#define EFA_ADMIN_MODIFY_QP_CMD_RNR_RETRY_MASK BIT(5)
/* reg_mr_cmd */
#define EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT_MASK GENMASK(4, 0)
#define EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT_MASK GENMASK(5, 0)
#define EFA_ADMIN_REG_MR_CMD_MEM_ADDR_PHY_MODE_EN_MASK BIT(7)
#define EFA_ADMIN_REG_MR_CMD_LOCAL_WRITE_ENABLE_MASK BIT(0)
#define EFA_ADMIN_REG_MR_CMD_REMOTE_WRITE_ENABLE_MASK BIT(1)
@ -1119,6 +1218,8 @@ struct efa_admin_host_info {
#define EFA_ADMIN_CREATE_CQ_CMD_VIRT_MASK BIT(6)
#define EFA_ADMIN_CREATE_CQ_CMD_CQ_ENTRY_SIZE_WORDS_MASK GENMASK(4, 0)
#define EFA_ADMIN_CREATE_CQ_CMD_SET_SRC_ADDR_MASK BIT(5)
#define EFA_ADMIN_CREATE_CQ_CMD_SQ_COMP_64_BIT_REQ_ID_SHIFT 6
#define EFA_ADMIN_CREATE_CQ_CMD_SQ_COMP_64_BIT_REQ_ID_MASK BIT(6)
/* create_cq_resp */
#define EFA_ADMIN_CREATE_CQ_RESP_DB_VALID_MASK BIT(0)
@ -1129,6 +1230,21 @@ struct efa_admin_host_info {
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_DATA_POLLING_128_MASK BIT(2)
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_RDMA_WRITE_MASK BIT(3)
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_UNSOLICITED_WRITE_RECV_MASK BIT(4)
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_EVENT_COUNTERS_MASK BIT(5)
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_SQ_64_BIT_REQ_ID_SHIFT 10
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_SQ_64_BIT_REQ_ID_MASK BIT(10)
/* feature_queue_attr_desc_2 */
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_SEND_COMP_MASK BIT(0)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_SEND_COMP_ERR_MASK BIT(1)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_RECV_COMP_MASK BIT(2)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_RECV_COMP_ERR_MASK BIT(3)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_READ_COMP_MASK BIT(4)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_READ_COMP_ERR_MASK BIT(5)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_WRITE_COMP_MASK BIT(6)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_WRITE_COMP_ERR_MASK BIT(7)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_REMOTE_READ_COMP_MASK BIT(8)
#define EFA_ADMIN_FEATURE_QUEUE_ATTR_DESC_2_REMOTE_WRITE_COMP_MASK BIT(9)
/* create_eq_cmd */
#define EFA_ADMIN_CREATE_EQ_CMD_ENTRY_SIZE_WORDS_MASK GENMASK(4, 0)
@ -1147,4 +1263,16 @@ struct efa_admin_host_info {
#define EFA_ADMIN_HOST_INFO_INTREE_MASK BIT(0)
#define EFA_ADMIN_HOST_INFO_GDR_MASK BIT(1)
/* counter_attach_qp_events */
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_SEND_COMP_MASK BIT(0)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_SEND_COMP_ERR_MASK BIT(1)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_RECV_COMP_MASK BIT(2)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_RECV_COMP_ERR_MASK BIT(3)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_READ_COMP_MASK BIT(4)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_READ_COMP_ERR_MASK BIT(5)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_WRITE_COMP_MASK BIT(6)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_WRITE_COMP_ERR_MASK BIT(7)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_REMOTE_READ_COMP_MASK BIT(8)
#define EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_REMOTE_WRITE_COMP_MASK BIT(9)
#endif /* _EFA_ADMIN_CMDS_H_ */

33
drivers/infiniband/hw/efa/efa_admin_defs.h
View File

@ -7,7 +7,7 @@
#define _EFA_ADMIN_H_
#define EFA_ADMIN_API_VERSION_MAJOR 0
#define EFA_ADMIN_API_VERSION_MINOR 2
#define EFA_ADMIN_API_VERSION_MINOR 3
enum efa_admin_aq_completion_status {
EFA_ADMIN_SUCCESS = 0,
@ -41,6 +41,21 @@ struct efa_admin_aq_common_desc {
u8 flags;
};
struct efa_admin_aq_common_desc_v2 {
struct efa_admin_aq_common_desc common;
/*
* Poly 0x8005 CRC16 with initial value 0xFFFF and final XOR of
* 0xFFFF. The checksum covers the entire admin command entry
* including the zeroed checksum field.
*/
u16 checksum;
u8 payload_ver;
u8 reserved[5];
};
/*
* used in efa_admin_aq_entry. Can point directly to control data, or to a
* page list chunk. Used also at the end of indirect mode page list chunks,
@ -55,13 +70,13 @@ struct efa_admin_ctrl_buff_info {
struct efa_admin_aq_entry {
struct efa_admin_aq_common_desc aq_common_descriptor;
union {
u32 inline_data_w1[3];
u32 request_payload[15];
};
struct efa_admin_ctrl_buff_info control_buffer;
} u;
struct efa_admin_aq_entry_v2 {
struct efa_admin_aq_common_desc_v2 aq_common_descriptor;
u32 inline_data_w4[12];
u32 request_payload[29];
};
struct efa_admin_acq_common_desc {
@ -80,7 +95,11 @@ struct efa_admin_acq_common_desc {
*/
u8 flags;
/* Poly 0x8005 CRC16 with initial value 0xFFFF and final XOR of 0xFFFF */
/*
* Poly 0x8005 CRC16 with initial value 0xFFFF and final XOR of 0xFFFF.
* The checksum covers the entire admin completion entry including the
* zeroed checksum field.
*/
u16 checksum;
u16 reserved;

135
drivers/infiniband/hw/efa/efa_ah_cache.c Normal file
View File

@ -0,0 +1,135 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
/*
* Copyright 2026 Amazon.com, Inc. or its affiliates. All rights reserved.
*/
#include <linux/slab.h>
#include "efa_ah_cache.h"
static const struct rhashtable_params ah_cache_params = {
.key_len = sizeof(struct efa_ah_cache_key),
.key_offset = offsetof(struct efa_ah_cache_entry, key),
.head_offset = offsetof(struct efa_ah_cache_entry, linkage),
};
int efa_ah_cache_init(struct efa_ah_cache *ah_cache)
{
int err;
mutex_init(&ah_cache->lock);
err = rhashtable_init(&ah_cache->hashtable, &ah_cache_params);
if (err)
mutex_destroy(&ah_cache->lock);
return err;
}
static void efa_ah_cache_entry_free(void *ptr, void *arg)
{
struct efa_ah_cache_entry *entry = ptr;
WARN_ON(entry->usecnt);
mutex_destroy(&entry->lock);
kfree(entry);
}
void efa_ah_cache_destroy(struct efa_ah_cache *ah_cache)
{
rhashtable_free_and_destroy(&ah_cache->hashtable, efa_ah_cache_entry_free, NULL);
mutex_destroy(&ah_cache->lock);
}
static struct efa_ah_cache_entry *efa_ah_cache_lookup_locked(struct efa_ah_cache *ah_cache, u16 pd,
u8 *gid)
__must_hold(&ah_cache->lock)
{
struct efa_ah_cache_key key = {};
memcpy(key.gid, gid, sizeof(key.gid));
key.pd = pd;
return rhashtable_lookup_fast(&ah_cache->hashtable, &key, ah_cache_params);
}
struct efa_ah_cache_entry *efa_ah_cache_lookup(struct efa_ah_cache *ah_cache, u16 pd, u8 *gid)
{
struct efa_ah_cache_entry *entry;
mutex_lock(&ah_cache->lock);
entry = efa_ah_cache_lookup_locked(ah_cache, pd, gid);
mutex_unlock(&ah_cache->lock);
return entry;
}
/**
* efa_ah_cache_get - Get or create an AH cache entry
* @ah_cache: AH cache
* @pd: Protection domain number
* @gid: GID address
*
* Look up an AH cache entry by PD and GID. If found, take a reference and
* return it. If not found, allocate a new entry and insert it. The caller must lock
* the entry mutex and check usecnt to determine whether a device create
* command is needed.
*
* Return: Pointer to the entry on success, ERR_PTR on failure.
*/
struct efa_ah_cache_entry *efa_ah_cache_get(struct efa_ah_cache *ah_cache, u16 pd, u8 *gid)
{
struct efa_ah_cache_entry *entry;
int err;
mutex_lock(&ah_cache->lock);
entry = efa_ah_cache_lookup_locked(ah_cache, pd, gid);
if (entry) {
refcount_inc(&entry->refcount);
mutex_unlock(&ah_cache->lock);
return entry;
}
entry = kzalloc_obj(*entry);
if (!entry) {
mutex_unlock(&ah_cache->lock);
return ERR_PTR(-ENOMEM);
}
memcpy(entry->key.gid, gid, sizeof(entry->key.gid));
entry->key.pd = pd;
refcount_set(&entry->refcount, 1);
mutex_init(&entry->lock);
err = rhashtable_insert_fast(&ah_cache->hashtable, &entry->linkage, ah_cache_params);
if (err) {
mutex_destroy(&entry->lock);
kfree(entry);
mutex_unlock(&ah_cache->lock);
return ERR_PTR(err);
}
mutex_unlock(&ah_cache->lock);
return entry;
}
/**
* efa_ah_cache_put - Put a refcount of an AH cache entry
* @ah_cache: AH cache
* @entry: AH cache entry
*
* Drop the refcount. If it reaches zero, remove the entry from the hashtable
* and free it.
*/
void efa_ah_cache_put(struct efa_ah_cache *ah_cache, struct efa_ah_cache_entry *entry)
{
if (!refcount_dec_and_mutex_lock(&entry->refcount, &ah_cache->lock))
return;
/* AH cache lock is held here */
rhashtable_remove_fast(&ah_cache->hashtable, &entry->linkage, ah_cache_params);
mutex_unlock(&ah_cache->lock);
mutex_destroy(&entry->lock);
kfree(entry);
}

39
drivers/infiniband/hw/efa/efa_ah_cache.h Normal file
View File

@ -0,0 +1,39 @@
/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
/*
* Copyright 2026 Amazon.com, Inc. or its affiliates. All rights reserved.
*/
#ifndef _EFA_AH_CACHE_H_
#define _EFA_AH_CACHE_H_
#include <linux/refcount.h>
#include <linux/rhashtable.h>
#define EFA_AH_GID_SIZE 16
struct efa_ah_cache_key {
u8 gid[EFA_AH_GID_SIZE];
u16 pd;
};
struct efa_ah_cache_entry {
struct efa_ah_cache_key key;
u16 ah;
unsigned int usecnt;
refcount_t refcount;
struct rhash_head linkage;
struct mutex lock; /* Serializes device commands per cache entry */
};
struct efa_ah_cache {
struct rhashtable hashtable;
struct mutex lock; /* Protects AH cache hashtable */
};
int efa_ah_cache_init(struct efa_ah_cache *ah_cache);
void efa_ah_cache_destroy(struct efa_ah_cache *ah_cache);
struct efa_ah_cache_entry *efa_ah_cache_get(struct efa_ah_cache *ah_cache, u16 pd, u8 *gid);
struct efa_ah_cache_entry *efa_ah_cache_lookup(struct efa_ah_cache *ah_cache, u16 pd, u8 *gid);
void efa_ah_cache_put(struct efa_ah_cache *ah_cache, struct efa_ah_cache_entry *entry);
#endif /* _EFA_AH_CACHE_H_ */

206
drivers/infiniband/hw/efa/efa_com.c
View File

@ -25,11 +25,16 @@
#define EFA_CRC16_INIT_VAL 0xffff
#define EFA_CRC_MIN_ADMIN_API_VERSION_MAJOR 0
#define EFA_CRC_MIN_ADMIN_API_VERSION_MINOR 2
#define EFA_ADMIN_SQ_MAX_ENT_SIZE sizeof(struct efa_admin_aq_entry_v2)
#define EFA_MIN_ADMIN_API_VERSION_MAJOR 0
#define EFA_MIN_ADMIN_API_VERSION_MINOR 1
#define EFA_CRC_MIN_API_VERSION_MAJOR 0
#define EFA_CRC_MIN_API_VERSION_MINOR 2
#define EFA_ADMIN_V2_MIN_API_VERSION_MAJOR 0
#define EFA_ADMIN_V2_MIN_API_VERSION_MINOR 3
#define EFA_MIN_API_VERSION_MAJOR 0
#define EFA_MIN_API_VERSION_MINOR 1
enum efa_cmd_status {
EFA_CMD_UNUSED,
@ -82,6 +87,16 @@ void efa_com_set_dma_addr(dma_addr_t addr, u32 *addr_high, u32 *addr_low)
*addr_high = upper_32_bits(addr);
}
static u32 efa_com_construct_ver(u32 major, u32 minor)
{
u32 ver = 0;
EFA_SET(&ver, EFA_REGS_VERSION_MAJOR_VERSION, major);
EFA_SET(&ver, EFA_REGS_VERSION_MINOR_VERSION, minor);
return ver;
}
static u32 efa_com_reg_read32(struct efa_com_dev *edev, u16 offset)
{
struct efa_com_mmio_read *mmio_read = &edev->mmio_read;
@ -138,14 +153,26 @@ static int efa_com_admin_init_sq(struct efa_com_dev *edev)
{
struct efa_com_admin_queue *aq = &edev->aq;
struct efa_com_admin_sq *sq = &aq->sq;
u16 size = aq->depth * sizeof(*sq->entries);
u32 aq_caps = 0;
u32 addr_high;
u32 addr_low;
u32 aq_caps = 0, admin_v2_min_ver = 0;
u32 addr_high, addr_low;
sq->entries =
dma_alloc_coherent(aq->dmadev, size, &sq->dma_addr, GFP_KERNEL);
if (!sq->entries)
admin_v2_min_ver = efa_com_construct_ver(EFA_ADMIN_V2_MIN_API_VERSION_MAJOR,
EFA_ADMIN_V2_MIN_API_VERSION_MINOR);
if (edev->dev_api_ver >= admin_v2_min_ver) {
sq->entry_size = sizeof(struct efa_admin_aq_entry_v2);
sq->payload_offset = offsetof(struct efa_admin_aq_entry_v2, request_payload);
sq->proto_ver = EFA_ADMIN_V2_PROTO_VER;
} else {
sq->entry_size = sizeof(struct efa_admin_aq_entry);
sq->payload_offset = offsetof(struct efa_admin_aq_entry, request_payload);
sq->proto_ver = EFA_ADMIN_V1_PROTO_VER;
}
sq->max_payload_size = sq->entry_size - sq->payload_offset;
sq->buffer = dma_alloc_coherent(aq->dmadev, aq->depth * sq->entry_size,
&sq->dma_addr, GFP_KERNEL);
if (!sq->buffer)
return -ENOMEM;
spin_lock_init(&sq->lock);
@ -163,8 +190,7 @@ static int efa_com_admin_init_sq(struct efa_com_dev *edev)
writel(addr_high, edev->reg_bar + EFA_REGS_AQ_BASE_HI_OFF);
EFA_SET(&aq_caps, EFA_REGS_AQ_CAPS_AQ_DEPTH, aq->depth);
EFA_SET(&aq_caps, EFA_REGS_AQ_CAPS_AQ_ENTRY_SIZE,
sizeof(struct efa_admin_aq_entry));
EFA_SET(&aq_caps, EFA_REGS_AQ_CAPS_AQ_ENTRY_SIZE, sq->entry_size);
writel(aq_caps, edev->reg_bar + EFA_REGS_AQ_CAPS_OFF);
@ -186,8 +212,8 @@ static int efa_com_admin_init_cq(struct efa_com_dev *edev)
spin_lock_init(&cq->lock);
EFA_SET(&crc_min_ver, EFA_REGS_VERSION_MAJOR_VERSION, EFA_CRC_MIN_ADMIN_API_VERSION_MAJOR);
EFA_SET(&crc_min_ver, EFA_REGS_VERSION_MINOR_VERSION, EFA_CRC_MIN_ADMIN_API_VERSION_MINOR);
crc_min_ver = efa_com_construct_ver(EFA_CRC_MIN_API_VERSION_MAJOR,
EFA_CRC_MIN_API_VERSION_MINOR);
if (edev->dev_api_ver >= crc_min_ver)
cq->validate_checksum = true;
@ -323,52 +349,80 @@ static inline struct efa_comp_ctx *efa_com_get_comp_ctx_by_cmd_id(struct efa_com
return &aq->comp_ctx[ctx_id];
}
static u16 efa_com_calc_crc16_checksum(u8 *buff, u32 buff_size)
{
return crc16(EFA_CRC16_INIT_VAL, buff, buff_size) ^ EFA_CRC16_INIT_VAL;
}
static void efa_com_construct_aq_entry(struct efa_com_admin_queue *aq, u8 *aq_entry, u16 cmd_id,
u8 opcode, u8 flags, void *payload, size_t payload_size)
{
struct efa_admin_aq_common_desc_v2 *common_v2 = NULL;
struct efa_admin_aq_common_desc *common;
struct efa_com_admin_sq *sq = &aq->sq;
if (sq->proto_ver == EFA_ADMIN_V1_PROTO_VER) {
common = (struct efa_admin_aq_common_desc *)aq_entry;
} else {
common_v2 = (struct efa_admin_aq_common_desc_v2 *)aq_entry;
common = &common_v2->common;
}
common->command_id = cmd_id;
common->opcode = opcode;
common->flags = flags;
EFA_SET(&common->flags, EFA_ADMIN_AQ_COMMON_DESC_PHASE, sq->phase);
if (payload)
memcpy(aq_entry + sq->payload_offset, payload, payload_size);
if (common_v2)
common_v2->checksum = efa_com_calc_crc16_checksum(aq_entry, sq->entry_size);
}
static void __efa_com_submit_admin_cmd(struct efa_com_admin_queue *aq,
struct efa_comp_ctx *comp_ctx,
struct efa_admin_aq_entry *cmd,
size_t cmd_size_in_bytes,
u8 opcode, u8 flags,
void *payload, size_t payload_size,
struct efa_admin_acq_entry *comp,
size_t comp_size_in_bytes)
{
struct efa_admin_aq_entry *aqe;
u16 queue_size_mask;
u16 cmd_id;
u16 ctx_id;
u16 pi;
u8 aq_entry[EFA_ADMIN_SQ_MAX_ENT_SIZE] __aligned(sizeof(u64)) = {};
u16 queue_size_mask, cmd_id, ctx_id, pi;
struct efa_com_admin_sq *sq = &aq->sq;
u8 *aqe;
queue_size_mask = aq->depth - 1;
pi = aq->sq.pc & queue_size_mask;
pi = sq->pc & queue_size_mask;
ctx_id = efa_com_get_comp_ctx_id(aq, comp_ctx);
/* cmd_id LSBs are the ctx_id and MSBs are entropy bits from pc */
cmd_id = ctx_id & queue_size_mask;
cmd_id |= aq->sq.pc << ilog2(aq->depth);
cmd_id |= sq->pc << ilog2(aq->depth);
cmd_id &= EFA_ADMIN_AQ_COMMON_DESC_COMMAND_ID_MASK;
cmd->aq_common_descriptor.command_id = cmd_id;
EFA_SET(&cmd->aq_common_descriptor.flags,
EFA_ADMIN_AQ_COMMON_DESC_PHASE, aq->sq.phase);
efa_com_construct_aq_entry(aq, aq_entry, cmd_id, opcode, flags, payload, payload_size);
comp_ctx->status = EFA_CMD_SUBMITTED;
comp_ctx->comp_size = comp_size_in_bytes;
comp_ctx->user_cqe = comp;
comp_ctx->cmd_opcode = cmd->aq_common_descriptor.opcode;
comp_ctx->cmd_opcode = opcode;
comp_ctx->cmd_id = cmd_id;
reinit_completion(&comp_ctx->wait_event);
aqe = &aq->sq.entries[pi];
memset(aqe, 0, sizeof(*aqe));
memcpy(aqe, cmd, cmd_size_in_bytes);
aqe = sq->buffer + sq->entry_size * pi;
memset(aqe, 0, sq->entry_size);
memcpy(aqe, aq_entry, sq->entry_size);
aq->sq.pc++;
sq->pc++;
atomic64_inc(&aq->stats.submitted_cmd);
if ((aq->sq.pc & queue_size_mask) == 0)
aq->sq.phase = !aq->sq.phase;
if ((sq->pc & queue_size_mask) == 0)
sq->phase = !sq->phase;
/* barrier not needed in case of writel */
writel(aq->sq.pc, aq->sq.db_addr);
writel(sq->pc, sq->db_addr);
}
static inline int efa_com_init_comp_ctxt(struct efa_com_admin_queue *aq)
@ -403,8 +457,8 @@ static inline int efa_com_init_comp_ctxt(struct efa_com_admin_queue *aq)
static int efa_com_submit_admin_cmd(struct efa_com_admin_queue *aq,
struct efa_comp_ctx *comp_ctx,
struct efa_admin_aq_entry *cmd,
size_t cmd_size_in_bytes,
u8 opcode, u8 flags,
void *payload, size_t payload_size,
struct efa_admin_acq_entry *comp,
size_t comp_size_in_bytes)
{
@ -415,8 +469,8 @@ static int efa_com_submit_admin_cmd(struct efa_com_admin_queue *aq,
return -ENODEV;
}
__efa_com_submit_admin_cmd(aq, comp_ctx, cmd, cmd_size_in_bytes, comp,
comp_size_in_bytes);
__efa_com_submit_admin_cmd(aq, comp_ctx, opcode, flags, payload,
payload_size, comp, comp_size_in_bytes);
spin_unlock(&aq->sq.lock);
return 0;
@ -430,7 +484,7 @@ static bool efa_com_cqe_checksum_valid(struct efa_com_admin_queue *aq,
cqe->acq_common_descriptor.checksum = 0;
calc_checksum = crc16(EFA_CRC16_INIT_VAL, (u8 *)cqe, sizeof(*cqe)) ^ EFA_CRC16_INIT_VAL;
calc_checksum = efa_com_calc_crc16_checksum((u8 *)cqe, sizeof(*cqe));
if (calc_checksum != cqe_checksum) {
ibdev_err(aq->efa_dev,
"Received completion with invalid checksum, cqe[%u], calc[%u], sq producer[%d], sq consumer[%d], cq consumer[%d]\n",
@ -634,8 +688,10 @@ static int efa_com_wait_and_process_admin_cq(struct efa_comp_ctx *comp_ctx,
/**
* efa_com_cmd_exec - Execute admin command
* @aq: admin queue.
* @cmd: the admin command to execute.
* @cmd_size: the command size.
* @opcode: the admin command opcode.
* @flags: the admin command header flags.
* @payload: the admin command payload.
* @payload_size: the payload size.
* @comp: command completion return entry.
* @comp_size: command completion size.
* Submit an admin command and then wait until the device will return a
@ -645,22 +701,23 @@ static int efa_com_wait_and_process_admin_cq(struct efa_comp_ctx *comp_ctx,
* @return - 0 on success, negative value on failure.
*/
int efa_com_cmd_exec(struct efa_com_admin_queue *aq,
struct efa_admin_aq_entry *cmd,
size_t cmd_size,
struct efa_admin_acq_entry *comp,
size_t comp_size)
u8 opcode, u8 flags,
void *payload, size_t payload_size,
struct efa_admin_acq_entry *comp, size_t comp_size)
{
struct efa_comp_ctx *comp_ctx;
int err;
if (payload_size > aq->sq.max_payload_size)
return -EINVAL;
might_sleep();
/* In case of queue FULL */
down(&aq->avail_cmds);
ibdev_dbg(aq->efa_dev, "%s (opcode %d)\n",
efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
cmd->aq_common_descriptor.opcode);
ibdev_dbg(aq->efa_dev, "%s (opcode %d)\n", efa_com_cmd_str(opcode),
opcode);
comp_ctx = efa_com_alloc_comp_ctx(aq);
if (!comp_ctx) {
@ -669,13 +726,13 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq,
return -EINVAL;
}
err = efa_com_submit_admin_cmd(aq, comp_ctx, cmd, cmd_size, comp, comp_size);
err = efa_com_submit_admin_cmd(aq, comp_ctx, opcode, flags, payload, payload_size, comp,
comp_size);
if (err) {
ibdev_err_ratelimited(
aq->efa_dev,
"Failed to submit command %s (opcode %u) err %d\n",
efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
cmd->aq_common_descriptor.opcode, err);
efa_com_cmd_str(opcode), opcode, err);
efa_com_dealloc_comp_ctx(aq, comp_ctx);
up(&aq->avail_cmds);
@ -688,8 +745,7 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq,
ibdev_err_ratelimited(
aq->efa_dev,
"Failed to process command %s (opcode %u) err %d\n",
efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
cmd->aq_common_descriptor.opcode, err);
efa_com_cmd_str(opcode), opcode, err);
atomic64_inc(&aq->stats.cmd_err);
}
@ -716,14 +772,16 @@ void efa_com_admin_destroy(struct efa_com_dev *edev)
devm_kfree(edev->dmadev, aq->comp_ctx_pool);
devm_kfree(edev->dmadev, aq->comp_ctx);
size = aq->depth * sizeof(*sq->entries);
dma_free_coherent(edev->dmadev, size, sq->entries, sq->dma_addr);
size = aq->depth * sq->entry_size;
dma_free_coherent(edev->dmadev, size, sq->buffer, sq->dma_addr);
size = aq->depth * sizeof(*cq->entries);
dma_free_coherent(edev->dmadev, size, cq->entries, cq->dma_addr);
size = aenq->depth * sizeof(*aenq->entries);
dma_free_coherent(edev->dmadev, size, aenq->entries, aenq->dma_addr);
efa_ah_cache_destroy(&edev->ah_cache);
}
/**
@ -782,6 +840,12 @@ int efa_com_admin_init(struct efa_com_dev *edev,
return -ENODEV;
}
err = efa_ah_cache_init(&edev->ah_cache);
if (err) {
ibdev_err(edev->efa_dev, "Failed to init AH cache\n");
return err;
}
aq->depth = EFA_ADMIN_QUEUE_DEPTH;
aq->dmadev = edev->dmadev;
@ -794,7 +858,7 @@ int efa_com_admin_init(struct efa_com_dev *edev,
err = efa_com_init_comp_ctxt(aq);
if (err)
return err;
goto err_destroy_ah_cache;
err = efa_com_admin_init_sq(edev);
if (err)
@ -828,10 +892,12 @@ int efa_com_admin_init(struct efa_com_dev *edev,
dma_free_coherent(edev->dmadev, aq->depth * sizeof(*aq->cq.entries),
aq->cq.entries, aq->cq.dma_addr);
err_destroy_sq:
dma_free_coherent(edev->dmadev, aq->depth * sizeof(*aq->sq.entries),
aq->sq.entries, aq->sq.dma_addr);
dma_free_coherent(edev->dmadev, aq->depth * aq->sq.entry_size,
aq->sq.buffer, aq->sq.dma_addr);
err_destroy_comp_ctxt:
devm_kfree(edev->dmadev, aq->comp_ctx);
err_destroy_ah_cache:
efa_ah_cache_destroy(&edev->ah_cache);
return err;
}
@ -990,8 +1056,8 @@ int efa_com_validate_version(struct efa_com_dev *edev)
EFA_GET(&ver, EFA_REGS_VERSION_MAJOR_VERSION),
EFA_GET(&ver, EFA_REGS_VERSION_MINOR_VERSION));
EFA_SET(&min_ver, EFA_REGS_VERSION_MAJOR_VERSION, EFA_MIN_ADMIN_API_VERSION_MAJOR);
EFA_SET(&min_ver, EFA_REGS_VERSION_MINOR_VERSION, EFA_MIN_ADMIN_API_VERSION_MINOR);
min_ver = efa_com_construct_ver(EFA_MIN_API_VERSION_MAJOR,
EFA_MIN_API_VERSION_MINOR);
if (ver < min_ver) {
ibdev_err(edev->efa_dev,
"EFA version is lower than the minimal version the driver supports\n");
@ -1146,7 +1212,6 @@ static int efa_com_create_eq(struct efa_com_dev *edev,
struct efa_admin_create_eq_cmd cmd = {};
int err;
cmd.aq_common_descriptor.opcode = EFA_ADMIN_CREATE_EQ;
EFA_SET(&cmd.caps, EFA_ADMIN_CREATE_EQ_CMD_ENTRY_SIZE_WORDS,
params->entry_size_in_bytes / 4);
cmd.depth = params->depth;
@ -1156,11 +1221,9 @@ static int efa_com_create_eq(struct efa_com_dev *edev,
efa_com_set_dma_addr(params->dma_addr, &cmd.ba.mem_addr_high,
&cmd.ba.mem_addr_low);
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_CREATE_EQ, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to create eq[%d]\n", err);
@ -1180,14 +1243,11 @@ static void efa_com_destroy_eq(struct efa_com_dev *edev,
struct efa_admin_destroy_eq_cmd cmd = {};
int err;
cmd.aq_common_descriptor.opcode = EFA_ADMIN_DESTROY_EQ;
cmd.eqn = params->eqn;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_DESTROY_EQ, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err)
ibdev_err_ratelimited(edev->efa_dev,
"Failed to destroy EQ-%u [%d]\n", cmd.eqn,

19
drivers/infiniband/hw/efa/efa_com.h
View File

@ -14,6 +14,7 @@
#include <rdma/ib_verbs.h>
#include "efa_ah_cache.h"
#include "efa_common_defs.h"
#include "efa_admin_defs.h"
#include "efa_admin_cmds_defs.h"
@ -21,6 +22,9 @@
#define EFA_MAX_HANDLERS 256
#define EFA_ADMIN_V1_PROTO_VER 0
#define EFA_ADMIN_V2_PROTO_VER 1
struct efa_com_admin_cq {
struct efa_admin_acq_entry *entries;
dma_addr_t dma_addr;
@ -32,9 +36,13 @@ struct efa_com_admin_cq {
};
struct efa_com_admin_sq {
struct efa_admin_aq_entry *entries;
u8 *buffer;
u16 entry_size;
u16 payload_offset;
u16 max_payload_size;
dma_addr_t dma_addr;
spinlock_t lock; /* Protects ASQ */
u8 proto_ver;
u32 __iomem *db_addr;
@ -113,6 +121,8 @@ struct efa_com_dev {
u32 supported_features;
u32 dma_addr_bits;
struct efa_ah_cache ah_cache;
u32 dev_api_ver;
struct efa_com_mmio_read mmio_read;
};
@ -171,10 +181,9 @@ int efa_com_validate_version(struct efa_com_dev *edev);
int efa_com_get_dma_width(struct efa_com_dev *edev);
int efa_com_cmd_exec(struct efa_com_admin_queue *aq,
struct efa_admin_aq_entry *cmd,
size_t cmd_size,
struct efa_admin_acq_entry *comp,
size_t comp_size);
u8 opcode, u8 flags,
void *payload, size_t payload_size,
struct efa_admin_acq_entry *comp, size_t comp_size);
void efa_com_aenq_intr_handler(struct efa_com_dev *edev, void *data);
void efa_com_eq_comp_intr_handler(struct efa_com_dev *edev,
struct efa_com_eq *eeq);

329
drivers/infiniband/hw/efa/efa_com_cmd.c
View File

@ -17,8 +17,6 @@ int efa_com_create_qp(struct efa_com_dev *edev,
struct efa_com_admin_queue *aq = &edev->aq;
int err;
create_qp_cmd.aq_common_desc.opcode = EFA_ADMIN_CREATE_QP;
create_qp_cmd.pd = params->pd;
create_qp_cmd.qp_type = params->qp_type;
create_qp_cmd.rq_base_addr = params->rq_base_addr;
@ -38,9 +36,11 @@ int efa_com_create_qp(struct efa_com_dev *edev,
if (params->unsolicited_write_recv)
EFA_SET(&create_qp_cmd.flags, EFA_ADMIN_CREATE_QP_CMD_UNSOLICITED_WRITE_RECV, 1);
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&create_qp_cmd,
sizeof(create_qp_cmd),
if (params->sq_64_bit_req_id)
EFA_SET(&create_qp_cmd.flags, EFA_ADMIN_CREATE_QP_CMD_SQ_64_BIT_REQ_ID, 1);
err = efa_com_cmd_exec(aq, EFA_ADMIN_CREATE_QP, 0,
&create_qp_cmd, sizeof(create_qp_cmd),
(struct efa_admin_acq_entry *)&cmd_completion,
sizeof(cmd_completion));
if (err) {
@ -68,7 +68,6 @@ int efa_com_modify_qp(struct efa_com_dev *edev,
struct efa_admin_modify_qp_resp resp;
int err;
cmd.aq_common_desc.opcode = EFA_ADMIN_MODIFY_QP;
cmd.modify_mask = params->modify_mask;
cmd.qp_handle = params->qp_handle;
cmd.qp_state = params->qp_state;
@ -78,11 +77,9 @@ int efa_com_modify_qp(struct efa_com_dev *edev,
cmd.sq_drained_async_notify = params->sq_drained_async_notify;
cmd.rnr_retry = params->rnr_retry;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_MODIFY_QP, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(
edev->efa_dev,
@ -103,14 +100,11 @@ int efa_com_query_qp(struct efa_com_dev *edev,
struct efa_admin_query_qp_resp resp;
int err;
cmd.aq_common_desc.opcode = EFA_ADMIN_QUERY_QP;
cmd.qp_handle = params->qp_handle;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_QUERY_QP, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to query qp-%u [%d]\n",
@ -135,12 +129,10 @@ int efa_com_destroy_qp(struct efa_com_dev *edev,
struct efa_com_admin_queue *aq = &edev->aq;
int err;
qp_cmd.aq_common_desc.opcode = EFA_ADMIN_DESTROY_QP;
qp_cmd.qp_handle = params->qp_handle;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&qp_cmd,
sizeof(qp_cmd),
err = efa_com_cmd_exec(aq, EFA_ADMIN_DESTROY_QP, 0,
&qp_cmd, sizeof(qp_cmd),
(struct efa_admin_acq_entry *)&cmd_completion,
sizeof(cmd_completion));
if (err) {
@ -162,7 +154,6 @@ int efa_com_create_cq(struct efa_com_dev *edev,
struct efa_com_admin_queue *aq = &edev->aq;
int err;
create_cmd.aq_common_desc.opcode = EFA_ADMIN_CREATE_CQ;
EFA_SET(&create_cmd.cq_caps_2,
EFA_ADMIN_CREATE_CQ_CMD_CQ_ENTRY_SIZE_WORDS,
params->entry_size_in_bytes / 4);
@ -178,13 +169,17 @@ int efa_com_create_cq(struct efa_com_dev *edev,
EFA_SET(&create_cmd.cq_caps_2,
EFA_ADMIN_CREATE_CQ_CMD_SET_SRC_ADDR, 1);
}
if (params->sq_comp_64_bit_req_id) {
EFA_SET(&create_cmd.cq_caps_2,
EFA_ADMIN_CREATE_CQ_CMD_SQ_COMP_64_BIT_REQ_ID, 1);
}
efa_com_set_dma_addr(params->dma_addr,
&create_cmd.cq_ba.mem_addr_high,
&create_cmd.cq_ba.mem_addr_low);
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&create_cmd,
sizeof(create_cmd),
err = efa_com_cmd_exec(aq, EFA_ADMIN_CREATE_CQ, 0,
&create_cmd, sizeof(create_cmd),
(struct efa_admin_acq_entry *)&cmd_completion,
sizeof(cmd_completion));
if (err) {
@ -211,11 +206,9 @@ int efa_com_destroy_cq(struct efa_com_dev *edev,
int err;
destroy_cmd.cq_idx = params->cq_idx;
destroy_cmd.aq_common_desc.opcode = EFA_ADMIN_DESTROY_CQ;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&destroy_cmd,
sizeof(destroy_cmd),
err = efa_com_cmd_exec(aq, EFA_ADMIN_DESTROY_CQ, 0,
&destroy_cmd, sizeof(destroy_cmd),
(struct efa_admin_acq_entry *)&destroy_resp,
sizeof(destroy_resp));
@ -236,9 +229,9 @@ int efa_com_register_mr(struct efa_com_dev *edev,
struct efa_admin_reg_mr_resp cmd_completion;
struct efa_com_admin_queue *aq = &edev->aq;
struct efa_admin_reg_mr_cmd mr_cmd = {};
u8 flags = 0;
int err;
mr_cmd.aq_common_desc.opcode = EFA_ADMIN_REG_MR;
mr_cmd.pd = params->pd;
mr_cmd.mr_length = params->mr_length_in_bytes;
EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT,
@ -256,16 +249,13 @@ int efa_com_register_mr(struct efa_com_dev *edev,
params->pbl.pbl.address.mem_addr_low;
mr_cmd.pbl.pbl.address.mem_addr_high =
params->pbl.pbl.address.mem_addr_high;
EFA_SET(&mr_cmd.aq_common_desc.flags,
EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA, 1);
EFA_SET(&flags, EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA, 1);
if (params->indirect)
EFA_SET(&mr_cmd.aq_common_desc.flags,
EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_INDIRECT, 1);
EFA_SET(&flags, EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_INDIRECT, 1);
}
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&mr_cmd,
sizeof(mr_cmd),
err = efa_com_cmd_exec(aq, EFA_ADMIN_REG_MR, flags,
&mr_cmd, sizeof(mr_cmd),
(struct efa_admin_acq_entry *)&cmd_completion,
sizeof(cmd_completion));
if (err) {
@ -297,12 +287,10 @@ int efa_com_dereg_mr(struct efa_com_dev *edev,
struct efa_admin_dereg_mr_cmd mr_cmd = {};
int err;
mr_cmd.aq_common_desc.opcode = EFA_ADMIN_DEREG_MR;
mr_cmd.l_key = params->l_key;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&mr_cmd,
sizeof(mr_cmd),
err = efa_com_cmd_exec(aq, EFA_ADMIN_DEREG_MR, 0,
&mr_cmd, sizeof(mr_cmd),
(struct efa_admin_acq_entry *)&cmd_completion,
sizeof(cmd_completion));
if (err) {
@ -322,26 +310,41 @@ int efa_com_create_ah(struct efa_com_dev *edev,
struct efa_admin_create_ah_resp cmd_completion;
struct efa_com_admin_queue *aq = &edev->aq;
struct efa_admin_create_ah_cmd ah_cmd = {};
struct efa_ah_cache_entry *entry;
int err;
ah_cmd.aq_common_desc.opcode = EFA_ADMIN_CREATE_AH;
entry = efa_ah_cache_get(&edev->ah_cache, params->pdn, params->dest_addr);
if (IS_ERR(entry))
return PTR_ERR(entry);
mutex_lock(&entry->lock);
if (entry->usecnt) {
result->ah = entry->ah;
entry->usecnt++;
mutex_unlock(&entry->lock);
return 0;
}
memcpy(ah_cmd.dest_addr, params->dest_addr, sizeof(ah_cmd.dest_addr));
ah_cmd.pd = params->pdn;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&ah_cmd,
sizeof(ah_cmd),
err = efa_com_cmd_exec(aq, EFA_ADMIN_CREATE_AH, 0,
&ah_cmd, sizeof(ah_cmd),
(struct efa_admin_acq_entry *)&cmd_completion,
sizeof(cmd_completion));
if (err) {
mutex_unlock(&entry->lock);
efa_ah_cache_put(&edev->ah_cache, entry);
ibdev_err_ratelimited(edev->efa_dev,
"Failed to create ah for %pI6 [%d]\n",
ah_cmd.dest_addr, err);
return err;
}
entry->ah = cmd_completion.ah;
result->ah = cmd_completion.ah;
entry->usecnt++;
mutex_unlock(&entry->lock);
return 0;
}
@ -352,25 +355,38 @@ int efa_com_destroy_ah(struct efa_com_dev *edev,
struct efa_admin_destroy_ah_resp cmd_completion;
struct efa_admin_destroy_ah_cmd ah_cmd = {};
struct efa_com_admin_queue *aq = &edev->aq;
int err;
struct efa_ah_cache_entry *entry;
int err = 0;
ah_cmd.aq_common_desc.opcode = EFA_ADMIN_DESTROY_AH;
ah_cmd.ah = params->ah;
ah_cmd.pd = params->pdn;
entry = efa_ah_cache_lookup(&edev->ah_cache, params->pdn, params->gid);
if (!entry)
return -EINVAL;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&ah_cmd,
sizeof(ah_cmd),
mutex_lock(&entry->lock);
if (entry->usecnt > 1)
goto out_put;
ah_cmd.ah = entry->ah;
ah_cmd.pd = entry->key.pd;
err = efa_com_cmd_exec(aq, EFA_ADMIN_DESTROY_AH, 0,
&ah_cmd, sizeof(ah_cmd),
(struct efa_admin_acq_entry *)&cmd_completion,
sizeof(cmd_completion));
if (err) {
mutex_unlock(&entry->lock);
ibdev_err_ratelimited(edev->efa_dev,
"Failed to destroy ah-%d pd-%d [%d]\n",
ah_cmd.ah, ah_cmd.pd, err);
return err;
}
return 0;
out_put:
entry->usecnt--;
mutex_unlock(&entry->lock);
efa_ah_cache_put(&edev->ah_cache, entry);
return err;
}
bool
@ -395,6 +411,7 @@ static int efa_com_get_feature_ex(struct efa_com_dev *edev,
{
struct efa_admin_get_feature_cmd get_cmd = {};
struct efa_com_admin_queue *aq;
u8 flags = 0;
int err;
if (!efa_com_check_supported_feature_id(edev, feature_id)) {
@ -406,11 +423,8 @@ static int efa_com_get_feature_ex(struct efa_com_dev *edev,
aq = &edev->aq;
get_cmd.aq_common_descriptor.opcode = EFA_ADMIN_GET_FEATURE;
if (control_buff_size)
EFA_SET(&get_cmd.aq_common_descriptor.flags,
EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA, 1);
EFA_SET(&flags, EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA, 1);
efa_com_set_dma_addr(control_buf_dma_addr,
&get_cmd.control_buffer.address.mem_addr_high,
@ -418,12 +432,9 @@ static int efa_com_get_feature_ex(struct efa_com_dev *edev,
get_cmd.control_buffer.length = control_buff_size;
get_cmd.feature_common.feature_id = feature_id;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)
&get_cmd,
sizeof(get_cmd),
(struct efa_admin_acq_entry *)
get_resp,
err = efa_com_cmd_exec(aq, EFA_ADMIN_GET_FEATURE, flags,
&get_cmd, sizeof(get_cmd),
(struct efa_admin_acq_entry *)get_resp,
sizeof(*get_resp));
if (err) {
@ -520,6 +531,10 @@ int efa_com_get_device_attr(struct efa_com_dev *edev,
}
result->inline_buf_size_ex = resp.u.queue_attr_2.inline_buf_size_ex;
result->max_event_counters = resp.u.queue_attr_2.max_event_counters;
result->event_counter_max_val = resp.u.queue_attr_2.event_counter_max_val;
result->supported_event_counter_qp_events =
resp.u.queue_attr_2.supported_event_counter_qp_events;
} else {
result->inline_buf_size_ex = result->inline_buf_size;
}
@ -585,6 +600,7 @@ int efa_com_set_feature_ex(struct efa_com_dev *edev,
u32 control_buff_size)
{
struct efa_com_admin_queue *aq;
u8 flags = 0;
int err;
if (!efa_com_check_supported_feature_id(edev, feature_id)) {
@ -596,11 +612,8 @@ int efa_com_set_feature_ex(struct efa_com_dev *edev,
aq = &edev->aq;
set_cmd->aq_common_descriptor.opcode = EFA_ADMIN_SET_FEATURE;
if (control_buff_size) {
set_cmd->aq_common_descriptor.flags = 0;
EFA_SET(&set_cmd->aq_common_descriptor.flags,
EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA, 1);
EFA_SET(&flags, EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA, 1);
efa_com_set_dma_addr(control_buf_dma_addr,
&set_cmd->control_buffer.address.mem_addr_high,
&set_cmd->control_buffer.address.mem_addr_low);
@ -608,9 +621,8 @@ int efa_com_set_feature_ex(struct efa_com_dev *edev,
set_cmd->control_buffer.length = control_buff_size;
set_cmd->feature_common.feature_id = feature_id;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)set_cmd,
sizeof(*set_cmd),
err = efa_com_cmd_exec(aq, EFA_ADMIN_SET_FEATURE, flags,
set_cmd, sizeof(*set_cmd),
(struct efa_admin_acq_entry *)set_resp,
sizeof(*set_resp));
@ -681,17 +693,12 @@ int efa_com_alloc_pd(struct efa_com_dev *edev,
struct efa_com_alloc_pd_result *result)
{
struct efa_com_admin_queue *aq = &edev->aq;
struct efa_admin_alloc_pd_cmd cmd = {};
struct efa_admin_alloc_pd_resp resp;
int err;
cmd.aq_common_descriptor.opcode = EFA_ADMIN_ALLOC_PD;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_ALLOC_PD, 0,
NULL, 0,
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to allocate pd[%d]\n", err);
@ -711,14 +718,11 @@ int efa_com_dealloc_pd(struct efa_com_dev *edev,
struct efa_admin_dealloc_pd_resp resp;
int err;
cmd.aq_common_descriptor.opcode = EFA_ADMIN_DEALLOC_PD;
cmd.pd = params->pdn;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_DEALLOC_PD, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to deallocate pd-%u [%d]\n",
@ -733,17 +737,12 @@ int efa_com_alloc_uar(struct efa_com_dev *edev,
struct efa_com_alloc_uar_result *result)
{
struct efa_com_admin_queue *aq = &edev->aq;
struct efa_admin_alloc_uar_cmd cmd = {};
struct efa_admin_alloc_uar_resp resp;
int err;
cmd.aq_common_descriptor.opcode = EFA_ADMIN_ALLOC_UAR;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_ALLOC_UAR, 0,
NULL, 0,
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to allocate uar[%d]\n", err);
@ -763,14 +762,11 @@ int efa_com_dealloc_uar(struct efa_com_dev *edev,
struct efa_admin_dealloc_uar_resp resp;
int err;
cmd.aq_common_descriptor.opcode = EFA_ADMIN_DEALLOC_UAR;
cmd.uar = params->uarn;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_DEALLOC_UAR, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to deallocate uar-%u [%d]\n",
@ -795,16 +791,13 @@ int efa_com_get_stats(struct efa_com_dev *edev,
struct efa_admin_basic_stats *bs;
int err;
cmd.aq_common_descriptor.opcode = EFA_ADMIN_GET_STATS;
cmd.type = params->type;
cmd.scope = params->scope;
cmd.scope_modifier = params->scope_modifier;
err = efa_com_cmd_exec(aq,
(struct efa_admin_aq_entry *)&cmd,
sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
err = efa_com_cmd_exec(aq, EFA_ADMIN_GET_STATS, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp, sizeof(resp));
if (err) {
ibdev_err_ratelimited(
edev->efa_dev,
@ -855,3 +848,131 @@ int efa_com_get_stats(struct efa_com_dev *edev,
return 0;
}
int efa_com_create_event_counter(struct efa_com_dev *edev,
struct efa_com_create_event_counter_params *params,
struct efa_com_create_event_counter_result *result)
{
struct efa_admin_create_event_counter_cmd cmd = {};
struct efa_admin_create_event_counter_resp resp;
struct efa_com_admin_queue *aq = &edev->aq;
int err;
cmd.uar = params->uarn;
cmd.paddr = params->dma_addr;
err = efa_com_cmd_exec(aq, EFA_ADMIN_CREATE_EVENT_COUNTER, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to create event counter [%d]\n",
err);
return err;
}
result->cntr_handle = resp.cntr_handle;
return 0;
}
int efa_com_destroy_event_counter(struct efa_com_dev *edev,
struct efa_com_destroy_event_counter_params *params)
{
struct efa_admin_destroy_event_counter_cmd cmd = {};
struct efa_admin_destroy_event_counter_resp resp;
struct efa_com_admin_queue *aq = &edev->aq;
int err;
cmd.cntr_handle = params->cntr_handle;
err = efa_com_cmd_exec(aq, EFA_ADMIN_DESTROY_EVENT_COUNTER, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to destroy event counter [%d]\n",
err);
return err;
}
return 0;
}
static int efa_com_attach_detach_event_counter(struct efa_com_dev *edev, u8 opcode,
u32 cntr_handle, u32 qp_handle,
u32 events)
{
struct efa_admin_attach_detach_event_counter_cmd cmd = {};
struct efa_admin_attach_detach_event_counter_resp resp;
struct efa_com_admin_queue *aq = &edev->aq;
int err;
cmd.cntr_handle = cntr_handle;
cmd.attach_type = EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS;
cmd.u.qp_events.qp_handle = qp_handle;
cmd.u.qp_events.events = events;
err = efa_com_cmd_exec(aq, opcode, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to %s event counter [%d]\n",
opcode == EFA_ADMIN_ATTACH_EVENT_COUNTER
? "attach"
: "detach",
err);
return err;
}
return 0;
}
int efa_com_attach_event_counter(struct efa_com_dev *edev,
struct efa_com_attach_event_counter_params *params)
{
return efa_com_attach_detach_event_counter(edev,
EFA_ADMIN_ATTACH_EVENT_COUNTER,
params->cntr_handle,
params->qp_handle,
params->events);
}
int efa_com_detach_event_counter(struct efa_com_dev *edev,
struct efa_com_detach_event_counter_params *params)
{
return efa_com_attach_detach_event_counter(edev,
EFA_ADMIN_DETACH_EVENT_COUNTER,
params->cntr_handle,
params->qp_handle,
params->events);
}
int efa_com_modify_event_counter(struct efa_com_dev *edev,
struct efa_com_modify_event_counter_params *params)
{
struct efa_admin_modify_event_counter_cmd cmd = {};
struct efa_admin_modify_event_counter_resp resp;
struct efa_com_admin_queue *aq = &edev->aq;
int err;
cmd.cntr_handle = params->cntr_handle;
cmd.operation = params->operation;
cmd.value = params->value;
err = efa_com_cmd_exec(aq, EFA_ADMIN_MODIFY_EVENT_COUNTER, 0,
&cmd, sizeof(cmd),
(struct efa_admin_acq_entry *)&resp,
sizeof(resp));
if (err) {
ibdev_err_ratelimited(edev->efa_dev,
"Failed to modify event counter [%d]\n",
err);
return err;
}
return 0;
}

48
drivers/infiniband/hw/efa/efa_com_cmd.h
View File

@ -29,6 +29,7 @@ struct efa_com_create_qp_params {
u8 qp_type;
u8 sl;
u8 unsolicited_write_recv : 1;
u8 sq_64_bit_req_id : 1;
};
struct efa_com_create_qp_result {
@ -79,6 +80,7 @@ struct efa_com_create_cq_params {
u8 entry_size_in_bytes;
u8 interrupt_mode_enabled : 1;
u8 set_src_addr : 1;
u8 sq_comp_64_bit_req_id : 1;
};
struct efa_com_create_cq_result {
@ -106,6 +108,7 @@ struct efa_com_create_ah_result {
struct efa_com_destroy_ah_params {
u16 ah;
u8 gid[EFA_GID_SIZE];
u16 pdn;
};
@ -145,6 +148,9 @@ struct efa_com_get_device_attr_result {
u16 min_sq_depth;
u16 max_link_speed_gbps;
u8 db_bar;
u32 max_event_counters;
u64 event_counter_max_val;
u32 supported_event_counter_qp_events;
};
struct efa_com_get_hw_hints_result {
@ -300,6 +306,37 @@ union efa_com_get_stats_result {
struct efa_com_network_stats network_stats;
};
struct efa_com_create_event_counter_params {
dma_addr_t dma_addr;
u16 uarn;
};
struct efa_com_create_event_counter_result {
u32 cntr_handle;
};
struct efa_com_destroy_event_counter_params {
u32 cntr_handle;
};
struct efa_com_attach_event_counter_params {
u32 cntr_handle;
u32 qp_handle;
u32 events;
};
struct efa_com_detach_event_counter_params {
u32 cntr_handle;
u32 qp_handle;
u32 events;
};
struct efa_com_modify_event_counter_params {
u32 cntr_handle;
u8 operation;
u64 value;
};
int efa_com_create_qp(struct efa_com_dev *edev,
struct efa_com_create_qp_params *params,
struct efa_com_create_qp_result *res);
@ -350,5 +387,16 @@ int efa_com_dealloc_uar(struct efa_com_dev *edev,
int efa_com_get_stats(struct efa_com_dev *edev,
struct efa_com_get_stats_params *params,
union efa_com_get_stats_result *result);
int efa_com_create_event_counter(struct efa_com_dev *edev,
struct efa_com_create_event_counter_params *params,
struct efa_com_create_event_counter_result *result);
int efa_com_destroy_event_counter(struct efa_com_dev *edev,
struct efa_com_destroy_event_counter_params *params);
int efa_com_attach_event_counter(struct efa_com_dev *edev,
struct efa_com_attach_event_counter_params *params);
int efa_com_detach_event_counter(struct efa_com_dev *edev,
struct efa_com_detach_event_counter_params *params);
int efa_com_modify_event_counter(struct efa_com_dev *edev,
struct efa_com_modify_event_counter_params *params);
#endif /* _EFA_COM_CMD_H_ */

32
drivers/infiniband/hw/efa/efa_io_defs.h
View File

@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
/*
* Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved.
* Copyright 2018-2026 Amazon.com, Inc. or its affiliates. All rights reserved.
*/
#ifndef _EFA_IO_H_
@ -65,6 +65,8 @@ enum efa_io_comp_status {
EFA_IO_COMP_STATUS_REMOTE_ERROR_UNKNOWN_PEER = 14,
/* Unreachable remote - never received a response */
EFA_IO_COMP_STATUS_LOCAL_ERROR_UNREACH_REMOTE = 15,
/* Remote feature mismatch */
EFA_IO_COMP_STATUS_REMOTE_ERROR_FEATURE_MISMATCH = 18,
};
enum efa_io_frwr_pbl_mode {
@ -72,6 +74,15 @@ enum efa_io_frwr_pbl_mode {
EFA_IO_FRWR_DIRECT_PBL = 1,
};
enum efa_io_processing_hint {
/* Optimize for throughput */
EFA_IO_PROCESSING_HINT_BURST_PPS_SENSITIVE = 1 << 0,
};
struct efa_io_req_id_ex {
u16 w[3];
};
struct efa_io_tx_meta_desc {
/* Verbs-generated Request ID */
u16 req_id;
@ -121,12 +132,22 @@ struct efa_io_tx_meta_desc {
u16 ah;
u16 reserved;
/*
* control flags
* 1:0 : processing_hints - Bitmask of enum
* efa_io_processing_hint
* 7:2 : reserved - MBZ
*/
u8 ctrl3;
u8 reserved;
/* Queue key */
u32 qkey;
u8 reserved2[12];
u8 reserved2[6];
struct efa_io_req_id_ex req_id_ex;
};
/*
@ -312,8 +333,10 @@ struct efa_io_tx_cdesc {
/* Common completion info */
struct efa_io_cdesc_common common;
struct efa_io_req_id_ex req_id_ex;
/* MBZ */
u16 reserved16;
u8 reserved[4];
};
/* Rx Completion Descriptor */
@ -365,6 +388,7 @@ struct efa_io_rx_cdesc_ex {
#define EFA_IO_TX_META_DESC_FIRST_MASK BIT(2)
#define EFA_IO_TX_META_DESC_LAST_MASK BIT(3)
#define EFA_IO_TX_META_DESC_COMP_REQ_MASK BIT(4)
#define EFA_IO_TX_META_DESC_PROCESSING_HINTS_MASK GENMASK(1, 0)
/* tx_buf_desc */
#define EFA_IO_TX_BUF_DESC_LKEY_MASK GENMASK(23, 0)

10
drivers/infiniband/hw/efa/efa_main.c
View File

@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
/*
* Copyright 2018-2025 Amazon.com, Inc. or its affiliates. All rights reserved.
* Copyright 2018-2026 Amazon.com, Inc. or its affiliates. All rights reserved.
*/
#include <linux/module.h>
@ -17,12 +17,14 @@
#define PCI_DEV_ID_EFA1_VF 0xefa1
#define PCI_DEV_ID_EFA2_VF 0xefa2
#define PCI_DEV_ID_EFA3_VF 0xefa3
#define PCI_DEV_ID_EFA4_VF 0xefa4
static const struct pci_device_id efa_pci_tbl[] = {
{ PCI_VDEVICE(AMAZON, PCI_DEV_ID_EFA0_VF) },
{ PCI_VDEVICE(AMAZON, PCI_DEV_ID_EFA1_VF) },
{ PCI_VDEVICE(AMAZON, PCI_DEV_ID_EFA2_VF) },
{ PCI_VDEVICE(AMAZON, PCI_DEV_ID_EFA3_VF) },
{ PCI_VDEVICE(AMAZON, PCI_DEV_ID_EFA4_VF) },
{ }
};
@ -372,20 +374,25 @@ static const struct ib_device_ops efa_dev_ops = {
.alloc_pd = efa_alloc_pd,
.alloc_ucontext = efa_alloc_ucontext,
.create_user_cq = efa_create_user_cq,
.create_comp_cntr = efa_create_comp_cntr,
.create_qp = efa_create_qp,
.create_user_ah = efa_create_ah,
.dealloc_pd = efa_dealloc_pd,
.dealloc_ucontext = efa_dealloc_ucontext,
.dereg_mr = efa_dereg_mr,
.destroy_ah = efa_destroy_ah,
.destroy_comp_cntr = efa_destroy_comp_cntr,
.destroy_cq = efa_destroy_cq,
.destroy_qp = efa_destroy_qp,
.get_hw_stats = efa_get_hw_stats,
.get_link_layer = efa_port_link_layer,
.get_port_immutable = efa_get_port_immutable,
.modify_comp_cntr = efa_modify_comp_cntr,
.mmap = efa_mmap,
.mmap_free = efa_mmap_free,
.modify_qp = efa_modify_qp,
.qp_attach_comp_cntr = efa_qp_attach_comp_cntr,
.query_comp_cntr_caps = efa_query_comp_cntr_caps,
.query_device = efa_query_device,
.query_gid = efa_query_gid,
.query_pkey = efa_query_pkey,
@ -397,6 +404,7 @@ static const struct ib_device_ops efa_dev_ops = {
INIT_RDMA_OBJ_SIZE(ib_ah, efa_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_cq, efa_cq, ibcq),
INIT_RDMA_OBJ_SIZE(ib_comp_cntr, efa_comp_cntr, ibcc),
INIT_RDMA_OBJ_SIZE(ib_pd, efa_pd, ibpd),
INIT_RDMA_OBJ_SIZE(ib_qp, efa_qp, ibqp),
INIT_RDMA_OBJ_SIZE(ib_ucontext, efa_ucontext, ibucontext),

272
drivers/infiniband/hw/efa/efa_verbs.c
View File

@ -167,6 +167,11 @@ static inline struct efa_ah *to_eah(struct ib_ah *ibah)
return container_of(ibah, struct efa_ah, ibah);
}
static inline struct efa_comp_cntr *to_ecc(struct ib_comp_cntr *ibcc)
{
return container_of(ibcc, struct efa_comp_cntr, ibcc);
}
static inline struct efa_user_mmap_entry *
to_emmap(struct rdma_user_mmap_entry *rdma_entry)
{
@ -222,7 +227,6 @@ int efa_query_device(struct ib_device *ibdev,
dev_attr = &dev->dev_attr;
memset(props, 0, sizeof(*props));
props->max_mr_size = dev_attr->max_mr_pages * PAGE_SIZE;
props->page_size_cap = dev_attr->page_size_cap;
props->vendor_id = dev->pdev->vendor;
@ -265,6 +269,12 @@ int efa_query_device(struct ib_device *ibdev,
if (EFA_DEV_CAP(dev, UNSOLICITED_WRITE_RECV))
resp.device_caps |= EFA_QUERY_DEVICE_CAPS_UNSOLICITED_WRITE_RECV;
if (EFA_DEV_CAP(dev, EVENT_COUNTERS))
resp.device_caps |= EFA_QUERY_DEVICE_CAPS_COMP_CNTR;
if (EFA_DEV_CAP(dev, SQ_64_BIT_REQ_ID))
resp.device_caps |= EFA_QUERY_DEVICE_CAPS_SQ_64_BIT_REQ_ID;
if (dev->neqs)
resp.device_caps |= EFA_QUERY_DEVICE_CAPS_CQ_NOTIFICATIONS;
@ -712,6 +722,9 @@ int efa_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
if (EFA_DEV_CAP(dev, UNSOLICITED_WRITE_RECV))
supported_efa_flags |= EFA_CREATE_QP_WITH_UNSOLICITED_WRITE_RECV;
if (EFA_DEV_CAP(dev, SQ_64_BIT_REQ_ID))
supported_efa_flags |= EFA_CREATE_QP_WITH_SQ_64_BIT_REQ_ID;
if (cmd.flags & ~supported_efa_flags) {
ibdev_dbg(&dev->ibdev, "Unsupported EFA QP create flags[%#x], supported[%#x]\n",
cmd.flags, supported_efa_flags);
@ -771,6 +784,9 @@ int efa_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
if (cmd.flags & EFA_CREATE_QP_WITH_UNSOLICITED_WRITE_RECV)
create_qp_params.unsolicited_write_recv = true;
if (cmd.flags & EFA_CREATE_QP_WITH_SQ_64_BIT_REQ_ID)
create_qp_params.sq_64_bit_req_id = true;
err = efa_com_create_qp(&dev->edev, &create_qp_params,
&create_qp_resp);
if (err)
@ -1205,6 +1221,7 @@ int efa_create_user_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
params.entry_size_in_bytes = cmd.cq_entry_size;
params.num_sub_cqs = cmd.num_sub_cqs;
params.set_src_addr = set_src_addr;
params.sq_comp_64_bit_req_id = !!(cmd.flags & EFA_CREATE_CQ_WITH_SQ_COMP_64_BIT_REQ_ID);
if (cmd.flags & EFA_CREATE_CQ_WITH_COMPLETION_CHANNEL) {
cq->eq = efa_vec2eq(dev, attr->comp_vector);
params.eqn = cq->eq->eeq.eqn;
@ -1345,9 +1362,11 @@ static int pbl_chunk_list_create(struct efa_dev *dev, struct pbl_context *pbl)
chunk_list->chunks[i].length = EFA_CHUNK_USED_SIZE;
}
chunk_list->chunks[chunk_list_size - 1].length =
((page_cnt % EFA_PTRS_PER_CHUNK) * EFA_CHUNK_PAYLOAD_PTR_SIZE) +
EFA_CHUNK_PTR_SIZE;
if (page_cnt % EFA_PTRS_PER_CHUNK != 0)
chunk_list->chunks[chunk_list_size - 1].length =
((page_cnt % EFA_PTRS_PER_CHUNK) * EFA_CHUNK_PAYLOAD_PTR_SIZE) +
EFA_CHUNK_PTR_SIZE;
/* fill the dma addresses of sg list pages to chunks: */
chunk_idx = 0;
@ -1359,9 +1378,12 @@ static int pbl_chunk_list_create(struct efa_dev *dev, struct pbl_context *pbl)
rdma_block_iter_dma_address(&biter);
if (payload_idx == EFA_PTRS_PER_CHUNK) {
chunk_idx++;
cur_chunk_buf = chunk_list->chunks[chunk_idx].buf;
payload_idx = 0;
chunk_idx++;
if (chunk_idx >= chunk_list_size)
break;
cur_chunk_buf = chunk_list->chunks[chunk_idx].buf;
}
}
@ -2054,10 +2076,11 @@ int efa_mmap(struct ib_ucontext *ibucontext,
static int efa_ah_destroy(struct efa_dev *dev, struct efa_ah *ah)
{
struct efa_com_destroy_ah_params params = {
.ah = ah->ah,
.pdn = to_epd(ah->ibah.pd)->pdn,
};
struct efa_com_destroy_ah_params params = {};
params.ah = ah->ah;
memcpy(params.gid, ah->id, sizeof(params.gid));
params.pdn = to_epd(ah->ibah.pd)->pdn;
return efa_com_destroy_ah(&dev->edev, &params);
}
@ -2260,6 +2283,220 @@ enum rdma_link_layer efa_port_link_layer(struct ib_device *ibdev,
return IB_LINK_LAYER_UNSPECIFIED;
}
int efa_query_comp_cntr_caps(struct ib_device *ibdev,
struct ib_comp_cntr_caps *caps,
struct uverbs_attr_bundle *attrs)
{
struct efa_dev *dev = to_edev(ibdev);
u32 dev_ops = dev->dev_attr.supported_event_counter_qp_events;
caps->max_counters = dev->dev_attr.max_event_counters / 2;
caps->max_value = dev->dev_attr.event_counter_max_val;
caps->supported_qp_attach_ops = 0;
if (EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_SEND_COMP) &&
EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_SEND_COMP_ERR))
caps->supported_qp_attach_ops |= IB_QP_ATTACH_COMP_CNTR_OP_SEND;
if (EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_RECV_COMP) &&
EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_RECV_COMP_ERR))
caps->supported_qp_attach_ops |= IB_QP_ATTACH_COMP_CNTR_OP_RECV;
if (EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_READ_COMP) &&
EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_READ_COMP_ERR))
caps->supported_qp_attach_ops |= IB_QP_ATTACH_COMP_CNTR_OP_RDMA_READ;
if (EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_WRITE_COMP) &&
EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_WRITE_COMP_ERR))
caps->supported_qp_attach_ops |= IB_QP_ATTACH_COMP_CNTR_OP_RDMA_WRITE;
if (EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_REMOTE_READ_COMP))
caps->supported_qp_attach_ops |= IB_QP_ATTACH_COMP_CNTR_OP_REMOTE_RDMA_READ;
if (EFA_GET(&dev_ops, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_REMOTE_WRITE_COMP))
caps->supported_qp_attach_ops |= IB_QP_ATTACH_COMP_CNTR_OP_REMOTE_RDMA_WRITE;
return 0;
}
static int efa_create_event_counter(struct efa_dev *dev, u16 uarn, dma_addr_t addr, u32 *handle)
{
struct efa_com_create_event_counter_params params = {};
struct efa_com_create_event_counter_result result;
int err;
params.uarn = uarn;
params.dma_addr = addr;
err = efa_com_create_event_counter(&dev->edev, &params, &result);
if (err)
return err;
*handle = result.cntr_handle;
return 0;
}
static int efa_destroy_event_counter(struct efa_dev *dev, u32 handle)
{
struct efa_com_destroy_event_counter_params params = {
.cntr_handle = handle,
};
return efa_com_destroy_event_counter(&dev->edev, &params);
}
int efa_create_comp_cntr(struct ib_comp_cntr *ibcc, struct uverbs_attr_bundle *attrs)
{
struct efa_dev *dev = to_edev(ibcc->device);
struct efa_comp_cntr *cc = to_ecc(ibcc);
struct efa_ucontext *ucontext;
struct ib_umem *comp_umem;
struct ib_umem *err_umem;
dma_addr_t comp_addr;
dma_addr_t err_addr;
int err;
ucontext = rdma_udata_to_drv_context(&attrs->driver_udata, struct efa_ucontext,
ibucontext);
comp_umem = ib_umem_get_attr(ibcc->device, attrs, EFA_IB_ATTR_CREATE_COMP_CNTR_COMP_BUFFER,
sizeof(u64), IB_ACCESS_LOCAL_WRITE);
if (IS_ERR(comp_umem))
return PTR_ERR(comp_umem);
err_umem = ib_umem_get_attr(ibcc->device, attrs, EFA_IB_ATTR_CREATE_COMP_CNTR_ERR_BUFFER,
sizeof(u64), IB_ACCESS_LOCAL_WRITE);
if (IS_ERR(err_umem)) {
err = PTR_ERR(err_umem);
goto err_comp_umem;
}
comp_addr = ib_umem_start_dma_addr(comp_umem);
err_addr = ib_umem_start_dma_addr(err_umem);
if (!IS_ALIGNED(comp_addr, sizeof(u64)) || !IS_ALIGNED(err_addr, sizeof(u64))) {
ibdev_dbg(&dev->ibdev, "Completion Counter memory is unaligned\n");
err = -EINVAL;
goto err_err_umem;
}
err = efa_create_event_counter(dev, ucontext->uarn, comp_addr, &cc->comp_handle);
if (err) {
ibdev_dbg(&dev->ibdev, "Failed to create comp event counter [%d]\n", err);
goto err_err_umem;
}
err = efa_create_event_counter(dev, ucontext->uarn, err_addr, &cc->err_handle);
if (err) {
ibdev_dbg(&dev->ibdev, "Failed to create err event counter [%d]\n", err);
goto err_destroy_comp_event_cntr;
}
cc->comp_umem = comp_umem;
cc->err_umem = err_umem;
return 0;
err_destroy_comp_event_cntr:
efa_destroy_event_counter(dev, cc->comp_handle);
err_err_umem:
ib_umem_release(err_umem);
err_comp_umem:
ib_umem_release(comp_umem);
return err;
}
int efa_destroy_comp_cntr(struct ib_comp_cntr *ibcc)
{
struct efa_dev *dev = to_edev(ibcc->device);
struct efa_comp_cntr *cc = to_ecc(ibcc);
efa_destroy_event_counter(dev, cc->comp_handle);
efa_destroy_event_counter(dev, cc->err_handle);
ib_umem_release(cc->comp_umem);
ib_umem_release(cc->err_umem);
return 0;
}
int efa_modify_comp_cntr(struct ib_comp_cntr *ibcc, enum ib_comp_cntr_entry entry,
enum ib_comp_cntr_modify_op op, u64 value)
{
struct efa_com_modify_event_counter_params params = {};
struct efa_comp_cntr *cc = to_ecc(ibcc);
params.cntr_handle = entry == IB_COMP_CNTR_ENTRY_ERR ? cc->err_handle : cc->comp_handle;
params.operation = op == IB_COMP_CNTR_MODIFY_OP_SET ?
EFA_ADMIN_EVENT_COUNTER_MODIFY_SET : EFA_ADMIN_EVENT_COUNTER_MODIFY_ADD;
params.value = value;
return efa_com_modify_event_counter(&to_edev(ibcc->device)->edev, &params);
}
static u32 efa_comp_cntr_op_to_comp_events(u32 op_mask)
{
u32 events = 0;
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_SEND)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_SEND_COMP, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_RECV)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_RECV_COMP, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_RDMA_READ)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_READ_COMP, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_REMOTE_RDMA_READ)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_REMOTE_READ_COMP, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_RDMA_WRITE)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_WRITE_COMP, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_REMOTE_RDMA_WRITE)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_REMOTE_WRITE_COMP, 1);
return events;
}
static u32 efa_comp_cntr_op_to_err_events(u32 op_mask)
{
u32 events = 0;
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_SEND)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_SEND_COMP_ERR, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_RECV)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_RECV_COMP_ERR, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_RDMA_READ)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_READ_COMP_ERR, 1);
if (op_mask & IB_QP_ATTACH_COMP_CNTR_OP_RDMA_WRITE)
EFA_SET(&events, EFA_ADMIN_EVENT_COUNTER_ATTACH_QP_EVENTS_WRITE_COMP_ERR, 1);
return events;
}
int efa_qp_attach_comp_cntr(struct ib_qp *ibqp, struct ib_comp_cntr *ibcc,
struct ib_qp_attach_comp_cntr_attr *attr)
{
struct efa_com_detach_event_counter_params detach_params = {};
struct efa_com_attach_event_counter_params params = {};
struct efa_dev *dev = to_edev(ibqp->device);
struct efa_comp_cntr *cc = to_ecc(ibcc);
struct efa_qp *qp = to_eqp(ibqp);
int err;
params.cntr_handle = cc->comp_handle;
params.qp_handle = qp->qp_handle;
params.events = efa_comp_cntr_op_to_comp_events(attr->op_mask);
err = efa_com_attach_event_counter(&dev->edev, &params);
if (err)
return err;
params.cntr_handle = cc->err_handle;
params.events = efa_comp_cntr_op_to_err_events(attr->op_mask);
err = efa_com_attach_event_counter(&dev->edev, &params);
if (err) {
detach_params.cntr_handle = cc->comp_handle;
detach_params.qp_handle = qp->qp_handle;
detach_params.events = efa_comp_cntr_op_to_comp_events(attr->op_mask);
efa_com_detach_event_counter(&dev->edev, &detach_params);
return err;
}
return 0;
}
DECLARE_UVERBS_NAMED_METHOD(EFA_IB_METHOD_MR_QUERY,
UVERBS_ATTR_IDR(EFA_IB_ATTR_QUERY_MR_HANDLE,
UVERBS_OBJECT_MR,
@ -2282,8 +2519,23 @@ ADD_UVERBS_METHODS(efa_mr,
UVERBS_OBJECT_MR,
&UVERBS_METHOD(EFA_IB_METHOD_MR_QUERY));
ADD_UVERBS_ATTRIBUTES_SIMPLE(
efa_comp_cntr_create,
UVERBS_OBJECT_COMP_CNTR,
UVERBS_METHOD_COMP_CNTR_CREATE,
UVERBS_ATTR_PTR_IN(
EFA_IB_ATTR_CREATE_COMP_CNTR_COMP_BUFFER,
UVERBS_ATTR_STRUCT(struct ib_uverbs_buffer_desc, length),
UA_MANDATORY),
UVERBS_ATTR_PTR_IN(
EFA_IB_ATTR_CREATE_COMP_CNTR_ERR_BUFFER,
UVERBS_ATTR_STRUCT(struct ib_uverbs_buffer_desc, length),
UA_MANDATORY));
const struct uapi_definition efa_uapi_defs[] = {
UAPI_DEF_CHAIN_OBJ_TREE(UVERBS_OBJECT_MR,
&efa_mr),
UAPI_DEF_CHAIN_OBJ_TREE(UVERBS_OBJECT_COMP_CNTR,
&efa_comp_cntr_create),
{},
};

2
drivers/infiniband/hw/erdma/Kconfig
View File

@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
config INFINIBAND_ERDMA
tristate "Alibaba Elastic RDMA Adapter (ERDMA) support"
depends on PCI_MSI && 64BIT
depends on PCI_MSI && 64BIT && !CPU_BIG_ENDIAN
depends on INFINIBAND_ADDR_TRANS
depends on INFINIBAND_USER_ACCESS
help

6
drivers/infiniband/hw/erdma/erdma_cm.c
View File

@ -1021,10 +1021,9 @@ int erdma_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
if (laddr->sa_family != AF_INET || raddr->sa_family != AF_INET)
return -EAFNOSUPPORT;
qp = find_qp_by_qpn(dev, params->qpn);
qp = erdma_qp_get_by_qpn(dev, params->qpn);
if (!qp)
return -ENOENT;
erdma_qp_get(qp);
ret = sock_create(AF_INET, SOCK_STREAM, IPPROTO_TCP, &s);
if (ret < 0)
@ -1154,10 +1153,9 @@ int erdma_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
return -ECONNRESET;
}
qp = find_qp_by_qpn(dev, params->qpn);
qp = erdma_qp_get_by_qpn(dev, params->qpn);
if (!qp)
return -ENOENT;
erdma_qp_get(qp);
down_write(&qp->state_lock);
if (qp->attrs.iwarp.state > ERDMA_QPS_IWARP_RTR) {

10
drivers/infiniband/hw/erdma/erdma_eq.c
View File

@ -52,7 +52,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev)
if (FIELD_GET(ERDMA_AEQE_HDR_TYPE_MASK,
le32_to_cpu(aeqe->hdr)) == ERDMA_AE_TYPE_CQ_ERR) {
cqn = le32_to_cpu(aeqe->event_data0);
cq = find_cq_by_cqn(dev, cqn);
cq = erdma_cq_get_by_cqn(dev, cqn);
if (!cq)
continue;
@ -62,9 +62,10 @@ void erdma_aeq_event_handler(struct erdma_dev *dev)
if (cq->ibcq.event_handler)
cq->ibcq.event_handler(&event,
cq->ibcq.cq_context);
erdma_cq_put(cq);
} else {
qpn = le32_to_cpu(aeqe->event_data0);
qp = find_qp_by_qpn(dev, qpn);
qp = erdma_qp_get_by_qpn(dev, qpn);
if (!qp)
continue;
@ -74,6 +75,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev)
if (qp->ibqp.event_handler)
qp->ibqp.event_handler(&event,
qp->ibqp.qp_context);
erdma_qp_put(qp);
}
}
@ -157,7 +159,7 @@ void erdma_ceq_completion_handler(struct erdma_eq_cb *ceq_cb)
poll_cnt++;
cqn = FIELD_GET(ERDMA_CEQE_HDR_CQN_MASK, READ_ONCE(*ceqe));
cq = find_cq_by_cqn(dev, cqn);
cq = erdma_cq_get_by_cqn(dev, cqn);
if (!cq)
continue;
@ -166,6 +168,7 @@ void erdma_ceq_completion_handler(struct erdma_eq_cb *ceq_cb)
if (cq->ibcq.comp_handler)
cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
erdma_cq_put(cq);
}
notify_eq(&ceq_cb->eq);
@ -220,6 +223,7 @@ static void erdma_free_ceq_irq(struct erdma_dev *dev, u16 ceqn)
irq_set_affinity_hint(eqc->irq.msix_vector, NULL);
free_irq(eqc->irq.msix_vector, eqc);
tasklet_kill(&eqc->tasklet);
}
static int create_eq_cmd(struct erdma_dev *dev, u32 eqn, struct erdma_eq *eq)

45
drivers/infiniband/hw/erdma/erdma_verbs.c
View File

@ -320,12 +320,10 @@ int erdma_query_device(struct ib_device *ibdev, struct ib_device_attr *attr,
struct erdma_dev *dev = to_edev(ibdev);
int err;
err = ib_is_udata_in_empty(udata);
err = ib_no_udata_io(udata);
if (err)
return err;
memset(attr, 0, sizeof(*attr));
attr->max_mr_size = dev->attrs.max_mr_size;
attr->vendor_id = PCI_VENDOR_ID_ALIBABA;
attr->vendor_part_id = dev->pdev->device;
@ -363,7 +361,7 @@ int erdma_query_device(struct ib_device *ibdev, struct ib_device_attr *attr,
addrconf_addr_eui48((u8 *)&attr->sys_image_guid,
dev->netdev->dev_addr);
return ib_respond_empty_udata(udata);
return 0;
}
int erdma_query_gid(struct ib_device *ibdev, u32 port, int idx,
@ -1304,8 +1302,15 @@ int erdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
true);
/*
* A timeout disables the command queue, so retry cannot succeed. Treat
* terminal command failures as diagnostic; propagating them can make
* forced uverbs cleanup discard the last software resource pointers.
*/
if (ret)
return ret;
ibdev_warn_ratelimited(&dev->ibdev,
"failed to deregister MR 0x%x: %d\n",
ibmr->lkey, ret);
erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_STAG_IDX], ibmr->lkey >> 8);
@ -1321,6 +1326,7 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
struct erdma_dev *dev = to_edev(ibcq->device);
struct erdma_ucontext *ctx = rdma_udata_to_drv_context(
udata, struct erdma_ucontext, ibucontext);
unsigned long flags;
int err;
struct erdma_cmdq_destroy_cq_req req;
@ -1331,7 +1337,16 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
true);
if (err)
return err;
ibdev_warn_ratelimited(&dev->ibdev,
"failed to destroy CQ %u: %d\n",
cq->cqn, err);
xa_lock_irqsave(&dev->cq_xa, flags);
__xa_erase(&dev->cq_xa, cq->cqn);
xa_unlock_irqrestore(&dev->cq_xa, flags);
erdma_cq_put(cq);
wait_for_completion(&cq->free);
if (rdma_is_kernel_res(&cq->ibcq.res)) {
dma_free_coherent(&dev->pdev->dev, cq->depth << CQE_SHIFT,
@ -1343,8 +1358,6 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
put_mtt_entries(dev, &cq->user_cq.qbuf_mem);
}
xa_erase(&dev->cq_xa, cq->cqn);
return 0;
}
@ -1356,6 +1369,7 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
udata, struct erdma_ucontext, ibucontext);
struct erdma_cmdq_destroy_qp_req req;
union erdma_mod_qp_params params;
unsigned long flags;
int err;
down_write(&qp->state_lock);
@ -1379,7 +1393,13 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
true);
if (err)
return err;
ibdev_warn_ratelimited(&dev->ibdev,
"failed to destroy QP %u: %d\n",
QP_ID(qp), err);
xa_lock_irqsave(&dev->qp_xa, flags);
__xa_erase(&dev->qp_xa, QP_ID(qp));
xa_unlock_irqrestore(&dev->qp_xa, flags);
erdma_qp_put(qp);
wait_for_completion(&qp->safe_free);
@ -1394,7 +1414,6 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
if (qp->cep)
erdma_cep_put(qp->cep);
xa_erase(&dev->qp_xa, QP_ID(qp));
return 0;
}
@ -1971,6 +1990,8 @@ int erdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
cq->ibcq.cqe = depth;
cq->depth = depth;
cq->assoc_eqn = attr->comp_vector + 1;
refcount_set(&cq->refcount, 1);
init_completion(&cq->free);
ret = xa_alloc_cyclic(&dev->cq_xa, &cq->cqn, cq,
XA_LIMIT(1, dev->attrs.max_cq - 1),
@ -2281,7 +2302,9 @@ int erdma_destroy_ah(struct ib_ah *ibah, u32 flags)
ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
flags & RDMA_DESTROY_AH_SLEEPABLE);
if (ret)
return ret;
ibdev_warn_ratelimited(&dev->ibdev,
"failed to destroy AH %u: %d\n",
ah->ahn, ret);
erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_AH], ah->ahn);

40
drivers/infiniband/hw/erdma/erdma_verbs.h
View File

@ -7,6 +7,9 @@
#ifndef __ERDMA_VERBS_H__
#define __ERDMA_VERBS_H__
#include <linux/completion.h>
#include <linux/refcount.h>
#include "erdma.h"
/* RDMA Capability. */
@ -341,6 +344,8 @@ struct erdma_cq {
u32 depth;
u32 assoc_eqn;
refcount_t refcount;
struct completion free;
union {
struct erdma_kcq_info kern_cq;
@ -355,9 +360,40 @@ static inline struct erdma_qp *find_qp_by_qpn(struct erdma_dev *dev, int id)
return (struct erdma_qp *)xa_load(&dev->qp_xa, id);
}
static inline struct erdma_cq *find_cq_by_cqn(struct erdma_dev *dev, int id)
static inline struct erdma_qp *erdma_qp_get_by_qpn(struct erdma_dev *dev,
int id)
{
return (struct erdma_cq *)xa_load(&dev->cq_xa, id);
struct erdma_qp *qp;
unsigned long flags;
xa_lock_irqsave(&dev->qp_xa, flags);
qp = xa_load(&dev->qp_xa, id);
if (qp && !kref_get_unless_zero(&qp->ref))
qp = NULL;
xa_unlock_irqrestore(&dev->qp_xa, flags);
return qp;
}
static inline struct erdma_cq *erdma_cq_get_by_cqn(struct erdma_dev *dev,
int id)
{
struct erdma_cq *cq;
unsigned long flags;
xa_lock_irqsave(&dev->cq_xa, flags);
cq = xa_load(&dev->cq_xa, id);
if (cq && !refcount_inc_not_zero(&cq->refcount))
cq = NULL;
xa_unlock_irqrestore(&dev->cq_xa, flags);
return cq;
}
static inline void erdma_cq_put(struct erdma_cq *cq)
{
if (refcount_dec_and_test(&cq->refcount))
complete(&cq->free);
}
void erdma_qp_get(struct erdma_qp *qp);

21
drivers/infiniband/hw/hfi1/chip.c
View File

@ -11929,26 +11929,27 @@ u8 encode_rcv_header_entry_size(u8 size)
/**
* hfi1_validate_rcvhdrcnt - validate hdrcnt
* @dd: the device data
* @pdev: the PCI device
* @thecnt: the header count
*/
int hfi1_validate_rcvhdrcnt(struct hfi1_devdata *dd, uint thecnt)
int hfi1_validate_rcvhdrcnt(struct pci_dev *pdev, uint thecnt)
{
if (thecnt <= HFI1_MIN_HDRQ_EGRBUF_CNT) {
dd_dev_err(dd, "Receive header queue count too small\n");
dev_err(&pdev->dev, "Receive header queue count too small\n");
return -EINVAL;
}
if (thecnt > HFI1_MAX_HDRQ_EGRBUF_CNT) {
dd_dev_err(dd,
"Receive header queue count cannot be greater than %u\n",
HFI1_MAX_HDRQ_EGRBUF_CNT);
dev_err(&pdev->dev,
"Receive header queue count cannot be greater than %u\n",
HFI1_MAX_HDRQ_EGRBUF_CNT);
return -EINVAL;
}
if (thecnt % HDRQ_INCREMENT) {
dd_dev_err(dd, "Receive header queue count %d must be divisible by %lu\n",
thecnt, HDRQ_INCREMENT);
dev_err(&pdev->dev,
"Receive header queue count %u must be divisible by %lu\n",
thecnt, HDRQ_INCREMENT);
return -EINVAL;
}
@ -15007,7 +15008,7 @@ int hfi1_init_dd(struct hfi1_devdata *dd)
*/
ret = hfi1_pcie_ddinit(dd, pdev);
if (ret < 0)
goto bail_free;
goto bail;
/* Save PCI space registers to rewrite after device reset */
ret = save_pci_variables(dd);
@ -15262,8 +15263,6 @@ int hfi1_init_dd(struct hfi1_devdata *dd)
bail_cleanup:
hfi1_free_rx(dd);
hfi1_pcie_ddcleanup(dd);
bail_free:
hfi1_free_devdata(dd);
bail:
return ret;
}

2
drivers/infiniband/hw/hfi1/chip.h
View File

@ -660,7 +660,7 @@ static inline u32 chip_rcv_array_count(struct hfi1_devdata *dd)
}
u8 encode_rcv_header_entry_size(u8 size);
int hfi1_validate_rcvhdrcnt(struct hfi1_devdata *dd, uint thecnt);
int hfi1_validate_rcvhdrcnt(struct pci_dev *pdev, uint thecnt);
void set_hdrq_regs(struct hfi1_devdata *dd, u8 ctxt, u8 entsize, u16 hdrcnt);
u64 create_pbc(struct hfi1_pportdata *ppd, u64 flags, int srate_mbs, u32 vl,

32
drivers/infiniband/hw/hfi1/device.c
View File

@ -10,18 +10,6 @@
#include "hfi.h"
#include "device.h"
static char *hfi1_devnode(const struct device *dev, umode_t *mode)
{
if (mode)
*mode = 0600;
return kasprintf(GFP_KERNEL, "%s", dev_name(dev));
}
static const struct class class = {
.name = "hfi1",
.devnode = hfi1_devnode,
};
static char *hfi1_user_devnode(const struct device *dev, umode_t *mode)
{
if (mode)
@ -38,7 +26,6 @@ static dev_t hfi1_dev;
int hfi1_cdev_init(int minor, const char *name,
const struct file_operations *fops,
struct cdev *cdev, struct device **devp,
bool user_accessible,
struct kobject *parent)
{
const dev_t dev = MKDEV(MAJOR(hfi1_dev), minor);
@ -57,10 +44,7 @@ int hfi1_cdev_init(int minor, const char *name,
goto done;
}
if (user_accessible)
device = device_create(&user_class, NULL, dev, NULL, "%s", name);
else
device = device_create(&class, NULL, dev, NULL, "%s", name);
device = device_create(&user_class, NULL, dev, NULL, "%s", name);
if (IS_ERR(device)) {
ret = PTR_ERR(device);
@ -100,33 +84,21 @@ int __init dev_init(void)
ret = alloc_chrdev_region(&hfi1_dev, 0, HFI1_NMINORS, DRIVER_NAME);
if (ret < 0) {
pr_err("Could not allocate chrdev region (err %d)\n", -ret);
goto done;
}
ret = class_register(&class);
if (ret) {
pr_err("Could not create device class (err %d)\n", -ret);
unregister_chrdev_region(hfi1_dev, HFI1_NMINORS);
goto done;
return ret;
}
ret = class_register(&user_class);
if (ret) {
pr_err("Could not create device class for user accessible files (err %d)\n",
-ret);
class_unregister(&class);
unregister_chrdev_region(hfi1_dev, HFI1_NMINORS);
goto done;
}
done:
return ret;
}
void dev_cleanup(void)
{
class_unregister(&class);
class_unregister(&user_class);
unregister_chrdev_region(hfi1_dev, HFI1_NMINORS);
}

1
drivers/infiniband/hw/hfi1/device.h
View File

@ -9,7 +9,6 @@
int hfi1_cdev_init(int minor, const char *name,
const struct file_operations *fops,
struct cdev *cdev, struct device **devp,
bool user_accessible,
struct kobject *parent);
void hfi1_cdev_cleanup(struct cdev *cdev, struct device **devp);
const char *class_name(void);

2
drivers/infiniband/hw/hfi1/file_ops.c
View File

@ -1689,7 +1689,7 @@ static int user_add(struct hfi1_devdata *dd)
snprintf(name, sizeof(name), "%s_%d", class_name(), dd->unit);
ret = hfi1_cdev_init(dd->unit, name, &hfi1_file_ops,
&dd->user_cdev, &dd->user_device,
true, &dd->verbs_dev.rdi.ibdev.dev.kobj);
&dd->verbs_dev.rdi.ibdev.dev.kobj);
if (ret)
user_remove(dd);

4
drivers/infiniband/hw/hfi1/hfi.h
View File

@ -2024,9 +2024,7 @@ struct cc_state *get_cc_state_protected(struct hfi1_pportdata *ppd)
/* waiting for an urgent packet to arrive */
#define HFI1_CTXT_WAITING_URG 4
/* free up any allocated data at closes */
int hfi1_init_dd(struct hfi1_devdata *dd);
void hfi1_free_devdata(struct hfi1_devdata *dd);
/* LED beaconing functions */
void hfi1_start_led_override(struct hfi1_pportdata *ppd, unsigned int timeon,
@ -2132,7 +2130,7 @@ void hfi1_verbs_unregister_sysfs(struct hfi1_devdata *dd);
/* Hook for sysfs read of QSFP */
int qsfp_dump(struct hfi1_pportdata *ppd, char *buf, int len);
int hfi1_pcie_init(struct hfi1_devdata *dd);
int hfi1_pcie_init(struct pci_dev *pdev);
void hfi1_pcie_cleanup(struct pci_dev *pdev);
int hfi1_pcie_ddinit(struct hfi1_devdata *dd, struct pci_dev *pdev);
void hfi1_pcie_ddcleanup(struct hfi1_devdata *);

196
drivers/infiniband/hw/hfi1/init.c
View File

@ -629,8 +629,6 @@ void hfi1_init_pportdata(struct pci_dev *pdev, struct hfi1_pportdata *ppd,
ppd->sm_trap_qp = 0x0;
ppd->sa_qp = 0x1;
ppd->hfi1_wq = NULL;
spin_lock_init(&ppd->cca_timer_lock);
for (i = 0; i < OPA_MAX_SLS; i++) {
@ -740,31 +738,27 @@ static int create_workqueues(struct hfi1_devdata *dd)
for (pidx = 0; pidx < dd->num_pports; ++pidx) {
ppd = dd->pport + pidx;
if (!ppd->hfi1_wq) {
ppd->hfi1_wq =
alloc_workqueue(
"hfi%d_%d",
WQ_SYSFS | WQ_HIGHPRI | WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM |
WQ_PERCPU,
HFI1_MAX_ACTIVE_WORKQUEUE_ENTRIES,
dd->unit, pidx);
if (!ppd->hfi1_wq)
goto wq_error;
}
if (!ppd->link_wq) {
/*
* Make the link workqueue single-threaded to enforce
* serialization.
*/
ppd->link_wq =
alloc_workqueue(
"hfi_link_%d_%d",
WQ_SYSFS | WQ_MEM_RECLAIM | WQ_UNBOUND,
1, /* max_active */
dd->unit, pidx);
if (!ppd->link_wq)
goto wq_error;
}
ppd->hfi1_wq =
alloc_workqueue(
"hfi%d_%d",
WQ_SYSFS | WQ_HIGHPRI | WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM |
WQ_PERCPU,
HFI1_MAX_ACTIVE_WORKQUEUE_ENTRIES,
dd->unit, pidx);
if (!ppd->hfi1_wq)
goto wq_error;
/*
* Make the link workqueue single-threaded to enforce
* serialization.
*/
ppd->link_wq =
alloc_workqueue(
"hfi_link_%d_%d",
WQ_SYSFS | WQ_MEM_RECLAIM | WQ_UNBOUND,
1, /* max_active */
dd->unit, pidx);
if (!ppd->link_wq)
goto wq_error;
}
return 0;
wq_error:
@ -1161,7 +1155,7 @@ static void finalize_asic_data(struct hfi1_devdata *dd,
* It cleans up and frees all data structures set up by
* by hfi1_alloc_devdata().
*/
void hfi1_free_devdata(struct hfi1_devdata *dd)
static void hfi1_free_devdata(struct hfi1_devdata *dd)
{
struct hfi1_asic_data *ad;
unsigned long flags;
@ -1225,8 +1219,9 @@ static struct hfi1_devdata *hfi1_alloc_devdata(struct pci_dev *pdev,
GFP_KERNEL);
if (ret < 0) {
dev_err(&pdev->dev,
"Could not allocate unit ID: error %d\n", -ret);
goto bail;
"Could not allocate unit ID: error %pe\n", ERR_PTR(ret));
rvt_dealloc_device(&dd->verbs_dev.rdi);
return ERR_PTR(ret);
}
/*
@ -1554,50 +1549,28 @@ static void postinit_cleanup(struct hfi1_devdata *dd)
hfi1_dev_affinity_clean_up(dd);
hfi1_pcie_ddcleanup(dd);
hfi1_pcie_cleanup(dd->pcidev);
cleanup_device_data(dd);
hfi1_free_devdata(dd);
}
static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
{
int ret = 0, j, pidx, initfail;
int ret;
struct hfi1_devdata *dd;
struct hfi1_pportdata *ppd;
/* First, lock the non-writable module parameters */
HFI1_CAP_LOCK();
/* Validate dev ids */
if (!(ent->device == PCI_DEVICE_ID_INTEL0 ||
ent->device == PCI_DEVICE_ID_INTEL1)) {
dev_err(&pdev->dev, "Failing on unknown Intel deviceid 0x%x\n",
ent->device);
ret = -ENODEV;
goto bail;
}
/* Allocate the dd so we can get to work */
dd = hfi1_alloc_devdata(pdev, NUM_IB_PORTS *
sizeof(struct hfi1_pportdata));
if (IS_ERR(dd)) {
ret = PTR_ERR(dd);
goto bail;
}
/* Validate some global module parameters */
ret = hfi1_validate_rcvhdrcnt(dd, rcvhdrcnt);
ret = hfi1_validate_rcvhdrcnt(pdev, rcvhdrcnt);
if (ret)
goto bail;
return ret;
/* use the encoding function as a sanitization check */
if (!encode_rcv_header_entry_size(hfi1_hdrq_entsize)) {
dd_dev_err(dd, "Invalid HdrQ Entry size %u\n",
hfi1_hdrq_entsize);
ret = -EINVAL;
goto bail;
dev_err(&pdev->dev, "Invalid HdrQ Entry size %u\n",
hfi1_hdrq_entsize);
return -EINVAL;
}
/* The receive eager buffer size must be set before the receive
@ -1617,87 +1590,75 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
clamp_val(eager_buffer_size,
MIN_EAGER_BUFFER * 8,
MAX_EAGER_BUFFER_TOTAL);
dd_dev_info(dd, "Eager buffer size %u\n",
eager_buffer_size);
pci_info(pdev, "Eager buffer size %u\n", eager_buffer_size);
} else {
dd_dev_err(dd, "Invalid Eager buffer size of 0\n");
ret = -EINVAL;
goto bail;
dev_err(&pdev->dev, "Invalid Eager buffer size of 0\n");
return -EINVAL;
}
/* restrict value of hfi1_rcvarr_split */
hfi1_rcvarr_split = clamp_val(hfi1_rcvarr_split, 0, 100);
ret = hfi1_pcie_init(dd);
ret = hfi1_pcie_init(pdev);
if (ret)
goto bail;
return ret;
/*
* Do device-specific initialization, function table setup, dd
* allocation, etc.
*/
ret = hfi1_init_dd(dd);
if (ret)
goto clean_bail; /* error already printed */
/* Allocate the dd so we can get to work */
dd = hfi1_alloc_devdata(pdev, NUM_IB_PORTS *
sizeof(struct hfi1_pportdata));
if (IS_ERR(dd)) {
ret = PTR_ERR(dd);
goto clean_pcie;
}
ret = create_workqueues(dd);
if (ret)
goto clean_bail;
goto free_devdata;
/*
* Do device-specific initialization, function table setup, etc.
*/
ret = hfi1_init_dd(dd);
if (ret)
goto destroy_workqueues; /* error already printed */
/* do the generic initialization */
initfail = hfi1_init(dd, 0);
ret = hfi1_init(dd, 0);
if (ret)
goto free_rx;
ret = hfi1_register_ib_device(dd);
if (ret)
goto free_rx;
/*
* Now ready for use. this should be cleared whenever we
* detect a reset, or initiate one. If earlier failure,
* we still create devices, so diags, etc. can be used
* to determine cause of problem.
* detect a reset, or initiate one.
*/
if (!initfail && !ret) {
dd->flags |= HFI1_INITTED;
/* create debufs files after init and ib register */
hfi1_dbg_ibdev_init(&dd->verbs_dev);
}
dd->flags |= HFI1_INITTED;
j = hfi1_device_create(dd);
if (j)
dd_dev_err(dd, "Failed to create /dev devices: %d\n", -j);
if (initfail || ret) {
msix_clean_up_interrupts(dd);
stop_timers(dd);
flush_workqueue(ib_wq);
for (pidx = 0; pidx < dd->num_pports; ++pidx) {
hfi1_quiet_serdes(dd->pport + pidx);
ppd = dd->pport + pidx;
if (ppd->hfi1_wq) {
destroy_workqueue(ppd->hfi1_wq);
ppd->hfi1_wq = NULL;
}
if (ppd->link_wq) {
destroy_workqueue(ppd->link_wq);
ppd->link_wq = NULL;
}
}
if (!j)
hfi1_device_remove(dd);
if (!ret)
hfi1_unregister_ib_device(dd);
postinit_cleanup(dd);
if (initfail)
ret = initfail;
goto bail; /* everything already cleaned */
}
ret = hfi1_device_create(dd);
if (ret)
dd_dev_err(dd, "Failed to create /dev devices: %pe\n",
ERR_PTR(ret));
sdma_start(dd);
hfi1_dbg_ibdev_init(&dd->verbs_dev);
return 0;
clean_bail:
free_rx:
hfi1_free_rx(dd);
shutdown_device(dd);
stop_timers(dd);
postinit_cleanup(dd);
destroy_workqueues:
destroy_workqueues(dd);
free_devdata:
hfi1_free_devdata(dd);
clean_pcie:
hfi1_pcie_cleanup(pdev);
bail:
return ret;
}
@ -1737,14 +1698,11 @@ static void remove_one(struct pci_dev *pdev)
* clear dma engines, etc.
*/
shutdown_device(dd);
destroy_workqueues(dd);
stop_timers(dd);
/* wait until all of our (qsfp) queue_work() calls complete */
flush_workqueue(ib_wq);
postinit_cleanup(dd);
destroy_workqueues(dd);
hfi1_free_devdata(dd);
hfi1_pcie_cleanup(pdev);
}
static void shutdown_one(struct pci_dev *pdev)

32
drivers/infiniband/hw/hfi1/pcie.c
View File

@ -21,10 +21,9 @@
/*
* Do all the common PCIe setup and initialization.
*/
int hfi1_pcie_init(struct hfi1_devdata *dd)
int hfi1_pcie_init(struct pci_dev *pdev)
{
int ret;
struct pci_dev *pdev = dd->pcidev;
ret = pci_enable_device(pdev);
if (ret) {
@ -40,13 +39,15 @@ int hfi1_pcie_init(struct hfi1_devdata *dd)
* about that, it appears. If the original BAR was retained
* in the kernel data structures, this may be OK.
*/
dd_dev_err(dd, "pci enable failed: error %d\n", -ret);
dev_err(&pdev->dev, "pci enable failed: error %pe\n",
ERR_PTR(ret));
return ret;
}
ret = pci_request_regions(pdev, DRIVER_NAME);
if (ret) {
dd_dev_err(dd, "pci_request_regions fails: err %d\n", -ret);
dev_err(&pdev->dev, "pci_request_regions fails: err %pe\n",
ERR_PTR(ret));
goto bail;
}
@ -59,7 +60,8 @@ int hfi1_pcie_init(struct hfi1_devdata *dd)
*/
ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
if (ret) {
dd_dev_err(dd, "Unable to set DMA mask: %d\n", ret);
dev_err(&pdev->dev, "Unable to set DMA mask: %pe\n",
ERR_PTR(ret));
goto bail;
}
}
@ -512,29 +514,28 @@ pci_error_detected(struct pci_dev *pdev, pci_channel_state_t state)
switch (state) {
case pci_channel_io_normal:
dd_dev_info(dd, "State Normal, ignoring\n");
dev_info(&pdev->dev, "State Normal, ignoring\n");
break;
case pci_channel_io_frozen:
dd_dev_info(dd, "State Frozen, requesting reset\n");
dev_info(&pdev->dev, "State Frozen, requesting reset\n");
pci_disable_device(pdev);
ret = PCI_ERS_RESULT_NEED_RESET;
break;
case pci_channel_io_perm_failure:
dev_info(&pdev->dev, "State Permanent Failure, disabling\n");
if (dd) {
dd_dev_info(dd, "State Permanent Failure, disabling\n");
/* no more register accesses! */
dd->flags &= ~HFI1_PRESENT;
hfi1_disable_after_error(dd);
}
/* else early, or other problem */
ret = PCI_ERS_RESULT_DISCONNECT;
break;
default: /* shouldn't happen */
dd_dev_info(dd, "HFI1 PCI errors detected (state %d)\n",
state);
dev_info(&pdev->dev, "HFI1 PCI errors detected (state %d)\n",
state);
break;
}
return ret;
@ -561,9 +562,7 @@ pci_mmio_enabled(struct pci_dev *pdev)
static pci_ers_result_t
pci_slot_reset(struct pci_dev *pdev)
{
struct hfi1_devdata *dd = pci_get_drvdata(pdev);
dd_dev_info(dd, "HFI1 slot_reset function called, ignored\n");
dev_info(&pdev->dev, "HFI1 slot_reset function called, ignored\n");
return PCI_ERS_RESULT_CAN_RECOVER;
}
@ -572,7 +571,10 @@ pci_resume(struct pci_dev *pdev)
{
struct hfi1_devdata *dd = pci_get_drvdata(pdev);
dd_dev_info(dd, "HFI1 resume function called\n");
dev_info(&pdev->dev, "HFI1 resume function called\n");
if (!dd)
return;
/*
* Running jobs will fail, since it's asynchronous
* unlike sysfs-requested reset. Better than

13
drivers/infiniband/hw/hfi1/user_sdma.c
View File

@ -1026,6 +1026,7 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
struct user_sdma_txreq *tx, u32 datalen)
{
u32 ahg[AHG_KDETH_ARRAY_SIZE];
int ret;
int idx = 0;
u8 omfactor; /* KDETH.OM */
struct hfi1_user_sdma_pkt_q *pq = req->pq;
@ -1130,11 +1131,13 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
trace_hfi1_sdma_user_header_ahg(pq->dd, pq->ctxt, pq->subctxt,
req->info.comp_idx, req->sde->this_idx,
req->ahg_idx, ahg, idx, tidval);
sdma_txinit_ahg(&tx->txreq,
SDMA_TXREQ_F_USE_AHG,
datalen, req->ahg_idx, idx,
ahg, sizeof(req->hdr),
user_sdma_txreq_cb);
ret = sdma_txinit_ahg(&tx->txreq,
SDMA_TXREQ_F_USE_AHG,
datalen, req->ahg_idx, idx,
ahg, sizeof(req->hdr),
user_sdma_txreq_cb);
if (ret)
return ret;
return idx;
}

2
drivers/infiniband/hw/hns/hns_roce_hw_v2.c
View File

@ -4393,7 +4393,7 @@ static int hns_roce_v2_set_hem(struct hns_roce_dev *hr_dev,
struct hns_roce_hem_mhop mhop;
struct hns_roce_hem *hem;
unsigned long mhop_obj = obj;
int i, j, k;
u64 i, j, k;
int ret = 0;
u64 hem_idx = 0;
u64 l1_idx = 0;

6
drivers/infiniband/hw/hns/hns_roce_main.c
View File

@ -223,12 +223,10 @@ static int hns_roce_query_device(struct ib_device *ib_dev,
struct hns_roce_dev *hr_dev = to_hr_dev(ib_dev);
int ret;
ret = ib_is_udata_in_empty(uhw);
ret = ib_no_udata_io(uhw);
if (ret)
return ret;
memset(props, 0, sizeof(*props));
props->fw_ver = hr_dev->caps.fw_ver;
props->sys_image_guid = cpu_to_be64(hr_dev->sys_image_guid);
props->max_mr_size = (u64)(~(0ULL));
@ -279,7 +277,7 @@ static int hns_roce_query_device(struct ib_device *ib_dev,
if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_XRC)
props->device_cap_flags |= IB_DEVICE_XRC;
return ib_respond_empty_udata(uhw);
return 0;
}
static int hns_roce_query_port(struct ib_device *ib_dev, u32 port_num,

72
drivers/infiniband/hw/ionic/ionic_controlpath.c
View File

@ -391,6 +391,16 @@ int ionic_alloc_ucontext(struct ib_ucontext *ibctx, struct ib_udata *udata)
goto err_mmap_dbell;
}
if (dev->lif_cfg.phc_state) {
ctx->mmap_phc = ionic_mmap_entry_insert(ctx, PAGE_SIZE, 0,
IONIC_MMAP_PHC,
&resp.phc_offset);
if (!ctx->mmap_phc) {
rc = -ENOMEM;
goto err_mmap_phc;
}
}
resp.page_shift = PAGE_SHIFT;
resp.dbell_offset = db_phys & ~PAGE_MASK;
@ -421,6 +431,8 @@ int ionic_alloc_ucontext(struct ib_ucontext *ibctx, struct ib_udata *udata)
return 0;
err_resp:
rdma_user_mmap_entry_remove(ctx->mmap_phc);
err_mmap_phc:
rdma_user_mmap_entry_remove(ctx->mmap_dbell);
err_mmap_dbell:
ionic_put_dbid(dev, ctx->dbid);
@ -433,10 +445,26 @@ void ionic_dealloc_ucontext(struct ib_ucontext *ibctx)
struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device);
struct ionic_ctx *ctx = to_ionic_ctx(ibctx);
rdma_user_mmap_entry_remove(ctx->mmap_phc);
rdma_user_mmap_entry_remove(ctx->mmap_dbell);
ionic_put_dbid(dev, ctx->dbid);
}
static int ionic_mmap_phc_state(struct ionic_ibdev *dev,
struct vm_area_struct *vma)
{
if (!(vma->vm_flags & VM_SHARED))
return -EINVAL;
if (vma->vm_flags & (VM_WRITE | VM_EXEC))
return -EPERM;
vm_flags_clear(vma, VM_MAYWRITE);
return vm_insert_page(vma, vma->vm_start,
virt_to_page(dev->lif_cfg.phc_state));
}
int ionic_mmap(struct ib_ucontext *ibctx, struct vm_area_struct *vma)
{
struct ionic_ibdev *dev = to_ionic_ibdev(ibctx->device);
@ -455,6 +483,12 @@ int ionic_mmap(struct ib_ucontext *ibctx, struct vm_area_struct *vma)
ionic_entry = container_of(rdma_entry, struct ionic_mmap_entry,
rdma_entry);
if (ionic_entry->mmap_flags & IONIC_MMAP_PHC) {
rc = ionic_mmap_phc_state(dev, vma);
rdma_user_mmap_entry_put(rdma_entry);
return rc;
}
ibdev_dbg(&dev->ibdev, "writecombine? %d\n",
ionic_entry->mmap_flags & IONIC_MMAP_WC);
if (ionic_entry->mmap_flags & IONIC_MMAP_WC)
@ -487,6 +521,11 @@ int ionic_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device);
struct ionic_pd *pd = to_ionic_pd(ibpd);
int rc;
rc = ib_no_udata_io(udata);
if (rc)
return rc;
return ionic_get_pdid(dev, &pd->pdid);
}
@ -495,6 +534,11 @@ int ionic_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ionic_ibdev *dev = to_ionic_ibdev(ibpd->device);
struct ionic_pd *pd = to_ionic_pd(ibpd);
int rc;
rc = ib_no_udata_io(udata);
if (rc)
return rc;
ionic_put_pdid(dev, pd->pdid);
@ -741,6 +785,10 @@ int ionic_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,
u32 flags = init_attr->flags;
int rc;
rc = ib_is_udata_in_empty(udata);
if (rc)
return rc;
rc = ionic_get_ahid(dev, &ah->ahid);
if (rc)
return rc;
@ -877,6 +925,10 @@ struct ib_mr *ionic_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
unsigned long pg_sz;
int rc;
rc = ib_no_udata_io(udata);
if (rc)
return ERR_PTR(rc);
if (dmah)
return ERR_PTR(-EOPNOTSUPP);
@ -1008,6 +1060,10 @@ int ionic_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
struct ionic_mr *mr = to_ionic_mr(ibmr);
int rc;
rc = ib_no_udata_io(udata);
if (rc)
return rc;
if (!mr->ibmr.lkey)
goto out;
@ -1120,6 +1176,10 @@ int ionic_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)
struct ionic_mr *mr = to_ionic_mw(ibmw);
int rc;
rc = ib_no_udata_io(udata);
if (rc)
return rc;
rc = ionic_get_mrid(dev, &mr->mrid);
if (rc)
return rc;
@ -1292,6 +1352,10 @@ int ionic_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
struct ionic_vcq *vcq = to_ionic_vcq(ibcq);
int udma_idx, rc_tmp, rc = 0;
rc = ib_no_udata_io(udata);
if (rc)
return rc;
for (udma_idx = dev->lif_cfg.udma_count; udma_idx; ) {
--udma_idx;
@ -2585,6 +2649,10 @@ int ionic_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask,
struct ionic_qp *qp = to_ionic_qp(ibqp);
int rc;
rc = ib_no_udata_io(udata);
if (rc)
return rc;
rc = ionic_check_modify_qp(qp, attr, mask);
if (rc)
return rc;
@ -2658,6 +2726,10 @@ int ionic_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
struct ionic_cq *cq;
int rc;
rc = ib_no_udata_io(udata);
if (rc)
return rc;
rc = ionic_destroy_qp_cmd(dev, qp->qpid);
if (rc)
return rc;

43
drivers/infiniband/hw/ionic/ionic_datapath.c
View File

@ -32,6 +32,7 @@ static int ionic_flush_recv(struct ionic_qp *qp, struct ib_wc *wc)
{
struct ionic_rq_meta *meta;
struct ionic_v1_wqe *wqe;
u64 wqe_idx;
if (!qp->rq_flush)
return 0;
@ -40,21 +41,22 @@ static int ionic_flush_recv(struct ionic_qp *qp, struct ib_wc *wc)
return 0;
wqe = ionic_queue_at_cons(&qp->rq);
wqe_idx = le64_to_cpu(wqe->base.wqe_idx);
/* wqe_id must be a valid queue index */
if (unlikely(wqe->base.wqe_id >> qp->rq.depth_log2)) {
/* wqe_idx must be a valid queue index */
if (unlikely(wqe_idx >> qp->rq.depth_log2)) {
ibdev_warn(qp->ibqp.device,
"flush qp %u recv index %llu invalid\n",
qp->qpid, (unsigned long long)wqe->base.wqe_id);
qp->qpid, (unsigned long long)wqe_idx);
return -EIO;
}
/* wqe_id must indicate a request that is outstanding */
meta = &qp->rq_meta[wqe->base.wqe_id];
/* wqe_idx must indicate a request that is outstanding */
meta = &qp->rq_meta[wqe_idx];
if (unlikely(meta->next != IONIC_META_POSTED)) {
ibdev_warn(qp->ibqp.device,
"flush qp %u recv index %llu not posted\n",
qp->qpid, (unsigned long long)wqe->base.wqe_id);
qp->qpid, (unsigned long long)wqe_idx);
return -EIO;
}
@ -133,8 +135,8 @@ static int ionic_poll_recv(struct ionic_ibdev *dev, struct ionic_cq *cq,
{
struct ionic_qp *qp = NULL;
struct ionic_rq_meta *meta;
u16 vlan_tag, wqe_idx;
u32 src_qpn, st_len;
u16 vlan_tag;
u8 op;
if (cqe_qp->rq_flush)
@ -144,7 +146,7 @@ static int ionic_poll_recv(struct ionic_ibdev *dev, struct ionic_cq *cq,
st_len = be32_to_cpu(cqe->status_length);
/* ignore wqe_id in case of flush error */
/* ignore wqe_idx in case of flush error */
if (ionic_v1_cqe_error(cqe) && st_len == IONIC_STS_WQE_FLUSHED_ERR) {
cqe_qp->rq_flush = true;
cq->flush = true;
@ -160,20 +162,19 @@ static int ionic_poll_recv(struct ionic_ibdev *dev, struct ionic_cq *cq,
return -EIO;
}
/* wqe_id must be a valid queue index */
if (unlikely(cqe->recv.wqe_id >> qp->rq.depth_log2)) {
wqe_idx = le64_to_cpu(cqe->recv.wqe_idx_timestamp) & IONIC_V1_CQE_WQE_IDX_MASK;
/* wqe_idx must be a valid queue index */
if (unlikely(wqe_idx >> qp->rq.depth_log2)) {
ibdev_warn(&dev->ibdev,
"qp %u recv index %llu invalid\n",
qp->qpid, (unsigned long long)cqe->recv.wqe_id);
"qp %u recv index %u invalid\n", qp->qpid, wqe_idx);
return -EIO;
}
/* wqe_id must indicate a request that is outstanding */
meta = &qp->rq_meta[cqe->recv.wqe_id];
/* wqe_idx must indicate a request that is outstanding */
meta = &qp->rq_meta[wqe_idx];
if (unlikely(meta->next != IONIC_META_POSTED)) {
ibdev_warn(&dev->ibdev,
"qp %u recv index %llu not posted\n",
qp->qpid, (unsigned long long)cqe->recv.wqe_id);
"qp %u recv index %u not posted\n", qp->qpid, wqe_idx);
return -EIO;
}
@ -408,7 +409,7 @@ static int ionic_comp_msn(struct ionic_qp *qp, struct ionic_v1_cqe *cqe)
static int ionic_comp_npg(struct ionic_qp *qp, struct ionic_v1_cqe *cqe)
{
struct ionic_sq_meta *meta;
u16 cqe_idx;
u16 wqe_idx;
u32 st_len;
if (qp->sq_flush)
@ -430,8 +431,8 @@ static int ionic_comp_npg(struct ionic_qp *qp, struct ionic_v1_cqe *cqe)
return 0;
}
cqe_idx = cqe->send.npg_wqe_id & qp->sq.mask;
meta = &qp->sq_meta[cqe_idx];
wqe_idx = le64_to_cpu(cqe->send.npg_wqe_idx_timestamp) & qp->sq.mask;
meta = &qp->sq_meta[wqe_idx];
meta->local_comp = true;
if (ionic_v1_cqe_error(cqe)) {
@ -811,7 +812,7 @@ static void ionic_prep_base(struct ionic_qp *qp,
meta->signal = false;
meta->local_comp = false;
wqe->base.wqe_id = qp->sq.prod;
wqe->base.wqe_idx = cpu_to_le64(qp->sq.prod);
if (wr->send_flags & IB_SEND_FENCE)
wqe->base.flags |= cpu_to_be16(IONIC_V1_FLAG_FENCE);
@ -1205,7 +1206,7 @@ static int ionic_prep_recv(struct ionic_qp *qp,
meta->wrid = wr->wr_id;
wqe->base.wqe_id = meta - qp->rq_meta;
wqe->base.wqe_idx = cpu_to_le64(meta - qp->rq_meta);
wqe->base.num_sge_key = wr->num_sge;
/* total length for recv goes in base imm_data_key */

12
drivers/infiniband/hw/ionic/ionic_fw.h
View File

@ -332,7 +332,7 @@ struct ionic_v1_cqe {
__le16 old_rq_cq_cindex;
} admin;
struct {
__u64 wqe_id;
__le64 wqe_idx_timestamp;
__be32 src_qpn_op;
__u8 src_mac[6];
__be16 vlan_tag;
@ -342,13 +342,19 @@ struct ionic_v1_cqe {
__u8 rsvd[4];
__be32 msg_msn;
__u8 rsvd2[8];
__u64 npg_wqe_id;
__le64 npg_wqe_idx_timestamp;
} send;
};
__be32 status_length;
__be32 qid_type_flags;
};
/* bits for cqe wqe_idx and timestamp */
enum ionic_v1_cqe_wqe_idx_timestamp_bits {
IONIC_V1_CQE_WQE_IDX_MASK = 0xffff,
IONIC_V1_CQE_TIMESTAMP_SHIFT = 16,
};
/* bits for cqe recv */
enum ionic_v1_cqe_src_qpn_bits {
IONIC_V1_CQE_RECV_QPN_MASK = 0xffffff,
@ -423,7 +429,7 @@ static inline u32 ionic_v1_cqe_qtf_qid(u32 qtf)
/* v1 base wqe header */
struct ionic_v1_base_hdr {
__u64 wqe_id;
__le64 wqe_idx;
__u8 op;
__u8 num_sge_key;
__be16 flags;

95
drivers/infiniband/hw/ionic/ionic_hw_stats.c
View File

@ -235,35 +235,34 @@ static int ionic_get_hw_stats(struct ib_device *ibdev,
static struct rdma_hw_stats *
ionic_counter_alloc_stats(struct rdma_counter *counter)
{
struct ionic_rdma_counter *cntr = to_ionic_rdma_counter(counter);
struct ionic_ibdev *dev = to_ionic_ibdev(counter->device);
struct ionic_counter *cntr;
int err;
struct rdma_hw_stats *stats;
int id;
cntr = kzalloc_obj(*cntr);
if (!cntr)
return NULL;
/* buffer for current values from the device */
cntr->vals = kzalloc(PAGE_SIZE, GFP_KERNEL);
if (!cntr->vals)
goto err_vals;
return NULL;
err = xa_alloc(&dev->counter_stats->xa_counters, &counter->id,
cntr,
XA_LIMIT(0, IONIC_MAX_QPID),
GFP_KERNEL);
if (err)
goto err_xa;
id = ida_alloc_max(&dev->counter_stats->counter_ida,
IONIC_MAX_QPID, GFP_KERNEL);
if (id < 0)
goto err_ida;
INIT_LIST_HEAD(&cntr->qp_list);
counter->id = id;
return rdma_alloc_hw_stats_struct(dev->counter_stats->stats_hdrs,
dev->counter_stats->queue_stats_count,
RDMA_HW_STATS_DEFAULT_LIFESPAN);
err_xa:
stats = rdma_alloc_hw_stats_struct(dev->counter_stats->stats_hdrs,
dev->counter_stats->queue_stats_count,
RDMA_HW_STATS_DEFAULT_LIFESPAN);
if (!stats)
goto err_hw_stats;
return stats;
err_hw_stats:
ida_free(&dev->counter_stats->counter_ida, id);
err_ida:
kfree(cntr->vals);
err_vals:
kfree(cntr);
return NULL;
}
@ -271,14 +270,10 @@ ionic_counter_alloc_stats(struct rdma_counter *counter)
static int ionic_counter_dealloc(struct rdma_counter *counter)
{
struct ionic_ibdev *dev = to_ionic_ibdev(counter->device);
struct ionic_counter *cntr;
cntr = xa_erase(&dev->counter_stats->xa_counters, counter->id);
if (!cntr)
return -EINVAL;
struct ionic_rdma_counter *cntr = to_ionic_rdma_counter(counter);
ida_free(&dev->counter_stats->counter_ida, counter->id);
kfree(cntr->vals);
kfree(cntr);
return 0;
}
@ -287,13 +282,8 @@ static int ionic_counter_bind_qp(struct rdma_counter *counter,
struct ib_qp *ibqp,
u32 port)
{
struct ionic_ibdev *dev = to_ionic_ibdev(counter->device);
struct ionic_rdma_counter *cntr = to_ionic_rdma_counter(counter);
struct ionic_qp *qp = to_ionic_qp(ibqp);
struct ionic_counter *cntr;
cntr = xa_load(&dev->counter_stats->xa_counters, counter->id);
if (!cntr)
return -EINVAL;
list_add_tail(&qp->qp_list_counter, &cntr->qp_list);
ibqp->counter = counter;
@ -313,29 +303,23 @@ static int ionic_counter_unbind_qp(struct ib_qp *ibqp, u32 port)
return 0;
}
static int ionic_get_qp_stats(struct ib_device *ibdev,
struct rdma_hw_stats *hw_stats,
u32 counter_id)
static int ionic_counter_update_stats(struct rdma_counter *counter)
{
struct ionic_ibdev *dev = to_ionic_ibdev(ibdev);
struct ionic_counter_stats *cs;
struct ionic_counter *cntr;
struct ionic_rdma_counter *cntr = to_ionic_rdma_counter(counter);
struct ionic_ibdev *dev = to_ionic_ibdev(counter->device);
struct ionic_counter_stats *cs = dev->counter_stats;
dma_addr_t hw_stats_dma;
struct ionic_qp *qp;
int rc, stat_i = 0;
cs = dev->counter_stats;
cntr = xa_load(&cs->xa_counters, counter_id);
if (!cntr)
return -EINVAL;
hw_stats_dma = dma_map_single(dev->lif_cfg.hwdev, cntr->vals,
PAGE_SIZE, DMA_FROM_DEVICE);
rc = dma_mapping_error(dev->lif_cfg.hwdev, hw_stats_dma);
if (rc)
return rc;
memset(hw_stats->value, 0, sizeof(u64) * hw_stats->num_counters);
memset(counter->stats->value, 0,
sizeof(u64) * counter->stats->num_counters);
list_for_each_entry(qp, &cntr->qp_list, qp_list_counter) {
rc = ionic_hw_stats_cmd(dev, hw_stats_dma, PAGE_SIZE,
@ -345,7 +329,7 @@ static int ionic_get_qp_stats(struct ib_device *ibdev,
goto err_cmd;
for (stat_i = 0; stat_i < cs->queue_stats_count; ++stat_i)
hw_stats->value[stat_i] +=
counter->stats->value[stat_i] +=
ionic_v1_stat_val(&cs->hdr[stat_i],
cntr->vals,
PAGE_SIZE);
@ -360,11 +344,6 @@ static int ionic_get_qp_stats(struct ib_device *ibdev,
return rc;
}
static int ionic_counter_update_stats(struct rdma_counter *counter)
{
return ionic_get_qp_stats(counter->device, counter->stats, counter->id);
}
static int ionic_alloc_counters(struct ionic_ibdev *dev)
{
struct ionic_counter_stats *cs = dev->counter_stats;
@ -424,12 +403,22 @@ static const struct ib_device_ops ionic_hw_stats_ops = {
.get_hw_stats = ionic_get_hw_stats,
};
static void ionic_counter_init(struct rdma_counter *counter)
{
struct ionic_rdma_counter *cntr = to_ionic_rdma_counter(counter);
INIT_LIST_HEAD(&cntr->qp_list);
}
static const struct ib_device_ops ionic_counter_stats_ops = {
.counter_alloc_stats = ionic_counter_alloc_stats,
.counter_dealloc = ionic_counter_dealloc,
.counter_bind_qp = ionic_counter_bind_qp,
.counter_unbind_qp = ionic_counter_unbind_qp,
.counter_update_stats = ionic_counter_update_stats,
.counter_init = ionic_counter_init,
INIT_RDMA_OBJ_SIZE(rdma_counter, ionic_rdma_counter, rdma_counter),
};
void ionic_stats_init(struct ionic_ibdev *dev)
@ -458,7 +447,7 @@ void ionic_stats_init(struct ionic_ibdev *dev)
return;
}
xa_init_flags(&dev->counter_stats->xa_counters, XA_FLAGS_ALLOC);
ida_init(&dev->counter_stats->counter_ida);
ib_set_device_ops(&dev->ibdev, &ionic_counter_stats_ops);
}
@ -467,7 +456,7 @@ void ionic_stats_init(struct ionic_ibdev *dev)
void ionic_stats_cleanup(struct ionic_ibdev *dev)
{
if (dev->counter_stats) {
xa_destroy(&dev->counter_stats->xa_counters);
ida_destroy(&dev->counter_stats->counter_ida);
kfree(dev->counter_stats->hdr);
kfree(dev->counter_stats->stats_hdrs);
kfree(dev->counter_stats);

5
drivers/infiniband/hw/ionic/ionic_ibdev.c
View File

@ -27,7 +27,7 @@ static int ionic_query_device(struct ib_device *ibdev,
struct net_device *ndev;
int err;
err = ib_is_udata_in_empty(udata);
err = ib_no_udata_io(udata);
if (err)
return err;
@ -74,7 +74,7 @@ static int ionic_query_device(struct ib_device *ibdev,
attr->max_fast_reg_page_list_len = dev->lif_cfg.npts_per_lif / 2;
attr->max_pkeys = IONIC_PKEY_TBL_LEN;
return ib_respond_empty_udata(udata);
return 0;
}
static int ionic_query_port(struct ib_device *ibdev, u32 port,
@ -216,6 +216,7 @@ static const struct ib_device_ops ionic_dev_ops = {
.owner = THIS_MODULE,
.driver_id = RDMA_DRIVER_IONIC,
.uverbs_abi_ver = IONIC_ABI_VERSION,
.uverbs_robust_udata = true,
.alloc_ucontext = ionic_alloc_ucontext,
.dealloc_ucontext = ionic_dealloc_ucontext,

14
drivers/infiniband/hw/ionic/ionic_ibdev.h
View File

@ -33,7 +33,6 @@
#define IONIC_MAX_QPID 0xffffff
#define IONIC_SPEC_HIGH 8
#define IONIC_MAX_PD 1024
#define IONIC_SPEC_HIGH 8
#define IONIC_SQCMB_ORDER 5
#define IONIC_RQCMB_ORDER 0
@ -72,6 +71,7 @@ enum ionic_admin_flags {
enum ionic_mmap_flag {
IONIC_MMAP_WC = BIT(0),
IONIC_MMAP_PHC = BIT(1),
};
struct ionic_mmap_entry {
@ -173,6 +173,7 @@ struct ionic_ctx {
struct ib_ucontext ibctx;
u32 dbid;
struct rdma_user_mmap_entry *mmap_dbell;
struct rdma_user_mmap_entry *mmap_phc;
};
struct ionic_tbl_buf {
@ -331,14 +332,21 @@ struct ionic_counter_stats {
int queue_stats_count;
struct ionic_v1_stat *hdr;
struct rdma_stat_desc *stats_hdrs;
struct xarray xa_counters;
struct ida counter_ida;
};
struct ionic_counter {
struct ionic_rdma_counter {
struct rdma_counter rdma_counter;
void *vals;
struct list_head qp_list;
};
static inline struct ionic_rdma_counter *
to_ionic_rdma_counter(struct rdma_counter *counter)
{
return container_of(counter, struct ionic_rdma_counter, rdma_counter);
}
static inline struct ionic_ibdev *to_ionic_ibdev(struct ib_device *ibdev)
{
return container_of(ibdev, struct ionic_ibdev, ibdev);

4
drivers/infiniband/hw/ionic/ionic_lif_cfg.c
View File

@ -40,6 +40,8 @@ void ionic_fill_lif_cfg(struct ionic_lif *lif, struct ionic_lif_cfg *cfg)
cfg->dbid_count = le32_to_cpu(lif->ionic->ident.dev.ndbpgs_per_lif);
cfg->dbpage = lif->kern_dbpage;
cfg->intr_ctrl = lif->ionic->idev.intr_ctrl;
if (lif->phc)
cfg->phc_state = lif->phc->state_page;
cfg->db_phys = lif->ionic->bars[IONIC_PCI_BAR_DBELL].bus_addr;
@ -70,7 +72,7 @@ void ionic_fill_lif_cfg(struct ionic_lif *lif, struct ionic_lif_cfg *cfg)
* eq_count is tunable; see ionic_eq_count
*/
cfg->aq_count = le32_to_cpu(ident->rdma.aq_qtype.qid_count);
cfg->eq_count = le32_to_cpu(ident->rdma.eq_qtype.qid_count);
cfg->eq_count = lif->ionic->neqs_per_lif;
cfg->cq_count = le32_to_cpu(ident->rdma.cq_qtype.qid_count);
cfg->qp_count = le32_to_cpu(ident->rdma.sq_qtype.qid_count);
cfg->dbid_count = le32_to_cpu(lif->ionic->ident.dev.ndbpgs_per_lif);

1
drivers/infiniband/hw/ionic/ionic_lif_cfg.h
View File

@ -23,6 +23,7 @@ struct ionic_lif_cfg {
u64 __iomem *dbpage;
struct ionic_intr __iomem *intr_ctrl;
phys_addr_t db_phys;
void *phc_state;
u64 page_size_supported;
u32 npts_per_lif;

4
drivers/infiniband/hw/irdma/i40iw_hw.c
View File

@ -29,7 +29,9 @@ static u32 i40iw_regs[IRDMA_MAX_REGS] = {
I40E_PFHMC_PDINV,
I40E_GLHMC_VFPDINV(0),
I40E_GLPE_CRITERR,
0xffffffff /* PFINT_RATEN not used in FPK */
0xffffffff, /* PFINT_RATEN not used in FPK */
0xffffffff, /* PFHMC_ERRORINFO not used in FPK */
0xffffffff /* PFHMC_ERRORDATA not used in FPK */
};
static u32 i40iw_stat_offsets[] = {

2
drivers/infiniband/hw/irdma/icrdma_hw.c
View File

@ -29,6 +29,8 @@ static u32 icrdma_regs[IRDMA_MAX_REGS] = {
GLHMC_VFPDINV(0),
GLPE_CRITERR,
GLINT_RATE(0),
PFHMC_ERRORINFO,
PFHMC_ERRORDATA,
};
static u64 icrdma_masks[IRDMA_MAX_MASKS] = {

2
drivers/infiniband/hw/irdma/icrdma_hw.h
View File

@ -40,6 +40,8 @@
#define GLHMC_VFPDINV(_i) (0x00528300 + ((_i) * 4)) /* _i=0...31 */
#define GLPE_CRITERR 0x00534000
#define GLINT_RATE(_INT) (0x0015A000 + ((_INT) * 4)) /* _i=0...2047 */ /* Reset Source: CORER */
#define PFHMC_ERRORINFO 0x00520400
#define PFHMC_ERRORDATA 0x00520500
#define ICRDMA_DB_ADDR_OFFSET (8 * 1024 * 1024 - 64 * 1024)

8
drivers/infiniband/hw/irdma/icrdma_if.c
View File

@ -91,8 +91,12 @@ static void icrdma_iidc_event_handler(struct iidc_rdma_core_dev_info *cdev_info,
}
}
if (event->reg & IRDMAPFINT_OICR_HMC_ERR_M) {
ibdev_err(&iwdev->ibdev, "HMC Error\n");
iwdev->rf->reset = true;
u32 hmc_errinfo = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_PFHMC_ERRORINFO]);
u32 hmc_errdata = readl(iwdev->rf->sc_dev.hw_regs[IRDMA_PFHMC_ERRORDATA]);
/* Log diagnostics; do not reset here. */
ibdev_warn(&iwdev->ibdev, "HMC Error: errinfo=0x%08x errdata=0x%08x\n",
hmc_errinfo, hmc_errdata);
}
if (event->reg & IRDMAPFINT_OICR_PE_PUSH_M) {
ibdev_err(&iwdev->ibdev, "PE Push Error\n");

2
drivers/infiniband/hw/irdma/irdma.h
View File

@ -66,6 +66,8 @@ enum irdma_registers {
IRDMA_GLHMC_VFPDINV,
IRDMA_GLPE_CRITERR,
IRDMA_GLINT_RATE,
IRDMA_PFHMC_ERRORINFO,
IRDMA_PFHMC_ERRORDATA,
IRDMA_MAX_REGS, /* Must be last entry */
};

6
drivers/infiniband/hw/irdma/utils.c
View File

@ -1168,6 +1168,12 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp)
iwqp->kqp.dma_mem.va = NULL;
kfree(iwqp->kqp.sq_wrid_mem);
kfree(iwqp->kqp.rq_wrid_mem);
if (iwqp->user_mode && iwqp->iwpbl) {
struct irdma_mr *iwmr = iwqp->iwpbl->iwmr;
refcount_dec(&iwmr->user_ring_refs);
}
}
/**

368
drivers/infiniband/hw/irdma/verbs.c
View File

@ -18,11 +18,10 @@ static int irdma_query_device(struct ib_device *ibdev,
struct irdma_hw_attrs *hw_attrs = &rf->sc_dev.hw_attrs;
int err;
err = ib_is_udata_in_empty(udata);
err = ib_no_udata_io(udata);
if (err)
return err;
memset(props, 0, sizeof(*props));
addrconf_addr_eui48((u8 *)&props->sys_image_guid,
iwdev->netdev->dev_addr);
props->fw_ver = (u64)irdma_fw_major_ver(&rf->sc_dev) << 32 |
@ -76,7 +75,7 @@ static int irdma_query_device(struct ib_device *ibdev,
if (hw_attrs->uk_attrs.hw_rev >= IRDMA_GEN_3)
props->device_cap_flags |= IB_DEVICE_MEM_WINDOW_TYPE_2B;
return ib_respond_empty_udata(udata);
return 0;
}
/**
@ -407,6 +406,10 @@ static int irdma_alloc_pd(struct ib_pd *pd, struct ib_udata *udata)
u32 pd_id = 0;
int err;
err = ib_is_udata_in_empty(udata);
if (err)
return err;
if (udata && udata->outlen < IRDMA_ALLOC_PD_MIN_RESP_LEN)
return -EINVAL;
@ -445,6 +448,11 @@ static int irdma_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct irdma_pd *iwpd = to_iwpd(ibpd);
struct irdma_device *iwdev = to_iwdev(ibpd->device);
int ret;
ret = ib_no_udata_io(udata);
if (ret)
return ret;
irdma_free_rsrc(iwdev->rf, iwdev->rf->allocated_pds, iwpd->sc_pd.pd_id);
@ -464,6 +472,9 @@ static struct irdma_pbl *irdma_get_pbl(unsigned long va,
list_for_each_entry (iwpbl, pbl_list, list) {
if (iwpbl->user_base == va) {
struct irdma_mr *iwmr = iwpbl->iwmr;
refcount_inc(&iwmr->user_ring_refs);
list_del(&iwpbl->list);
iwpbl->on_list = false;
return iwpbl;
@ -534,11 +545,10 @@ static int irdma_setup_push_mmap_entries(struct irdma_ucontext *ucontext,
}
/**
* irdma_destroy_qp - destroy qp
* _irdma_destroy_qp - destroy qp
* @ibqp: qp's ib pointer also to get to device's qp address
* @udata: user data
*/
static int irdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
static void _irdma_destroy_qp(struct ib_qp *ibqp)
{
struct irdma_qp *iwqp = to_iwqp(ibqp);
struct irdma_device *iwdev = iwqp->iwdev;
@ -570,6 +580,22 @@ static int irdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
if (iwqp->sc_qp.qp_uk.qp_id == 1)
iwdev->rf->hwqp1_rsvd = false;
irdma_free_qp_rsrc(iwqp);
}
/**
* irdma_destroy_qp - destroy qp
* @ibqp: qp's ib pointer also to get to device's qp address
* @udata: user data
*/
static int irdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
int ret;
ret = ib_no_udata_io(udata);
if (ret)
return ret;
_irdma_destroy_qp(ibqp);
return 0;
}
@ -605,37 +631,29 @@ static void irdma_setup_virt_qp(struct irdma_device *iwdev,
/**
* irdma_setup_umode_qp - setup sq and rq size in user mode qp
* @udata: udata
* @ucontext: user context
* @req: user request pointer
* @iwdev: iwarp device
* @iwqp: qp ptr (user or kernel)
* @info: initialize info to return
* @init_attr: Initial QP create attributes
*/
static int irdma_setup_umode_qp(struct ib_udata *udata,
static int irdma_setup_umode_qp(struct irdma_ucontext *ucontext,
struct irdma_create_qp_req *req,
struct irdma_device *iwdev,
struct irdma_qp *iwqp,
struct irdma_qp_init_info *info,
struct ib_qp_init_attr *init_attr)
{
struct irdma_ucontext *ucontext = rdma_udata_to_drv_context(udata,
struct irdma_ucontext, ibucontext);
struct irdma_qp_uk_init_info *ukinfo = &info->qp_uk_init_info;
struct irdma_create_qp_req req;
unsigned long flags;
int ret;
ret = ib_copy_from_udata(&req, udata,
min(sizeof(req), udata->inlen));
if (ret) {
ibdev_dbg(&iwdev->ibdev, "VERBS: ib_copy_from_data fail\n");
return ret;
}
iwqp->ctx_info.qp_compl_ctx = req.user_compl_ctx;
iwqp->ctx_info.qp_compl_ctx = req->user_compl_ctx;
iwqp->user_mode = 1;
spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs,
iwqp->iwpbl = irdma_get_pbl((unsigned long)req->user_wqe_bufs,
&ucontext->qp_reg_mem_list);
spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
@ -962,6 +980,7 @@ static int irdma_create_qp(struct ib_qp *ibqp,
struct irdma_uk_attrs *uk_attrs = &dev->hw_attrs.uk_attrs;
struct irdma_qp_init_info init_info = {};
struct irdma_qp_host_ctx_info *ctx_info;
struct irdma_create_qp_req ureq = {};
struct irdma_srq *iwsrq;
bool srq_valid = false;
u32 srq_id = 0;
@ -979,9 +998,14 @@ static int irdma_create_qp(struct ib_qp *ibqp,
if (err_code)
return err_code;
if (udata && (udata->inlen < IRDMA_CREATE_QP_MIN_REQ_LEN ||
udata->outlen < IRDMA_CREATE_QP_MIN_RESP_LEN))
return -EINVAL;
if (udata) {
if (udata->outlen < IRDMA_CREATE_QP_MIN_RESP_LEN)
return -EINVAL;
err_code = ib_copy_validate_udata_in(udata, ureq, user_compl_ctx);
if (err_code)
return err_code;
}
init_info.vsi = &iwdev->vsi;
init_info.qp_uk_init_info.uk_attrs = uk_attrs;
@ -1040,9 +1064,14 @@ static int irdma_create_qp(struct ib_qp *ibqp,
init_waitqueue_head(&iwqp->mod_qp_waitq);
if (udata) {
struct irdma_ucontext *ucontext =
rdma_udata_to_drv_context(udata,
struct irdma_ucontext,
ibucontext);
init_info.qp_uk_init_info.abi_ver = iwpd->sc_pd.abi_ver;
err_code = irdma_setup_umode_qp(udata, iwdev, iwqp, &init_info,
init_attr);
err_code = irdma_setup_umode_qp(ucontext, &ureq, iwdev, iwqp,
&init_info, init_attr);
} else {
INIT_DELAYED_WORK(&iwqp->dwork_flush, irdma_flush_worker);
init_info.qp_uk_init_info.abi_ver = IRDMA_ABI_VER;
@ -1109,8 +1138,12 @@ static int irdma_create_qp(struct ib_qp *ibqp,
init_completion(&iwqp->free_qp);
if (udata) {
/* GEN_1 legacy support with libi40iw does not have expanded uresp struct */
if (udata->outlen < sizeof(uresp)) {
/* GEN_1 legacy support with libi40iw does not have expanded
* uresp struct. Check for the exact legacy size (20 bytes) to
* ensure that newer expanded uresp structs don't accidentally
* trigger the legacy fallback.
*/
if (udata->outlen == IRDMA_CREATE_QP_MIN_RESP_LEN) {
uresp.lsmm = 1;
uresp.push_idx = IRDMA_INVALID_PUSH_PAGE_INDEX_GEN_1;
} else {
@ -1124,7 +1157,7 @@ static int irdma_create_qp(struct ib_qp *ibqp,
err_code = ib_respond_udata(udata, uresp);
if (err_code) {
irdma_destroy_qp(&iwqp->ibqp, udata);
_irdma_destroy_qp(&iwqp->ibqp);
return err_code;
}
}
@ -1273,7 +1306,6 @@ static int irdma_wait_for_suspend(struct irdma_qp *iwqp)
int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_udata *udata)
{
#define IRDMA_MODIFY_QP_MIN_REQ_LEN offsetofend(struct irdma_modify_qp_req, rq_flush)
#define IRDMA_MODIFY_QP_MIN_RESP_LEN offsetofend(struct irdma_modify_qp_resp, push_valid)
struct irdma_pd *iwpd = to_iwpd(ibqp->pd);
struct irdma_qp *iwqp = to_iwqp(ibqp);
@ -1289,15 +1321,26 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
u8 issue_modify_qp = 0;
int ret = 0;
/* Clear the response buffer (if any). It may be updated again later. */
ret = ib_respond_empty_udata(udata);
if (ret)
return ret;
ctx_info = &iwqp->ctx_info;
roce_info = &iwqp->roce_info;
udp_info = &iwqp->udp_info;
if (udata) {
/* udata inlen/outlen can be 0 when supporting legacy libi40iw */
if ((udata->inlen && udata->inlen < IRDMA_MODIFY_QP_MIN_REQ_LEN) ||
(udata->outlen && udata->outlen < IRDMA_MODIFY_QP_MIN_RESP_LEN))
if (udata->outlen && udata->outlen < IRDMA_MODIFY_QP_MIN_RESP_LEN)
return -EINVAL;
/* For current irdma, validate against ABI def. */
if (udata->inlen) {
ret = ib_copy_validate_udata_in(udata, ureq, rsvd);
if (ret)
return ret;
}
}
if (attr_mask & ~IB_QP_ATTR_STANDARD_BITS)
@ -1540,10 +1583,6 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
iwqp->ibqp_state = attr->qp_state;
spin_unlock_irqrestore(&iwqp->lock, flags);
if (udata && udata->inlen) {
if (ib_copy_from_udata(&ureq, udata,
min(sizeof(ureq), udata->inlen)))
return -EINVAL;
irdma_flush_wqes(iwqp,
(ureq.sq_flush ? IRDMA_FLUSH_SQ : 0) |
(ureq.rq_flush ? IRDMA_FLUSH_RQ : 0) |
@ -1633,7 +1672,6 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int irdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
struct ib_udata *udata)
{
#define IRDMA_MODIFY_QP_MIN_REQ_LEN offsetofend(struct irdma_modify_qp_req, rq_flush)
#define IRDMA_MODIFY_QP_MIN_RESP_LEN offsetofend(struct irdma_modify_qp_resp, push_valid)
struct irdma_qp *iwqp = to_iwqp(ibqp);
struct irdma_device *iwdev = iwqp->iwdev;
@ -1649,11 +1687,20 @@ int irdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
int err;
unsigned long flags;
err = ib_respond_empty_udata(udata);
if (err)
return err;
if (udata) {
/* udata inlen/outlen can be 0 when supporting legacy libi40iw */
if ((udata->inlen && udata->inlen < IRDMA_MODIFY_QP_MIN_REQ_LEN) ||
(udata->outlen && udata->outlen < IRDMA_MODIFY_QP_MIN_RESP_LEN))
if (udata->outlen && udata->outlen < IRDMA_MODIFY_QP_MIN_RESP_LEN)
return -EINVAL;
if (udata->inlen) {
err = ib_copy_validate_udata_in(udata, ureq, rsvd);
if (err)
return err;
}
}
if (attr_mask & ~IB_QP_ATTR_STANDARD_BITS)
@ -1743,10 +1790,6 @@ int irdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
iwqp->ibqp_state = attr->qp_state;
spin_unlock_irqrestore(&iwqp->lock, flags);
if (udata && udata->inlen) {
if (ib_copy_from_udata(&ureq, udata,
min(sizeof(ureq), udata->inlen)))
return -EINVAL;
irdma_flush_wqes(iwqp,
(ureq.sq_flush ? IRDMA_FLUSH_SQ : 0) |
(ureq.rq_flush ? IRDMA_FLUSH_RQ : 0) |
@ -1880,6 +1923,11 @@ static void irdma_srq_free_rsrc(struct irdma_pci_f *rf, struct irdma_srq *iwsrq)
dma_free_coherent(rf->sc_dev.hw->device, iwsrq->kmem.size,
iwsrq->kmem.va, iwsrq->kmem.pa);
iwsrq->kmem.va = NULL;
} else {
/* Not called in any failure path, so iwpbl is valid. */
struct irdma_mr *iwmr = iwsrq->iwpbl->iwmr;
refcount_dec(&iwmr->user_ring_refs);
}
irdma_free_rsrc(rf, rf->allocated_srqs, srq->srq_uk.srq_id);
@ -1902,6 +1950,21 @@ static void irdma_cq_free_rsrc(struct irdma_pci_f *rf, struct irdma_cq *iwcq)
iwcq->kmem_shadow.size,
iwcq->kmem_shadow.va, iwcq->kmem_shadow.pa);
iwcq->kmem_shadow.va = NULL;
} else {
struct irdma_mr *iwmr;
/* May be called in a failure path before iwpbl is valid. */
if (iwcq->iwpbl) {
iwmr = iwcq->iwpbl->iwmr;
refcount_dec(&iwmr->user_ring_refs);
}
if (iwcq->iwpbl_shadow) {
iwmr = iwcq->iwpbl_shadow->iwmr;
refcount_dec(&iwmr->user_ring_refs);
}
}
irdma_free_rsrc(rf, rf->allocated_cqs, cq->cq_uk.cq_id);
@ -1958,6 +2021,11 @@ static int irdma_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
struct irdma_device *iwdev = to_iwdev(ibsrq->device);
struct irdma_srq *iwsrq = to_iwsrq(ibsrq);
struct irdma_sc_srq *srq = &iwsrq->sc_srq;
int ret;
ret = ib_no_udata_io(udata);
if (ret)
return ret;
irdma_srq_wq_destroy(iwdev->rf, srq);
irdma_srq_free_rsrc(iwdev->rf, iwsrq);
@ -1978,6 +2046,11 @@ static int irdma_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
struct irdma_sc_ceq *ceq = dev->ceq[cq->ceq_id];
struct irdma_ceq *iwceq = container_of(ceq, struct irdma_ceq, sc_ceq);
unsigned long flags;
int ret;
ret = ib_no_udata_io(udata);
if (ret)
return ret;
spin_lock_irqsave(&iwcq->lock, flags);
if (!list_empty(&iwcq->cmpl_generated))
@ -2008,7 +2081,6 @@ static int irdma_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
static int irdma_resize_cq(struct ib_cq *ibcq, unsigned int entries,
struct ib_udata *udata)
{
#define IRDMA_RESIZE_CQ_MIN_REQ_LEN offsetofend(struct irdma_resize_cq_req, user_cq_buffer)
struct irdma_cq *iwcq = to_iwcq(ibcq);
struct irdma_sc_dev *dev = iwcq->sc_cq.dev;
struct irdma_cqp_request *cqp_request;
@ -2017,7 +2089,7 @@ static int irdma_resize_cq(struct ib_cq *ibcq, unsigned int entries,
struct irdma_modify_cq_info info = {};
struct irdma_dma_mem kmem_buf;
struct irdma_cq_mr *cqmr_buf;
struct irdma_pbl *iwpbl_buf;
struct irdma_pbl *iwpbl_buf = NULL;
struct irdma_device *iwdev;
struct irdma_pci_f *rf;
struct irdma_cq_buf *cq_buf = NULL;
@ -2032,12 +2104,13 @@ static int irdma_resize_cq(struct ib_cq *ibcq, unsigned int entries,
IRDMA_FEATURE_CQ_RESIZE))
return -EOPNOTSUPP;
if (udata && udata->inlen < IRDMA_RESIZE_CQ_MIN_REQ_LEN)
return -EINVAL;
if (entries > rf->max_cqe)
return -EINVAL;
ret = ib_respond_empty_udata(udata);
if (ret)
return ret;
if (!iwcq->user_mode) {
entries += 2;
@ -2064,9 +2137,9 @@ static int irdma_resize_cq(struct ib_cq *ibcq, unsigned int entries,
rdma_udata_to_drv_context(udata, struct irdma_ucontext,
ibucontext);
if (ib_copy_from_udata(&req, udata,
min(sizeof(req), udata->inlen)))
return -EINVAL;
ret = ib_copy_validate_udata_in(udata, req, user_cq_buffer);
if (ret)
return ret;
spin_lock_irqsave(&ucontext->cq_reg_mem_list_lock, flags);
iwpbl_buf = irdma_get_pbl((unsigned long)req.user_cq_buffer,
@ -2128,6 +2201,19 @@ static int irdma_resize_cq(struct ib_cq *ibcq, unsigned int entries,
goto error;
spin_lock_irqsave(&iwcq->lock, flags);
if (udata) {
struct irdma_pbl *old_iwpbl = iwcq->iwpbl;
/* Only update if the resize was successful. Otherwise, HW is
* still pointing to the old PBL.
*/
iwcq->iwpbl = iwpbl_buf;
if (old_iwpbl) {
struct irdma_mr *old_iwmr = old_iwpbl->iwmr;
refcount_dec(&old_iwmr->user_ring_refs);
}
}
if (cq_buf) {
cq_buf->kmem_buf = iwcq->kmem;
cq_buf->hw = dev->hw;
@ -2143,6 +2229,11 @@ static int irdma_resize_cq(struct ib_cq *ibcq, unsigned int entries,
return 0;
error:
if (iwpbl_buf) {
struct irdma_mr *iwmr = iwpbl_buf->iwmr;
refcount_dec(&iwmr->user_ring_refs);
}
if (!udata) {
dma_free_coherent(dev->hw->device, kmem_buf.size, kmem_buf.va,
kmem_buf.pa);
@ -2194,6 +2285,10 @@ static int irdma_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
struct cqp_cmds_info *cqp_info;
int status;
status = ib_no_udata_io(udata);
if (status)
return status;
if (attr_mask & IB_SRQ_MAX_WR)
return -EINVAL;
@ -2236,24 +2331,20 @@ static int irdma_setup_umode_srq(struct irdma_device *iwdev,
struct irdma_srq_init_info *info,
struct ib_udata *udata)
{
#define IRDMA_CREATE_SRQ_MIN_REQ_LEN \
offsetofend(struct irdma_create_srq_req, user_shadow_area)
struct irdma_create_srq_req req = {};
struct irdma_ucontext *ucontext;
struct irdma_srq_mr *srqmr;
struct irdma_pbl *iwpbl;
unsigned long flags;
int ret;
iwsrq->user_mode = true;
ucontext = rdma_udata_to_drv_context(udata, struct irdma_ucontext,
ibucontext);
if (udata->inlen < IRDMA_CREATE_SRQ_MIN_REQ_LEN)
return -EINVAL;
if (ib_copy_from_udata(&req, udata,
min(sizeof(req), udata->inlen)))
return -EFAULT;
ret = ib_copy_validate_udata_in(udata, req, user_shadow_area);
if (ret)
return ret;
spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags);
iwpbl = irdma_get_pbl((unsigned long)req.user_srq_buf,
@ -2316,6 +2407,7 @@ static int irdma_create_srq(struct ib_srq *ibsrq,
struct ib_srq_init_attr *initattrs,
struct ib_udata *udata)
{
#define IRDMA_CREATE_SRQ_MIN_RESP_LEN offsetofend(struct irdma_create_srq_resp, srq_size)
struct irdma_device *iwdev = to_iwdev(ibsrq->device);
struct ib_srq_attr *attr = &initattrs->attr;
struct irdma_pd *iwpd = to_iwpd(ibsrq->pd);
@ -2336,6 +2428,9 @@ static int irdma_create_srq(struct ib_srq *ibsrq,
if (initattrs->srq_type != IB_SRQT_BASIC)
return -EOPNOTSUPP;
if (udata && udata->outlen < IRDMA_CREATE_SRQ_MIN_RESP_LEN)
return -EINVAL;
if (!(uk_attrs->feature_flags & IRDMA_FEATURE_SRQ) ||
attr->max_sge > uk_attrs->max_hw_wq_frags)
return -EINVAL;
@ -2419,6 +2514,11 @@ static int irdma_create_srq(struct ib_srq *ibsrq,
dma_free_coherent(rf->hw.device, iwsrq->kmem.size,
iwsrq->kmem.va, iwsrq->kmem.pa);
free_rsrc:
if (iwsrq->user_mode && iwsrq->iwpbl) {
struct irdma_mr *iwmr = iwsrq->iwpbl->iwmr;
refcount_dec(&iwmr->user_ring_refs);
}
irdma_free_rsrc(rf, rf->allocated_srqs, iwsrq->srq_num);
return err_code;
}
@ -2458,7 +2558,6 @@ static int irdma_create_cq(struct ib_cq *ibcq,
const struct ib_cq_init_attr *attr,
struct uverbs_attr_bundle *attrs)
{
#define IRDMA_CREATE_CQ_MIN_REQ_LEN offsetofend(struct irdma_create_cq_req, user_cq_buf)
#define IRDMA_CREATE_CQ_MIN_RESP_LEN offsetofend(struct irdma_create_cq_resp, cq_size)
struct ib_udata *udata = &attrs->driver_udata;
struct ib_device *ibdev = ibcq->device;
@ -2482,8 +2581,7 @@ static int irdma_create_cq(struct ib_cq *ibcq,
if (err_code)
return err_code;
if (udata && (udata->inlen < IRDMA_CREATE_CQ_MIN_REQ_LEN ||
udata->outlen < IRDMA_CREATE_CQ_MIN_RESP_LEN))
if (udata && udata->outlen < IRDMA_CREATE_CQ_MIN_RESP_LEN)
return -EINVAL;
err_code = irdma_alloc_rsrc(rf, rf->allocated_cqs, rf->max_cq, &cq_num,
@ -2498,6 +2596,8 @@ static int irdma_create_cq(struct ib_cq *ibcq,
INIT_LIST_HEAD(&iwcq->resize_list);
INIT_LIST_HEAD(&iwcq->cmpl_generated);
iwcq->cq_num = cq_num;
iwcq->iwpbl = NULL;
iwcq->iwpbl_shadow = NULL;
info.dev = dev;
ukinfo->cq_size = max(entries, 4);
ukinfo->cq_id = cq_num;
@ -2517,49 +2617,50 @@ static int irdma_create_cq(struct ib_cq *ibcq,
struct irdma_ucontext *ucontext;
struct irdma_create_cq_req req = {};
struct irdma_cq_mr *cqmr;
struct irdma_pbl *iwpbl;
struct irdma_pbl *iwpbl_shadow;
struct irdma_cq_mr *cqmr_shadow;
iwcq->user_mode = true;
ucontext =
rdma_udata_to_drv_context(udata, struct irdma_ucontext,
ibucontext);
if (ib_copy_from_udata(&req, udata,
min(sizeof(req), udata->inlen))) {
err_code = -EFAULT;
/* Even though the last member of struct irdma_create_cq_req
* was always user_shadow_area, we need backwards compat with
* the legacy i40iw struct i40iw_ucreate_cq which stopped
* at user_cq_buffer.
*/
err_code = ib_copy_validate_udata_in(udata, req, user_cq_buf);
if (err_code)
goto cq_free_rsrc;
}
spin_lock_irqsave(&ucontext->cq_reg_mem_list_lock, flags);
iwpbl = irdma_get_pbl((unsigned long)req.user_cq_buf,
&ucontext->cq_reg_mem_list);
iwcq->iwpbl = irdma_get_pbl((unsigned long)req.user_cq_buf,
&ucontext->cq_reg_mem_list);
spin_unlock_irqrestore(&ucontext->cq_reg_mem_list_lock, flags);
if (!iwpbl) {
if (!iwcq->iwpbl) {
err_code = -EPROTO;
goto cq_free_rsrc;
}
cqmr = &iwpbl->cq_mr;
cqmr = &iwcq->iwpbl->cq_mr;
if (rf->sc_dev.hw_attrs.uk_attrs.feature_flags &
IRDMA_FEATURE_CQ_RESIZE) {
spin_lock_irqsave(&ucontext->cq_reg_mem_list_lock, flags);
iwpbl_shadow = irdma_get_pbl(
iwcq->iwpbl_shadow = irdma_get_pbl(
(unsigned long)req.user_shadow_area,
&ucontext->cq_reg_mem_list);
spin_unlock_irqrestore(&ucontext->cq_reg_mem_list_lock, flags);
if (!iwpbl_shadow) {
if (!iwcq->iwpbl_shadow) {
err_code = -EPROTO;
goto cq_free_rsrc;
}
cqmr_shadow = &iwpbl_shadow->cq_mr;
cqmr_shadow = &iwcq->iwpbl_shadow->cq_mr;
info.shadow_area_pa = cqmr_shadow->cq_pbl.addr;
} else {
info.shadow_area_pa = cqmr->shadow;
}
if (iwpbl->pbl_allocated) {
if (iwcq->iwpbl->pbl_allocated) {
info.virtual_map = true;
info.pbl_chunk_size = 1;
info.first_pm_pbl_idx = cqmr->cq_pbl.idx;
@ -3034,6 +3135,10 @@ static int irdma_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)
int err_code;
u32 stag;
err_code = ib_no_udata_io(udata);
if (err_code)
return err_code;
stag = irdma_create_stag(iwdev);
if (!stag)
return -ENOMEM;
@ -3362,6 +3467,7 @@ static struct irdma_mr *irdma_alloc_iwmr(struct ib_umem *region,
if (!iwmr)
return ERR_PTR(-ENOMEM);
refcount_set(&iwmr->user_ring_refs, 1);
iwpbl = &iwmr->iwpbl;
iwpbl->iwmr = iwmr;
iwmr->region = region;
@ -3508,7 +3614,6 @@ static struct ib_mr *irdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 len,
struct ib_dmah *dmah,
struct ib_udata *udata)
{
#define IRDMA_MEM_REG_MIN_REQ_LEN offsetofend(struct irdma_mem_reg_req, sq_pages)
struct irdma_device *iwdev = to_iwdev(pd->device);
struct irdma_mem_reg_req req = {};
struct ib_umem *region = NULL;
@ -3518,10 +3623,15 @@ static struct ib_mr *irdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 len,
if (dmah)
return ERR_PTR(-EOPNOTSUPP);
if (len > iwdev->rf->sc_dev.hw_attrs.max_mr_size)
return ERR_PTR(-EINVAL);
err = ib_copy_validate_udata_in(udata, req, sq_pages);
if (err)
return ERR_PTR(err);
if (udata->inlen < IRDMA_MEM_REG_MIN_REQ_LEN)
err = ib_respond_empty_udata(udata);
if (err)
return ERR_PTR(err);
if (len > iwdev->rf->sc_dev.hw_attrs.max_mr_size)
return ERR_PTR(-EINVAL);
region = ib_umem_get_va(pd->device, start, len, access);
@ -3532,11 +3642,6 @@ static struct ib_mr *irdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 len,
return (struct ib_mr *)region;
}
if (ib_copy_from_udata(&req, udata, min(sizeof(req), udata->inlen))) {
ib_umem_release(region);
return ERR_PTR(-EFAULT);
}
iwmr = irdma_alloc_iwmr(region, pd, virt, req.reg_type);
if (IS_ERR(iwmr)) {
ib_umem_release(region);
@ -3780,10 +3885,12 @@ static struct ib_mr *irdma_rereg_user_mr(struct ib_mr *ib_mr, int flags,
struct irdma_device *iwdev = to_iwdev(ib_mr->device);
struct irdma_mr *iwmr = to_iwmr(ib_mr);
struct irdma_pbl *iwpbl = &iwmr->iwpbl;
bool dmabuf_revocable = iwmr->region && iwmr->region->is_dmabuf;
struct ib_umem_dmabuf *umem_dmabuf;
int ret;
ret = ib_no_udata_io(udata);
if (ret)
return ERR_PTR(ret);
if (len > iwdev->rf->sc_dev.hw_attrs.max_mr_size)
return ERR_PTR(-EINVAL);
@ -3797,26 +3904,9 @@ static struct ib_mr *irdma_rereg_user_mr(struct ib_mr *ib_mr, int flags,
if (ret)
return ERR_PTR(ret);
if (dmabuf_revocable) {
umem_dmabuf = to_ib_umem_dmabuf(iwmr->region);
ib_umem_dmabuf_revoke_lock(umem_dmabuf);
/* If the dmabuf has been revoked, it means that the region has
* been invalidated in HW. We must not allow it to become valid
* again unless the user is requesting a change in translation
* which will end up dropping the umem dmabuf and allocating an
* entirely new umem anyway.
*/
if (umem_dmabuf->revoked && !(flags & IB_MR_REREG_TRANS)) {
ret = -EINVAL;
goto err_unlock;
}
}
ret = irdma_hwdereg_mr(ib_mr);
if (ret)
goto err_unlock;
return ERR_PTR(ret);
if (flags & IB_MR_REREG_ACCESS)
iwmr->access = new_access;
@ -3833,12 +3923,6 @@ static struct ib_mr *irdma_rereg_user_mr(struct ib_mr *ib_mr, int flags,
iwpbl->pbl_allocated = false;
}
if (dmabuf_revocable) {
/* Must unlock before release to prevent deadlock */
ib_umem_dmabuf_revoke_unlock(umem_dmabuf);
dmabuf_revocable = false;
}
if (iwmr->region) {
ib_umem_release(iwmr->region);
iwmr->region = NULL;
@ -3849,10 +3933,6 @@ static struct ib_mr *irdma_rereg_user_mr(struct ib_mr *ib_mr, int flags,
ret = irdma_hwreg_mr(iwdev, iwmr, iwmr->access);
}
err_unlock:
if (dmabuf_revocable)
ib_umem_dmabuf_revoke_unlock(umem_dmabuf);
return ret ? ERR_PTR(ret) : NULL;
}
@ -3929,41 +4009,41 @@ static struct ib_mr *irdma_get_dma_mr(struct ib_pd *pd, int acc)
* irdma_del_memlist - Deleting pbl list entries for CQ/QP
* @iwmr: iwmr for IB's user page addresses
* @ucontext: ptr to user context
*
* Return: True if the MR is currently in-use by a QP/CQ/SRQ ring.
*/
static void irdma_del_memlist(struct irdma_mr *iwmr,
static bool irdma_del_memlist(struct irdma_mr *iwmr,
struct irdma_ucontext *ucontext)
{
struct irdma_pbl *iwpbl = &iwmr->iwpbl;
unsigned long flags;
spinlock_t *lock;
bool in_use = false;
switch (iwmr->type) {
case IRDMA_MEMREG_TYPE_CQ:
spin_lock_irqsave(&ucontext->cq_reg_mem_list_lock, flags);
if (iwpbl->on_list) {
iwpbl->on_list = false;
list_del(&iwpbl->list);
}
spin_unlock_irqrestore(&ucontext->cq_reg_mem_list_lock, flags);
lock = &ucontext->cq_reg_mem_list_lock;
break;
case IRDMA_MEMREG_TYPE_QP:
spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
if (iwpbl->on_list) {
iwpbl->on_list = false;
list_del(&iwpbl->list);
}
spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
lock = &ucontext->qp_reg_mem_list_lock;
break;
case IRDMA_MEMREG_TYPE_SRQ:
spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags);
if (iwpbl->on_list) {
iwpbl->on_list = false;
list_del(&iwpbl->list);
}
spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags);
lock = &ucontext->srq_reg_mem_list_lock;
break;
default:
break;
return false;
}
spin_lock_irqsave(lock, flags);
if (!refcount_dec_if_one(&iwmr->user_ring_refs)) {
in_use = true;
} else if (iwpbl->on_list) {
iwpbl->on_list = false;
list_del(&iwpbl->list);
}
spin_unlock_irqrestore(lock, flags);
return in_use;
}
/**
@ -3979,6 +4059,10 @@ static int irdma_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
bool dmabuf_revocable = iwmr->region && iwmr->region->is_dmabuf;
int ret;
ret = ib_no_udata_io(udata);
if (ret)
return ret;
if (iwmr->type != IRDMA_MEMREG_TYPE_MEM) {
if (iwmr->region) {
struct irdma_ucontext *ucontext;
@ -3986,7 +4070,12 @@ static int irdma_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
ucontext = rdma_udata_to_drv_context(udata,
struct irdma_ucontext,
ibucontext);
irdma_del_memlist(iwmr, ucontext);
/* Do not allow the MR to be unpinned if it is still
* backing a user ring.
*/
if (irdma_del_memlist(iwmr, ucontext))
return -EBUSY;
}
goto done;
}
@ -5295,6 +5384,10 @@ static int irdma_create_user_ah(struct ib_ah *ibah,
struct irdma_ah *parent_ah;
int err;
err = ib_is_udata_in_empty(udata);
if (err)
return err;
if (udata->outlen < IRDMA_CREATE_AH_MIN_RESP_LEN)
return -EINVAL;
@ -5346,6 +5439,10 @@ static int irdma_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *attr,
struct irdma_device *iwdev = to_iwdev(ibah->pd->device);
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
err = irdma_setup_ah(ibah, attr);
if (err)
return err;
@ -5428,6 +5525,7 @@ static const struct ib_device_ops irdma_dev_ops = {
.owner = THIS_MODULE,
.driver_id = RDMA_DRIVER_IRDMA,
.uverbs_abi_ver = IRDMA_ABI_VER,
.uverbs_robust_udata = true,
.alloc_hw_port_stats = irdma_alloc_hw_port_stats,
.alloc_mr = irdma_alloc_mr,

3
drivers/infiniband/hw/irdma/verbs.h
View File

@ -120,6 +120,7 @@ struct irdma_mr {
u64 len;
u64 pgaddrmem[IRDMA_MAX_SAVED_PHY_PGADDR];
struct irdma_pbl iwpbl;
refcount_t user_ring_refs;
};
struct irdma_srq {
@ -152,6 +153,8 @@ struct irdma_cq {
struct list_head resize_list;
struct irdma_cq_poll_info cur_cqe;
struct list_head cmpl_generated;
struct irdma_pbl *iwpbl;
struct irdma_pbl *iwpbl_shadow;
};
struct irdma_cmpl_gen {

8
drivers/infiniband/hw/mana/cq.c
View File

@ -27,7 +27,8 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
is_rnic_cq = mana_ib_is_rnic(mdev);
if (udata) {
err = ib_copy_validate_udata_in(udata, ucmd, buf_addr);
err = ib_copy_validate_udata_in_cm(udata, ucmd, buf_addr,
MANA_IB_CREATE_RNIC_CQ);
if (err)
return err;
@ -105,6 +106,11 @@ int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq);
struct ib_device *ibdev = ibcq->device;
struct mana_ib_dev *mdev;
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);

1
drivers/infiniband/hw/mana/device.c
View File

@ -15,6 +15,7 @@ static const struct ib_device_ops mana_ib_dev_ops = {
.owner = THIS_MODULE,
.driver_id = RDMA_DRIVER_MANA,
.uverbs_abi_ver = MANA_IB_UVERBS_ABI_VERSION,
.uverbs_robust_udata = true,
.add_gid = mana_ib_gd_add_gid,
.alloc_mw = mana_ib_alloc_mw,

129
drivers/infiniband/hw/mana/main.c
View File

@ -90,17 +90,42 @@ int mana_ib_cfg_vport(struct mana_ib_dev *dev, u32 port, struct mana_ib_pd *pd,
return err;
}
static int mana_gd_destroy_pd(struct mana_ib_dev *mdev, u64 pd_handle)
{
struct gdma_destroy_pd_resp resp = {};
struct gdma_destroy_pd_req req = {};
mana_gd_init_req_hdr(&req.hdr, GDMA_DESTROY_PD, sizeof(req),
sizeof(resp));
req.pd_handle = pd_handle;
return mana_gd_send_request(mdev_to_gc(mdev), sizeof(req), &req, sizeof(resp), &resp);
}
int mana_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct mana_ib_pd *pd = container_of(ibpd, struct mana_ib_pd, ibpd);
struct mana_ib_alloc_pd_resp ucmd_resp = {};
struct ib_device *ibdev = ibpd->device;
struct gdma_create_pd_resp resp = {};
struct gdma_create_pd_req req = {};
struct mana_ib_alloc_pd ucmd;
enum gdma_pd_flags flags = 0;
struct mana_ib_dev *dev;
struct gdma_context *gc;
int err;
if (udata && udata->inlen) {
err = ib_copy_validate_udata_in_cm(udata, ucmd, reserved,
MANA_IB_PD_SHORT_PDN);
if (err)
return err;
if (ucmd.comp_mask & MANA_IB_PD_SHORT_PDN)
flags |= GDMA_PD_FLAG_SHORT_PDN;
}
dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
gc = mdev_to_gc(dev);
@ -117,32 +142,41 @@ int mana_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
pd->pd_handle = resp.pd_handle;
pd->pdn = resp.pd_id;
ibdev_dbg(&dev->ib_dev, "pd_handle 0x%llx pd_id %d\n",
pd->pd_handle, pd->pdn);
mutex_init(&pd->vport_mutex);
pd->vport_use_count = 0;
if (udata) {
ucmd_resp.pdn = pd->pdn;
err = ib_respond_udata(udata, ucmd_resp);
if (err)
goto destroy_pd;
}
return 0;
destroy_pd:
mana_gd_destroy_pd(dev, pd->pd_handle);
return err;
}
int mana_ib_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct mana_ib_pd *pd = container_of(ibpd, struct mana_ib_pd, ibpd);
struct ib_device *ibdev = ibpd->device;
struct gdma_destory_pd_resp resp = {};
struct gdma_destroy_pd_req req = {};
struct mana_ib_dev *dev;
struct gdma_context *gc;
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
gc = mdev_to_gc(dev);
err = mana_gd_destroy_pd(dev, pd->pd_handle);
if (err)
return err;
mana_gd_init_req_hdr(&req.hdr, GDMA_DESTROY_PD, sizeof(req),
sizeof(resp));
req.pd_handle = pd->pd_handle;
return mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
return 0;
}
static int mana_gd_destroy_doorbell_page(struct gdma_context *gc,
@ -192,25 +226,30 @@ int mana_ib_alloc_ucontext(struct ib_ucontext *ibcontext,
{
struct mana_ib_ucontext *ucontext =
container_of(ibcontext, struct mana_ib_ucontext, ibucontext);
struct mana_ib_alloc_ucontext_resp ucmd_resp = {};
struct ib_device *ibdev = ibcontext->device;
struct mana_ib_dev *mdev;
struct gdma_context *gc;
int doorbell_page;
int ret;
ret = ib_is_udata_in_empty(udata);
if (ret)
return ret;
mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);
gc = mdev_to_gc(mdev);
/* Allocate a doorbell page index */
ret = mana_gd_allocate_doorbell_page(gc, &doorbell_page);
if (ret) {
ibdev_dbg(ibdev, "Failed to allocate doorbell page %d\n", ret);
if (ret)
return ret;
}
ibdev_dbg(ibdev, "Doorbell page allocated %d\n", doorbell_page);
ucontext->doorbell = doorbell_page;
ucmd_resp.comp_mask = MANA_IB_UCNTX_ALLOC_PDN_SUPPORT;
ret = ib_respond_udata(udata, ucmd_resp);
if (ret)
return ret;
return 0;
}
@ -575,11 +614,10 @@ int mana_ib_query_device(struct ib_device *ibdev, struct ib_device_attr *props,
struct pci_dev *pdev = to_pci_dev(mdev_to_gc(dev)->dev);
int err;
err = ib_is_udata_in_empty(uhw);
err = ib_no_udata_io(uhw);
if (err)
return err;
memset(props, 0, sizeof(*props));
props->vendor_id = pdev->vendor;
props->vendor_part_id = dev->gdma_dev->dev_id.type;
props->max_mr_size = MANA_IB_MAX_MR_SIZE;
@ -605,7 +643,7 @@ int mana_ib_query_device(struct ib_device *ibdev, struct ib_device_attr *props,
if (!mana_ib_is_rnic(dev))
props->raw_packet_caps = IB_RAW_PACKET_CAP_IP_CSUM;
return ib_respond_empty_udata(uhw);
return 0;
}
int mana_ib_query_port(struct ib_device *ibdev, u32 port,
@ -1034,16 +1072,55 @@ int mana_ib_gd_create_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
return 0;
}
int mana_ib_gd_destroy_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
int mana_ib_gd_create_uc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
struct ib_qp_init_attr *attr, u32 doorbell, u64 flags)
{
struct mana_rnic_destroy_rc_qp_resp resp = {0};
struct mana_rnic_destroy_rc_qp_req req = {0};
struct mana_ib_cq *send_cq = container_of(qp->ibqp.send_cq, struct mana_ib_cq, ibcq);
struct mana_ib_cq *recv_cq = container_of(qp->ibqp.recv_cq, struct mana_ib_cq, ibcq);
struct mana_ib_pd *pd = container_of(qp->ibqp.pd, struct mana_ib_pd, ibpd);
struct gdma_context *gc = mdev_to_gc(mdev);
struct mana_rnic_create_uc_qp_resp resp = {};
struct mana_rnic_create_uc_qp_req req = {};
int err, i;
mana_gd_init_req_hdr(&req.hdr, MANA_IB_DESTROY_RC_QP, sizeof(req), sizeof(resp));
mana_gd_init_req_hdr(&req.hdr, MANA_IB_CREATE_UC_QP, sizeof(req), sizeof(resp));
req.hdr.dev_id = mdev->gdma_dev->dev_id;
req.adapter = mdev->adapter_handle;
req.rc_qp_handle = qp->qp_handle;
req.pd_handle = pd->pd_handle;
req.send_cq_handle = send_cq->cq_handle;
req.recv_cq_handle = recv_cq->cq_handle;
for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; i++)
req.dma_region[i] = qp->uc_qp.queues[i].gdma_region;
req.doorbell_page = doorbell;
req.max_send_wr = attr->cap.max_send_wr;
req.max_recv_wr = attr->cap.max_recv_wr;
req.max_send_sge = attr->cap.max_send_sge;
req.max_recv_sge = attr->cap.max_recv_sge;
req.flags = flags;
err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
if (err)
return err;
qp->qp_handle = resp.qp_handle;
for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; i++) {
qp->uc_qp.queues[i].id = resp.queue_ids[i];
/* The GDMA regions are now owned by the RNIC QP handle */
qp->uc_qp.queues[i].gdma_region = GDMA_INVALID_DMA_REGION;
}
return 0;
}
int mana_ib_gd_destroy_rnic_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
{
struct mana_rnic_destroy_rnic_qp_resp resp = {0};
struct mana_rnic_destroy_rnic_qp_req req = {0};
struct gdma_context *gc = mdev_to_gc(mdev);
mana_gd_init_req_hdr(&req.hdr, MANA_IB_DESTROY_RNIC_QP, sizeof(req), sizeof(resp));
req.hdr.dev_id = mdev->gdma_dev->dev_id;
req.adapter = mdev->adapter_handle;
req.qp_handle = qp->qp_handle;
return mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
}

83
drivers/infiniband/hw/mana/mana_ib.h
View File

@ -26,6 +26,8 @@
/* Send queue ID mask */
#define MANA_SENDQ_MASK BIT(31)
/* Queue ID encodes type in the lower 2 bits */
#define MANA_QID_SUBTYPE_MASK 0x3
/*
* The hardware limit of number of MRs is greater than maximum number of MRs
@ -180,6 +182,17 @@ struct mana_ib_rc_qp {
struct mana_ib_queue queues[MANA_RC_QUEUE_TYPE_MAX];
};
enum mana_uc_queue_type {
MANA_UC_SEND_QUEUE_REQUESTER = 0,
MANA_UC_RECV_QUEUE_RESPONDER,
MANA_UC_SEND_QUEUE_MMQ,
MANA_UC_QUEUE_TYPE_MAX,
};
struct mana_ib_uc_qp {
struct mana_ib_queue queues[MANA_UC_QUEUE_TYPE_MAX];
};
enum mana_ud_queue_type {
MANA_UD_SEND_QUEUE = 0,
MANA_UD_RECV_QUEUE,
@ -198,6 +211,7 @@ struct mana_ib_qp {
union {
struct mana_ib_queue raw_sq;
struct mana_ib_rc_qp rc_qp;
struct mana_ib_uc_qp uc_qp;
struct mana_ib_ud_qp ud_qp;
};
@ -233,8 +247,9 @@ enum mana_ib_command_code {
MANA_IB_CREATE_CQ = 0x30008,
MANA_IB_DESTROY_CQ = 0x30009,
MANA_IB_CREATE_RC_QP = 0x3000a,
MANA_IB_DESTROY_RC_QP = 0x3000b,
MANA_IB_DESTROY_RNIC_QP = 0x3000b,
MANA_IB_SET_QP_STATE = 0x3000d,
MANA_IB_CREATE_UC_QP = 0x30020,
MANA_IB_QUERY_VF_COUNTERS = 0x30022,
MANA_IB_QUERY_DEVICE_COUNTERS = 0x30023,
};
@ -384,16 +399,39 @@ struct mana_rnic_create_qp_resp {
u32 reserved;
}; /* HW Data*/
struct mana_rnic_destroy_rc_qp_req {
struct mana_rnic_destroy_rnic_qp_req {
struct gdma_req_hdr hdr;
mana_handle_t adapter;
mana_handle_t rc_qp_handle;
mana_handle_t qp_handle;
}; /* HW Data */
struct mana_rnic_destroy_rc_qp_resp {
struct mana_rnic_destroy_rnic_qp_resp {
struct gdma_resp_hdr hdr;
}; /* HW Data */
struct mana_rnic_create_uc_qp_req {
struct gdma_req_hdr hdr;
mana_handle_t adapter;
mana_handle_t pd_handle;
mana_handle_t send_cq_handle;
mana_handle_t recv_cq_handle;
u64 dma_region[MANA_UC_QUEUE_TYPE_MAX];
u64 flags;
u32 doorbell_page;
u32 max_send_wr;
u32 max_recv_wr;
u32 max_send_sge;
u32 max_recv_sge;
u32 reserved;
}; /* HW Data */
struct mana_rnic_create_uc_qp_resp {
struct gdma_resp_hdr hdr;
mana_handle_t qp_handle;
u32 queue_ids[MANA_UC_QUEUE_TYPE_MAX];
u32 reserved;
}; /* HW Data*/
struct mana_rnic_create_udqp_req {
struct gdma_req_hdr hdr;
mana_handle_t adapter;
@ -582,12 +620,44 @@ static inline struct gdma_context *mdev_to_gc(struct mana_ib_dev *mdev)
return mdev->gdma_dev->gdma_context;
}
static inline struct mana_ib_queue *mana_qp_get_sq(struct mana_ib_qp *qp)
{
switch (qp->ibqp.qp_type) {
case IB_QPT_RC:
return &qp->rc_qp.queues[MANA_RC_SEND_QUEUE_REQUESTER];
case IB_QPT_UC:
return &qp->uc_qp.queues[MANA_UC_SEND_QUEUE_REQUESTER];
case IB_QPT_UD:
case IB_QPT_GSI:
return &qp->ud_qp.queues[MANA_UD_SEND_QUEUE];
default:
return NULL;
}
}
static inline struct mana_ib_queue *mana_qp_get_rq(struct mana_ib_qp *qp)
{
switch (qp->ibqp.qp_type) {
case IB_QPT_RC:
return &qp->rc_qp.queues[MANA_RC_RECV_QUEUE_RESPONDER];
case IB_QPT_UC:
return &qp->uc_qp.queues[MANA_UC_RECV_QUEUE_RESPONDER];
case IB_QPT_UD:
case IB_QPT_GSI:
return &qp->ud_qp.queues[MANA_UD_RECV_QUEUE];
default:
return NULL;
}
}
static inline struct mana_ib_qp *mana_get_qp_ref(struct mana_ib_dev *mdev,
u32 qid, bool is_sq)
{
struct mana_ib_qp *qp;
unsigned long flag;
/* Remove subtype bits */
qid &= ~MANA_QID_SUBTYPE_MASK;
if (is_sq)
qid |= MANA_SENDQ_MASK;
@ -736,8 +806,9 @@ int mana_ib_gd_destroy_cq(struct mana_ib_dev *mdev, struct mana_ib_cq *cq);
int mana_ib_gd_create_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
struct ib_qp_init_attr *attr, u32 doorbell, u64 flags);
int mana_ib_gd_destroy_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp);
int mana_ib_gd_destroy_rnic_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp);
int mana_ib_gd_create_uc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
struct ib_qp_init_attr *attr, u32 doorbell, u64 flags);
int mana_ib_gd_create_ud_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
struct ib_qp_init_attr *attr, u32 doorbell, u32 type);
int mana_ib_gd_destroy_ud_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp);

13
drivers/infiniband/hw/mana/mr.c
View File

@ -113,6 +113,10 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
if (dmah)
return ERR_PTR(-EOPNOTSUPP);
err = ib_no_udata_io(udata);
if (err)
return ERR_PTR(err);
dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
ibdev_dbg(ibdev,
@ -327,6 +331,11 @@ int mana_ib_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)
{
struct mana_ib_dev *mdev = container_of(ibmw->device, struct mana_ib_dev, ib_dev);
struct mana_ib_pd *pd = container_of(ibmw->pd, struct mana_ib_pd, ibpd);
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
return mana_ib_gd_create_mw(mdev, pd, ibmw);
}
@ -346,6 +355,10 @@ int mana_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
struct mana_ib_dev *dev;
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
dev = container_of(ibdev, struct mana_ib_dev, ib_dev);
err = mana_ib_gd_destroy_mr(dev, mr->mr_handle);

229
drivers/infiniband/hw/mana/qp.c
View File

@ -461,84 +461,75 @@ static enum gdma_queue_type mana_ib_queue_type(struct ib_qp_init_attr *attr, u32
return type;
}
static int mana_table_store_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
static void mana_table_drain_qp_ref(struct mana_ib_qp *qp)
{
return xa_insert_irq(&mdev->qp_table_wq, qp->ibqp.qp_num, qp,
GFP_KERNEL);
mana_put_qp_ref(qp);
wait_for_completion(&qp->free);
}
static void mana_table_remove_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
static int mana_table_store_qp_qid(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
u32 qid, bool is_sq)
{
xa_erase_irq(&mdev->qp_table_wq, qp->ibqp.qp_num);
/* Remove subtype bits */
qid &= ~MANA_QID_SUBTYPE_MASK;
if (is_sq)
qid |= MANA_SENDQ_MASK;
return xa_insert_irq(&mdev->qp_table_wq, qid, qp, GFP_KERNEL);
}
static int mana_table_store_ud_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
static void mana_table_remove_qp_qid(struct mana_ib_dev *mdev, u32 qid, bool is_sq)
{
u32 qids = qp->ud_qp.queues[MANA_UD_SEND_QUEUE].id | MANA_SENDQ_MASK;
u32 qidr = qp->ud_qp.queues[MANA_UD_RECV_QUEUE].id;
int err;
/* Remove subtype bits */
qid &= ~MANA_QID_SUBTYPE_MASK;
if (is_sq)
qid |= MANA_SENDQ_MASK;
err = xa_insert_irq(&mdev->qp_table_wq, qids, qp, GFP_KERNEL);
if (err)
return err;
err = xa_insert_irq(&mdev->qp_table_wq, qidr, qp, GFP_KERNEL);
if (err)
goto remove_sq;
return 0;
remove_sq:
xa_erase_irq(&mdev->qp_table_wq, qids);
return err;
}
static void mana_table_remove_ud_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
{
u32 qids = qp->ud_qp.queues[MANA_UD_SEND_QUEUE].id | MANA_SENDQ_MASK;
u32 qidr = qp->ud_qp.queues[MANA_UD_RECV_QUEUE].id;
xa_erase_irq(&mdev->qp_table_wq, qids);
xa_erase_irq(&mdev->qp_table_wq, qidr);
xa_erase_irq(&mdev->qp_table_wq, qid);
}
static int mana_table_store_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
{
struct mana_ib_queue *sq, *rq;
int err;
refcount_set(&qp->refcount, 1);
init_completion(&qp->free);
sq = mana_qp_get_sq(qp);
rq = mana_qp_get_rq(qp);
switch (qp->ibqp.qp_type) {
case IB_QPT_RC:
return mana_table_store_rc_qp(mdev, qp);
case IB_QPT_UD:
case IB_QPT_GSI:
return mana_table_store_ud_qp(mdev, qp);
default:
ibdev_dbg(&mdev->ib_dev, "Unknown QP type for storing in mana table, %d\n",
qp->ibqp.qp_type);
}
if (!sq || !rq)
return -EINVAL;
return -EINVAL;
err = mana_table_store_qp_qid(mdev, qp, sq->id, true);
if (err)
return err;
err = mana_table_store_qp_qid(mdev, qp, rq->id, false);
if (err)
goto err_remove_sq;
return 0;
err_remove_sq:
mana_table_remove_qp_qid(mdev, sq->id, true);
mana_table_drain_qp_ref(qp);
return err;
}
static void mana_table_remove_qp(struct mana_ib_dev *mdev,
struct mana_ib_qp *qp)
static void mana_table_remove_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
{
switch (qp->ibqp.qp_type) {
case IB_QPT_RC:
mana_table_remove_rc_qp(mdev, qp);
break;
case IB_QPT_UD:
case IB_QPT_GSI:
mana_table_remove_ud_qp(mdev, qp);
break;
default:
ibdev_dbg(&mdev->ib_dev, "Unknown QP type for removing from mana table, %d\n",
qp->ibqp.qp_type);
struct mana_ib_queue *sq, *rq;
sq = mana_qp_get_sq(qp);
rq = mana_qp_get_rq(qp);
if (!sq || !rq)
return;
}
mana_put_qp_ref(qp);
wait_for_completion(&qp->free);
mana_table_remove_qp_qid(mdev, sq->id, true);
mana_table_remove_qp_qid(mdev, rq->id, false);
mana_table_drain_qp_ref(qp);
}
static int mana_ib_create_rc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
@ -606,13 +597,67 @@ static int mana_ib_create_rc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
return 0;
destroy_qp:
mana_ib_gd_destroy_rc_qp(mdev, qp);
mana_ib_gd_destroy_rnic_qp(mdev, qp);
destroy_queues:
while (i-- > 0)
mana_ib_destroy_queue(mdev, &qp->rc_qp.queues[i]);
return err;
}
static int mana_ib_create_uc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
struct ib_qp_init_attr *attr, struct ib_udata *udata)
{
struct mana_ib_dev *mdev = container_of(ibpd->device, struct mana_ib_dev, ib_dev);
struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
struct mana_ib_create_uc_qp_resp resp = {};
struct mana_ib_ucontext *mana_ucontext;
struct mana_ib_create_uc_qp ucmd;
u64 flags = 0;
u32 doorbell;
int err, i;
if (!udata)
return -EINVAL;
mana_ucontext = rdma_udata_to_drv_context(udata, struct mana_ib_ucontext, ibucontext);
doorbell = mana_ucontext->doorbell;
err = ib_copy_validate_udata_in_cm(udata, ucmd, comp_mask, 0);
if (err)
return err;
for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; ++i) {
err = mana_ib_create_queue(mdev, ucmd.queue_buf[i], ucmd.queue_size[i],
&qp->uc_qp.queues[i]);
if (err)
goto destroy_queues;
}
err = mana_ib_gd_create_uc_qp(mdev, qp, attr, doorbell, flags);
if (err)
goto destroy_queues;
qp->ibqp.qp_num = qp->uc_qp.queues[MANA_UC_RECV_QUEUE_RESPONDER].id;
qp->port = attr->port_num;
for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; ++i)
resp.queue_id[i] = qp->uc_qp.queues[i].id;
err = ib_respond_udata(udata, resp);
if (err)
goto destroy_qp;
err = mana_table_store_qp(mdev, qp);
if (err)
goto destroy_qp;
return 0;
destroy_qp:
mana_ib_gd_destroy_rnic_qp(mdev, qp);
destroy_queues:
while (i-- > 0)
mana_ib_destroy_queue(mdev, &qp->uc_qp.queues[i]);
return err;
}
static void mana_add_qp_to_cqs(struct mana_ib_qp *qp)
{
struct mana_ib_cq *send_cq = container_of(qp->ibqp.send_cq, struct mana_ib_cq, ibcq);
@ -651,10 +696,8 @@ static int mana_ib_create_ud_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
u32 doorbell, queue_size;
int i, err;
if (udata) {
ibdev_dbg(&mdev->ib_dev, "User-level UD QPs are not supported\n");
if (udata)
return -EOPNOTSUPP;
}
for (i = 0; i < MANA_UD_QUEUE_TYPE_MAX; ++i) {
queue_size = mana_ib_queue_size(attr, i);
@ -724,6 +767,8 @@ int mana_ib_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attr,
return mana_ib_create_qp_raw(ibqp, ibqp->pd, attr, udata);
case IB_QPT_RC:
return mana_ib_create_rc_qp(ibqp, ibqp->pd, attr, udata);
case IB_QPT_UC:
return mana_ib_create_uc_qp(ibqp, ibqp->pd, attr, udata);
case IB_QPT_UD:
case IB_QPT_GSI:
return mana_ib_create_ud_qp(ibqp, ibqp->pd, attr, udata);
@ -745,6 +790,11 @@ static int mana_ib_gd_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
struct gdma_context *gc = mdev_to_gc(mdev);
struct mana_port_context *mpc;
struct net_device *ndev;
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
mana_gd_init_req_hdr(&req.hdr, MANA_IB_SET_QP_STATE, sizeof(req), sizeof(resp));
@ -797,7 +847,11 @@ static int mana_ib_gd_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
req.ah_attr.flow_label = attr->ah_attr.grh.flow_label;
}
return mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
if (err)
return err;
return 0;
}
int mana_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
@ -805,6 +859,7 @@ int mana_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
{
switch (ibqp->qp_type) {
case IB_QPT_RC:
case IB_QPT_UC:
case IB_QPT_UD:
case IB_QPT_GSI:
return mana_ib_gd_modify_qp(ibqp, attr, attr_mask, udata);
@ -826,7 +881,11 @@ static int mana_ib_destroy_qp_rss(struct mana_ib_qp *qp,
struct mana_ib_pd *pd;
struct mana_ib_wq *wq;
struct ib_wq *ibwq;
int i;
int i, err;
err = ib_no_udata_io(udata);
if (err)
return err;
ndev = mana_ib_get_netdev(qp->ibqp.device, qp->port);
mpc = netdev_priv(ndev);
@ -872,6 +931,11 @@ static int mana_ib_destroy_qp_raw(struct mana_ib_qp *qp, struct ib_udata *udata)
struct mana_port_context *mpc;
struct net_device *ndev;
struct mana_ib_pd *pd;
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
ndev = mana_ib_get_netdev(qp->ibqp.device, qp->port);
mpc = netdev_priv(ndev);
@ -890,25 +954,54 @@ static int mana_ib_destroy_rc_qp(struct mana_ib_qp *qp, struct ib_udata *udata)
{
struct mana_ib_dev *mdev =
container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
int i;
int i, err;
err = ib_no_udata_io(udata);
if (err)
return err;
mana_table_remove_qp(mdev, qp);
/* Ignore return code as there is not much we can do about it.
* The error message is printed inside.
*/
mana_ib_gd_destroy_rc_qp(mdev, qp);
mana_ib_gd_destroy_rnic_qp(mdev, qp);
for (i = 0; i < MANA_RC_QUEUE_TYPE_MAX; ++i)
mana_ib_destroy_queue(mdev, &qp->rc_qp.queues[i]);
return 0;
}
static int mana_ib_destroy_uc_qp(struct mana_ib_qp *qp, struct ib_udata *udata)
{
struct mana_ib_dev *mdev =
container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
int i, err;
err = ib_no_udata_io(udata);
if (err)
return err;
mana_table_remove_qp(mdev, qp);
/* Ignore return code as there is not much we can do about it.
* The error message is printed inside.
*/
mana_ib_gd_destroy_rnic_qp(mdev, qp);
for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; ++i)
mana_ib_destroy_queue(mdev, &qp->uc_qp.queues[i]);
return 0;
}
static int mana_ib_destroy_ud_qp(struct mana_ib_qp *qp, struct ib_udata *udata)
{
struct mana_ib_dev *mdev =
container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
int i;
int i, err;
err = ib_no_udata_io(udata);
if (err)
return err;
mana_remove_qp_from_cqs(qp);
mana_table_remove_qp(mdev, qp);
@ -939,6 +1032,8 @@ int mana_ib_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
return mana_ib_destroy_qp_raw(qp, udata);
case IB_QPT_RC:
return mana_ib_destroy_rc_qp(qp, udata);
case IB_QPT_UC:
return mana_ib_destroy_uc_qp(qp, udata);
case IB_QPT_UD:
case IB_QPT_GSI:
return mana_ib_destroy_ud_qp(qp, udata);

12
drivers/infiniband/hw/mana/wq.c
View File

@ -55,6 +55,11 @@ int mana_ib_destroy_wq(struct ib_wq *ibwq, struct ib_udata *udata)
struct mana_ib_wq *wq = container_of(ibwq, struct mana_ib_wq, ibwq);
struct ib_device *ib_dev = ibwq->device;
struct mana_ib_dev *mdev;
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
mdev = container_of(ib_dev, struct mana_ib_dev, ib_dev);
@ -69,10 +74,17 @@ int mana_ib_create_rwq_ind_table(struct ib_rwq_ind_table *ib_rwq_ind_table,
struct ib_rwq_ind_table_init_attr *init_attr,
struct ib_udata *udata)
{
int err;
err = ib_no_udata_io(udata);
if (err)
return err;
/*
* There is no additional data in ind_table to be maintained by this
* driver, do nothing
*/
return 0;
}

120
drivers/infiniband/hw/mlx4/cm.c
View File

@ -40,6 +40,7 @@
#include "mlx4_ib.h"
#define CM_CLEANUP_CACHE_TIMEOUT (30 * HZ)
#define CM_RTU_TIMEOUT (60 * HZ)
struct id_map_entry {
struct rb_node node;
@ -48,6 +49,7 @@ struct id_map_entry {
u32 pv_cm_id;
int slave_id;
int scheduled_delete;
bool rtu_timeout;
struct mlx4_ib_dev *dev;
struct list_head list;
@ -184,6 +186,10 @@ static void id_map_ent_timeout(struct work_struct *work)
struct rb_root *sl_id_map = &sriov->sl_id_map;
spin_lock(&sriov->id_map_lock);
if (!ent->scheduled_delete) {
spin_unlock(&sriov->id_map_lock);
return;
}
if (!xa_erase(&sriov->pv_id_table, ent->pv_cm_id))
goto out;
found_ent = id_map_find_by_sl_id(&dev->ib_dev, ent->slave_id, ent->sl_cm_id);
@ -228,8 +234,12 @@ static void sl_id_map_add(struct ib_device *ibdev, struct id_map_entry *new)
rb_insert_color(&new->node, sl_id_map);
}
static void schedule_delayed(struct ib_device *ibdev, struct id_map_entry *id,
unsigned long timeout, bool rtu_timeout);
static struct id_map_entry *
id_map_alloc(struct ib_device *ibdev, int slave_id, u32 sl_cm_id)
id_map_alloc(struct ib_device *ibdev, int slave_id, u32 sl_cm_id,
u32 *pv_cm_id)
{
int ret;
struct id_map_entry *ent;
@ -242,6 +252,7 @@ id_map_alloc(struct ib_device *ibdev, int slave_id, u32 sl_cm_id)
ent->sl_cm_id = sl_cm_id;
ent->slave_id = slave_id;
ent->scheduled_delete = 0;
ent->rtu_timeout = false;
ent->dev = to_mdev(ibdev);
INIT_DELAYED_WORK(&ent->timeout, id_map_ent_timeout);
@ -251,6 +262,8 @@ id_map_alloc(struct ib_device *ibdev, int slave_id, u32 sl_cm_id)
spin_lock(&sriov->id_map_lock);
sl_id_map_add(ibdev, ent);
list_add_tail(&ent->list, &sriov->cm_list);
*pv_cm_id = ent->pv_cm_id;
schedule_delayed(ibdev, ent, CM_RTU_TIMEOUT, true);
spin_unlock(&sriov->id_map_lock);
return ent;
}
@ -267,42 +280,41 @@ id_map_get(struct ib_device *ibdev, int *pv_cm_id, int slave_id, int sl_cm_id)
struct id_map_entry *ent;
struct mlx4_ib_sriov *sriov = &to_mdev(ibdev)->sriov;
spin_lock(&sriov->id_map_lock);
lockdep_assert_held(&sriov->id_map_lock);
if (*pv_cm_id == -1) {
ent = id_map_find_by_sl_id(ibdev, slave_id, sl_cm_id);
if (ent)
*pv_cm_id = (int) ent->pv_cm_id;
} else
ent = xa_load(&sriov->pv_id_table, *pv_cm_id);
spin_unlock(&sriov->id_map_lock);
return ent;
}
static void schedule_delayed(struct ib_device *ibdev, struct id_map_entry *id)
static void schedule_delayed(struct ib_device *ibdev, struct id_map_entry *id,
unsigned long timeout, bool rtu_timeout)
{
struct mlx4_ib_sriov *sriov = &to_mdev(ibdev)->sriov;
unsigned long flags;
spin_lock(&sriov->id_map_lock);
lockdep_assert_held(&sriov->id_map_lock);
spin_lock_irqsave(&sriov->going_down_lock, flags);
/*make sure that there is no schedule inside the scheduled work.*/
if (!sriov->is_going_down && !id->scheduled_delete) {
if (!sriov->is_going_down || id->scheduled_delete) {
id->scheduled_delete = 1;
queue_delayed_work(cm_wq, &id->timeout, CM_CLEANUP_CACHE_TIMEOUT);
} else if (id->scheduled_delete) {
/* Adjust timeout if already scheduled */
mod_delayed_work(cm_wq, &id->timeout, CM_CLEANUP_CACHE_TIMEOUT);
id->rtu_timeout = rtu_timeout;
mod_delayed_work(cm_wq, &id->timeout, timeout);
}
spin_unlock_irqrestore(&sriov->going_down_lock, flags);
spin_unlock(&sriov->id_map_lock);
}
#define REJ_REASON(m) be16_to_cpu(((struct cm_generic_msg *)(m))->rej_reason)
int mlx4_ib_multiplex_cm_handler(struct ib_device *ibdev, int port, int slave_id,
struct ib_mad *mad)
{
struct mlx4_ib_sriov *sriov = &to_mdev(ibdev)->sriov;
struct id_map_entry *id;
u32 pv_cm_id_to_set = 0;
u32 sl_cm_id;
int pv_cm_id = -1;
@ -312,21 +324,62 @@ int mlx4_ib_multiplex_cm_handler(struct ib_device *ibdev, int port, int slave_id
mad->mad_hdr.attr_id == CM_SIDR_REQ_ATTR_ID ||
(mad->mad_hdr.attr_id == CM_REJ_ATTR_ID && REJ_REASON(mad) == IB_CM_REJ_TIMEOUT)) {
sl_cm_id = get_local_comm_id(mad);
spin_lock(&sriov->id_map_lock);
id = id_map_get(ibdev, &pv_cm_id, slave_id, sl_cm_id);
if (id) {
pv_cm_id_to_set = id->pv_cm_id;
if (mad->mad_hdr.attr_id == CM_REJ_ATTR_ID)
schedule_delayed(ibdev, id,
CM_CLEANUP_CACHE_TIMEOUT,
false);
}
spin_unlock(&sriov->id_map_lock);
if (id)
goto cont;
id = id_map_alloc(ibdev, slave_id, sl_cm_id);
if (mad->mad_hdr.attr_id == CM_REJ_ATTR_ID)
return 0;
id = id_map_alloc(ibdev, slave_id, sl_cm_id,
&pv_cm_id_to_set);
if (IS_ERR(id)) {
mlx4_ib_warn(ibdev, "%s: id{slave: %d, sl_cm_id: 0x%x} Failed to id_map_alloc\n",
__func__, slave_id, sl_cm_id);
return PTR_ERR(id);
}
} else if (mad->mad_hdr.attr_id == CM_REJ_ATTR_ID ||
mad->mad_hdr.attr_id == CM_SIDR_REP_ATTR_ID) {
} else if (mad->mad_hdr.attr_id == CM_REJ_ATTR_ID) {
sl_cm_id = get_local_comm_id(mad);
spin_lock(&sriov->id_map_lock);
id = id_map_get(ibdev, &pv_cm_id, slave_id, sl_cm_id);
if (id) {
pv_cm_id_to_set = id->pv_cm_id;
schedule_delayed(ibdev, id, CM_CLEANUP_CACHE_TIMEOUT,
false);
}
spin_unlock(&sriov->id_map_lock);
if (!id)
return 0;
} else if (mad->mad_hdr.attr_id == CM_SIDR_REP_ATTR_ID) {
return 0;
} else {
sl_cm_id = get_local_comm_id(mad);
spin_lock(&sriov->id_map_lock);
id = id_map_get(ibdev, &pv_cm_id, slave_id, sl_cm_id);
if (id) {
if (mad->mad_hdr.attr_id == CM_RTU_ATTR_ID &&
id->rtu_timeout) {
id->rtu_timeout = false;
if (cancel_delayed_work(&id->timeout))
id->scheduled_delete = 0;
else
id = NULL;
}
if (id)
pv_cm_id_to_set = id->pv_cm_id;
if (id && mad->mad_hdr.attr_id == CM_DREQ_ATTR_ID)
schedule_delayed(ibdev, id,
CM_CLEANUP_CACHE_TIMEOUT,
false);
}
spin_unlock(&sriov->id_map_lock);
}
if (!id) {
@ -336,10 +389,7 @@ int mlx4_ib_multiplex_cm_handler(struct ib_device *ibdev, int port, int slave_id
}
cont:
set_local_comm_id(mad, id->pv_cm_id);
if (mad->mad_hdr.attr_id == CM_DREQ_ATTR_ID)
schedule_delayed(ibdev, id);
set_local_comm_id(mad, pv_cm_id_to_set);
return 0;
}
@ -429,7 +479,10 @@ int mlx4_ib_demux_cm_handler(struct ib_device *ibdev, int port, int *slave,
struct mlx4_ib_sriov *sriov = &to_mdev(ibdev)->sriov;
u32 rem_pv_cm_id = get_local_comm_id(mad);
u32 pv_cm_id;
u32 sl_cm_id = 0;
struct id_map_entry *id;
int pv_cm_id_int;
int slave_id = 0;
int sts;
if (mad->mad_hdr.attr_id == CM_REQ_ATTR_ID ||
@ -457,7 +510,28 @@ int mlx4_ib_demux_cm_handler(struct ib_device *ibdev, int port, int *slave,
}
pv_cm_id = get_remote_comm_id(mad);
id = id_map_get(ibdev, (int *)&pv_cm_id, -1, -1);
pv_cm_id_int = pv_cm_id;
spin_lock(&sriov->id_map_lock);
id = id_map_get(ibdev, &pv_cm_id_int, -1, -1);
if (id) {
if (mad->mad_hdr.attr_id == CM_RTU_ATTR_ID &&
id->rtu_timeout) {
id->rtu_timeout = false;
if (cancel_delayed_work(&id->timeout))
id->scheduled_delete = 0;
else
id = NULL;
}
if (id && slave)
slave_id = id->slave_id;
if (id)
sl_cm_id = id->sl_cm_id;
if (id && (mad->mad_hdr.attr_id == CM_DREQ_ATTR_ID ||
mad->mad_hdr.attr_id == CM_REJ_ATTR_ID))
schedule_delayed(ibdev, id,
CM_CLEANUP_CACHE_TIMEOUT, false);
}
spin_unlock(&sriov->id_map_lock);
if (!id) {
if (mad->mad_hdr.attr_id == CM_REJ_ATTR_ID &&
@ -472,12 +546,8 @@ int mlx4_ib_demux_cm_handler(struct ib_device *ibdev, int port, int *slave,
}
if (slave)
*slave = id->slave_id;
set_remote_comm_id(mad, id->sl_cm_id);
if (mad->mad_hdr.attr_id == CM_DREQ_ATTR_ID ||
mad->mad_hdr.attr_id == CM_REJ_ATTR_ID)
schedule_delayed(ibdev, id);
*slave = slave_id;
set_remote_comm_id(mad, sl_cm_id);
return 0;
}

2
drivers/infiniband/hw/mlx4/main.c
View File

@ -471,8 +471,6 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
if (err)
goto out;
memset(props, 0, sizeof *props);
have_ib_ports = num_ib_ports(dev->dev);
props->fw_ver = dev->dev->caps.fw_ver;

6
drivers/infiniband/hw/mlx4/mr.c
View File

@ -313,7 +313,7 @@ mlx4_alloc_priv_pages(struct ib_device *device,
MLX4_MR_PAGES_ALIGN);
/* Prevent cross page boundary allocation. */
mr->pages = (__be64 *)get_zeroed_page(GFP_KERNEL);
mr->pages = kzalloc(PAGE_SIZE, GFP_KERNEL);
if (!mr->pages)
return -ENOMEM;
@ -328,7 +328,7 @@ mlx4_alloc_priv_pages(struct ib_device *device,
return 0;
err:
free_page((unsigned long)mr->pages);
kfree(mr->pages);
return ret;
}
@ -340,7 +340,7 @@ mlx4_free_priv_pages(struct mlx4_ib_mr *mr)
dma_unmap_single(device->dev.parent, mr->page_map,
mr->page_map_size, DMA_TO_DEVICE);
free_page((unsigned long)mr->pages);
kfree(mr->pages);
mr->pages = NULL;
}
}

24
drivers/infiniband/hw/mlx5/cong.c
View File

@ -361,7 +361,7 @@ static int mlx5_ib_set_cc_params(struct mlx5_ib_dev *dev, u32 port_num,
MLX5_SET(field_select_r_roce_rp, field, field_select_r_roce_rp,
attr_mask);
err = mlx5_cmd_exec_in(dev->mdev, modify_cong_params, in);
err = mlx5_cmd_exec_in(mdev, modify_cong_params, in);
kvfree(in);
alloc_err:
mlx5_ib_put_native_port_mdev(dev, port_num + 1);
@ -373,22 +373,12 @@ static ssize_t set_param(struct file *filp, const char __user *buf,
{
struct mlx5_ib_dbg_param *param = filp->private_data;
int offset = param->offset;
char lbuf[11] = { };
u32 var;
int ret;
if (count > sizeof(lbuf))
return -EINVAL;
if (copy_from_user(lbuf, buf, count))
return -EFAULT;
lbuf[sizeof(lbuf) - 1] = '\0';
if (kstrtou32(lbuf, 0, &var))
return -EINVAL;
ret = mlx5_ib_set_cc_params(param->dev, param->port_num, offset, var);
ret = kstrtou32_from_user(buf, count, 0, &var);
if (!ret)
ret = mlx5_ib_set_cc_params(param->dev, param->port_num, offset, var);
return ret ? ret : count;
}
@ -399,15 +389,13 @@ static ssize_t get_param(struct file *filp, char __user *buf, size_t count,
int offset = param->offset;
u32 var = 0;
int ret;
char lbuf[11];
char lbuf[12];
ret = mlx5_ib_get_cc_params(param->dev, param->port_num, offset, &var);
if (ret)
return ret;
ret = snprintf(lbuf, sizeof(lbuf), "%d\n", var);
if (ret < 0)
return ret;
ret = scnprintf(lbuf, sizeof(lbuf), "%u\n", var);
return simple_read_from_buffer(buf, count, pos, lbuf, ret);
}

11
drivers/infiniband/hw/mlx5/counters.c
View File

@ -742,11 +742,9 @@ static void mlx5_ib_fill_counters(struct mlx5_ib_dev *dev,
names = is_vport ? vport_roce_accl_cnts : roce_accl_cnts;
size = is_vport ? ARRAY_SIZE(vport_roce_accl_cnts) :
ARRAY_SIZE(roce_accl_cnts);
if (MLX5_CAP_GEN(dev->mdev, roce_accl)) {
for (i = 0; i < size; i++, j++) {
descs[j].name = names[i].name;
offsets[j] = names[i].offset;
}
for (i = 0; i < size; i++, j++) {
descs[j].name = names[i].name;
offsets[j] = names[i].offset;
}
if (is_vport)
@ -826,8 +824,7 @@ static int __mlx5_ib_alloc_counters(struct mlx5_ib_dev *dev,
size = is_vport ? ARRAY_SIZE(vport_roce_accl_cnts) :
ARRAY_SIZE(roce_accl_cnts);
if (MLX5_CAP_GEN(dev->mdev, roce_accl))
num_counters += size;
num_counters += size;
cnts->num_q_counters = num_counters;

18
drivers/infiniband/hw/mlx5/main.c
View File

@ -969,7 +969,6 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
if (err)
return err;
memset(props, 0, sizeof(*props));
err = mlx5_query_system_image_guid(ibdev,
&props->sys_image_guid);
if (err)
@ -4477,6 +4476,7 @@ static const struct uapi_definition mlx5_ib_defs[] = {
UAPI_DEF_CHAIN(mlx5_ib_dm_defs),
UAPI_DEF_CHAIN(mlx5_ib_create_cq_defs),
UAPI_DEF_CHAIN(mlx5_ib_create_qp_defs),
UAPI_DEF_CHAIN(mlx5_ib_create_srq_defs),
UAPI_DEF_CHAIN_OBJ_TREE(UVERBS_OBJECT_DEVICE, &mlx5_ib_query_context),
UAPI_DEF_CHAIN_OBJ_TREE(UVERBS_OBJECT_MR, &mlx5_ib_reg_dmabuf_mr),
@ -4951,6 +4951,9 @@ static int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev)
if (err)
return err;
if (MLX5_CAP_GEN(dev->mdev, qp_latency_sensitive_disable))
return 0;
err = mlx5_alloc_bfreg(dev->mdev, &dev->fp_bfreg, false, true);
if (err)
mlx5_free_bfreg(dev->mdev, &dev->bfreg);
@ -4960,7 +4963,8 @@ static int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev)
static void mlx5_ib_stage_bfrag_cleanup(struct mlx5_ib_dev *dev)
{
mlx5_free_bfreg(dev->mdev, &dev->fp_bfreg);
if (!MLX5_CAP_GEN(dev->mdev, qp_latency_sensitive_disable))
mlx5_free_bfreg(dev->mdev, &dev->fp_bfreg);
mlx5_free_bfreg(dev->mdev, &dev->bfreg);
}
@ -5501,13 +5505,13 @@ static int __init mlx5_ib_init(void)
{
int ret;
xlt_emergency_page = (void *)__get_free_page(GFP_KERNEL);
xlt_emergency_page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!xlt_emergency_page)
return -ENOMEM;
mlx5_ib_event_wq = alloc_ordered_workqueue("mlx5_ib_event_wq", 0);
if (!mlx5_ib_event_wq) {
free_page((unsigned long)xlt_emergency_page);
kfree(xlt_emergency_page);
return -ENOMEM;
}
@ -5538,10 +5542,11 @@ static int __init mlx5_ib_init(void)
dd_err:
mlx5r_rep_cleanup();
rep_err:
rcu_barrier();
mlx5_ib_qp_event_cleanup();
qp_event_err:
destroy_workqueue(mlx5_ib_event_wq);
free_page((unsigned long)xlt_emergency_page);
kfree(xlt_emergency_page);
return ret;
}
@ -5551,10 +5556,11 @@ static void __exit mlx5_ib_cleanup(void)
auxiliary_driver_unregister(&mlx5r_driver);
auxiliary_driver_unregister(&mlx5r_mp_driver);
mlx5r_rep_cleanup();
rcu_barrier();
mlx5_ib_qp_event_cleanup();
destroy_workqueue(mlx5_ib_event_wq);
free_page((unsigned long)xlt_emergency_page);
kfree(xlt_emergency_page);
}
module_init(mlx5_ib_init);

Some files were not shown because too many files have changed in this diff Show More