github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-07-31 03:27:03 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge branch 'net-mlx5-hws-complex-matchers-and-rehash-mechanism-fixes'

Browse Source 
Tariq Toukan says:

====================
net/mlx5: HWS, Complex Matchers and rehash mechanism fixes

Motivation:
----------

A matcher can match a certain set of match parameters. However,
the number and size of match params for a single matcher are
limited — all the parameters must fit within a single definer.

A common example of this limitation is IPv6 address matching, where
matching both source and destination IPs requires more bits than a
single definer can support.

SW Steering addresses this limitation by chaining multiple Steering
Table Entries (STEs) within the same matcher, where each STE matches
on a subset of the parameters.

In HW Steering, such chaining is not possible — the matcher's STEs
are managed in a hash table, and a single definer is used to calculate
the hash index for STEs.

Overview:
--------

To address this limitation in HW Steering, we introduce
*Complex Matchers*, which consist of two chained matchers. This allows
matching on twice as many parameters. Complex Matchers are filled with
*Complex Rules* — rules that are split into two parts and inserted into
their respective matchers.

The first half of the Complex Matcher is a regular matcher and points
to the second half, which is an *Isolated Matcher*. An Isolated Matcher
has its own isolated table and is accessible only by traffic coming
from the first half of the Complex Matcher.

This splitting of matchers/rules into multiple parts is transparent to
users. It is hidden behind the BWC HWS API. It becomes visible only
when dumping steering debug information, where the Complex Matcher
appears as two separate matchers: one in the user-created table and
another in its isolated table.

Implementation Details:
----------------------

All user actions are performed on the second part of the rules only.
The first part handles matching and applies two actions: modify header
(set metadata, see details below) and go-to-table (directing traffic
to the isolated table containing the isolated matcher).

Rule updates (updating rule actions) are applied to the second part
of the rule since user-provided actions are not executed in the first
matcher.

We use REG_C_6 metadata register to set and match on unique per-rule
tag (see details below).

Splitting rules into two parts introduces new challenges:

1. Invalid Combinations

   Consider two rules with different matching values:
   - Rule 1: A+B
   - Rule 2: C+D

   Let's split the rules into two parts as follows:

   |-----Complex Matcher-------|
   |                           |
   | 1st matcher   2nd matcher |
   |    |---|        |---|     |
   |    | A |        | B |     |
   |    |---| -----> |---|     |
   |    | C |        | D |     |
   |    |---|        |---|     |
   |                           |
   |---------------------------|

   Splitting these rules results in invalid combinations: A+D and C+B:
   any packet that matched on A will be forwarded to the 2nd matcher,
   where it will try to match on B (which is legal, and it is what the
   user asked for), but it will also try to match on D (which is not
   what the user asked for). To resolve this, we assign unique tags
   to each rule on the first matcher and match on these tags on the
   second matcher:

   |----------|     |---------|
   |     A    |     | B, TagA |
   | action:  |     |         |
   | set TagA |     |         |
   |----------| --> |---------|
   |     C    |     | D, TagB |
   | action:  |     |         |
   | set TagB |     |         |
   |----------|     |---------|

2. Duplicated Entries:

   Consider two rules with overlapping values:
   - Rule 1: A+B
   - Rule 2: A+D

   Let's split the rules into two parts as follows:

    |---|     |---|
    | A |     | B |
    |---| --> |---|
    |   |     | D |
    |---|     |---|

   This leads to the duplicated entries on the first matcher, which HWS
   doesn't allow: subsequent delete of either of the rules will delete
   the only entry in the first matcher, leaving the remaining rule
   broken. To address this, we use a reference count for entries in the
   first matcher and delete STEs only when their refcount reaches zero.

Both challenges are resolved by having a per-matcher data structure
(implemented with rhashtable) that manages refcounts for the first part
of the rules and holds unique tags (managed via IDA) for these rules to
set and to match on the second matcher.

Limitations:
-----------

We utilize metadata register REG_C_6 in this implementation, so its
usage anywhere along the flow that might include the need for Complex
Matcher is prohibited.

The number and size of match parameters remain limited — now
constrained by what can be represented by two definers instead of one.
This architectural limitation arises from the structure of Complex
Matchers. If future requirements demand more parameters, Complex
Matchers can be extended beyond two matchers.

Additionally, there is an implementation limit of 32 match parameters
per matcher (disregarding parameter size). This limit can be lifted
if needed.

Patches:
-------

 - Patches 1-3: small additions/refactoring in preparation for
   Complex Matcher: exposed mlx5hws_table_ft_set_next_ft() in header,
   added definer function to convert field name enum to string,
   expose the polling function mlx5hws_bwc_queue_poll() in a header.
 - Patch 4: in preparation for Complex Matcher, this patch adds
   support for Isolated Matcher.
 - Patch 5: the main patch - Complex Matchers implementation.

[2]

Patch 6: fixing the usecase where rule insertion was failing,
but rehash couldn't be initiated if the number of rules in
the table is below the rehash threshold.

Patch 7: fixing the usecase where many rules in parallel
would require rehash, due to the way the counting of rules
was done.

Patch 8: fixing the case where rules were requiring action
template extension in parallel, leading to unneeded extensions
with the same templates.

Patch 9: refactor and simplify the rehash loop.

Patch 10: dump error completion details, which helps a lot
in trying to understand what went wrong, especially during
rehash.
====================

Link: https://patch.msgid.link/1746992290-568936-1-git-send-email-tariqt@nvidia.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
This commit is contained in:
Jakub Kicinski 2025-05-13 15:26:32 -07:00
commit c20219ee62
13 changed files with 2182 additions and 183 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

338
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.c
View File

@ -46,10 +46,14 @@ static void hws_bwc_unlock_all_queues(struct mlx5hws_context *ctx)
}
}
static void hws_bwc_matcher_init_attr(struct mlx5hws_matcher_attr *attr,
static void hws_bwc_matcher_init_attr(struct mlx5hws_bwc_matcher *bwc_matcher,
u32 priority,
u8 size_log)
u8 size_log,
struct mlx5hws_matcher_attr *attr)
{
struct mlx5hws_bwc_matcher *first_matcher =
bwc_matcher->complex_first_bwc_matcher;
memset(attr, 0, sizeof(*attr));
attr->priority = priority;
@ -61,6 +65,9 @@ static void hws_bwc_matcher_init_attr(struct mlx5hws_matcher_attr *attr,
attr->rule.num_log = size_log;
attr->resizable = true;
attr->max_num_of_at_attach = MLX5HWS_BWC_MATCHER_ATTACH_AT_NUM;
attr->isolated_matcher_end_ft_id =
first_matcher ? first_matcher->matcher->end_ft_id : 0;
}
int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher,
@ -83,9 +90,10 @@ int mlx5hws_bwc_matcher_create_simple(struct mlx5hws_bwc_matcher *bwc_matcher,
for (i = 0; i < bwc_queues; i++)
INIT_LIST_HEAD(&bwc_matcher->rules[i]);
hws_bwc_matcher_init_attr(&attr,
hws_bwc_matcher_init_attr(bwc_matcher,
priority,
MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG);
MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG,
&attr);
bwc_matcher->priority = priority;
bwc_matcher->size_log = MLX5HWS_BWC_MATCHER_INIT_SIZE_LOG;
@ -161,6 +169,7 @@ mlx5hws_bwc_matcher_create(struct mlx5hws_table *table,
return NULL;
atomic_set(&bwc_matcher->num_of_rules, 0);
atomic_set(&bwc_matcher->rehash_required, false);
/* Check if the required match params can be all matched
* in single STE, otherwise complex matcher is needed.
@ -217,16 +226,19 @@ int mlx5hws_bwc_matcher_destroy(struct mlx5hws_bwc_matcher *bwc_matcher)
"BWC matcher destroy: matcher still has %d rules\n",
num_of_rules);
mlx5hws_bwc_matcher_destroy_simple(bwc_matcher);
if (bwc_matcher->complex)
mlx5hws_bwc_matcher_destroy_complex(bwc_matcher);
else
mlx5hws_bwc_matcher_destroy_simple(bwc_matcher);
kfree(bwc_matcher);
return 0;
}
static int hws_bwc_queue_poll(struct mlx5hws_context *ctx,
u16 queue_id,
u32 *pending_rules,
bool drain)
int mlx5hws_bwc_queue_poll(struct mlx5hws_context *ctx,
u16 queue_id,
u32 *pending_rules,
bool drain)
{
unsigned long timeout = jiffies +
secs_to_jiffies(MLX5HWS_BWC_POLLING_TIMEOUT);
@ -329,16 +341,12 @@ static void hws_bwc_rule_list_add(struct mlx5hws_bwc_rule *bwc_rule, u16 idx)
{
struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher;
atomic_inc(&bwc_matcher->num_of_rules);
bwc_rule->bwc_queue_idx = idx;
list_add(&bwc_rule->list_node, &bwc_matcher->rules[idx]);
}
static void hws_bwc_rule_list_remove(struct mlx5hws_bwc_rule *bwc_rule)
{
struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher;
atomic_dec(&bwc_matcher->num_of_rules);
list_del_init(&bwc_rule->list_node);
}
@ -361,7 +369,8 @@ hws_bwc_rule_destroy_hws_sync(struct mlx5hws_bwc_rule *bwc_rule,
if (unlikely(ret))
return ret;
ret = hws_bwc_queue_poll(ctx, rule_attr->queue_id, &expected_completions, true);
ret = mlx5hws_bwc_queue_poll(ctx, rule_attr->queue_id,
&expected_completions, true);
if (unlikely(ret))
return ret;
@ -391,6 +400,7 @@ int mlx5hws_bwc_rule_destroy_simple(struct mlx5hws_bwc_rule *bwc_rule)
mutex_lock(queue_lock);
ret = hws_bwc_rule_destroy_hws_sync(bwc_rule, &attr);
atomic_dec(&bwc_matcher->num_of_rules);
hws_bwc_rule_list_remove(bwc_rule);
mutex_unlock(queue_lock);
@ -400,9 +410,13 @@ int mlx5hws_bwc_rule_destroy_simple(struct mlx5hws_bwc_rule *bwc_rule)
int mlx5hws_bwc_rule_destroy(struct mlx5hws_bwc_rule *bwc_rule)
{
int ret;
bool is_complex = !!bwc_rule->bwc_matcher->complex;
int ret = 0;
ret = mlx5hws_bwc_rule_destroy_simple(bwc_rule);
if (is_complex)
ret = mlx5hws_bwc_rule_destroy_complex(bwc_rule);
else
ret = mlx5hws_bwc_rule_destroy_simple(bwc_rule);
mlx5hws_bwc_rule_free(bwc_rule);
return ret;
@ -442,9 +456,8 @@ hws_bwc_rule_create_sync(struct mlx5hws_bwc_rule *bwc_rule,
if (unlikely(ret))
return ret;
ret = hws_bwc_queue_poll(ctx, rule_attr->queue_id, &expected_completions, true);
return ret;
return mlx5hws_bwc_queue_poll(ctx, rule_attr->queue_id,
&expected_completions, true);
}
static int
@ -465,7 +478,8 @@ hws_bwc_rule_update_sync(struct mlx5hws_bwc_rule *bwc_rule,
if (unlikely(ret))
return ret;
ret = hws_bwc_queue_poll(ctx, rule_attr->queue_id, &expected_completions, true);
ret = mlx5hws_bwc_queue_poll(ctx, rule_attr->queue_id,
&expected_completions, true);
if (unlikely(ret))
mlx5hws_err(ctx, "Failed updating BWC rule (%d)\n", ret);
@ -596,100 +610,79 @@ hws_bwc_matcher_find_at(struct mlx5hws_bwc_matcher *bwc_matcher,
static int hws_bwc_matcher_move_all_simple(struct mlx5hws_bwc_matcher *bwc_matcher)
{
bool move_error = false, poll_error = false, drain_error = false;
struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx;
struct mlx5hws_matcher *matcher = bwc_matcher->matcher;
u16 bwc_queues = mlx5hws_bwc_queues(ctx);
struct mlx5hws_bwc_rule **bwc_rules;
struct mlx5hws_rule_attr rule_attr;
u32 *pending_rules;
int i, j, ret = 0;
bool all_done;
u16 burst_th;
struct mlx5hws_bwc_rule *bwc_rule;
struct mlx5hws_send_engine *queue;
struct list_head *rules_list;
u32 pending_rules;
int i, ret = 0;
mlx5hws_bwc_rule_fill_attr(bwc_matcher, 0, 0, &rule_attr);
pending_rules = kcalloc(bwc_queues, sizeof(*pending_rules), GFP_KERNEL);
if (!pending_rules)
return -ENOMEM;
bwc_rules = kcalloc(bwc_queues, sizeof(*bwc_rules), GFP_KERNEL);
if (!bwc_rules) {
ret = -ENOMEM;
goto free_pending_rules;
}
for (i = 0; i < bwc_queues; i++) {
if (list_empty(&bwc_matcher->rules[i]))
bwc_rules[i] = NULL;
else
bwc_rules[i] = list_first_entry(&bwc_matcher->rules[i],
struct mlx5hws_bwc_rule,
list_node);
}
continue;
do {
all_done = true;
pending_rules = 0;
rule_attr.queue_id = mlx5hws_bwc_get_queue_id(ctx, i);
rules_list = &bwc_matcher->rules[i];
for (i = 0; i < bwc_queues; i++) {
rule_attr.queue_id = mlx5hws_bwc_get_queue_id(ctx, i);
burst_th = hws_bwc_get_burst_th(ctx, rule_attr.queue_id);
for (j = 0; j < burst_th && bwc_rules[i]; j++) {
rule_attr.burst = !!((j + 1) % burst_th);
ret = mlx5hws_matcher_resize_rule_move(bwc_matcher->matcher,
bwc_rules[i]->rule,
&rule_attr);
if (unlikely(ret)) {
mlx5hws_err(ctx,
"Moving BWC rule failed during rehash (%d)\n",
ret);
goto free_bwc_rules;
}
all_done = false;
pending_rules[i]++;
bwc_rules[i] = list_is_last(&bwc_rules[i]->list_node,
&bwc_matcher->rules[i]) ?
NULL : list_next_entry(bwc_rules[i], list_node);
ret = hws_bwc_queue_poll(ctx, rule_attr.queue_id,
&pending_rules[i], false);
if (unlikely(ret)) {
mlx5hws_err(ctx,
"Moving BWC rule failed during rehash (%d)\n",
ret);
goto free_bwc_rules;
}
}
}
} while (!all_done);
/* drain all the bwc queues */
for (i = 0; i < bwc_queues; i++) {
if (pending_rules[i]) {
u16 queue_id = mlx5hws_bwc_get_queue_id(ctx, i);
mlx5hws_send_engine_flush_queue(&ctx->send_queue[queue_id]);
ret = hws_bwc_queue_poll(ctx, queue_id,
&pending_rules[i], true);
if (unlikely(ret)) {
list_for_each_entry(bwc_rule, rules_list, list_node) {
ret = mlx5hws_matcher_resize_rule_move(matcher,
bwc_rule->rule,
&rule_attr);
if (unlikely(ret && !move_error)) {
mlx5hws_err(ctx,
"Moving BWC rule failed during rehash (%d)\n", ret);
goto free_bwc_rules;
"Moving BWC rule: move failed (%d), attempting to move rest of the rules\n",
ret);
move_error = true;
}
pending_rules++;
ret = mlx5hws_bwc_queue_poll(ctx,
rule_attr.queue_id,
&pending_rules,
false);
if (unlikely(ret && !poll_error)) {
mlx5hws_err(ctx,
"Moving BWC rule: poll failed (%d), attempting to move rest of the rules\n",
ret);
poll_error = true;
}
}
if (pending_rules) {
queue = &ctx->send_queue[rule_attr.queue_id];
mlx5hws_send_engine_flush_queue(queue);
ret = mlx5hws_bwc_queue_poll(ctx,
rule_attr.queue_id,
&pending_rules,
true);
if (unlikely(ret && !drain_error)) {
mlx5hws_err(ctx,
"Moving BWC rule: drain failed (%d), attempting to move rest of the rules\n",
ret);
drain_error = true;
}
}
}
free_bwc_rules:
kfree(bwc_rules);
free_pending_rules:
kfree(pending_rules);
if (move_error || poll_error || drain_error)
ret = -EINVAL;
return ret;
}
static int hws_bwc_matcher_move_all(struct mlx5hws_bwc_matcher *bwc_matcher)
{
return hws_bwc_matcher_move_all_simple(bwc_matcher);
if (!bwc_matcher->complex)
return hws_bwc_matcher_move_all_simple(bwc_matcher);
return mlx5hws_bwc_matcher_move_all_complex(bwc_matcher);
}
static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher)
@ -700,9 +693,10 @@ static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher)
struct mlx5hws_matcher *new_matcher;
int ret;
hws_bwc_matcher_init_attr(&matcher_attr,
hws_bwc_matcher_init_attr(bwc_matcher,
bwc_matcher->priority,
bwc_matcher->size_log);
bwc_matcher->size_log,
&matcher_attr);
old_matcher = bwc_matcher->matcher;
new_matcher = mlx5hws_matcher_create(old_matcher->tbl,
@ -722,15 +716,18 @@ static int hws_bwc_matcher_move(struct mlx5hws_bwc_matcher *bwc_matcher)
}
ret = hws_bwc_matcher_move_all(bwc_matcher);
if (ret) {
mlx5hws_err(ctx, "Rehash error: moving rules failed\n");
return -ENOMEM;
}
if (ret)
mlx5hws_err(ctx, "Rehash error: moving rules failed, attempting to remove the old matcher\n");
/* Error during rehash can't be rolled back.
* The best option here is to allow the rehash to complete and remove
* the old matcher - can't leave the matcher in the 'in_resize' state.
*/
bwc_matcher->matcher = new_matcher;
mlx5hws_matcher_destroy(old_matcher);
return 0;
return ret;
}
static int
@ -747,9 +744,9 @@ hws_bwc_matcher_rehash_size(struct mlx5hws_bwc_matcher *bwc_matcher)
/* It is possible that other rule has already performed rehash.
* Need to check again if we really need rehash.
* If the reason for rehash was size, but not any more - skip rehash.
*/
if (!hws_bwc_matcher_rehash_size_needed(bwc_matcher,
if (!atomic_read(&bwc_matcher->rehash_required) &&
!hws_bwc_matcher_rehash_size_needed(bwc_matcher,
atomic_read(&bwc_matcher->num_of_rules)))
return 0;
@ -760,6 +757,8 @@ hws_bwc_matcher_rehash_size(struct mlx5hws_bwc_matcher *bwc_matcher)
* - destroy the old matcher
*/
atomic_set(&bwc_matcher->rehash_required, false);
ret = hws_bwc_matcher_extend_size(bwc_matcher);
if (ret)
return ret;
@ -767,6 +766,53 @@ hws_bwc_matcher_rehash_size(struct mlx5hws_bwc_matcher *bwc_matcher)
return hws_bwc_matcher_move(bwc_matcher);
}
static int hws_bwc_rule_get_at_idx(struct mlx5hws_bwc_rule *bwc_rule,
struct mlx5hws_rule_action rule_actions[],
u16 bwc_queue_idx)
{
struct mlx5hws_bwc_matcher *bwc_matcher = bwc_rule->bwc_matcher;
struct mlx5hws_context *ctx = bwc_matcher->matcher->tbl->ctx;
struct mutex *queue_lock; /* Protect the queue */
int at_idx, ret;
/* check if rehash needed due to missing action template */
at_idx = hws_bwc_matcher_find_at(bwc_matcher, rule_actions);
if (likely(at_idx >= 0))
return at_idx;
/* we need to extend BWC matcher action templates array */
queue_lock = hws_bwc_get_queue_lock(ctx, bwc_queue_idx);
mutex_unlock(queue_lock);
hws_bwc_lock_all_queues(ctx);
/* check again - perhaps other thread already did extend_at */
at_idx = hws_bwc_matcher_find_at(bwc_matcher, rule_actions);
if (at_idx >= 0)
goto out;
ret = hws_bwc_matcher_extend_at(bwc_matcher, rule_actions);
if (unlikely(ret)) {
mlx5hws_err(ctx, "BWC rule: failed extending AT (%d)", ret);
at_idx = -EINVAL;
goto out;
}
/* action templates array was extended, we need the last idx */
at_idx = bwc_matcher->num_of_at - 1;
ret = mlx5hws_matcher_attach_at(bwc_matcher->matcher,
bwc_matcher->at[at_idx]);
if (unlikely(ret)) {
mlx5hws_err(ctx, "BWC rule: failed attaching new AT (%d)", ret);
at_idx = -EINVAL;
goto out;
}
out:
hws_bwc_unlock_all_queues(ctx);
mutex_lock(queue_lock);
return at_idx;
}
int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,
u32 *match_param,
struct mlx5hws_rule_action rule_actions[],
@ -787,35 +833,16 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,
mutex_lock(queue_lock);
/* check if rehash needed due to missing action template */
at_idx = hws_bwc_matcher_find_at(bwc_matcher, rule_actions);
at_idx = hws_bwc_rule_get_at_idx(bwc_rule, rule_actions, bwc_queue_idx);
if (unlikely(at_idx < 0)) {
/* we need to extend BWC matcher action templates array */
mutex_unlock(queue_lock);
hws_bwc_lock_all_queues(ctx);
ret = hws_bwc_matcher_extend_at(bwc_matcher, rule_actions);
if (unlikely(ret)) {
hws_bwc_unlock_all_queues(ctx);
return ret;
}
/* action templates array was extended, we need the last idx */
at_idx = bwc_matcher->num_of_at - 1;
ret = mlx5hws_matcher_attach_at(bwc_matcher->matcher,
bwc_matcher->at[at_idx]);
if (unlikely(ret)) {
hws_bwc_unlock_all_queues(ctx);
return ret;
}
hws_bwc_unlock_all_queues(ctx);
mutex_lock(queue_lock);
mlx5hws_err(ctx, "BWC rule create: failed getting AT (%d)",
ret);
return -EINVAL;
}
/* check if number of rules require rehash */
num_of_rules = atomic_read(&bwc_matcher->num_of_rules);
num_of_rules = atomic_inc_return(&bwc_matcher->num_of_rules);
if (unlikely(hws_bwc_matcher_rehash_size_needed(bwc_matcher, num_of_rules))) {
mutex_unlock(queue_lock);
@ -829,6 +856,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,
bwc_matcher->size_log - MLX5HWS_BWC_MATCHER_SIZE_LOG_STEP,
bwc_matcher->size_log,
ret);
atomic_dec(&bwc_matcher->num_of_rules);
return ret;
}
@ -853,6 +881,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,
* Try rehash by size and insert rule again - last chance.
*/
atomic_set(&bwc_matcher->rehash_required, true);
mutex_unlock(queue_lock);
hws_bwc_lock_all_queues(ctx);
@ -861,6 +890,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,
if (ret) {
mlx5hws_err(ctx, "BWC rule insertion: rehash failed (%d)\n", ret);
atomic_dec(&bwc_matcher->num_of_rules);
return ret;
}
@ -876,6 +906,7 @@ int mlx5hws_bwc_rule_create_simple(struct mlx5hws_bwc_rule *bwc_rule,
if (unlikely(ret)) {
mutex_unlock(queue_lock);
mlx5hws_err(ctx, "BWC rule insertion failed (%d)\n", ret);
atomic_dec(&bwc_matcher->num_of_rules);
return ret;
}
@ -907,11 +938,18 @@ mlx5hws_bwc_rule_create(struct mlx5hws_bwc_matcher *bwc_matcher,
bwc_queue_idx = hws_bwc_gen_queue_idx(ctx);
ret = mlx5hws_bwc_rule_create_simple(bwc_rule,
params->match_buf,
rule_actions,
flow_source,
bwc_queue_idx);
if (bwc_matcher->complex)
ret = mlx5hws_bwc_rule_create_complex(bwc_rule,
params,
flow_source,
rule_actions,
bwc_queue_idx);
else
ret = mlx5hws_bwc_rule_create_simple(bwc_rule,
params->match_buf,
rule_actions,
flow_source,
bwc_queue_idx);
if (unlikely(ret)) {
mlx5hws_bwc_rule_free(bwc_rule);
return NULL;
@ -938,36 +976,11 @@ hws_bwc_rule_action_update(struct mlx5hws_bwc_rule *bwc_rule,
mutex_lock(queue_lock);
/* check if rehash needed due to missing action template */
at_idx = hws_bwc_matcher_find_at(bwc_matcher, rule_actions);
at_idx = hws_bwc_rule_get_at_idx(bwc_rule, rule_actions, idx);
if (unlikely(at_idx < 0)) {
/* we need to extend BWC matcher action templates array */
mutex_unlock(queue_lock);
hws_bwc_lock_all_queues(ctx);
/* check again - perhaps other thread already did extend_at */
at_idx = hws_bwc_matcher_find_at(bwc_matcher, rule_actions);
if (likely(at_idx < 0)) {
ret = hws_bwc_matcher_extend_at(bwc_matcher, rule_actions);
if (unlikely(ret)) {
hws_bwc_unlock_all_queues(ctx);
mlx5hws_err(ctx, "BWC rule update: failed extending AT (%d)", ret);
return -EINVAL;
}
/* action templates array was extended, we need the last idx */
at_idx = bwc_matcher->num_of_at - 1;
ret = mlx5hws_matcher_attach_at(bwc_matcher->matcher,
bwc_matcher->at[at_idx]);
if (unlikely(ret)) {
hws_bwc_unlock_all_queues(ctx);
return ret;
}
}
hws_bwc_unlock_all_queues(ctx);
mutex_lock(queue_lock);
mlx5hws_err(ctx, "BWC rule update: failed getting AT\n");
return -EINVAL;
}
ret = hws_bwc_rule_update_sync(bwc_rule,
@ -993,5 +1006,10 @@ int mlx5hws_bwc_rule_action_update(struct mlx5hws_bwc_rule *bwc_rule,
return -EINVAL;
}
return hws_bwc_rule_action_update(bwc_rule, rule_actions);
/* For complex rule, the update should happen on the second matcher */
if (bwc_rule->isolated_bwc_rule)
return hws_bwc_rule_action_update(bwc_rule->isolated_bwc_rule,
rule_actions);
else
return hws_bwc_rule_action_update(bwc_rule, rule_actions);
}

11
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.h
View File

@ -18,21 +18,27 @@
#define MLX5HWS_BWC_POLLING_TIMEOUT 60
struct mlx5hws_bwc_matcher_complex_data;
struct mlx5hws_bwc_matcher {
struct mlx5hws_matcher *matcher;
struct mlx5hws_match_template *mt;
struct mlx5hws_action_template **at;
struct mlx5hws_bwc_matcher_complex_data *complex;
struct mlx5hws_bwc_matcher *complex_first_bwc_matcher;
u8 num_of_at;
u8 size_of_at_array;
u8 size_log;
u32 priority;
atomic_t num_of_rules;
atomic_t rehash_required;
struct list_head *rules;
};
struct mlx5hws_bwc_rule {
struct mlx5hws_bwc_matcher *bwc_matcher;
struct mlx5hws_rule *rule;
struct mlx5hws_bwc_rule *isolated_bwc_rule;
struct mlx5hws_bwc_complex_rule_hash_node *complex_hash_node;
u16 bwc_queue_idx;
struct list_head list_node;
};
@ -64,6 +70,11 @@ void mlx5hws_bwc_rule_fill_attr(struct mlx5hws_bwc_matcher *bwc_matcher,
u32 flow_source,
struct mlx5hws_rule_attr *rule_attr);
int mlx5hws_bwc_queue_poll(struct mlx5hws_context *ctx,
u16 queue_id,
u32 *pending_rules,
bool drain);
static inline u16 mlx5hws_bwc_queues(struct mlx5hws_context *ctx)
{
/* Besides the control queue, half of the queues are

1348
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc_complex.c
View File

File diff suppressed because it is too large Load Diff

21
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc_complex.h
View File

@ -4,6 +4,27 @@
#ifndef HWS_BWC_COMPLEX_H_
#define HWS_BWC_COMPLEX_H_
struct mlx5hws_bwc_complex_rule_hash_node {
u32 match_buf[MLX5_ST_SZ_DW_MATCH_PARAM];
u32 tag;
refcount_t refcount;
bool rtc_valid;
u32 rtc_0;
u32 rtc_1;
struct rhash_head hash_node;
};
struct mlx5hws_bwc_matcher_complex_data {
struct mlx5hws_table *isolated_tbl;
struct mlx5hws_bwc_matcher *isolated_bwc_matcher;
struct mlx5hws_action *action_metadata;
struct mlx5hws_action *action_go_to_tbl;
struct mlx5hws_action *action_last;
struct rhashtable refcount_hash;
struct mutex hash_lock; /* Protect the refcount rhashtable */
struct ida metadata_ida;
};
bool mlx5hws_bwc_match_params_is_complex(struct mlx5hws_context *ctx,
u8 match_criteria_enable,
struct mlx5hws_match_parameters *mask);

212
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/definer.c
View File

@ -158,6 +158,218 @@ struct mlx5hws_definer_conv_data {
u32 match_flags;
};
#define HWS_DEFINER_ENTRY(name)[MLX5HWS_DEFINER_FNAME_##name] = #name
static const char * const hws_definer_fname_to_str[] = {
HWS_DEFINER_ENTRY(ETH_SMAC_47_16_O),
HWS_DEFINER_ENTRY(ETH_SMAC_47_16_I),
HWS_DEFINER_ENTRY(ETH_SMAC_15_0_O),
HWS_DEFINER_ENTRY(ETH_SMAC_15_0_I),
HWS_DEFINER_ENTRY(ETH_DMAC_47_16_O),
HWS_DEFINER_ENTRY(ETH_DMAC_47_16_I),
HWS_DEFINER_ENTRY(ETH_DMAC_15_0_O),
HWS_DEFINER_ENTRY(ETH_DMAC_15_0_I),
HWS_DEFINER_ENTRY(ETH_TYPE_O),
HWS_DEFINER_ENTRY(ETH_TYPE_I),
HWS_DEFINER_ENTRY(ETH_L3_TYPE_O),
HWS_DEFINER_ENTRY(ETH_L3_TYPE_I),
HWS_DEFINER_ENTRY(VLAN_TYPE_O),
HWS_DEFINER_ENTRY(VLAN_TYPE_I),
HWS_DEFINER_ENTRY(VLAN_FIRST_PRIO_O),
HWS_DEFINER_ENTRY(VLAN_FIRST_PRIO_I),
HWS_DEFINER_ENTRY(VLAN_CFI_O),
HWS_DEFINER_ENTRY(VLAN_CFI_I),
HWS_DEFINER_ENTRY(VLAN_ID_O),
HWS_DEFINER_ENTRY(VLAN_ID_I),
HWS_DEFINER_ENTRY(VLAN_SECOND_TYPE_O),
HWS_DEFINER_ENTRY(VLAN_SECOND_TYPE_I),
HWS_DEFINER_ENTRY(VLAN_SECOND_PRIO_O),
HWS_DEFINER_ENTRY(VLAN_SECOND_PRIO_I),
HWS_DEFINER_ENTRY(VLAN_SECOND_CFI_O),
HWS_DEFINER_ENTRY(VLAN_SECOND_CFI_I),
HWS_DEFINER_ENTRY(VLAN_SECOND_ID_O),
HWS_DEFINER_ENTRY(VLAN_SECOND_ID_I),
HWS_DEFINER_ENTRY(IPV4_IHL_O),
HWS_DEFINER_ENTRY(IPV4_IHL_I),
HWS_DEFINER_ENTRY(IP_DSCP_O),
HWS_DEFINER_ENTRY(IP_DSCP_I),
HWS_DEFINER_ENTRY(IP_ECN_O),
HWS_DEFINER_ENTRY(IP_ECN_I),
HWS_DEFINER_ENTRY(IP_TTL_O),
HWS_DEFINER_ENTRY(IP_TTL_I),
HWS_DEFINER_ENTRY(IPV4_DST_O),
HWS_DEFINER_ENTRY(IPV4_DST_I),
HWS_DEFINER_ENTRY(IPV4_SRC_O),
HWS_DEFINER_ENTRY(IPV4_SRC_I),
HWS_DEFINER_ENTRY(IP_VERSION_O),
HWS_DEFINER_ENTRY(IP_VERSION_I),
HWS_DEFINER_ENTRY(IP_FRAG_O),
HWS_DEFINER_ENTRY(IP_FRAG_I),
HWS_DEFINER_ENTRY(IP_LEN_O),
HWS_DEFINER_ENTRY(IP_LEN_I),
HWS_DEFINER_ENTRY(IP_TOS_O),
HWS_DEFINER_ENTRY(IP_TOS_I),
HWS_DEFINER_ENTRY(IPV6_FLOW_LABEL_O),
HWS_DEFINER_ENTRY(IPV6_FLOW_LABEL_I),
HWS_DEFINER_ENTRY(IPV6_DST_127_96_O),
HWS_DEFINER_ENTRY(IPV6_DST_95_64_O),
HWS_DEFINER_ENTRY(IPV6_DST_63_32_O),
HWS_DEFINER_ENTRY(IPV6_DST_31_0_O),
HWS_DEFINER_ENTRY(IPV6_DST_127_96_I),
HWS_DEFINER_ENTRY(IPV6_DST_95_64_I),
HWS_DEFINER_ENTRY(IPV6_DST_63_32_I),
HWS_DEFINER_ENTRY(IPV6_DST_31_0_I),
HWS_DEFINER_ENTRY(IPV6_SRC_127_96_O),
HWS_DEFINER_ENTRY(IPV6_SRC_95_64_O),
HWS_DEFINER_ENTRY(IPV6_SRC_63_32_O),
HWS_DEFINER_ENTRY(IPV6_SRC_31_0_O),
HWS_DEFINER_ENTRY(IPV6_SRC_127_96_I),
HWS_DEFINER_ENTRY(IPV6_SRC_95_64_I),
HWS_DEFINER_ENTRY(IPV6_SRC_63_32_I),
HWS_DEFINER_ENTRY(IPV6_SRC_31_0_I),
HWS_DEFINER_ENTRY(IP_PROTOCOL_O),
HWS_DEFINER_ENTRY(IP_PROTOCOL_I),
HWS_DEFINER_ENTRY(L4_SPORT_O),
HWS_DEFINER_ENTRY(L4_SPORT_I),
HWS_DEFINER_ENTRY(L4_DPORT_O),
HWS_DEFINER_ENTRY(L4_DPORT_I),
HWS_DEFINER_ENTRY(TCP_FLAGS_I),
HWS_DEFINER_ENTRY(TCP_FLAGS_O),
HWS_DEFINER_ENTRY(TCP_SEQ_NUM),
HWS_DEFINER_ENTRY(TCP_ACK_NUM),
HWS_DEFINER_ENTRY(GTP_TEID),
HWS_DEFINER_ENTRY(GTP_MSG_TYPE),
HWS_DEFINER_ENTRY(GTP_EXT_FLAG),
HWS_DEFINER_ENTRY(GTP_NEXT_EXT_HDR),
HWS_DEFINER_ENTRY(GTP_EXT_HDR_PDU),
HWS_DEFINER_ENTRY(GTP_EXT_HDR_QFI),
HWS_DEFINER_ENTRY(GTPU_DW0),
HWS_DEFINER_ENTRY(GTPU_FIRST_EXT_DW0),
HWS_DEFINER_ENTRY(GTPU_DW2),
HWS_DEFINER_ENTRY(FLEX_PARSER_0),
HWS_DEFINER_ENTRY(FLEX_PARSER_1),
HWS_DEFINER_ENTRY(FLEX_PARSER_2),
HWS_DEFINER_ENTRY(FLEX_PARSER_3),
HWS_DEFINER_ENTRY(FLEX_PARSER_4),
HWS_DEFINER_ENTRY(FLEX_PARSER_5),
HWS_DEFINER_ENTRY(FLEX_PARSER_6),
HWS_DEFINER_ENTRY(FLEX_PARSER_7),
HWS_DEFINER_ENTRY(VPORT_REG_C_0),
HWS_DEFINER_ENTRY(VXLAN_FLAGS),
HWS_DEFINER_ENTRY(VXLAN_VNI),
HWS_DEFINER_ENTRY(VXLAN_GPE_FLAGS),
HWS_DEFINER_ENTRY(VXLAN_GPE_RSVD0),
HWS_DEFINER_ENTRY(VXLAN_GPE_PROTO),
HWS_DEFINER_ENTRY(VXLAN_GPE_VNI),
HWS_DEFINER_ENTRY(VXLAN_GPE_RSVD1),
HWS_DEFINER_ENTRY(GENEVE_OPT_LEN),
HWS_DEFINER_ENTRY(GENEVE_OAM),
HWS_DEFINER_ENTRY(GENEVE_PROTO),
HWS_DEFINER_ENTRY(GENEVE_VNI),
HWS_DEFINER_ENTRY(SOURCE_QP),
HWS_DEFINER_ENTRY(SOURCE_GVMI),
HWS_DEFINER_ENTRY(REG_0),
HWS_DEFINER_ENTRY(REG_1),
HWS_DEFINER_ENTRY(REG_2),
HWS_DEFINER_ENTRY(REG_3),
HWS_DEFINER_ENTRY(REG_4),
HWS_DEFINER_ENTRY(REG_5),
HWS_DEFINER_ENTRY(REG_6),
HWS_DEFINER_ENTRY(REG_7),
HWS_DEFINER_ENTRY(REG_8),
HWS_DEFINER_ENTRY(REG_9),
HWS_DEFINER_ENTRY(REG_10),
HWS_DEFINER_ENTRY(REG_11),
HWS_DEFINER_ENTRY(REG_A),
HWS_DEFINER_ENTRY(REG_B),
HWS_DEFINER_ENTRY(GRE_KEY_PRESENT),
HWS_DEFINER_ENTRY(GRE_C),
HWS_DEFINER_ENTRY(GRE_K),
HWS_DEFINER_ENTRY(GRE_S),
HWS_DEFINER_ENTRY(GRE_PROTOCOL),
HWS_DEFINER_ENTRY(GRE_OPT_KEY),
HWS_DEFINER_ENTRY(GRE_OPT_SEQ),
HWS_DEFINER_ENTRY(GRE_OPT_CHECKSUM),
HWS_DEFINER_ENTRY(INTEGRITY_O),
HWS_DEFINER_ENTRY(INTEGRITY_I),
HWS_DEFINER_ENTRY(ICMP_DW1),
HWS_DEFINER_ENTRY(ICMP_DW2),
HWS_DEFINER_ENTRY(ICMP_DW3),
HWS_DEFINER_ENTRY(IPSEC_SPI),
HWS_DEFINER_ENTRY(IPSEC_SEQUENCE_NUMBER),
HWS_DEFINER_ENTRY(IPSEC_SYNDROME),
HWS_DEFINER_ENTRY(MPLS0_O),
HWS_DEFINER_ENTRY(MPLS1_O),
HWS_DEFINER_ENTRY(MPLS2_O),
HWS_DEFINER_ENTRY(MPLS3_O),
HWS_DEFINER_ENTRY(MPLS4_O),
HWS_DEFINER_ENTRY(MPLS0_I),
HWS_DEFINER_ENTRY(MPLS1_I),
HWS_DEFINER_ENTRY(MPLS2_I),
HWS_DEFINER_ENTRY(MPLS3_I),
HWS_DEFINER_ENTRY(MPLS4_I),
HWS_DEFINER_ENTRY(FLEX_PARSER0_OK),
HWS_DEFINER_ENTRY(FLEX_PARSER1_OK),
HWS_DEFINER_ENTRY(FLEX_PARSER2_OK),
HWS_DEFINER_ENTRY(FLEX_PARSER3_OK),
HWS_DEFINER_ENTRY(FLEX_PARSER4_OK),
HWS_DEFINER_ENTRY(FLEX_PARSER5_OK),
HWS_DEFINER_ENTRY(FLEX_PARSER6_OK),
HWS_DEFINER_ENTRY(FLEX_PARSER7_OK),
HWS_DEFINER_ENTRY(OKS2_MPLS0_O),
HWS_DEFINER_ENTRY(OKS2_MPLS1_O),
HWS_DEFINER_ENTRY(OKS2_MPLS2_O),
HWS_DEFINER_ENTRY(OKS2_MPLS3_O),
HWS_DEFINER_ENTRY(OKS2_MPLS4_O),
HWS_DEFINER_ENTRY(OKS2_MPLS0_I),
HWS_DEFINER_ENTRY(OKS2_MPLS1_I),
HWS_DEFINER_ENTRY(OKS2_MPLS2_I),
HWS_DEFINER_ENTRY(OKS2_MPLS3_I),
HWS_DEFINER_ENTRY(OKS2_MPLS4_I),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_0),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_1),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_2),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_3),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_4),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_5),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_6),
HWS_DEFINER_ENTRY(GENEVE_OPT_OK_7),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_0),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_1),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_2),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_3),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_4),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_5),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_6),
HWS_DEFINER_ENTRY(GENEVE_OPT_DW_7),
HWS_DEFINER_ENTRY(IB_L4_OPCODE),
HWS_DEFINER_ENTRY(IB_L4_QPN),
HWS_DEFINER_ENTRY(IB_L4_A),
HWS_DEFINER_ENTRY(RANDOM_NUM),
HWS_DEFINER_ENTRY(PTYPE_L2_O),
HWS_DEFINER_ENTRY(PTYPE_L2_I),
HWS_DEFINER_ENTRY(PTYPE_L3_O),
HWS_DEFINER_ENTRY(PTYPE_L3_I),
HWS_DEFINER_ENTRY(PTYPE_L4_O),
HWS_DEFINER_ENTRY(PTYPE_L4_I),
HWS_DEFINER_ENTRY(PTYPE_L4_EXT_O),
HWS_DEFINER_ENTRY(PTYPE_L4_EXT_I),
HWS_DEFINER_ENTRY(PTYPE_FRAG_O),
HWS_DEFINER_ENTRY(PTYPE_FRAG_I),
HWS_DEFINER_ENTRY(TNL_HDR_0),
HWS_DEFINER_ENTRY(TNL_HDR_1),
HWS_DEFINER_ENTRY(TNL_HDR_2),
HWS_DEFINER_ENTRY(TNL_HDR_3),
[MLX5HWS_DEFINER_FNAME_MAX] = "DEFINER_FNAME_UNKNOWN",
};
const char *mlx5hws_definer_fname_to_str(enum mlx5hws_definer_fname fname)
{
if (fname > MLX5HWS_DEFINER_FNAME_MAX)
fname = MLX5HWS_DEFINER_FNAME_MAX;
return hws_definer_fname_to_str[fname];
}
static void
hws_definer_ones_set(struct mlx5hws_definer_fc *fc,
void *match_param,

2
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/definer.h
View File

@ -831,4 +831,6 @@ mlx5hws_definer_conv_match_params_to_compressed_fc(struct mlx5hws_context *ctx,
u32 *match_param,
int *fc_sz);
const char *mlx5hws_definer_fname_to_str(enum mlx5hws_definer_fname fname);
#endif /* HWS_DEFINER_H_ */

278
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.c
View File

@ -23,19 +23,199 @@ static void hws_matcher_destroy_end_ft(struct mlx5hws_matcher *matcher)
mlx5hws_table_destroy_default_ft(matcher->tbl, matcher->end_ft_id);
}
int mlx5hws_matcher_update_end_ft_isolated(struct mlx5hws_table *tbl,
u32 miss_ft_id)
{
struct mlx5hws_matcher *tmp_matcher;
if (list_empty(&tbl->matchers_list))
return -EINVAL;
/* Update isolated_matcher_end_ft_id attribute for all
* the matchers in isolated table.
*/
list_for_each_entry(tmp_matcher, &tbl->matchers_list, list_node)
tmp_matcher->attr.isolated_matcher_end_ft_id = miss_ft_id;
tmp_matcher = list_last_entry(&tbl->matchers_list,
struct mlx5hws_matcher,
list_node);
return mlx5hws_table_ft_set_next_ft(tbl->ctx,
tmp_matcher->end_ft_id,
tbl->fw_ft_type,
miss_ft_id);
}
static int hws_matcher_connect_end_ft_isolated(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_table *tbl = matcher->tbl;
u32 end_ft_id;
int ret;
/* Reset end_ft next RTCs */
ret = mlx5hws_table_ft_set_next_rtc(tbl->ctx,
matcher->end_ft_id,
matcher->tbl->fw_ft_type,
0, 0);
if (ret) {
mlx5hws_err(tbl->ctx, "Isolated matcher: failed to reset FT's next RTCs\n");
return ret;
}
/* Connect isolated matcher's end_ft to the complex matcher's end FT */
end_ft_id = matcher->attr.isolated_matcher_end_ft_id;
ret = mlx5hws_table_ft_set_next_ft(tbl->ctx,
matcher->end_ft_id,
matcher->tbl->fw_ft_type,
end_ft_id);
if (ret) {
mlx5hws_err(tbl->ctx, "Isolated matcher: failed to set FT's miss_ft_id\n");
return ret;
}
return 0;
}
static int hws_matcher_create_end_ft_isolated(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_table *tbl = matcher->tbl;
int ret;
ret = mlx5hws_table_create_default_ft(tbl->ctx->mdev,
tbl,
&matcher->end_ft_id);
if (ret) {
mlx5hws_err(tbl->ctx, "Isolated matcher: failed to create end flow table\n");
return ret;
}
ret = hws_matcher_connect_end_ft_isolated(matcher);
if (ret) {
mlx5hws_err(tbl->ctx, "Isolated matcher: failed to connect end FT\n");
goto destroy_default_ft;
}
return 0;
destroy_default_ft:
mlx5hws_table_destroy_default_ft(tbl, matcher->end_ft_id);
return ret;
}
static int hws_matcher_create_end_ft(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_table *tbl = matcher->tbl;
int ret;
ret = mlx5hws_table_create_default_ft(tbl->ctx->mdev, tbl, &matcher->end_ft_id);
if (mlx5hws_matcher_is_isolated(matcher))
ret = hws_matcher_create_end_ft_isolated(matcher);
else
ret = mlx5hws_table_create_default_ft(tbl->ctx->mdev, tbl,
&matcher->end_ft_id);
if (ret) {
mlx5hws_err(tbl->ctx, "Failed to create matcher end flow table\n");
return ret;
}
return 0;
}
static int hws_matcher_connect_isolated_first(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_table *tbl = matcher->tbl;
struct mlx5hws_context *ctx = tbl->ctx;
int ret;
/* Isolated matcher's end_ft is already pointing to the end_ft
* of the complex matcher - it was set at creation of end_ft,
* so no need to connect it.
* We still need to connect the isolated table's start FT to
* this matcher's RTC.
*/
ret = mlx5hws_table_ft_set_next_rtc(ctx,
tbl->ft_id,
tbl->fw_ft_type,
matcher->match_ste.rtc_0_id,
matcher->match_ste.rtc_1_id);
if (ret) {
mlx5hws_err(ctx, "Isolated matcher: failed to connect start FT to match RTC\n");
return ret;
}
/* Reset table's FT default miss (drop refcount) */
ret = mlx5hws_table_ft_set_default_next_ft(tbl, tbl->ft_id);
if (ret) {
mlx5hws_err(ctx, "Isolated matcher: failed to reset table ft default miss\n");
return ret;
}
list_add(&matcher->list_node, &tbl->matchers_list);
return ret;
}
static int hws_matcher_connect_isolated_last(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_table *tbl = matcher->tbl;
struct mlx5hws_context *ctx = tbl->ctx;
struct mlx5hws_matcher *last;
int ret;
last = list_last_entry(&tbl->matchers_list,
struct mlx5hws_matcher,
list_node);
/* New matcher's end_ft is already pointing to the end_ft of
* the complex matcher.
* Connect previous matcher's end_ft to this new matcher RTC.
*/
ret = mlx5hws_table_ft_set_next_rtc(ctx,
last->end_ft_id,
tbl->fw_ft_type,
matcher->match_ste.rtc_0_id,
matcher->match_ste.rtc_1_id);
if (ret) {
mlx5hws_err(ctx,
"Isolated matcher: failed to connect matcher end_ft to new match RTC\n");
return ret;
}
/* Reset prev matcher FT default miss (drop refcount) */
ret = mlx5hws_table_ft_set_default_next_ft(tbl, last->end_ft_id);
if (ret) {
mlx5hws_err(ctx, "Isolated matcher: failed to reset matcher ft default miss\n");
return ret;
}
/* Insert after the last matcher */
list_add(&matcher->list_node, &last->list_node);
return 0;
}
static int hws_matcher_connect_isolated(struct mlx5hws_matcher *matcher)
{
/* Isolated matcher is expected to be the only one in its table.
* However, it can have a collision matcher, and it can go through
* rehash process, in which case we will temporary have both old and
* new matchers in the isolated table.
* Check if this is the first matcher in the isolated table.
*/
if (list_empty(&matcher->tbl->matchers_list))
return hws_matcher_connect_isolated_first(matcher);
/* If this wasn't the first matcher, then we have 3 possible cases:
* - this is a collision matcher for the first matcher
* - this is a new rehash dest matcher
* - this is a collision matcher for the new rehash dest matcher
* The logic to add new matcher is the same for all these cases.
*/
return hws_matcher_connect_isolated_last(matcher);
}
static int hws_matcher_connect(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_table *tbl = matcher->tbl;
@ -45,6 +225,9 @@ static int hws_matcher_connect(struct mlx5hws_matcher *matcher)
struct mlx5hws_matcher *tmp_matcher;
int ret;
if (mlx5hws_matcher_is_isolated(matcher))
return hws_matcher_connect_isolated(matcher);
/* Find location in matcher list */
if (list_empty(&tbl->matchers_list)) {
list_add(&matcher->list_node, &tbl->matchers_list);
@ -121,6 +304,92 @@ static int hws_matcher_connect(struct mlx5hws_matcher *matcher)
return ret;
}
static int hws_matcher_disconnect_isolated(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_matcher *first, *last, *prev, *next;
struct mlx5hws_table *tbl = matcher->tbl;
struct mlx5hws_context *ctx = tbl->ctx;
u32 end_ft_id;
int ret;
first = list_first_entry(&tbl->matchers_list,
struct mlx5hws_matcher,
list_node);
last = list_last_entry(&tbl->matchers_list,
struct mlx5hws_matcher,
list_node);
prev = list_prev_entry(matcher, list_node);
next = list_next_entry(matcher, list_node);
list_del_init(&matcher->list_node);
if (first == last) {
/* This was the only matcher in the list.
* Reset isolated table FT next RTCs and connect it
* to the whole complex matcher end FT instead.
*/
ret = mlx5hws_table_ft_set_next_rtc(ctx,
tbl->ft_id,
tbl->fw_ft_type,
0, 0);
if (ret) {
mlx5hws_err(tbl->ctx, "Isolated matcher: failed to reset FT's next RTCs\n");
return ret;
}
end_ft_id = matcher->attr.isolated_matcher_end_ft_id;
ret = mlx5hws_table_ft_set_next_ft(tbl->ctx,
tbl->ft_id,
tbl->fw_ft_type,
end_ft_id);
if (ret) {
mlx5hws_err(tbl->ctx, "Isolated matcher: failed to set FT's miss_ft_id\n");
return ret;
}
return 0;
}
/* At this point we know that there are more matchers in the list */
if (matcher == first) {
/* We've disconnected the first matcher.
* Now update isolated table default FT.
*/
if (!next)
return -EINVAL;
return mlx5hws_table_ft_set_next_rtc(ctx,
tbl->ft_id,
tbl->fw_ft_type,
next->match_ste.rtc_0_id,
next->match_ste.rtc_1_id);
}
if (matcher == last) {
/* If we've disconnected the last matcher - update prev
* matcher's end_ft to point to the complex matcher end_ft.
*/
if (!prev)
return -EINVAL;
return hws_matcher_connect_end_ft_isolated(prev);
}
/* This wasn't the first or the last matcher, which means that it has
* both prev and next matchers. Note that this only happens if we're
* disconnecting collision matcher of the old matcher during rehash.
*/
if (!prev || !next ||
!(matcher->flags & MLX5HWS_MATCHER_FLAGS_COLLISION))
return -EINVAL;
/* Update prev end FT to point to next match RTC */
return mlx5hws_table_ft_set_next_rtc(ctx,
prev->end_ft_id,
tbl->fw_ft_type,
next->match_ste.rtc_0_id,
next->match_ste.rtc_1_id);
}
static int hws_matcher_disconnect(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_matcher *next = NULL, *prev = NULL;
@ -128,6 +397,9 @@ static int hws_matcher_disconnect(struct mlx5hws_matcher *matcher)
u32 prev_ft_id = tbl->ft_id;
int ret;
if (mlx5hws_matcher_is_isolated(matcher))
return hws_matcher_disconnect_isolated(matcher);
if (!list_is_first(&matcher->list_node, &tbl->matchers_list)) {
prev = list_prev_entry(matcher, list_node);
prev_ft_id = prev->end_ft_id;
@ -531,6 +803,8 @@ hws_matcher_process_attr(struct mlx5hws_cmd_query_caps *caps,
attr->table.sz_col_log = hws_matcher_rules_to_tbl_depth(attr->rule.num_log);
matcher->flags |= attr->resizable ? MLX5HWS_MATCHER_FLAGS_RESIZABLE : 0;
matcher->flags |= attr->isolated_matcher_end_ft_id ?
MLX5HWS_MATCHER_FLAGS_ISOLATED : 0;
return hws_matcher_check_attr_sz(caps, matcher);
}
@ -617,6 +891,8 @@ hws_matcher_create_col_matcher(struct mlx5hws_matcher *matcher)
col_matcher->attr.table.sz_row_log -= MLX5HWS_MATCHER_ASSURED_ROW_RATIO;
col_matcher->attr.max_num_of_at_attach = matcher->attr.max_num_of_at_attach;
col_matcher->attr.isolated_matcher_end_ft_id =
matcher->attr.isolated_matcher_end_ft_id;
ret = hws_matcher_process_attr(ctx->caps, col_matcher);
if (ret)

9
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.h
View File

@ -34,6 +34,7 @@ enum mlx5hws_matcher_offset {
enum mlx5hws_matcher_flags {
MLX5HWS_MATCHER_FLAGS_COLLISION = 1 << 2,
MLX5HWS_MATCHER_FLAGS_RESIZABLE = 1 << 3,
MLX5HWS_MATCHER_FLAGS_ISOLATED = 1 << 4,
};
struct mlx5hws_match_template {
@ -96,9 +97,17 @@ static inline bool mlx5hws_matcher_is_in_resize(struct mlx5hws_matcher *matcher)
return !!matcher->resize_dst;
}
static inline bool mlx5hws_matcher_is_isolated(struct mlx5hws_matcher *matcher)
{
return !!(matcher->flags & MLX5HWS_MATCHER_FLAGS_ISOLATED);
}
static inline bool mlx5hws_matcher_is_insert_by_idx(struct mlx5hws_matcher *matcher)
{
return matcher->attr.insert_mode == MLX5HWS_MATCHER_INSERT_BY_INDEX;
}
int mlx5hws_matcher_update_end_ft_isolated(struct mlx5hws_table *tbl,
u32 miss_ft_id);
#endif /* HWS_MATCHER_H_ */

2
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws.h
View File

@ -119,6 +119,8 @@ struct mlx5hws_matcher_attr {
};
/* Optional AT attach configuration - Max number of additional AT */
u8 max_num_of_at_attach;
/* Optional end FT (miss FT ID) for match RTC (for isolated matcher) */
u32 isolated_matcher_end_ft_id;
};
struct mlx5hws_rule_attr {

122
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.c
View File

@ -344,18 +344,133 @@ hws_send_engine_update_rule_resize(struct mlx5hws_send_engine *queue,
}
}
static void hws_send_engine_dump_error_cqe(struct mlx5hws_send_engine *queue,
struct mlx5hws_send_ring_priv *priv,
struct mlx5_cqe64 *cqe)
{
u8 wqe_opcode = cqe ? be32_to_cpu(cqe->sop_drop_qpn) >> 24 : 0;
struct mlx5hws_context *ctx = priv->rule->matcher->tbl->ctx;
u32 opcode = cqe ? get_cqe_opcode(cqe) : 0;
struct mlx5hws_rule *rule = priv->rule;
/* If something bad happens and lots of rules are failing, we don't
* want to pollute dmesg. Print only the first bad cqe per engine,
* the one that started the avalanche.
*/
if (queue->error_cqe_printed)
return;
queue->error_cqe_printed = true;
if (mlx5hws_rule_move_in_progress(rule))
mlx5hws_err(ctx,
"--- rule 0x%08llx: error completion moving rule: phase %s, wqes left %d\n",
HWS_PTR_TO_ID(rule),
rule->resize_info->state ==
MLX5HWS_RULE_RESIZE_STATE_WRITING ? "WRITING" :
rule->resize_info->state ==
MLX5HWS_RULE_RESIZE_STATE_DELETING ? "DELETING" :
"UNKNOWN",
rule->pending_wqes);
else
mlx5hws_err(ctx,
"--- rule 0x%08llx: error completion %s (%d), wqes left %d\n",
HWS_PTR_TO_ID(rule),
rule->status ==
MLX5HWS_RULE_STATUS_CREATING ? "CREATING" :
rule->status ==
MLX5HWS_RULE_STATUS_DELETING ? "DELETING" :
rule->status ==
MLX5HWS_RULE_STATUS_FAILING ? "FAILING" :
rule->status ==
MLX5HWS_RULE_STATUS_UPDATING ? "UPDATING" : "NA",
rule->status,
rule->pending_wqes);
mlx5hws_err(ctx, " rule 0x%08llx: matcher 0x%llx %s\n",
HWS_PTR_TO_ID(rule),
HWS_PTR_TO_ID(rule->matcher),
(rule->matcher->flags & MLX5HWS_MATCHER_FLAGS_ISOLATED) ?
"(isolated)" : "");
if (!cqe) {
mlx5hws_err(ctx, " rule 0x%08llx: no CQE\n",
HWS_PTR_TO_ID(rule));
return;
}
mlx5hws_err(ctx, " rule 0x%08llx: cqe->opcode = %d %s\n",
HWS_PTR_TO_ID(rule), opcode,
opcode == MLX5_CQE_REQ ? "(MLX5_CQE_REQ)" :
opcode == MLX5_CQE_REQ_ERR ? "(MLX5_CQE_REQ_ERR)" : " ");
if (opcode == MLX5_CQE_REQ_ERR) {
struct mlx5_err_cqe *err_cqe = (struct mlx5_err_cqe *)cqe;
mlx5hws_err(ctx,
" rule 0x%08llx: |--- hw_error_syndrome = 0x%x\n",
HWS_PTR_TO_ID(rule),
err_cqe->rsvd1[16]);
mlx5hws_err(ctx,
" rule 0x%08llx: |--- hw_syndrome_type = 0x%x\n",
HWS_PTR_TO_ID(rule),
err_cqe->rsvd1[17] >> 4);
mlx5hws_err(ctx,
" rule 0x%08llx: |--- vendor_err_synd = 0x%x\n",
HWS_PTR_TO_ID(rule),
err_cqe->vendor_err_synd);
mlx5hws_err(ctx,
" rule 0x%08llx: |--- syndrome = 0x%x\n",
HWS_PTR_TO_ID(rule),
err_cqe->syndrome);
}
mlx5hws_err(ctx,
" rule 0x%08llx: cqe->byte_cnt = 0x%08x\n",
HWS_PTR_TO_ID(rule), be32_to_cpu(cqe->byte_cnt));
mlx5hws_err(ctx,
" rule 0x%08llx: |-- UPDATE STATUS = %s\n",
HWS_PTR_TO_ID(rule),
(be32_to_cpu(cqe->byte_cnt) & 0x80000000) ?
"FAILURE" : "SUCCESS");
mlx5hws_err(ctx,
" rule 0x%08llx: |------- SYNDROME = %s\n",
HWS_PTR_TO_ID(rule),
((be32_to_cpu(cqe->byte_cnt) & 0x00000003) == 1) ?
"SET_FLOW_FAIL" :
((be32_to_cpu(cqe->byte_cnt) & 0x00000003) == 2) ?
"DISABLE_FLOW_FAIL" : "UNKNOWN");
mlx5hws_err(ctx,
" rule 0x%08llx: cqe->sop_drop_qpn = 0x%08x\n",
HWS_PTR_TO_ID(rule), be32_to_cpu(cqe->sop_drop_qpn));
mlx5hws_err(ctx,
" rule 0x%08llx: |-send wqe opcode = 0x%02x %s\n",
HWS_PTR_TO_ID(rule), wqe_opcode,
wqe_opcode == MLX5HWS_WQE_OPCODE_TBL_ACCESS ?
"(MLX5HWS_WQE_OPCODE_TBL_ACCESS)" : "(UNKNOWN)");
mlx5hws_err(ctx,
" rule 0x%08llx: |------------ qpn = 0x%06x\n",
HWS_PTR_TO_ID(rule),
be32_to_cpu(cqe->sop_drop_qpn) & 0xffffff);
}
static void hws_send_engine_update_rule(struct mlx5hws_send_engine *queue,
struct mlx5hws_send_ring_priv *priv,
u16 wqe_cnt,
enum mlx5hws_flow_op_status *status)
enum mlx5hws_flow_op_status *status,
struct mlx5_cqe64 *cqe)
{
priv->rule->pending_wqes--;
if (*status == MLX5HWS_FLOW_OP_ERROR) {
if (unlikely(*status == MLX5HWS_FLOW_OP_ERROR)) {
if (priv->retry_id) {
/* If there is a retry_id, then it's not an error yet,
* retry to insert this rule in the collision RTC.
*/
hws_send_engine_retry_post_send(queue, priv, wqe_cnt);
return;
}
hws_send_engine_dump_error_cqe(queue, priv, cqe);
/* Some part of the rule failed */
priv->rule->status = MLX5HWS_RULE_STATUS_FAILING;
*priv->used_id = 0;
@ -420,7 +535,8 @@ static void hws_send_engine_update(struct mlx5hws_send_engine *queue,
if (priv->user_data) {
if (priv->rule) {
hws_send_engine_update_rule(queue, priv, wqe_cnt, &status);
hws_send_engine_update_rule(queue, priv, wqe_cnt,
&status, cqe);
/* Completion is provided on the last rule WQE */
if (priv->rule->pending_wqes)
return;

1
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.h
View File

@ -140,6 +140,7 @@ struct mlx5hws_send_engine {
u16 used_entries;
u16 num_entries;
bool err;
bool error_cqe_printed;
struct mutex lock; /* Protects the send engine */
};

16
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/table.c
View File

@ -342,10 +342,10 @@ int mlx5hws_table_ft_set_next_rtc(struct mlx5hws_context *ctx,
return mlx5hws_cmd_flow_table_modify(ctx->mdev, &ft_attr, ft_id);
}
static int hws_table_ft_set_next_ft(struct mlx5hws_context *ctx,
u32 ft_id,
u32 fw_ft_type,
u32 next_ft_id)
int mlx5hws_table_ft_set_next_ft(struct mlx5hws_context *ctx,
u32 ft_id,
u32 fw_ft_type,
u32 next_ft_id)
{
struct mlx5hws_cmd_ft_modify_attr ft_attr = {0};
@ -389,10 +389,10 @@ int mlx5hws_table_connect_to_miss_table(struct mlx5hws_table *src_tbl,
if (dst_tbl) {
if (list_empty(&dst_tbl->matchers_list)) {
/* Connect src_tbl last_ft to dst_tbl start anchor */
ret = hws_table_ft_set_next_ft(src_tbl->ctx,
last_ft_id,
src_tbl->fw_ft_type,
dst_tbl->ft_id);
ret = mlx5hws_table_ft_set_next_ft(src_tbl->ctx,
last_ft_id,
src_tbl->fw_ft_type,
dst_tbl->ft_id);
if (ret)
return ret;

5
drivers/net/ethernet/mellanox/mlx5/core/steering/hws/table.h
View File

@ -65,4 +65,9 @@ int mlx5hws_table_ft_set_next_rtc(struct mlx5hws_context *ctx,
u32 rtc_0_id,
u32 rtc_1_id);
int mlx5hws_table_ft_set_next_ft(struct mlx5hws_context *ctx,
u32 ft_id,
u32 fw_ft_type,
u32 next_ft_id);
#endif /* MLX5HWS_TABLE_H_ */