github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-07-29 02:31:27 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge branch 'selftests-bpf-add-xdp-load-balancer-benchmark'

Browse Source 
Puranjay Mohan says:

====================
selftests/bpf: Add XDP load-balancer benchmark

Changelog:
RFC: https://lore.kernel.org/all/20260420111726.2118636-1-puranjay@kernel.org/
Changes in v1:
- Replace bpf_get_cpu_time_counter() with bpf_ktime_get_ns()
- Replace bpf_repeat() with plain for loop and may_goto
- Refactor collect_measurements() to reuse bench_force_done()
- Remove histogram, verbose calibration output, and per-scenario status prints
- Trim run script table to p50/stddev/p99
- Set env.quiet when --machine-readable is passed
- Add || true to run script benchmark invocation for set -e safety
- Add bpf-nop benchmark as timing overhead baseline (patch 3)
- Use named struct for LRU inner map to fix build on older toolchains

This series adds an XDP load-balancer benchmark (based on Katran) to the BPF
selftest bench framework.

Motivation
----------

Existing BPF bench tests measure individual operations (map lookups,
kprobes, ring buffers) in isolation.  Production BPF programs combine
parsing, map lookups, branching, and packet rewriting in a single call
chain.  The performance characteristics of such programs depend on the
interaction of these operations -- register pressure, spills, inlining
decisions, branch layout -- which isolated micro-benchmarks do not
capture.

This benchmark implements a simplified L4 load-balancer modeled after
katran [1].  The BPF program reproduces katran's core datapath:

  L3/L4 parsing -> VIP hash lookup -> per-CPU LRU connection table
  with consistent-hash fallback -> real server selection -> per-VIP
  and per-real stats -> IPIP/IP6IP6 encapsulation

The BPF code exercises hash maps, array-of-maps (per-CPU LRU),
percpu arrays, jhash, bpf_xdp_adjust_head(), bpf_ktime_get_ns(),
and bpf_get_smp_processor_id() in a single pipeline.

This is intended as the first in a series of BPF workload benchmarks
covering other use cases (sched_ext, etc.).

Design
------

A userspace loop calling bpf_prog_test_run_opts(repeat=1) would
measure syscall overhead, not BPF program cost -- the ~4 ns early-exit
paths would be buried under kernel entry/exit.  Using repeat=N is
also unsuitable: the kernel re-runs the same packet without resetting
state between iterations, so the second iteration of an encap scenario
would process an already-encapsulated packet.

Instead, timing is measured inside the BPF program using
bpf_ktime_get_ns().  BENCH_BPF_LOOP() brackets N iterations with
timestamp reads using a plain for loop with may_goto, runs a
caller-supplied reset block between iterations to undo side effects
(e.g. strip encapsulation), and records the elapsed time per batch.
One extra untimed iteration runs afterward for output validation.

Auto-calibration picks a batch size targeting ~10 ms per invocation.
A proportionality sanity check verifies that 2N iterations take ~2x
as long as N.

24 scenarios cover the code-path matrix:

  - Protocol: TCP, UDP
  - Address family: IPv4, IPv6, cross-AF (IPv4-in-IPv6)
  - LRU state: hit, miss (16M flow space), diverse (4K flows), cold
  - Consistent-hash: direct (LRU bypass)
  - TCP flags: SYN (skip LRU, force CH), RST (skip LRU insert)
  - Early exits: unknown VIP, non-IP, ICMP, fragments, IP options

Each scenario validates correctness before benchmarking by comparing
the output packet byte-for-byte against a pre-built expected packet
and checking BPF map counters.

Sample single-scenario output:

  $ sudo ./bench xdp-lb --scenario tcp-v4-lru-hit
  Setting up benchmark 'xdp-lb'...
  Benchmark 'xdp-lb' started.
  tcp-v4-lru-hit: median 74.51 ns/op, stddev 0.11, p99 74.81 (202 samples)

Sample run script output:

  $ ./benchs/run_bench_xdp_lb.sh

  XDP load-balancer benchmark
  ===========================
  +----------------------------------+----------+---------+----------+
  | Single-flow baseline             |      p50 |  stddev |      p99 |
  +----------------------------------+----------+---------+----------+
  | tcp-v4-lru-hit                   |    74.30 |    0.08 |    74.48 |
  | tcp-v4-ch                        |   101.73 |    0.11 |   102.01 |
  | tcp-v6-lru-hit                   |    76.77 |    0.14 |    77.04 |
  | tcp-v6-ch                        |   121.40 |    0.10 |   121.65 |
  | udp-v4-lru-hit                   |   107.42 |    0.22 |   107.90 |
  | udp-v6-lru-hit                   |   110.21 |    0.12 |   110.45 |
  | tcp-v4v6-lru-hit                 |    74.82 |    0.35 |    75.43 |
  +----------------------------------+----------+---------+----------+
  | Diverse flows (4K src addrs)     |      p50 |  stddev |      p99 |
  +----------------------------------+----------+---------+----------+
  | tcp-v4-lru-diverse               |    86.63 |    0.37 |    89.04 |
  | tcp-v4-ch-diverse                |   104.09 |    0.19 |   105.67 |
  | tcp-v6-lru-diverse               |    89.34 |    0.42 |    90.70 |
  | tcp-v6-ch-diverse                |   122.20 |    0.21 |   123.78 |
  | udp-v4-lru-diverse               |   119.37 |    0.58 |   123.10 |
  +----------------------------------+----------+---------+----------+
  | TCP flags                        |      p50 |  stddev |      p99 |
  +----------------------------------+----------+---------+----------+
  | tcp-v4-syn                       |   165.52 |   15.68 |   198.34 |
  | tcp-v4-rst-miss                  |   161.34 |    2.69 |   172.64 |
  +----------------------------------+----------+---------+----------+
  | LRU stress                       |      p50 |  stddev |      p99 |
  +----------------------------------+----------+---------+----------+
  | tcp-v4-lru-miss                  |   440.39 |   35.75 |   550.62 |
  | udp-v4-lru-miss                  |   571.88 |   57.38 |   680.61 |
  | tcp-v4-lru-warmup                |   317.75 |    9.55 |   356.20 |
  +----------------------------------+----------+---------+----------+
  | Early exits                      |      p50 |  stddev |      p99 |
  +----------------------------------+----------+---------+----------+
  | pass-v4-no-vip                   |    18.26 |    0.13 |    18.66 |
  | pass-v6-no-vip                   |    19.08 |    0.01 |    19.10 |
  | pass-v4-icmp                     |     6.81 |    0.02 |     6.86 |
  | pass-non-ip                      |     5.71 |    0.03 |     5.76 |
  | drop-v4-frag                     |     6.09 |    0.01 |     6.10 |
  | drop-v4-options                  |     5.88 |    0.00 |     5.89 |
  | drop-v6-frag                     |     6.00 |    0.03 |     6.04 |
  +----------------------------------+----------+---------+----------+

Patches
-------

Patch 1 adds bench_force_done() to the bench framework so benchmarks
can signal early completion when enough samples have been collected.

Patch 2 adds the shared BPF batch-timing library (BPF-side timing
arrays, BENCH_BPF_LOOP macro, userspace statistics and calibration).

Patch 3 adds a bpf-nop benchmark as a timing overhead baseline and
usage example for the timing library.

Patch 4 adds the common header shared between the BPF program and
userspace (flow_key, vip_definition, real_definition, encap helpers).

Patch 5 adds the XDP load-balancer BPF program.

Patch 6 adds the userspace benchmark driver with 24 scenarios,
packet construction, validation, and bench framework integration.

Patch 7 adds the run script for running all scenarios.

[1] https://github.com/facebookincubator/katran
====================

Link: https://patch.msgid.link/20260427232313.1582588-1-puranjay@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit is contained in:
Alexei Starovoitov 2026-05-11 15:25:24 -07:00
commit a982dda833
12 changed files with 2462 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
tools/testing/selftests/bpf/Makefile
View File

@ -906,6 +906,9 @@ $(OUTPUT)/bench_htab_mem.o: $(OUTPUT)/htab_mem_bench.skel.h
$(OUTPUT)/bench_bpf_crypto.o: $(OUTPUT)/crypto_bench.skel.h
$(OUTPUT)/bench_sockmap.o: $(OUTPUT)/bench_sockmap_prog.skel.h
$(OUTPUT)/bench_lpm_trie_map.o: $(OUTPUT)/lpm_trie_bench.skel.h $(OUTPUT)/lpm_trie_map.skel.h
$(OUTPUT)/bench_bpf_nop.o: $(OUTPUT)/bpf_nop_bench.skel.h bench_bpf_timing.h
$(OUTPUT)/bench_xdp_lb.o: $(OUTPUT)/xdp_lb_bench.skel.h bench_bpf_timing.h
$(OUTPUT)/bench_bpf_timing.o: bench_bpf_timing.h
$(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ)
$(OUTPUT)/bench: LDLIBS += -lm
$(OUTPUT)/bench: $(OUTPUT)/bench.o \
@ -928,6 +931,9 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
$(OUTPUT)/bench_bpf_crypto.o \
$(OUTPUT)/bench_sockmap.o \
$(OUTPUT)/bench_lpm_trie_map.o \
$(OUTPUT)/bench_bpf_timing.o \
$(OUTPUT)/bench_bpf_nop.o \
$(OUTPUT)/bench_xdp_lb.o \
$(OUTPUT)/usdt_1.o \
$(OUTPUT)/usdt_2.o \
#

20
tools/testing/selftests/bpf/bench.c
View File

@ -286,6 +286,7 @@ extern struct argp bench_trigger_batch_argp;
extern struct argp bench_crypto_argp;
extern struct argp bench_sockmap_argp;
extern struct argp bench_lpm_trie_map_argp;
extern struct argp bench_xdp_lb_argp;
static const struct argp_child bench_parsers[] = {
{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
@ -302,6 +303,7 @@ static const struct argp_child bench_parsers[] = {
{ &bench_crypto_argp, 0, "bpf crypto benchmark", 0 },
{ &bench_sockmap_argp, 0, "bpf sockmap benchmark", 0 },
{ &bench_lpm_trie_map_argp, 0, "LPM trie map benchmark", 0 },
{ &bench_xdp_lb_argp, 0, "XDP load-balancer benchmark", 0 },
{},
};
@ -575,6 +577,8 @@ extern const struct bench bench_lpm_trie_insert;
extern const struct bench bench_lpm_trie_update;
extern const struct bench bench_lpm_trie_delete;
extern const struct bench bench_lpm_trie_free;
extern const struct bench bench_bpf_nop;
extern const struct bench bench_xdp_lb;
static const struct bench *benchs[] = {
&bench_count_global,
@ -653,6 +657,8 @@ static const struct bench *benchs[] = {
&bench_lpm_trie_update,
&bench_lpm_trie_delete,
&bench_lpm_trie_free,
&bench_bpf_nop,
&bench_xdp_lb,
};
static void find_benchmark(void)
@ -741,6 +747,13 @@ static void setup_benchmark(void)
static pthread_mutex_t bench_done_mtx = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t bench_done = PTHREAD_COND_INITIALIZER;
void bench_force_done(void)
{
pthread_mutex_lock(&bench_done_mtx);
pthread_cond_signal(&bench_done);
pthread_mutex_unlock(&bench_done_mtx);
}
static void collect_measurements(long delta_ns) {
int iter = state.res_cnt++;
struct bench_res *res = &state.results[iter];
@ -750,11 +763,8 @@ static void collect_measurements(long delta_ns) {
if (bench->report_progress)
bench->report_progress(iter, res, delta_ns);
if (iter == env.duration_sec + env.warmup_sec) {
pthread_mutex_lock(&bench_done_mtx);
pthread_cond_signal(&bench_done);
pthread_mutex_unlock(&bench_done_mtx);
}
if (iter == env.duration_sec + env.warmup_sec)
bench_force_done();
}
int main(int argc, char **argv)

1
tools/testing/selftests/bpf/bench.h
View File

@ -70,6 +70,7 @@ extern struct env env;
extern const struct bench *bench;
void setup_libbpf(void);
void bench_force_done(void);
void hits_drops_report_progress(int iter, struct bench_res *res, long delta_ns);
void hits_drops_report_final(struct bench_res res[], int res_cnt);
void false_hits_report_progress(int iter, struct bench_res *res, long delta_ns);

50
tools/testing/selftests/bpf/bench_bpf_timing.h Normal file
View File

@ -0,0 +1,50 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#ifndef __BENCH_BPF_TIMING_H__
#define __BENCH_BPF_TIMING_H__
#include <stdbool.h>
#include <linux/types.h>
#include "bench.h"
#ifndef BENCH_NR_SAMPLES
#define BENCH_NR_SAMPLES 4096
#endif
#ifndef BENCH_NR_CPUS
#define BENCH_NR_CPUS 256
#endif
typedef void (*bpf_bench_run_fn)(void *ctx);
struct bpf_bench_timing {
__u64 (*samples)[BENCH_NR_SAMPLES]; /* skel->bss->timing_samples */
__u32 *idx; /* skel->bss->timing_idx */
volatile __u32 *timing_enabled; /* &skel->bss->timing_enabled */
volatile __u32 *batch_iters_bss; /* &skel->bss->batch_iters */
__u32 batch_iters;
__u32 target_samples;
__u32 nr_cpus;
int warmup_ticks;
bool done;
bool machine_readable;
};
#define BENCH_TIMING_INIT(t, skel, iters) do { \
(t)->samples = (skel)->bss->timing_samples; \
(t)->idx = (skel)->bss->timing_idx; \
(t)->timing_enabled = &(skel)->bss->timing_enabled; \
(t)->batch_iters_bss = &(skel)->bss->batch_iters; \
(t)->batch_iters = (iters); \
(t)->target_samples = 200; \
(t)->nr_cpus = env.nr_cpus; \
(t)->warmup_ticks = 0; \
(t)->done = false; \
(t)->machine_readable = false; \
} while (0)
void bpf_bench_timing_measure(struct bpf_bench_timing *t, struct bench_res *res);
void bpf_bench_timing_report(struct bpf_bench_timing *t, const char *name, const char *desc);
void bpf_bench_calibrate(struct bpf_bench_timing *t, bpf_bench_run_fn run_fn, void *ctx);
#endif /* __BENCH_BPF_TIMING_H__ */

84
tools/testing/selftests/bpf/benchs/bench_bpf_nop.c Normal file
View File

@ -0,0 +1,84 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include "bench.h"
#include "bench_bpf_timing.h"
#include "bpf_nop_bench.skel.h"
#include "bpf_util.h"
static struct ctx {
struct bpf_nop_bench *skel;
struct bpf_bench_timing timing;
int prog_fd;
} ctx;
static void nop_validate(void)
{
if (env.consumer_cnt != 0) {
fprintf(stderr, "benchmark doesn't support consumers\n");
exit(1);
}
}
static void nop_run_once(void *unused __always_unused)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
bpf_prog_test_run_opts(ctx.prog_fd, &topts);
}
static void nop_setup(void)
{
struct bpf_nop_bench *skel;
int err;
setup_libbpf();
skel = bpf_nop_bench__open();
if (!skel) {
fprintf(stderr, "failed to open skeleton\n");
exit(1);
}
err = bpf_nop_bench__load(skel);
if (err) {
fprintf(stderr, "failed to load skeleton: %s\n", strerror(-err));
bpf_nop_bench__destroy(skel);
exit(1);
}
ctx.skel = skel;
ctx.prog_fd = bpf_program__fd(skel->progs.bench_nop);
BENCH_TIMING_INIT(&ctx.timing, skel, 0);
bpf_bench_calibrate(&ctx.timing, nop_run_once, NULL);
env.duration_sec = 600;
}
static void *nop_producer(void *input)
{
while (true)
nop_run_once(NULL);
return NULL;
}
static void nop_measure(struct bench_res *res)
{
bpf_bench_timing_measure(&ctx.timing, res);
}
static void nop_report_final(struct bench_res res[], int res_cnt)
{
bpf_bench_timing_report(&ctx.timing, "bpf-nop", NULL);
}
const struct bench bench_bpf_nop = {
.name = "bpf-nop",
.validate = nop_validate,
.setup = nop_setup,
.producer_thread = nop_producer,
.measure = nop_measure,
.report_final = nop_report_final,
};

272
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c Normal file
View File

@ -0,0 +1,272 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <math.h>
#include "bench_bpf_timing.h"
#include "bpf_util.h"
struct timing_stats {
double min, max;
double median, p99;
double mean, stddev;
int count;
};
static int cmp_double(const void *a, const void *b)
{
double da = *(const double *)a;
double db = *(const double *)b;
if (da < db)
return -1;
if (da > db)
return 1;
return 0;
}
static double percentile(const double *sorted, int n, double pct)
{
int idx = (int)(n * pct / 100.0);
if (idx >= n)
idx = n - 1;
return sorted[idx];
}
static int collect_samples(struct bpf_bench_timing *t,
double *out, int max_out)
{
unsigned int nr_cpus = bpf_num_possible_cpus();
__u32 timed_iters = t->batch_iters;
int total = 0;
if (nr_cpus > BENCH_NR_CPUS)
nr_cpus = BENCH_NR_CPUS;
for (unsigned int cpu = 0; cpu < nr_cpus; cpu++) {
__u32 count = t->idx[cpu];
if (count > BENCH_NR_SAMPLES)
count = BENCH_NR_SAMPLES;
for (__u32 i = 0; i < count && total < max_out; i++) {
__u64 sample = t->samples[cpu][i];
if (sample == 0)
continue;
out[total++] = (double)sample / timed_iters;
}
}
qsort(out, total, sizeof(double), cmp_double);
return total;
}
static void compute_stats(const double *sorted, int n,
struct timing_stats *s)
{
double sum = 0, var_sum = 0;
memset(s, 0, sizeof(*s));
s->count = n;
if (n == 0)
return;
s->min = sorted[0];
s->max = sorted[n - 1];
s->median = sorted[n / 2];
s->p99 = percentile(sorted, n, 99);
for (int i = 0; i < n; i++)
sum += sorted[i];
s->mean = sum / n;
for (int i = 0; i < n; i++) {
double d = sorted[i] - s->mean;
var_sum += d * d;
}
s->stddev = n > 1 ? sqrt(var_sum / (n - 1)) : 0;
}
void bpf_bench_timing_measure(struct bpf_bench_timing *t, struct bench_res *res)
{
unsigned int nr_cpus;
__u32 total_samples;
int i;
t->warmup_ticks++;
if (t->warmup_ticks < env.warmup_sec)
return;
if (t->warmup_ticks == env.warmup_sec) {
*t->timing_enabled = 1;
return;
}
nr_cpus = bpf_num_possible_cpus();
if (nr_cpus > BENCH_NR_CPUS)
nr_cpus = BENCH_NR_CPUS;
total_samples = 0;
for (i = 0; i < (int)nr_cpus; i++) {
__u32 cnt = t->idx[i];
if (cnt > BENCH_NR_SAMPLES)
cnt = BENCH_NR_SAMPLES;
total_samples += cnt;
}
if (total_samples >= (__u32)env.producer_cnt * t->target_samples && !t->done) {
t->done = true;
*t->timing_enabled = 0;
bench_force_done();
}
}
void bpf_bench_timing_report(struct bpf_bench_timing *t, const char *name, const char *description)
{
int max_out = BENCH_NR_CPUS * BENCH_NR_SAMPLES;
struct timing_stats s;
double *all;
int total;
all = calloc(max_out, sizeof(*all));
if (!all) {
fprintf(stderr, "failed to allocate timing buffer\n");
return;
}
total = collect_samples(t, all, max_out);
if (total == 0) {
printf("No timing samples collected.\n");
free(all);
return;
}
compute_stats(all, total, &s);
if (t->machine_readable) {
printf("RESULT scenario=%s samples=%d median=%.2f stddev=%.2f cv=%.2f min=%.2f "
"p99=%.2f max=%.2f\n", name, total, s.median, s.stddev,
s.mean > 0 ? s.stddev / s.mean * 100.0 : 0.0, s.min, s.p99, s.max);
} else {
printf("%s: median %.2f ns/op, stddev %.2f, p99 %.2f (%d samples)\n", name,
s.median, s.stddev, s.p99, total);
}
free(all);
}
#define CALIBRATE_SEED_BATCH 100
#define CALIBRATE_MIN_BATCH 100
#define CALIBRATE_MAX_BATCH 10000000
#define CALIBRATE_TARGET_MS 10
#define CALIBRATE_RUNS 5
#define PROPORTIONALITY_TOL 0.05 /* 5% */
static void reset_timing(struct bpf_bench_timing *t)
{
*t->timing_enabled = 0;
memset(t->samples, 0, sizeof(__u64) * BENCH_NR_CPUS * BENCH_NR_SAMPLES);
memset(t->idx, 0, sizeof(__u32) * BENCH_NR_CPUS);
}
static __u64 measure_elapsed(struct bpf_bench_timing *t, bpf_bench_run_fn run_fn, void *run_ctx,
__u32 iters, int runs)
{
__u64 buf[CALIBRATE_RUNS];
int n = 0, i, j;
reset_timing(t);
*t->batch_iters_bss = iters;
*t->timing_enabled = 1;
for (i = 0; i < runs; i++)
run_fn(run_ctx);
*t->timing_enabled = 0;
for (i = 0; i < BENCH_NR_CPUS && n < runs; i++) {
__u32 cnt = t->idx[i];
for (j = 0; j < (int)cnt && n < runs; j++)
buf[n++] = t->samples[i][j];
}
if (n == 0)
return 0;
for (i = 1; i < n; i++) {
__u64 key = buf[i];
j = i - 1;
while (j >= 0 && buf[j] > key) {
buf[j + 1] = buf[j];
j--;
}
buf[j + 1] = key;
}
return buf[n / 2];
}
static __u32 compute_batch_iters(__u64 per_op_ns)
{
__u64 target_ns = (__u64)CALIBRATE_TARGET_MS * 1000000ULL;
__u32 iters;
if (per_op_ns == 0)
return CALIBRATE_MIN_BATCH;
iters = target_ns / per_op_ns;
if (iters < CALIBRATE_MIN_BATCH)
iters = CALIBRATE_MIN_BATCH;
if (iters > CALIBRATE_MAX_BATCH)
iters = CALIBRATE_MAX_BATCH;
return iters;
}
void bpf_bench_calibrate(struct bpf_bench_timing *t, bpf_bench_run_fn run_fn, void *run_ctx)
{
__u64 elapsed, per_op_ns;
__u64 time_n, time_2n;
double ratio;
elapsed = measure_elapsed(t, run_fn, run_ctx, CALIBRATE_SEED_BATCH, CALIBRATE_RUNS);
if (elapsed == 0) {
fprintf(stderr, "calibration: no timing samples, using default\n");
t->batch_iters = 10000;
*t->batch_iters_bss = t->batch_iters;
reset_timing(t);
return;
}
per_op_ns = elapsed / CALIBRATE_SEED_BATCH;
t->batch_iters = compute_batch_iters(per_op_ns);
time_n = measure_elapsed(t, run_fn, run_ctx, t->batch_iters, CALIBRATE_RUNS);
time_2n = measure_elapsed(t, run_fn, run_ctx, t->batch_iters * 2, CALIBRATE_RUNS);
if (time_n > 0 && time_2n > 0) {
ratio = (double)time_2n / (double)time_n;
if (fabs(ratio - 2.0) / 2.0 > PROPORTIONALITY_TOL)
fprintf(stderr,
"WARNING: proportionality check failed (2N/N ratio=%.3f, "
"expected=2.000, error=%.1f%%)\n System noise may be affecting "
"results.\n",
ratio, fabs(ratio - 2.0) / 2.0 * 100.0);
}
*t->batch_iters_bss = t->batch_iters;
reset_timing(t);
}

1113
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c Normal file
View File

File diff suppressed because it is too large Load Diff

79
tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh Executable file
View File

@ -0,0 +1,79 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
source ./benchs/run_common.sh
set -eufo pipefail
WARMUP=${WARMUP:-3}
RUN="sudo ./bench -q -w${WARMUP} -a xdp-lb --machine-readable"
SEP=" +----------------------------------+----------+---------+----------+"
HDR=" | %-32s | %8s | %7s | %8s |\n"
ROW=" | %-32s | %8s | %7s | %8s |\n"
function group_header()
{
printf "%s\n" "$SEP"
printf "$HDR" "$1" "p50" "stddev" "p99"
printf "%s\n" "$SEP"
}
function rval()
{
echo "$1" | sed -nE "s/.*$2=([^ ]+).*/\1/p"
}
function run_scenario()
{
local sc="$1"
shift
local output rline
output=$($RUN --scenario "$sc" "$@" 2>&1) || true
rline=$(echo "$output" | grep '^RESULT ' || true)
if [ -z "$rline" ]; then
printf "$ROW" "$sc" "ERR" "-" "-"
return
fi
printf "$ROW" "$sc" \
"$(rval "$rline" median)" \
"$(rval "$rline" stddev)" \
"$(rval "$rline" p99)"
}
header "XDP load-balancer benchmark"
group_header "Single-flow baseline"
for sc in tcp-v4-lru-hit tcp-v4-ch \
tcp-v6-lru-hit tcp-v6-ch \
udp-v4-lru-hit udp-v6-lru-hit \
tcp-v4v6-lru-hit; do
run_scenario "$sc"
done
group_header "Diverse flows (4K src addrs)"
for sc in tcp-v4-lru-diverse tcp-v4-ch-diverse \
tcp-v6-lru-diverse tcp-v6-ch-diverse \
udp-v4-lru-diverse; do
run_scenario "$sc"
done
group_header "TCP flags"
run_scenario tcp-v4-syn
run_scenario tcp-v4-rst-miss
group_header "LRU stress"
run_scenario tcp-v4-lru-miss
run_scenario udp-v4-lru-miss
run_scenario tcp-v4-lru-warmup
group_header "Early exits"
for sc in pass-v4-no-vip pass-v6-no-vip pass-v4-icmp pass-non-ip drop-v4-frag drop-v4-options \
drop-v6-frag; do
run_scenario "$sc"
done
printf "%s\n" "$SEP"

69
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h Normal file
View File

@ -0,0 +1,69 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#ifndef __BENCH_BPF_TIMING_BPF_H__
#define __BENCH_BPF_TIMING_BPF_H__
#include <stdbool.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf_may_goto.h>
#ifndef BENCH_NR_SAMPLES
#define BENCH_NR_SAMPLES 4096
#endif
#ifndef BENCH_NR_CPUS
#define BENCH_NR_CPUS 256
#endif
#define BENCH_CPU_MASK (BENCH_NR_CPUS - 1)
__u64 timing_samples[BENCH_NR_CPUS][BENCH_NR_SAMPLES];
__u32 timing_idx[BENCH_NR_CPUS];
volatile __u32 batch_iters;
volatile __u32 timing_enabled;
static __always_inline void bench_record_sample(__u64 elapsed_ns)
{
__u32 cpu, idx;
if (!timing_enabled)
return;
cpu = bpf_get_smp_processor_id() & BENCH_CPU_MASK;
idx = timing_idx[cpu];
if (idx >= BENCH_NR_SAMPLES)
return;
timing_samples[cpu][idx] = elapsed_ns;
timing_idx[cpu] = idx + 1;
}
/*
* @body: expression to time; return value (int) stored in __bench_result.
* @reset: undo body's side-effects so each iteration starts identically.
* May reference __bench_result. Use ({}) for empty reset.
*
* Runs batch_iters timed iterations, then one untimed iteration whose
* return value the macro evaluates to (for validation).
*/
#define BENCH_BPF_LOOP(body, reset) ({ \
__u64 __bench_start = bpf_ktime_get_ns(); \
__u32 __bench_i; \
int __bench_result; \
\
for (__bench_i = 0; \
__bench_i < batch_iters && can_loop; \
__bench_i++) { \
__bench_result = (body); \
reset; \
} \
\
bench_record_sample(bpf_ktime_get_ns() - __bench_start); \
\
__bench_result = (body); \
__bench_result; \
})
#endif /* __BENCH_BPF_TIMING_BPF_H__ */

14
tools/testing/selftests/bpf/progs/bpf_nop_bench.c Normal file
View File

@ -0,0 +1,14 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include "bench_bpf_timing.bpf.h"
SEC("syscall")
int bench_nop(void *ctx)
{
return BENCH_BPF_LOOP(0, ({}));
}
char _license[] SEC("license") = "GPL";

647
tools/testing/selftests/bpf/progs/xdp_lb_bench.c Normal file
View File

@ -0,0 +1,647 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <stddef.h>
#include <stdbool.h>
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/in.h>
#include <linux/tcp.h>
#include <linux/udp.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
#include "bpf_compiler.h"
#include "xdp_lb_bench_common.h"
#include "bench_bpf_timing.bpf.h"
#ifndef IPPROTO_FRAGMENT
#define IPPROTO_FRAGMENT 44
#endif
/* jhash helpers */
static inline __u32 rol32(__u32 word, unsigned int shift)
{
return (word << shift) | (word >> ((-shift) & 31));
}
#define __jhash_mix(a, b, c) \
{ \
a -= c; a ^= rol32(c, 4); c += b; \
b -= a; b ^= rol32(a, 6); a += c; \
c -= b; c ^= rol32(b, 8); b += a; \
a -= c; a ^= rol32(c, 16); c += b; \
b -= a; b ^= rol32(a, 19); a += c; \
c -= b; c ^= rol32(b, 4); b += a; \
}
#define __jhash_final(a, b, c) \
{ \
c ^= b; c -= rol32(b, 14); \
a ^= c; a -= rol32(c, 11); \
b ^= a; b -= rol32(a, 25); \
c ^= b; c -= rol32(b, 16); \
a ^= c; a -= rol32(c, 4); \
b ^= a; b -= rol32(a, 14); \
c ^= b; c -= rol32(b, 24); \
}
#define JHASH_INITVAL 0xdeadbeef
static inline __u32 __jhash_nwords(__u32 a, __u32 b, __u32 c, __u32 initval)
{
a += initval;
b += initval;
c += initval;
__jhash_final(a, b, c);
return c;
}
static inline __u32 jhash_2words(__u32 a, __u32 b, __u32 initval)
{
return __jhash_nwords(a, b, 0, initval + JHASH_INITVAL + (2 << 2));
}
static inline __u32 jhash2_4words(const __u32 *k, __u32 initval)
{
__u32 a, b, c;
a = b = c = JHASH_INITVAL + (4 << 2) + initval;
a += k[0]; b += k[1]; c += k[2];
__jhash_mix(a, b, c);
a += k[3];
__jhash_final(a, b, c);
return c;
}
static __always_inline void ipv4_csum(struct iphdr *iph)
{
__u16 *next_iph = (__u16 *)iph;
__u32 csum = 0;
int i;
__pragma_loop_unroll_full
for (i = 0; i < (int)(sizeof(*iph) >> 1); i++)
csum += *next_iph++;
csum = (csum & 0xffff) + (csum >> 16);
csum = (csum & 0xffff) + (csum >> 16);
iph->check = ~csum;
}
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 64);
__type(key, struct vip_definition);
__type(value, struct vip_meta);
} vip_map SEC(".maps");
struct lru_inner_map {
__uint(type, BPF_MAP_TYPE_LRU_HASH);
__type(key, struct flow_key);
__type(value, struct real_pos_lru);
__uint(max_entries, DEFAULT_LRU_SIZE);
} lru_inner SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
__type(key, __u32);
__type(value, __u32);
__uint(max_entries, BENCH_NR_CPUS);
__array(values, struct lru_inner_map);
} lru_mapping SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, CH_RINGS_SIZE);
__type(key, __u32);
__type(value, __u32);
} ch_rings SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, MAX_REALS);
__type(key, __u32);
__type(value, struct real_definition);
} reals SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(max_entries, STATS_SIZE);
__type(key, __u32);
__type(value, struct lb_stats);
} stats SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(max_entries, MAX_REALS);
__type(key, __u32);
__type(value, struct lb_stats);
} reals_stats SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 1);
__type(key, __u32);
__type(value, struct ctl_value);
} ctl_array SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 1);
__type(key, __u32);
__type(value, struct vip_definition);
} vip_miss_stats SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(max_entries, MAX_REALS);
__type(key, __u32);
__type(value, __u32);
} lru_miss_stats SEC(".maps");
volatile __u32 flow_mask;
volatile __u32 cold_lru;
__u32 batch_gen;
/*
* old_eth MUST be read BEFORE writing the outer header because
* bpf_xdp_adjust_head makes them overlap.
*/
static __always_inline int encap_v4(struct xdp_md *xdp, __be32 saddr, __be32 daddr,
__u16 payload_len, const __u8 *dst_mac)
{
struct ethhdr *new_eth, *old_eth;
void *data, *data_end;
struct iphdr *iph;
if (bpf_xdp_adjust_head(xdp, -(int)sizeof(struct iphdr)))
return -1;
data = (void *)(long)xdp->data;
data_end = (void *)(long)xdp->data_end;
new_eth = data;
iph = data + sizeof(struct ethhdr);
old_eth = data + sizeof(struct iphdr);
if (new_eth + 1 > data_end || old_eth + 1 > data_end || iph + 1 > data_end)
return -1;
__builtin_memcpy(new_eth->h_source, old_eth->h_dest, sizeof(new_eth->h_source));
__builtin_memcpy(new_eth->h_dest, dst_mac, sizeof(new_eth->h_dest));
new_eth->h_proto = bpf_htons(ETH_P_IP);
__builtin_memset(iph, 0, sizeof(*iph));
iph->version = 4;
iph->ihl = sizeof(*iph) >> 2;
iph->protocol = IPPROTO_IPIP;
iph->tot_len = bpf_htons(payload_len + sizeof(*iph));
iph->ttl = 64;
iph->saddr = saddr;
iph->daddr = daddr;
ipv4_csum(iph);
return 0;
}
static __always_inline int encap_v6(struct xdp_md *xdp, const __be32 saddr[4],
const __be32 daddr[4], __u8 nexthdr, __u16 payload_len,
const __u8 *dst_mac)
{
struct ethhdr *new_eth, *old_eth;
void *data, *data_end;
struct ipv6hdr *ip6h;
if (bpf_xdp_adjust_head(xdp, -(int)sizeof(struct ipv6hdr)))
return -1;
data = (void *)(long)xdp->data;
data_end = (void *)(long)xdp->data_end;
new_eth = data;
ip6h = data + sizeof(struct ethhdr);
old_eth = data + sizeof(struct ipv6hdr);
if (new_eth + 1 > data_end || old_eth + 1 > data_end || ip6h + 1 > data_end)
return -1;
__builtin_memcpy(new_eth->h_source, old_eth->h_dest, sizeof(new_eth->h_source));
__builtin_memcpy(new_eth->h_dest, dst_mac, sizeof(new_eth->h_dest));
new_eth->h_proto = bpf_htons(ETH_P_IPV6);
__builtin_memset(ip6h, 0, sizeof(*ip6h));
ip6h->version = 6;
ip6h->nexthdr = nexthdr;
ip6h->payload_len = bpf_htons(payload_len);
ip6h->hop_limit = 64;
__builtin_memcpy(&ip6h->saddr, saddr, sizeof(ip6h->saddr));
__builtin_memcpy(&ip6h->daddr, daddr, sizeof(ip6h->daddr));
return 0;
}
static __always_inline void update_stats(void *map, __u32 key, __u16 bytes)
{
struct lb_stats *st = bpf_map_lookup_elem(map, &key);
if (st) {
st->v1 += 1;
st->v2 += bytes;
}
}
static __always_inline void count_action(int action)
{
struct lb_stats *st;
__u32 key;
if (action == XDP_TX)
key = STATS_XDP_TX;
else if (action == XDP_PASS)
key = STATS_XDP_PASS;
else
key = STATS_XDP_DROP;
st = bpf_map_lookup_elem(&stats, &key);
if (st)
st->v1 += 1;
}
static __always_inline bool is_under_flood(void)
{
__u32 key = STATS_NEW_CONN;
struct lb_stats *conn_st = bpf_map_lookup_elem(&stats, &key);
__u64 cur_time;
if (!conn_st)
return true;
cur_time = bpf_ktime_get_ns();
if ((cur_time - conn_st->v2) > ONE_SEC) {
conn_st->v1 = 1;
conn_st->v2 = cur_time;
} else {
conn_st->v1 += 1;
if (conn_st->v1 > MAX_CONN_RATE)
return true;
}
return false;
}
static __always_inline struct real_definition *connection_table_lookup(void *lru_map,
struct flow_key *flow,
__u32 *out_pos)
{
struct real_pos_lru *dst_lru;
struct real_definition *real;
__u32 key;
dst_lru = bpf_map_lookup_elem(lru_map, flow);
if (!dst_lru)
return NULL;
/* UDP connections use atime-based timeout instead of FIN/RST */
if (flow->proto == IPPROTO_UDP) {
__u64 cur_time = bpf_ktime_get_ns();
if (cur_time - dst_lru->atime > LRU_UDP_TIMEOUT)
return NULL;
dst_lru->atime = cur_time;
}
key = dst_lru->pos;
*out_pos = key;
real = bpf_map_lookup_elem(&reals, &key);
return real;
}
static __always_inline bool get_packet_dst(struct real_definition **real, struct flow_key *flow,
struct vip_meta *vip_info, bool is_v6, void *lru_map,
bool is_rst, __u32 *out_pos)
{
bool under_flood;
__u32 hash, ch_key;
__u32 *ch_val;
__u32 real_pos;
under_flood = is_under_flood();
if (is_v6) {
__u32 src_hash = jhash2_4words((__u32 *)flow->srcv6, MAX_VIPS);
hash = jhash_2words(src_hash, flow->ports, CH_RING_SIZE);
} else {
hash = jhash_2words(flow->src, flow->ports, CH_RING_SIZE);
}
ch_key = CH_RING_SIZE * vip_info->vip_num + hash % CH_RING_SIZE;
ch_val = bpf_map_lookup_elem(&ch_rings, &ch_key);
if (!ch_val)
return false;
real_pos = *ch_val;
*real = bpf_map_lookup_elem(&reals, &real_pos);
if (!(*real))
return false;
if (!(vip_info->flags & F_LRU_BYPASS) && !under_flood && !is_rst) {
struct real_pos_lru new_lru = { .pos = real_pos };
if (flow->proto == IPPROTO_UDP)
new_lru.atime = bpf_ktime_get_ns();
bpf_map_update_elem(lru_map, flow, &new_lru, BPF_ANY);
}
*out_pos = real_pos;
return true;
}
static __always_inline void update_vip_lru_miss_stats(struct vip_definition *vip, bool is_v6,
__u32 real_idx)
{
struct vip_definition *miss_vip;
__u32 key = 0;
__u32 *cnt;
miss_vip = bpf_map_lookup_elem(&vip_miss_stats, &key);
if (!miss_vip)
return;
if (is_v6) {
if (miss_vip->vipv6[0] != vip->vipv6[0] || miss_vip->vipv6[1] != vip->vipv6[1] ||
miss_vip->vipv6[2] != vip->vipv6[2] || miss_vip->vipv6[3] != vip->vipv6[3])
return;
} else {
if (miss_vip->vip != vip->vip)
return;
}
if (miss_vip->port != vip->port || miss_vip->proto != vip->proto)
return;
cnt = bpf_map_lookup_elem(&lru_miss_stats, &real_idx);
if (cnt)
*cnt += 1;
}
static __noinline int process_packet(struct xdp_md *xdp)
{
void *data = (void *)(long)xdp->data;
void *data_end = (void *)(long)xdp->data_end;
struct ethhdr *eth = data;
struct real_definition *dst = NULL;
struct vip_definition vip_def = {};
struct ctl_value *cval;
struct flow_key flow = {};
struct vip_meta *vip_info;
struct lb_stats *data_stats;
struct udphdr *uh;
__be32 tnl_src[4];
void *lru_map;
void *l4;
__u16 payload_len;
__u32 real_pos = 0, cpu_num, key;
__u8 proto;
int action = XDP_DROP;
bool is_v6, is_syn = false, is_rst = false;
if (eth + 1 > data_end)
goto out;
if (eth->h_proto == bpf_htons(ETH_P_IPV6)) {
is_v6 = true;
} else if (eth->h_proto == bpf_htons(ETH_P_IP)) {
is_v6 = false;
} else {
action = XDP_PASS;
goto out;
}
if (is_v6) {
struct ipv6hdr *ip6h = (void *)(eth + 1);
if (ip6h + 1 > data_end)
goto out;
if (ip6h->nexthdr == IPPROTO_FRAGMENT)
goto out;
payload_len = sizeof(struct ipv6hdr) + bpf_ntohs(ip6h->payload_len);
proto = ip6h->nexthdr;
__builtin_memcpy(flow.srcv6, &ip6h->saddr, sizeof(flow.srcv6));
__builtin_memcpy(flow.dstv6, &ip6h->daddr, sizeof(flow.dstv6));
__builtin_memcpy(vip_def.vipv6, &ip6h->daddr, sizeof(vip_def.vipv6));
l4 = (void *)(ip6h + 1);
} else {
struct iphdr *iph = (void *)(eth + 1);
if (iph + 1 > data_end)
goto out;
if (iph->ihl != 5)
goto out;
if (iph->frag_off & bpf_htons(PCKT_FRAGMENTED))
goto out;
payload_len = bpf_ntohs(iph->tot_len);
proto = iph->protocol;
flow.src = iph->saddr;
flow.dst = iph->daddr;
vip_def.vip = iph->daddr;
l4 = (void *)(iph + 1);
}
/* TCP and UDP share the same port layout at offset 0 */
if (proto != IPPROTO_TCP && proto != IPPROTO_UDP) {
action = XDP_PASS;
goto out;
}
uh = l4;
if ((void *)(uh + 1) > data_end)
goto out;
flow.port16[0] = uh->source;
flow.port16[1] = uh->dest;
if (proto == IPPROTO_TCP) {
struct tcphdr *th = l4;
if ((void *)(th + 1) > data_end)
goto out;
is_syn = th->syn;
is_rst = th->rst;
}
flow.proto = proto;
vip_def.port = flow.port16[1];
vip_def.proto = proto;
vip_info = bpf_map_lookup_elem(&vip_map, &vip_def);
if (!vip_info) {
action = XDP_PASS;
goto out;
}
key = STATS_LRU;
data_stats = bpf_map_lookup_elem(&stats, &key);
if (!data_stats)
goto out;
data_stats->v1 += 1;
cpu_num = bpf_get_smp_processor_id();
lru_map = bpf_map_lookup_elem(&lru_mapping, &cpu_num);
if (!lru_map)
goto out;
if (!(vip_info->flags & F_LRU_BYPASS) && !is_syn)
dst = connection_table_lookup(lru_map, &flow, &real_pos);
if (!dst) {
if (flow.proto == IPPROTO_TCP) {
struct lb_stats *miss_st;
key = STATS_LRU_MISS;
miss_st = bpf_map_lookup_elem(&stats, &key);
if (miss_st)
miss_st->v1 += 1;
}
if (!get_packet_dst(&dst, &flow, vip_info, is_v6, lru_map, is_rst, &real_pos))
goto out;
update_vip_lru_miss_stats(&vip_def, is_v6, real_pos);
data_stats->v2 += 1;
}
key = 0;
cval = bpf_map_lookup_elem(&ctl_array, &key);
if (!cval)
goto out;
update_stats(&stats, vip_info->vip_num, payload_len);
update_stats(&reals_stats, real_pos, payload_len);
if (is_v6) {
create_encap_ipv6_src(flow.port16[0], flow.srcv6[0], tnl_src);
if (encap_v6(xdp, tnl_src, dst->dstv6, IPPROTO_IPV6, payload_len, cval->mac))
goto out;
} else if (dst->flags & F_IPV6) {
create_encap_ipv6_src(flow.port16[0], flow.src, tnl_src);
if (encap_v6(xdp, tnl_src, dst->dstv6, IPPROTO_IPIP, payload_len, cval->mac))
goto out;
} else {
if (encap_v4(xdp, create_encap_ipv4_src(flow.port16[0], flow.src), dst->dst,
payload_len, cval->mac))
goto out;
}
action = XDP_TX;
out:
count_action(action);
return action;
}
static __always_inline int strip_encap(struct xdp_md *xdp, const struct ethhdr *saved_eth)
{
void *data = (void *)(long)xdp->data;
void *data_end = (void *)(long)xdp->data_end;
struct ethhdr *eth = data;
int hdr_sz;
if (eth + 1 > data_end)
return -1;
hdr_sz = (eth->h_proto == bpf_htons(ETH_P_IPV6)) ? (int)sizeof(struct ipv6hdr)
: (int)sizeof(struct iphdr);
if (bpf_xdp_adjust_head(xdp, hdr_sz))
return -1;
data = (void *)(long)xdp->data;
data_end = (void *)(long)xdp->data_end;
eth = data;
if (eth + 1 > data_end)
return -1;
__builtin_memcpy(eth, saved_eth, sizeof(*saved_eth));
return 0;
}
static __always_inline void randomize_src(struct xdp_md *xdp, int saddr_off, __u32 *rand_state)
{
void *data = (void *)(long)xdp->data;
void *data_end = (void *)(long)xdp->data_end;
__u32 *saddr = data + saddr_off;
*rand_state ^= *rand_state << 13;
*rand_state ^= *rand_state >> 17;
*rand_state ^= *rand_state << 5;
if ((void *)(saddr + 1) <= data_end)
*saddr = *rand_state & flow_mask;
}
SEC("xdp")
int xdp_lb_bench(struct xdp_md *xdp)
{
void *data = (void *)(long)xdp->data;
void *data_end = (void *)(long)xdp->data_end;
struct ethhdr *eth = data;
struct ethhdr saved_eth;
__u32 rand_state = 0;
__u32 batch_hash = 0;
int saddr_off = 0;
bool is_v6;
if (eth + 1 > data_end)
return XDP_DROP;
__builtin_memcpy(&saved_eth, eth, sizeof(saved_eth));
is_v6 = (saved_eth.h_proto == bpf_htons(ETH_P_IPV6));
saddr_off = sizeof(struct ethhdr) + (is_v6 ? offsetof(struct ipv6hdr, saddr) :
offsetof(struct iphdr, saddr));
if (flow_mask)
rand_state = bpf_get_prandom_u32() | 1;
if (cold_lru) {
__u32 *saddr = data + saddr_off;
batch_gen++;
batch_hash = (batch_gen ^ bpf_get_smp_processor_id()) * KNUTH_HASH_MULT;
if ((void *)(saddr + 1) <= data_end)
*saddr ^= batch_hash;
}
return BENCH_BPF_LOOP(
process_packet(xdp),
({
if (__bench_result == XDP_TX) {
if (strip_encap(xdp, &saved_eth))
return XDP_DROP;
if (rand_state)
randomize_src(xdp, saddr_off, &rand_state);
}
if (cold_lru) {
void *d = (void *)(long)xdp->data;
void *de = (void *)(long)xdp->data_end;
__u32 *__sa = d + saddr_off;
if ((void *)(__sa + 1) <= de)
*__sa ^= batch_hash;
}
})
);
}
char _license[] SEC("license") = "GPL";

112
tools/testing/selftests/bpf/xdp_lb_bench_common.h Normal file
View File

@ -0,0 +1,112 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#ifndef XDP_LB_BENCH_COMMON_H
#define XDP_LB_BENCH_COMMON_H
#define F_IPV6 (1 << 0)
#define F_LRU_BYPASS (1 << 1)
#define CH_RING_SIZE 65537 /* per-VIP consistent hash ring slots */
#define MAX_VIPS 16
#define CH_RINGS_SIZE (MAX_VIPS * CH_RING_SIZE)
#define MAX_REALS 512
#define DEFAULT_LRU_SIZE 100000 /* connection tracking cache size */
#define ONE_SEC 1000000000U /* 1 sec in nanosec */
#define MAX_CONN_RATE 100000000 /* high enough to never trigger in bench */
#define LRU_UDP_TIMEOUT 30000000000ULL /* 30 sec in nanosec */
#define PCKT_FRAGMENTED 0x3FFF
#define KNUTH_HASH_MULT 2654435761U
#define IPIP_V4_PREFIX 4268 /* 172.16/12 in network order */
#define IPIP_V6_PREFIX1 1 /* 0100::/64 (RFC 6666 discard) */
#define IPIP_V6_PREFIX2 0
#define IPIP_V6_PREFIX3 0
/* Stats indices (0..MAX_VIPS-1 are per-VIP packet/byte counters) */
#define STATS_LRU (MAX_VIPS + 0) /* v1: total VIP packets, v2: LRU misses */
#define STATS_XDP_TX (MAX_VIPS + 1)
#define STATS_XDP_PASS (MAX_VIPS + 2)
#define STATS_XDP_DROP (MAX_VIPS + 3)
#define STATS_NEW_CONN (MAX_VIPS + 4) /* v1: conn count, v2: last reset ts */
#define STATS_LRU_MISS (MAX_VIPS + 5) /* v1: TCP LRU misses */
#define STATS_SIZE (MAX_VIPS + 6)
#ifdef __BPF__
#define lb_htons(x) bpf_htons(x)
#define LB_INLINE static __always_inline
#else
#define lb_htons(x) htons(x)
#define LB_INLINE static inline
#endif
LB_INLINE __be32 create_encap_ipv4_src(__u16 port, __be32 src)
{
__u32 ip_suffix = lb_htons(port);
ip_suffix <<= 16;
ip_suffix ^= src;
return (0xFFFF0000 & ip_suffix) | IPIP_V4_PREFIX;
}
LB_INLINE void create_encap_ipv6_src(__u16 port, __be32 src, __be32 *saddr)
{
saddr[0] = IPIP_V6_PREFIX1;
saddr[1] = IPIP_V6_PREFIX2;
saddr[2] = IPIP_V6_PREFIX3;
saddr[3] = src ^ port;
}
struct flow_key {
union {
__be32 src;
__be32 srcv6[4];
};
union {
__be32 dst;
__be32 dstv6[4];
};
union {
__u32 ports;
__u16 port16[2];
};
__u8 proto;
__u8 pad[3];
};
struct vip_definition {
union {
__be32 vip;
__be32 vipv6[4];
};
__u16 port;
__u8 proto;
__u8 pad;
};
struct vip_meta {
__u32 flags;
__u32 vip_num;
};
struct real_pos_lru {
__u32 pos;
__u64 atime;
};
struct real_definition {
__be32 dst;
__be32 dstv6[4];
__u8 flags;
};
struct lb_stats {
__u64 v1;
__u64 v2;
};
struct ctl_value {
__u8 mac[6];
__u8 pad[2];
};
#endif /* XDP_LB_BENCH_COMMON_H */