linux/kernel/bpf
Alexander Lobakin ed16b8a4d1 bpf: cpumap: switch to napi_skb_cache_get_bulk()
Now that cpumap uses GRO, which drops unused skb heads to the NAPI
cache, use napi_skb_cache_get_bulk() to try to reuse cached entries
and lower MM layer pressure. Always disable the BH before checking and
running the cpumap-pinned XDP prog and don't re-enable it in between
that and allocating an skb bulk, as we can access the NAPI caches only
from the BH context.
The better GRO aggregates packets, the less new skbs will be allocated.
If an aggregated skb contains 16 frags, this means 15 skbs were returned
to the cache, so next 15 skbs will be built without allocating anything.

The same trafficgen UDP GRO test now shows:

                GRO off   GRO on
threaded GRO    2.3       4         Mpps
thr bulk GRO    2.4       4.7       Mpps

diff            +4        +17       %

Comparing to the baseline cpumap:

baseline        2.7       N/A       Mpps
thr bulk GRO    2.4       4.7       Mpps
diff            -11       +74       %

Tested-by: Daniel Xu <dxu@dxuuu.xyz>
Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-02-27 14:03:39 +01:00
..
preload bpf: make preloaded map iterators to display map elements count 2023-07-06 12:42:25 -07:00
arena.c The various patchsets are summarized below. Plus of course many 2025-01-26 18:36:23 -08:00
arraymap.c bpf: Remove migrate_{disable|enable} in ->map_for_each_callback 2025-01-08 18:06:35 -08:00
bloom_filter.c bpf: Check bloom filter map value size 2024-03-27 09:56:17 -07:00
bpf_cgrp_storage.c bpf: Remove migrate_{disable|enable} from bpf_cgrp_storage_lock helpers 2025-01-08 18:06:36 -08:00
bpf_inode_storage.c bpf: Disable migration when destroying inode storage 2025-01-08 18:06:36 -08:00
bpf_iter.c [tree-wide] finally take no_llseek out 2024-09-27 08:18:43 -07:00
bpf_local_storage.c bpf: Remove migrate_{disable|enable} from bpf_selem_free() 2025-01-08 18:06:37 -08:00
bpf_lru_list.c bpf: Address KCSAN report on bpf_lru_list 2023-05-12 12:01:03 -07:00
bpf_lru_list.h bpf: lru: Remove unused declaration bpf_lru_promote() 2023-08-08 17:21:42 -07:00
bpf_lsm.c bpf, lsm: Remove getlsmprop hooks BTF IDs 2024-11-25 14:14:17 -08:00
bpf_struct_ops.c bpf: Reject struct_ops registration that uses module ptr and the module btf_id is missing 2025-01-03 10:16:46 -08:00
bpf_task_storage.c bpf: Remove migrate_{disable|enable} from bpf_task_storage_lock helpers 2025-01-08 18:06:36 -08:00
btf_iter.c bpf: Remove custom build rule 2024-08-30 08:55:26 -07:00
btf_relocate.c bpf: Remove custom build rule 2024-08-30 08:55:26 -07:00
btf.c bpf: Support selective sampling for bpf timestamping 2025-02-20 14:30:02 -08:00
cgroup_iter.c bpf: Let verifier consider {task,cgroup} is trusted in bpf_iter_reg 2023-11-07 15:24:25 -08:00
cgroup.c Summary 2024-11-22 20:36:11 -08:00
core.c bpf: fix potential error return 2024-12-10 11:17:53 -08:00
cpumap.c bpf: cpumap: switch to napi_skb_cache_get_bulk() 2025-02-27 14:03:39 +01:00
cpumask.c bpf: Remove migrate_{disable,enable} in bpf_cpumask_release() 2025-01-08 18:06:37 -08:00
crypto.c bpf: crypto: make state and IV dynptr nullable 2024-06-13 16:33:04 -07:00
devmap.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2024-12-12 14:19:05 -08:00
disasm.c bpf: add special internal-only MOV instruction to resolve per-CPU addrs 2024-04-03 10:29:55 -07:00
disasm.h bpf: Relicense disassembler as GPL-2.0-only OR BSD-2-Clause 2021-09-02 14:49:23 +02:00
dispatcher.c bpf: Add kernel symbol for struct_ops trampoline 2024-11-12 17:13:46 -08:00
hashtab.c bpf: Free element after unlock in __htab_map_lookup_and_delete_elem() 2025-01-20 09:09:01 -08:00
helpers.c bpf: Cancel the running bpf_timer through kworker for PREEMPT_RT 2025-01-20 09:09:01 -08:00
inode.c bpf: Preserve param->string when parsing mount options 2024-10-22 12:56:38 -07:00
Kconfig bpf: remove CONFIG_BPF_JIT dependency on CONFIG_MODULES of 2024-05-14 00:36:29 -07:00
kmem_cache_iter.c bpf: Add open coded version of kmem_cache iterator 2024-11-01 11:08:32 -07:00
link_iter.c bpf: Add bpf_link iterator 2022-05-10 11:20:45 -07:00
local_storage.c bpf: Replace 8 seq_puts() calls by seq_putc() calls 2024-07-29 12:53:00 -07:00
log.c bpf: Introduce support for bpf_local_irq_{save,restore} 2024-12-04 08:38:29 -08:00
lpm_trie.c bpf: Remove migrate_{disable|enable} from LPM trie 2025-01-08 18:06:35 -08:00
Makefile bpf: Avoid deadlock caused by nested kprobe and fentry bpf programs 2024-12-14 09:49:27 -08:00
map_in_map.c bpf: switch maps to CLASS(fd, ...) 2024-08-13 15:58:17 -07:00
map_in_map.h bpf: Add map and need_defer parameters to .map_fd_put_ptr() 2023-12-04 17:50:26 -08:00
map_iter.c bpf: treewide: Annotate BPF kfuncs in BTF 2024-01-31 20:40:56 -08:00
memalloc.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf 2024-11-13 12:52:51 -08:00
mmap_unlock_work.h bpf: Introduce helper bpf_find_vma 2021-11-07 11:54:51 -08:00
mprog.c bpf: Handle bpf_mprog_query with NULL entry 2023-10-06 17:11:20 -07:00
net_namespace.c net: Add includes masked by netdevice.h including uapi/bpf.h 2021-12-29 20:03:05 -08:00
offload.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2023-09-21 21:49:45 +02:00
percpu_freelist.c bpf: Initialize same number of free nodes for each pcpu_freelist 2022-11-11 12:05:14 -08:00
percpu_freelist.h bpf: Use raw_spin_trylock() for pcpu_freelist_push/pop in NMI 2020-10-06 00:04:11 +02:00
prog_iter.c bpf: Refactor bpf_iter_reg to have separate seq_info member 2020-07-25 20:16:32 -07:00
queue_stack_maps.c bpf: Avoid deadlock when using queue and stack maps from NMI 2023-09-11 19:04:49 -07:00
range_tree.c bpf: Disable migration before calling ops->map_free() 2025-01-08 18:06:36 -08:00
range_tree.h bpf: Introduce range_tree data structure and use it in bpf arena 2024-11-13 13:52:45 -08:00
relo_core.c bpf: Remove custom build rule 2024-08-30 08:55:26 -07:00
reuseport_array.c bpf: Use sockfd_put() helper 2024-08-30 08:57:47 -07:00
ringbuf.c bpf: Add MEM_WRITE attribute 2024-10-22 15:42:56 -07:00
stackmap.c bpf: wire up sleepable bpf_get_stack() and bpf_get_task_stack() helpers 2024-09-11 09:58:31 -07:00
syscall.c treewide: const qualify ctl_tables where applicable 2025-01-28 13:48:37 +01:00
sysfs_btf.c btf: Switch vmlinux BTF attribute to sysfs_bin_attr_simple_read() 2025-01-09 10:44:06 +01:00
task_iter.c vfs-6.13.file 2024-11-18 10:30:29 -08:00
tcx.c bpf, tcx: Get rid of tcx_link_const 2023-10-23 15:01:53 -07:00
tnum.c bpf: simplify tnum output if a fully known constant 2023-12-02 11:36:51 -08:00
token.c remove pointless includes of <linux/fdtable.h> 2024-10-07 13:34:41 -04:00
trampoline.c bpf: Add kernel symbol for struct_ops trampoline 2024-11-12 17:13:46 -08:00
verifier.c bpf-next-6.14 2025-01-23 08:04:07 -08:00