github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-10-25 01:05:03 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph
ff6f5157e0
linux/include/net
History
Hamza Mahfooz 259c4a5191 net: mana: reserve RX buffer headroom to fix forwarding performance 
Commit 730ff06d3f ("net: mana: Use page pool fragments for RX buffers
instead of full pages to improve memory efficiency.") started handing
out RX buffers with zero headroom so that two buffers fit into one page
at the default MTU.

The MANA TX path, however, stores the per scatter-gather entry DMA
mappings in `struct mana_skb_head` at skb->head, and mana_start_xmit()
therefore calls skb_cow_head(skb, MANA_HEADROOM). The port advertises
this requirement as ndev->needed_headroom = MANA_HEADROOM.

As a result every packet that is received and then forwarded out of a
MANA port fails the skb_cow() in ip_forward() and gets reallocated and
copied by pskb_expand_head(). This is invisible to a plain RX or TX
workload, but it puts a full skb reallocation plus memcpy on the hot
path of every single forwarded packet, which is exactly what a
router/NVA workload does.

Restore the headroom. Note that reserving MANA_HEADROOM (232) is not
enough: ip_forward() asks for LL_RESERVED_SPACE(dev), which rounds
hard_header_len + needed_headroom up to HH_DATA_MOD and is 256 bytes on
ethernet. Use LL_RESERVED_SPACE() directly so the value keeps tracking
both constants. Also, since LL_RESERVED_SPACE() tracks MANA_HEADROOM,
it grows with MAX_SKB_FRAGS and for MAX_SKB_FRAGS >= 19 it is greater
than 256, so we have to account for that by using the headroom the RX
queue actually uses (instead of assuming XDP_PACKET_HEADROOM) and
turning MANA_XDP_MTU_MAX into MANA_XDP_MTU_MAX(ndev) (note that at the
default CONFIG_MAX_SKB_FRAGS=17 they are equivalent).

At the default MTU on a 4K page this means a buffer no longer fits twice
into a page (SKB_DATA_ALIGN(1500 + MANA_RXBUF_PAD + 256) = 2112), so the
frag-vs-single decision is now made by computing the real buffer size
instead of comparing the MTU against PAGE_SIZE / 2. The page_pool
fragment path is still used wherever at least two buffers genuinely fit,
e.g. on 16K and 64K page sizes.

Measured on an Azure VM with a MANA NIC acting as a forwarding NVA (UDP,
1400 byte payload, 4 streams, 8 Gbps offered, only the forwarding
node's kernel differs), 8 runs each, median:

                  forwarded pps    throughput
  before              272,830       3.06 Gbps
  after               390,560       4.37 Gbps   (+43%)

perf on the forwarding node, same workload:

                  memset_orig   __pi_memcpy   pskb_expand_head
  before             10.07%         3.96%         present
  after               0.94%         0.64%         gone

Cc: stable@vger.kernel.org
Fixes: 730ff06d3f ("net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency.")
Signed-off-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
Reviewed-by: Simon Horman <horms@kernel.org>
Link: https://patch.msgid.link/20261003013647.2051416-1-hamzamahfooz@linux.microsoft.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-10-08 10:42:03 -07:00
..
9p 9p: Cache negative dentries for lookup performance 2026-06-21 05:22:56 +00:00
bluetooth Bluetooth: RFCOMM: connect the session socket without rfcomm_mutex 2026-09-29 10:44:07 -04:00
iucv treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
libeth xsk: honor XDP_TX_METADATA in zero-copy path 2026-08-24 11:12:00 -07:00
mana net: mana: reserve RX buffer headroom to fix forwarding performance 2026-10-08 10:42:03 -07:00
netfilter netfilter pull request 26-09-30 2026-10-01 12:00:40 +02:00
netns ipv4: fib: Define fib_table_hash_lock under CONFIG_IP_MULTIPLE_TABLES. 2026-07-07 10:51:06 +02:00
nfc nfc: fix use-after-free in nfc_get_local_general_bytes 2026-09-23 22:13:17 +02:00
page_pool net: make page_pool_get_stats() void 2026-05-28 18:10:03 -07:00
phonet phonet: Convert phonet_routes.lock to spinlock_t. 2024-10-24 16:03:40 +02:00
phy net: phy: realtek: add dummy PHY driver for RTL8127ATF 2026-01-12 19:29:11 -08:00
psp psp: add new netlink cmd for dev-assoc and dev-disassoc 2026-06-12 18:31:32 -07:00
sctp sctp: fix stream->outcnt underflow on duplicate RECONF responses 2026-08-25 13:33:08 +02:00
tc_act net/sched: act_pedit: fix TOCTOU heap OOB write in tc offload 2026-07-06 12:52:01 +02:00
6lowpan.h
act_api.h net/sched: act_gact, act_police: range check the fallback control action 2026-08-10 16:00:19 -07:00
addrconf.h ipv6: prevent in6_dev_get() from resurrecting inet6_dev 2026-08-04 14:42:45 -07:00
af_ieee802154.h
af_rxrpc.h rxrpc: Remove deadcode 2025-04-24 17:03:45 -07:00
af_unix.h net: af_unix: useful handling of LSM denials on SCM_RIGHTS 2026-08-17 18:14:52 -07:00
af_vsock.h vsock/virtio: validate packet source for connected sockets 2026-08-31 16:50:54 -07:00
ah.h
aligned_data.h udp: move udp_memory_allocated into net_aligned_data 2025-07-02 14:22:02 -07:00
amt.h amt: send the relay's General Query directly from the receive path 2026-10-01 18:11:09 -07:00
arp.h
ax25.h net: remove ax25 and amateur radio (hamradio) subsystem 2026-04-23 10:24:02 -07:00
ax88796.h
bareudp.h
bond_3ad.h bonding: no longer rely on RTNL in bond_fill_info() 2026-06-30 17:00:03 -07:00
bond_alb.h bonding: Correct spelling in headers 2024-08-26 09:37:22 -07:00
bond_options.h bonding: 3ad: add lacp_strict configuration knob 2026-06-10 07:53:26 -07:00
bonding.h bonding: no longer rely on RTNL in bond_fill_info() 2026-06-30 17:00:03 -07:00
bpf_sk_storage.h
busy_poll.h net: gro: decouple GRO from the NAPI layer 2025-02-27 14:03:14 +01:00
calipso.h move asm/unaligned.h to linux/unaligned.h 2024-10-02 17:23:23 -04:00
can.h can: fix unique skb identifier regression under RPS 2026-10-01 17:05:03 +02:00
cfg80211-wext.h
cfg80211.h wifi: cfg80211: convert tx_control_port cookie to input parameter 2026-08-02 17:40:08 +02:00
cfg802154.h mac802154: fix use-after-free of sdata via queued RX frames 2026-09-03 11:00:54 +02:00
checksum.h net: Fix checksum update for ILA adj-transport 2025-05-30 19:53:51 -07:00
cipso_ipv4.h move asm/unaligned.h to linux/unaligned.h 2024-10-02 17:23:23 -04:00
cls_cgroup.h net/cls_cgroup: Fix task_get_classid() during qdisc run 2025-09-14 11:55:04 -07:00
codel_impl.h net/sched: codel: bound the dropping loop per dequeue call 2026-09-16 17:15:29 -07:00
codel_qdisc.h
codel.h net/sched: codel: bound the dropping loop per dequeue call 2026-09-16 17:15:29 -07:00
compat.h
datalink.h
dcbevent.h
dcbnl.h
devlink.h devlink: add generic device max_sfs parameter 2026-08-11 18:06:30 -07:00
dropreason-core.h ipv6: Implement limits on extension header parsing 2026-04-30 17:21:45 -07:00
dropreason-qdisc.h net: sched: sch_dualpi2: use qdisc_dequeue_drop() for dequeue drops 2026-02-28 15:31:35 -08:00
dropreason.h net: sched: introduce qdisc-specific drop reason tracing 2026-02-28 15:31:34 -08:00
dsa_stubs.h
dsa.h net: dsa: initial MT7628 tagging driver 2026-08-18 15:50:13 +02:00
dscp.h
dsfield.h
dst_cache.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
dst_metadata.h net: dst_metadata: fix false-positive memcpy overflow in tun_dst_unclone 2026-06-21 14:33:05 -07:00
dst_ops.h
dst.h net: ipv6: keep room for the mac header in dst_dev_overhead() 2026-09-23 19:04:47 -07:00
eee.h net: simplify eeecfg_mac_can_tx_lpi 2024-11-13 18:49:50 -08:00
erspan.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
esp.h
espintcp.h
ethoc.h
failover.h
fib_notifier.h net: do not acquire rtnl in fib_seq_sum() 2024-10-11 15:35:05 -07:00
fib_rules.h net: fib_rules: Only hold RTNL for the first IPv4 RTM_NEWRULE. 2026-07-01 18:42:34 +02:00
firewire.h
flow_dissector.h
flow_offload.h net: dsa: eliminate local type for tc policers 2026-02-10 15:30:11 +01:00
flow.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
fou.h
fq_impl.h Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
fq.h
garp.h
gen_stats.h net/sched: add const qualifiers to gnet_stats helpers 2026-05-11 18:28:08 -07:00
genetlink.h genetlink: free the skb on 'group >= family->n_mcgrps' 2026-05-08 15:43:29 -07:00
geneve.h net: geneve: remove unused geneve_dev_create_fb 2026-08-06 15:09:23 +02:00
gre.h net: gre: remove unused gretap_fb_dev_create 2026-08-06 15:09:23 +02:00
gro_cells.h
gro.h gro: inline tcp6_gro_complete() 2026-01-21 19:28:32 -08:00
gso.h net: gso: limit recursive IP-in-IP segmentation 2026-09-29 12:02:37 +02:00
gtp.h
gue.h net: gue: reject invalid REMCSUM offsets 2026-09-18 16:44:08 -07:00
handshake.h
hotdata.h net-sysfs: use rps_tag_ptr and remove metadata from rps_sock_flow_table 2026-03-04 16:54:09 -08:00
hwbm.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
icmp.h ipv4: icmp: Pass IPv4 control block structure as an argument to __icmp_send() 2025-09-11 12:22:38 +02:00
ieee8021q.h
ieee80211_radiotap.h wifi: radiotap: add definitions for the new UHR TLVs 2026-07-06 12:41:44 +02:00
ieee802154_netdev.h
if_inet6.h ipv6: mcast: use copy-on-write RCU updates in ip6_mc_source() 2026-08-31 17:54:45 -07:00
ife.h
inet_common.h net: remove addr_len argument of recvmsg() handlers 2026-03-02 18:17:17 -08:00
inet_connection_sock.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2026-02-26 10:23:00 -08:00
inet_dscp.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
inet_ecn.h tcp: ECT_1_NEGOTIATION and NEEDS_ACCECN identifiers 2026-02-03 15:13:24 +01:00
inet_frag.h inet: frags: flush pending skbs in fqdir_pre_exit() 2025-12-10 01:15:27 -08:00
inet_hashtables.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2026-03-26 12:09:57 -07:00
inet_sock.h ipv6: colocate inet6_cork in inet_cork_full 2026-02-02 17:49:30 -08:00
inet_timewait_sock.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet6_connection_sock.h tcp: move inet6_csk_update_pmtu() to tcp_ipv6.c 2026-02-24 17:47:27 -08:00
inet6_hashtables.h tcp: Initialise ehash secrets during connect() and listen(). 2026-03-05 18:50:05 -08:00
inetpeer.h inetpeer: randomize RB-tree node comparison using SipHash 2026-08-20 12:55:31 -07:00
ioam6.h ipv6: ioam: fix heap buffer overflow in __ioam6_fill_trace_data() 2026-02-13 12:24:05 -08:00
ip_fib.h ipv4: fib: Drop RTNL annotation for net->ipv4.fib_table_hash[]. 2026-07-01 18:42:33 +02:00
ip_tunnels.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2026-08-18 10:42:41 -07:00
ip_vs.h netfilter pull request 26-08-10 2026-08-14 12:23:12 -07:00
ip.h raw: annotate disconnect-side IPv4 match writers 2026-08-30 14:14:57 -07:00
ip6_checksum.h udp: move udp6_csum_init() back to net/ipv6/udp.c 2026-02-24 16:30:40 -08:00
ip6_fib.h ipv6: add inet6_rt_del_notify() 2026-08-13 12:30:27 +02:00
ip6_route.h ipv6: Fix dst leak for uncached routes. 2026-09-21 18:12:42 -07:00
ip6_tunnel.h net: dropreason: add SKB_DROP_REASON_RECURSION_LIMIT 2026-03-14 08:38:06 -07:00
ipcomp.h xfrm: ipcomp: Use crypto_acomp interface 2025-03-21 17:36:49 +08:00
ipconfig.h
ipv6_frag.h inet: frags: flush pending skbs in fqdir_pre_exit() 2025-12-10 01:15:27 -08:00
ipv6.h tcp: rehash onto different local ECMP path on retransmit timeout 2026-06-15 15:57:31 -07:00
iw_handler.h Revert "wifi: cfg80211: unexport wireless_nlevent_flush()" 2024-10-09 08:53:01 +02:00
kcm.h net: kcm: Fix race condition in kcm_unattach() 2025-08-13 18:18:33 -07:00
l3mdev.h net: l3mdev: use skb_dst_dev_rcu() in l3mdev_l3_out() 2026-02-02 17:09:11 -08:00
lag.h
lapb.h net: lapb: increase LAPB_HEADER_LEN 2024-12-06 17:43:08 -08:00
llc_c_ac.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_c_ev.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_c_st.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_conn.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_if.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_pdu.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_s_ac.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_s_ev.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_s_st.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc_sap.h llc: Add SPDX id lines to llc header files 2026-05-26 17:27:14 -07:00
llc.h llc: Add SPDX id lines to some llc source files 2026-05-26 17:26:57 -07:00
lwtunnel.h net: dst: annotate data-races around dst->output 2025-07-02 14:32:30 -07:00
mac80211.h wifi: mac80211: don't allow injecting frames wider than the chanctx 2026-09-09 14:51:30 +02:00
mac802154.h move asm/unaligned.h to linux/unaligned.h 2024-10-02 17:23:23 -04:00
macsec.h macsec: use rcu_work to defer TX SA crypto cleanup out of softirq 2026-05-13 19:03:05 -07:00
mctp.h net: mctp: fix don't require received header reserved bits to be zero 2026-04-20 11:46:57 -07:00
mctpdevice.h net: mctp: Expose transport binding identifier via IFLA attribute 2024-11-09 09:04:54 -08:00
mip6.h
mld.h ipv6: mld: encode multicast exponential fields 2026-05-04 19:09:41 -07:00
mpls_iptunnel.h
mpls.h
mptcp.h mptcp: remove unused data_ack from struct mptcp_ext 2026-08-17 17:25:49 -07:00
mrp.h
ncsi.h net: ncsi: Set ncsi_stop_dev() to inline while NET_NCSI not enabled 2026-06-09 17:18:51 -07:00
ndisc.h ipv6: ndisc: Add ndisc_check_ns_na() validation helper 2026-08-07 16:32:46 -07:00
neighbour_tables.h neighbour: Create netdev->neighbour association 2024-11-09 13:22:57 -08:00
neighbour.h netfilter: br_netfilter: Reallocate headroom if necessary in neigh_hh_bridge() 2026-05-16 13:22:50 +02:00
net_debug.h Kbuild updates for v6.13 2024-11-30 13:41:50 -08:00
net_failover.h
net_namespace.h net: enforce net sysctl registration 2026-08-13 13:12:21 +02:00
net_ratelimit.h
net_shaper.h net_shaper: fix kernel-doc list indentation 2026-08-17 13:54:16 -07:00
net_trackers.h
netdev_lock.h net: ethtool: make dev->hwprov ops-protected 2026-06-09 10:13:04 -07:00
netdev_netlink.h net: add granular lock for the netdev netlink socket 2025-03-12 13:32:35 -07:00
netdev_queues.h net: Proxy netdev_queue_get_dma_dev for leased queues 2026-04-09 18:21:46 -07:00
netdev_rx_queue.h net: remove the netif_get_rx_queue_lease_locked() helpers 2026-04-09 18:26:28 -07:00
netevent.h
netkit.h
netlabel.h Networking changes for 6.13. 2024-11-21 08:28:08 -08:00
netlink.h netlink: add one debug check in nla_nest_end() 2026-05-14 16:49:26 -07:00
netmem.h Revert "mm: introduce a new page type for page pool in page type" 2026-05-21 19:06:13 -07:00
netprio_cgroup.h
nexthop.h vxlan: honor per-nexthop fdb destination port 2026-07-30 16:09:22 -07:00
nl802154.h nl802154: fix some kernel-doc warnings 2025-10-20 17:13:40 -07:00
nsh.h net: nsh: fix incorrect header length macros 2026-05-08 15:32:59 -07:00
pfcp.h net: pfcp: fix typo in message_priority field name 2025-06-13 18:17:08 -07:00
pie.h net/sched: sch_pie: annotate data-races in pie_dump_stats() 2026-04-22 21:12:47 -07:00
ping.h net: remove addr_len argument of recvmsg() handlers 2026-03-02 18:17:17 -08:00
pkt_cls.h net/sched: Handle TC_ACT_REDIRECT from qdisc filter chains 2026-07-20 18:16:43 -07:00
pkt_sched.h net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup 2026-08-27 12:12:36 -07:00
pptp.h
proto_memory.h net: Allow opt-out from global protocol memory accounting. 2025-10-16 12:04:47 -07:00
protocol.h
psample.h
psnap.h
psp.h psp: base PSP device support 2025-09-18 12:32:06 +02:00
raw.h net: use NUMA drop counters for softnet_data.dropped 2025-09-14 11:35:17 -07:00
rawv6.h
red.h net: sched: Correct spelling in headers 2024-08-26 09:37:23 -07:00
regulatory.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
request_sock.h tcp: move __reqsk_free() out of line 2026-02-05 09:23:06 -08:00
rose.h net: remove ax25 and amateur radio (hamradio) subsystem 2026-04-23 10:24:02 -07:00
route.h ipv4: fix use-after-free in fib_nhc_update_mtu() 2026-08-11 11:25:10 +02:00
rpl.h
rps-types.h net: add rps_tag_ptr type and helpers 2026-03-04 16:54:09 -08:00
rps.h net-sysfs: use rps_tag_ptr and remove metadata from rps_dev_flow_table 2026-03-04 16:54:10 -08:00
rsi_91x.h
rstreason.h net: Retire DCCP socket. 2025-04-11 18:58:10 -07:00
rtnetlink.h net: ip_gre: require CAP_NET_ADMIN in the device netns for changelink 2026-06-17 16:01:52 -07:00
rtnh.h
sch_generic.h net: cap skb->queue_mapping when the tx queue is picked 2026-09-29 17:49:35 -07:00
sch_priv.h net/sched: Export mq functions for reuse 2026-01-13 11:54:29 +01:00
scm.h net: af_unix: useful handling of LSM denials on SCM_RIGHTS 2026-08-17 18:14:52 -07:00
secure_seq.h tcp: secure_seq: add back ports to TS offset 2026-03-04 17:44:35 -08:00
seg6_hmac.h ipv6: sr: Prepare HMAC key ahead of time 2025-08-26 18:11:29 -07:00
seg6_local.h
seg6.h
selftests.h net: selftests: export packet creation helpers for driver use 2025-11-06 13:38:11 +01:00
slhc_vj.h
smc.h smc: Use flexible array for SMCD connections 2026-05-20 17:13:18 -07:00
snmp.h net: snmp: remove SNMP_MIB_SENTINEL 2025-09-08 18:06:21 -07:00
sock_reuseport.h net: core: annotate socks of struct sock_reuseport with __counted_by 2024-08-02 17:16:59 -07:00
sock.h net: psp: avoid conflicts with skb->decrypted and sk_validate_xmit_skb() 2026-09-16 19:18:24 -07:00
stp.h
strparser.h strparser: make sure __strp_recv isn't running before tearing down the parser 2026-10-06 18:29:12 -07:00
switchdev.h bridge: No DEV_PATH_BR_VLAN_UNTAG_HW for dsa foreign 2026-03-19 13:14:00 +01:00
tc_wrapper.h net/sched: refine indirect call mitigation in tc_wrapper.h 2026-03-09 19:31:41 -07:00
tcp_ao.h tcp: restore RCU grace period in tcp_ao_destroy_sock 2026-06-29 18:14:30 -07:00
tcp_ecn.h tcp: annotate data-races around tp->delivered and tp->delivered_ce 2026-04-18 11:10:12 -07:00
tcp_states.h
tcp.h tcp: prevent collapsing skbs across boundary in rtx queue 2026-09-24 11:03:42 -07:00
tcx.h bpf: Remove location field in tcx_link 2025-07-11 11:00:57 -07:00
timewait_sock.h tcp: Remove timewait_sock_ops.twsk_destructor(). 2025-08-25 17:53:35 -07:00
tipc.h
tls_prot.h
tls.h tls: remove tls_toe and the related driver 2026-06-12 16:43:10 -07:00
transp_v6.h ipv6: Retire UDP-Lite. 2026-03-13 18:57:44 -07:00
tso.h net: tso: Introduce tso_dma_map and helpers 2026-04-12 10:54:31 -07:00
tun_proto.h
udp_tunnel.h udp_tunnel: Pass struct sock to udp_tunnel_notify_{add,del}_rx_port(). 2026-05-05 17:47:04 -07:00
udp.h udp: convert udp_lib_getsockopt to sockopt_t 2026-07-02 12:29:06 +02:00
vsock_addr.h net: Convert proto_ops connect() callbacks to use sockaddr_unsized 2025-11-04 19:10:32 -08:00
vxlan.h net: vxlan: remove unused vxlan_dev_create 2026-08-06 15:09:23 +02:00
wext.h
x25.h net/x25: Remove unused x25_terminate_link() 2025-07-14 17:19:13 -07:00
x25device.h
xdp_priv.h
xdp_sock_drv.h xsk: honor XDP_TX_METADATA in zero-copy path 2026-08-24 11:12:00 -07:00
xdp_sock.h xsk: move xsk_tx_metadata_request() to xdp_sock_drv.h 2026-08-03 16:43:54 -07:00
xdp.h net: mvneta: clear XDP pfmemalloc flag between frames 2026-09-30 17:39:11 -07:00
xfrm.h xfrm: cache the offload ifindex for netlink dumps 2026-07-02 09:12:58 +02:00
xsk_buff_pool.h xsk: honor XDP_TX_METADATA in zero-copy path 2026-08-24 11:12:00 -07:00