linux/include/net
Eric Dumazet b650bf0977 udp: remove busylock and add per NUMA queues
busylock was protecting UDP sockets against packet floods,
but unfortunately was not protecting the host itself.

Under stress, many cpus could spin while acquiring the busylock,
and NIC had to drop packets. Or packets would be dropped
in cpu backlog if RPS/RFS were in place.

This patch replaces the busylock by intermediate
lockless queues. (One queue per NUMA node).

This means that fewer number of cpus have to acquire
the UDP receive queue lock.

Most of the cpus can either:
- immediately drop the packet.
- or queue it in their NUMA aware lockless queue.

Then one of the cpu is chosen to process this lockless queue
in a batch.

The batch only contains packets that were cooked on the same
NUMA node, thus with very limited latency impact.

Tested:

DDOS targeting a victim UDP socket, on a platform with 6 NUMA nodes
(Intel(R) Xeon(R) 6985P-C)

Before:

nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1004179            0.0
Udp6InErrors                    3117               0.0
Udp6RcvbufErrors                3117               0.0

After:
nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1116633            0.0
Udp6InErrors                    14197275           0.0
Udp6RcvbufErrors                14197275           0.0

We can see this host can now proces 14.2 M more packets per second
while under attack, and the victim socket can receive 11 % more
packets.

I used a small bpftrace program measuring time (in us) spent in
__udp_enqueue_schedule_skb().

Before:

@udp_enqueue_us[398]:
[0]                24901 |@@@                                                 |
[1]                63512 |@@@@@@@@@                                           |
[2, 4)            344827 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            244673 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)            54022 |@@@@@@@@                                            |
[16, 32)          222134 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                   |
[32, 64)          232042 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[64, 128)           4219 |                                                    |
[128, 256)           188 |                                                    |

After:

@udp_enqueue_us[398]:
[0]              5608855 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]              1111277 |@@@@@@@@@@                                          |
[2, 4)            501439 |@@@@                                                |
[4, 8)            102921 |                                                    |
[8, 16)            29895 |                                                    |
[16, 32)           43500 |                                                    |
[32, 64)           31552 |                                                    |
[64, 128)            979 |                                                    |
[128, 256)            13 |                                                    |

Note that the remaining bottleneck for this platform is in
udp_drops_inc() because we limited struct numa_drop_counters
to only two nodes so far.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20250922104240.2182559-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-09-23 16:38:39 -07:00
..
9p
bluetooth Bluetooth: hci_sync: fix set_local_name race condition 2025-08-22 13:57:31 -04:00
caif
iucv
libeth xdp, libeth: make the xdp_init_buff() micro-optimization generic 2025-09-08 10:26:25 -07:00
mana net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency. 2025-08-19 14:42:44 +02:00
netfilter Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-11 17:40:13 -07:00
netns tcp: accecn: AccECN option send control 2025-09-18 08:47:52 +02:00
nfc net: nfc: nci: Increase NCI_DATA_TIMEOUT to 3000 ms 2025-09-03 17:02:12 -07:00
page_pool net: add helper to pre-check if PP for an Rx queue will be unreadable 2025-09-04 10:19:17 +02:00
phonet
psp psp: Fix typo in kdoc for struct psp_dev_caps.assoc_drv_spc. 2025-09-19 17:02:27 -07:00
sctp sctp: Convert cookie authentication to use HMAC-SHA256 2025-08-19 19:36:26 -07:00
tc_act net_sched: act_skbmod: use RCU in tcf_skbmod_dump() 2025-08-28 16:46:23 -07:00
6lowpan.h
act_api.h net_sched: act: remove tcfa_qstats 2025-09-02 15:52:24 -07:00
addrconf.h
af_ieee802154.h
af_rxrpc.h
af_unix.h af_unix: Introduce SO_INQ. 2025-07-08 18:05:25 -07:00
af_vsock.h vsock: fix vsock_proto declaration 2025-07-07 16:55:54 -07:00
ah.h
aligned_data.h udp: move udp_memory_allocated into net_aligned_data 2025-07-02 14:22:02 -07:00
amt.h
arp.h
atmclip.h
ax25.h
ax88796.h
bareudp.h
bond_3ad.h bonding: support aggregator selection based on port priority 2025-09-09 10:56:02 +02:00
bond_alb.h
bond_options.h bonding: add support for per-port LACP actor priority 2025-09-09 10:56:02 +02:00
bonding.h bonding: Remove support for use_carrier 2025-09-02 14:01:54 -07:00
bpf_sk_storage.h
busy_poll.h
calipso.h
cfg80211-wext.h
cfg80211.h wifi: nl80211: allow drivers to support subset of NL80211_CMD_SET_BSS 2025-09-04 11:19:02 +02:00
cfg802154.h
checksum.h net: Fix checksum update for ILA adj-transport 2025-05-30 19:53:51 -07:00
cipso_ipv4.h
cls_cgroup.h net/cls_cgroup: Fix task_get_classid() during qdisc run 2025-09-14 11:55:04 -07:00
codel_impl.h
codel_qdisc.h
codel.h
compat.h
datalink.h
dcbevent.h
dcbnl.h
devlink.h devlink: Add a 'num_doorbells' driverinit param 2025-09-17 18:30:51 -07:00
dropreason-core.h tcp: add datapath logic for PSP with inline key exchange 2025-09-18 12:32:06 +02:00
dropreason.h
dsa_stubs.h
dsa.h net: dsa: tag_brcm: add support for legacy FCS tags 2025-06-17 17:52:01 -07:00
dscp.h
dsfield.h
dst_cache.h
dst_metadata.h net: dst_metadata: fix IP_DF bit not extracted from tunnel headers 2025-09-14 14:28:12 -07:00
dst_ops.h
dst.h net: dst: introduce dst->dev_rcu 2025-08-29 19:36:31 -07:00
eee.h
erspan.h
esp.h
espintcp.h
ethoc.h
failover.h
fib_notifier.h
fib_rules.h
firewire.h
flow_dissector.h
flow_offload.h
flow.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
fou.h
fq_impl.h
fq.h
garp.h
gen_stats.h
genetlink.h genetlink: fix typo in comment 2025-09-03 15:16:49 -07:00
geneve.h
gre.h
gro_cells.h
gro.h net: Create separate gro_flush_normal function 2025-07-24 18:34:55 -07:00
gso.h
gtp.h
gue.h
handshake.h
hotdata.h
hwbm.h
icmp.h ipv4: icmp: Pass IPv4 control block structure as an argument to __icmp_send() 2025-09-11 12:22:38 +02:00
ieee8021q.h
ieee80211_radiotap.h
ieee802154_netdev.h
if_inet6.h
ife.h
inet_common.h
inet_connection_sock.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet_dscp.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
inet_ecn.h
inet_frag.h
inet_hashtables.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet_sock.h
inet_timewait_sock.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet6_connection_sock.h
inet6_hashtables.h tcp: Remove inet6_hash(). 2025-09-22 11:38:43 -07:00
inetpeer.h
ioam6.h
ip_fib.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
ip_tunnels.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
ip_vs.h ipvs: Fix estimator kthreads preferred affinity 2025-08-13 08:34:33 +02:00
ip.h net: snmp: remove SNMP_MIB_SENTINEL 2025-09-08 18:06:21 -07:00
ip6_checksum.h
ip6_fib.h
ip6_route.h ipv6: make ipv6_pinfo.daddr_cache a boolean 2025-09-18 10:17:09 +02:00
ip6_tunnel.h ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
ipcomp.h
ipconfig.h
ipv6_frag.h
ipv6_stubs.h
ipv6.h
iw_handler.h
kcm.h net: kcm: Fix race condition in kcm_unattach() 2025-08-13 18:18:33 -07:00
l3mdev.h
lag.h
lapb.h
llc_c_ac.h
llc_c_ev.h
llc_c_st.h
llc_conn.h
llc_if.h
llc_pdu.h
llc_s_ac.h
llc_s_ev.h
llc_s_st.h
llc_sap.h
llc.h
lwtunnel.h net: dst: annotate data-races around dst->output 2025-07-02 14:32:30 -07:00
mac80211.h wifi: mac80211: don't require cipher and keylen in gtk rekey 2025-07-22 10:43:19 +02:00
mac802154.h
macsec.h
mctp.h net: mctp: Allow limiting binds to a peer address 2025-07-15 12:08:39 +02:00
mctpdevice.h
mip6.h
mld.h
mpls_iptunnel.h
mpls.h
mptcp.h
mrp.h
ncsi.h
ndisc.h net: replace ND_PRINTK with dynamic debug 2025-07-10 15:27:32 -07:00
neighbour_tables.h
neighbour.h neighbour: Update pneigh_entry in pneigh_create(). 2025-07-17 16:25:22 -07:00
net_debug.h
net_failover.h
net_namespace.h
net_ratelimit.h
net_shaper.h
net_trackers.h
netdev_lock.h
netdev_netlink.h
netdev_queues.h net: add helper to pre-check if PP for an Rx queue will be unreadable 2025-09-04 10:19:17 +02:00
netdev_rx_queue.h
netevent.h
netkit.h
netlabel.h
netlink.h netlink: introduce type-checking attribute iteration for nlmsg 2025-07-02 15:39:04 -07:00
netmem.h netmem, mlx4: access ->pp_ref_count through netmem_desc instead of page 2025-07-23 17:46:54 -07:00
netprio_cgroup.h
netrom.h
nexthop.h
nl802154.h
nsh.h
pfcp.h net: pfcp: fix typo in message_priority field name 2025-06-13 18:17:08 -07:00
pie.h
ping.h inet: ping: remove ping_hash() 2025-09-01 13:15:14 -07:00
pkt_cls.h
pkt_sched.h net/sched: sch_qfq: Fix null-deref in agg_dequeue 2025-07-10 11:08:35 +02:00
pptp.h
proto_memory.h net-memcg: Pass struct sock to mem_cgroup_sk_under_memory_pressure(). 2025-08-19 19:20:59 -07:00
protocol.h
psample.h
psnap.h
psp.h psp: base PSP device support 2025-09-18 12:32:06 +02:00
raw.h net: use NUMA drop counters for softnet_data.dropped 2025-09-14 11:35:17 -07:00
rawv6.h
red.h
regulatory.h
request_sock.h tcp: reclaim 8 bytes in struct request_sock_queue 2025-09-22 17:55:25 -07:00
rose.h net: rose: convert 'use' field to refcount_t 2025-08-27 07:43:08 -07:00
route.h net: use dst_dev_rcu() in sk_setup_caps() 2025-08-29 19:36:32 -07:00
rpl.h
rps.h net: Add rfs_needed() helper 2025-09-03 15:08:20 -07:00
rsi_91x.h
rstreason.h
rtnetlink.h
rtnh.h
sch_generic.h net/sched: Fix backlog accounting in qdisc_dequeue_internal 2025-08-14 17:52:29 -07:00
scm.h af_unix/scm: fix whitespace errors 2025-07-04 09:32:35 +02:00
secure_seq.h
seg6_hmac.h ipv6: sr: Prepare HMAC key ahead of time 2025-08-26 18:11:29 -07:00
seg6_local.h
seg6.h
selftests.h
slhc_vj.h
smc.h dibs: Move event handling to dibs layer 2025-09-23 11:13:22 +02:00
snmp.h net: snmp: remove SNMP_MIB_SENTINEL 2025-09-08 18:06:21 -07:00
sock_reuseport.h
sock.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
Space.h
stp.h
strparser.h
switchdev.h
tc_wrapper.h
tcp_ao.h tcp: Free TCP-AO/TCP-MD5 info/keys without RCU 2025-09-11 19:05:56 -07:00
tcp_ecn.h tcp: accecn: AccECN option failure handling 2025-09-18 08:47:52 +02:00
tcp_states.h
tcp.h tcp: accecn: AccECN option ceb/cep and ACE field multi-wrap heuristics 2025-09-18 08:47:52 +02:00
tcx.h bpf: Remove location field in tcx_link 2025-07-11 11:00:57 -07:00
timewait_sock.h tcp: Remove timewait_sock_ops.twsk_destructor(). 2025-08-25 17:53:35 -07:00
tipc.h
tls_prot.h
tls_toe.h
tls.h
transp_v6.h
tso.h
tun_proto.h
udp_tunnel.h udp_tunnel: fix deadlock in udp_tunnel_nic_set_port_priv() 2025-06-24 16:31:36 -07:00
udp.h udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
udplite.h
vsock_addr.h
vxlan.h vxlan: Support MC routing in the underlay 2025-06-17 18:18:46 -07:00
wext.h
x25.h net/x25: Remove unused x25_terminate_link() 2025-07-14 17:19:13 -07:00
x25device.h
xdp_priv.h
xdp_sock_drv.h
xdp_sock.h net: xsk: introduce XDP_MAX_TX_SKB_BUDGET setsockopt 2025-07-10 14:48:29 +02:00
xdp.h net: xdp: handle frags with unreadable memory 2025-09-11 12:00:20 +02:00
xfrm.h Revert "xfrm: destroy xfrm_state synchronously on net exit path" 2025-07-08 13:28:29 +02:00
xsk_buff_pool.h