linux/net
Eric Dumazet b650bf0977 udp: remove busylock and add per NUMA queues
busylock was protecting UDP sockets against packet floods,
but unfortunately was not protecting the host itself.

Under stress, many cpus could spin while acquiring the busylock,
and NIC had to drop packets. Or packets would be dropped
in cpu backlog if RPS/RFS were in place.

This patch replaces the busylock by intermediate
lockless queues. (One queue per NUMA node).

This means that fewer number of cpus have to acquire
the UDP receive queue lock.

Most of the cpus can either:
- immediately drop the packet.
- or queue it in their NUMA aware lockless queue.

Then one of the cpu is chosen to process this lockless queue
in a batch.

The batch only contains packets that were cooked on the same
NUMA node, thus with very limited latency impact.

Tested:

DDOS targeting a victim UDP socket, on a platform with 6 NUMA nodes
(Intel(R) Xeon(R) 6985P-C)

Before:

nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1004179            0.0
Udp6InErrors                    3117               0.0
Udp6RcvbufErrors                3117               0.0

After:
nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1116633            0.0
Udp6InErrors                    14197275           0.0
Udp6RcvbufErrors                14197275           0.0

We can see this host can now proces 14.2 M more packets per second
while under attack, and the victim socket can receive 11 % more
packets.

I used a small bpftrace program measuring time (in us) spent in
__udp_enqueue_schedule_skb().

Before:

@udp_enqueue_us[398]:
[0]                24901 |@@@                                                 |
[1]                63512 |@@@@@@@@@                                           |
[2, 4)            344827 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            244673 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)            54022 |@@@@@@@@                                            |
[16, 32)          222134 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                   |
[32, 64)          232042 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[64, 128)           4219 |                                                    |
[128, 256)           188 |                                                    |

After:

@udp_enqueue_us[398]:
[0]              5608855 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]              1111277 |@@@@@@@@@@                                          |
[2, 4)            501439 |@@@@                                                |
[4, 8)            102921 |                                                    |
[8, 16)            29895 |                                                    |
[16, 32)           43500 |                                                    |
[32, 64)           31552 |                                                    |
[64, 128)            979 |                                                    |
[128, 256)            13 |                                                    |

Note that the remaining bottleneck for this platform is in
udp_drops_inc() because we limited struct numa_drop_counters
to only two nodes so far.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20250922104240.2182559-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-09-23 16:38:39 -07:00
..
6lowpan
9p
802
8021q
appletalk
atm net: atm: fix memory leak in atm_register_sysfs when device_register fail 2025-09-04 09:53:44 +02:00
ax25 ax25: properly unshare skbs in ax25_kiss_rcv() 2025-09-03 17:06:30 -07:00
batman-adv batman-adv: remove includes for extern declarations 2025-09-05 15:11:02 +02:00
bluetooth
bpf
bridge net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
caif
can can: j1939: j1939_local_ecu_get(): undo increment when j1939_local_ecu_get() fails 2025-09-10 17:12:05 +02:00
ceph net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
core net: move sk->sk_err_soft and sk->sk_sndbuf 2025-09-22 17:55:24 -07:00
dcb
devlink net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
dns_resolver
dsa
ethernet ethernet: Extend device_get_mac_address() to use NVMEM 2025-09-15 18:34:08 -07:00
ethtool Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-18 11:26:06 -07:00
handshake
hsr Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-11 17:40:13 -07:00
ieee802154
ife
ipv4 udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
ipv6 udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
iucv
kcm
key
l2tp
l3mdev
lapb
llc
mac80211 Plenty of things going on, notably: 2025-09-11 17:50:46 -07:00
mac802154
mctp net: mctp: fix typo in comment 2025-09-08 17:47:57 -07:00
mpls
mptcp mptcp: remove unused returned value of check_data_fin 2025-09-22 11:51:25 -07:00
ncsi
netfilter net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
netlabel
netlink Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-11 17:40:13 -07:00
netrom
nfc
nsh
openvswitch net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
packet net: af_packet: Use hrtimer to do the retire operation 2025-09-11 18:40:06 -07:00
phonet
psample
psp psp: clarify checksum behavior of psp_dev_rcv() 2025-09-19 17:02:57 -07:00
qrtr
rds net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
rfkill net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
rose
rxrpc net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
sched net_sched: act: remove tcfa_qstats 2025-09-02 15:52:24 -07:00
sctp sctp: snmp: do not use SNMP_MIB_SENTINEL anymore 2025-09-08 18:06:21 -07:00
shaper
smc dibs: Move event handling to dibs layer 2025-09-23 11:13:22 +02:00
strparser
sunrpc SUNRPC: call xs_sock_process_cmsg for all cmsg 2025-09-06 16:51:26 -04:00
switchdev
tipc
tls net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
unix net: replace use of system_unbound_wq with system_dfl_wq 2025-09-22 17:40:30 -07:00
vmw_vsock net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
wireless Plenty of things going on, notably: 2025-09-11 17:50:46 -07:00
x25
xdp xsk: Fix immature cq descriptor production 2025-09-09 15:09:55 -07:00
xfrm xfrm: snmp: do not use SNMP_MIB_SENTINEL anymore 2025-09-08 18:06:21 -07:00
compat.c
devres.c
Kconfig dibs: Create drivers/dibs 2025-09-23 11:13:21 +02:00
Kconfig.debug
Makefile psp: base PSP device support 2025-09-18 12:32:06 +02:00
socket.c
sysctl_net.c