linux/net
Eric Dumazet b650bf0977 udp: remove busylock and add per NUMA queues
busylock was protecting UDP sockets against packet floods,
but unfortunately was not protecting the host itself.

Under stress, many cpus could spin while acquiring the busylock,
and NIC had to drop packets. Or packets would be dropped
in cpu backlog if RPS/RFS were in place.

This patch replaces the busylock by intermediate
lockless queues. (One queue per NUMA node).

This means that fewer number of cpus have to acquire
the UDP receive queue lock.

Most of the cpus can either:
- immediately drop the packet.
- or queue it in their NUMA aware lockless queue.

Then one of the cpu is chosen to process this lockless queue
in a batch.

The batch only contains packets that were cooked on the same
NUMA node, thus with very limited latency impact.

Tested:

DDOS targeting a victim UDP socket, on a platform with 6 NUMA nodes
(Intel(R) Xeon(R) 6985P-C)

Before:

nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1004179            0.0
Udp6InErrors                    3117               0.0
Udp6RcvbufErrors                3117               0.0

After:
nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1116633            0.0
Udp6InErrors                    14197275           0.0
Udp6RcvbufErrors                14197275           0.0

We can see this host can now proces 14.2 M more packets per second
while under attack, and the victim socket can receive 11 % more
packets.

I used a small bpftrace program measuring time (in us) spent in
__udp_enqueue_schedule_skb().

Before:

@udp_enqueue_us[398]:
[0]                24901 |@@@                                                 |
[1]                63512 |@@@@@@@@@                                           |
[2, 4)            344827 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            244673 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)            54022 |@@@@@@@@                                            |
[16, 32)          222134 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                   |
[32, 64)          232042 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[64, 128)           4219 |                                                    |
[128, 256)           188 |                                                    |

After:

@udp_enqueue_us[398]:
[0]              5608855 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]              1111277 |@@@@@@@@@@                                          |
[2, 4)            501439 |@@@@                                                |
[4, 8)            102921 |                                                    |
[8, 16)            29895 |                                                    |
[16, 32)           43500 |                                                    |
[32, 64)           31552 |                                                    |
[64, 128)            979 |                                                    |
[128, 256)            13 |                                                    |

Note that the remaining bottleneck for this platform is in
udp_drops_inc() because we limited struct numa_drop_counters
to only two nodes so far.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20250922104240.2182559-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-09-23 16:38:39 -07:00
..
6lowpan net: replace ND_PRINTK with dynamic debug 2025-07-10 15:27:32 -07:00
9p netfs: Fix the request's work item to not require a ref 2025-05-21 14:35:20 +02:00
802 treewide, timers: Rename from_timer() to timer_container_of() 2025-06-08 09:07:37 +02:00
8021q net: s/dev_close_many/netif_close_many/ 2025-07-18 17:27:47 -07:00
appletalk Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-07-24 11:10:46 -07:00
atm net: atm: fix memory leak in atm_register_sysfs when device_register fail 2025-09-04 09:53:44 +02:00
ax25 ax25: properly unshare skbs in ax25_kiss_rcv() 2025-09-03 17:06:30 -07:00
batman-adv batman-adv: remove includes for extern declarations 2025-09-05 15:11:02 +02:00
bluetooth Bluetooth: Fix use-after-free in l2cap_sock_cleanup_listen() 2025-08-29 14:51:06 -04:00
bpf bpf: Add attach_type field to bpf_link 2025-07-11 10:51:55 -07:00
bridge net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
caif caif: Replace memset(0) + strscpy() with strscpy_pad() 2025-08-12 14:08:56 -07:00
can can: j1939: j1939_local_ecu_get(): undo increment when j1939_local_ecu_get() fails 2025-09-10 17:12:05 +02:00
ceph net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
core net: move sk->sk_err_soft and sk->sk_sndbuf 2025-09-22 17:55:24 -07:00
dcb
devlink net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
dns_resolver
dsa net: s/dev_close_many/netif_close_many/ 2025-07-18 17:27:47 -07:00
ethernet ethernet: Extend device_get_mac_address() to use NVMEM 2025-09-15 18:34:08 -07:00
ethtool Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-18 11:26:06 -07:00
handshake net/handshake: Add new parameter 'HANDSHAKE_A_ACCEPT_KEYRING' 2025-07-08 15:31:44 +02:00
hsr Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-11 17:40:13 -07:00
ieee802154 treewide, timers: Rename from_timer() to timer_container_of() 2025-06-08 09:07:37 +02:00
ife
ipv4 udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
ipv6 udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
iucv net: add sk_drops_read(), sk_drops_inc() and sk_drops_reset() helpers 2025-08-28 13:14:50 +02:00
kcm net: kcm: Fix race condition in kcm_unattach() 2025-08-13 18:18:33 -07:00
key Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-07-24 11:10:46 -07:00
l2tp l2tp: do not use sock_hold() in pppol2tp_session_get_sock() 2025-08-27 17:16:13 -07:00
l3mdev net: fib_rules: Fix iif / oif matching on L3 master device 2025-04-15 17:54:56 -07:00
lapb treewide, timers: Rename from_timer() to timer_container_of() 2025-06-08 09:07:37 +02:00
llc net: make sk->sk_rcvtimeo lockless 2025-06-23 17:05:12 -07:00
mac80211 Plenty of things going on, notably: 2025-09-11 17:50:46 -07:00
mac802154 mac802154: Switch to use hrtimer_setup() 2025-02-18 10:35:44 +01:00
mctp net: mctp: fix typo in comment 2025-09-08 17:47:57 -07:00
mpls net: s/dev_get_flags/netif_get_flags/ 2025-07-18 17:27:47 -07:00
mptcp mptcp: remove unused returned value of check_data_fin 2025-09-22 11:51:25 -07:00
ncsi net: ncsi: Fix buffer overflow in fetching version id 2025-06-12 18:21:59 -07:00
netfilter net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
netlabel calipso: unlock rcu before returning -EAFNOSUPPORT 2025-06-05 08:03:38 -07:00
netlink Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-11 17:40:13 -07:00
netrom treewide, timers: Rename from_timer() to timer_container_of() 2025-06-08 09:07:37 +02:00
nfc Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-06-19 13:00:24 -07:00
nsh
openvswitch net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
packet net: af_packet: Use hrtimer to do the retire operation 2025-09-11 18:40:06 -07:00
phonet net: add sk_drops_read(), sk_drops_inc() and sk_drops_reset() helpers 2025-08-28 13:14:50 +02:00
psample
psp psp: clarify checksum behavior of psp_dev_rcv() 2025-09-19 17:02:57 -07:00
qrtr
rds net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
rfkill net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
rose net: rose: fix a typo in rose_clear_routes() 2025-08-27 17:27:52 -07:00
rxrpc net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
sched net_sched: act: remove tcfa_qstats 2025-09-02 15:52:24 -07:00
sctp sctp: snmp: do not use SNMP_MIB_SENTINEL anymore 2025-09-08 18:06:21 -07:00
shaper
smc dibs: Move event handling to dibs layer 2025-09-23 11:13:22 +02:00
strparser net: make sk->sk_rcvtimeo lockless 2025-06-23 17:05:12 -07:00
sunrpc SUNRPC: call xs_sock_process_cmsg for all cmsg 2025-09-06 16:51:26 -04:00
switchdev net: switchdev: Convert blocking notification chain to a raw one 2025-03-11 11:30:28 +01:00
tipc net: add sk_drops_read(), sk_drops_inc() and sk_drops_reset() helpers 2025-08-28 13:14:50 +02:00
tls net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
unix net: replace use of system_unbound_wq with system_dfl_wq 2025-09-22 17:40:30 -07:00
vmw_vsock net: WQ_PERCPU added to alloc_workqueue users 2025-09-22 17:40:30 -07:00
wireless Plenty of things going on, notably: 2025-09-11 17:50:46 -07:00
x25 net/x25: Remove unused x25_terminate_link() 2025-07-14 17:19:13 -07:00
xdp xsk: Fix immature cq descriptor production 2025-09-09 15:09:55 -07:00
xfrm xfrm: snmp: do not use SNMP_MIB_SENTINEL anymore 2025-09-08 18:06:21 -07:00
compat.c
devres.c
Kconfig dibs: Create drivers/dibs 2025-09-23 11:13:21 +02:00
Kconfig.debug
Makefile psp: base PSP device support 2025-09-18 12:32:06 +02:00
socket.c net: annotate races around sk->sk_uid 2025-06-23 17:04:03 -07:00
sysctl_net.c