linux/net/ipv6
Eric Dumazet b650bf0977 udp: remove busylock and add per NUMA queues
busylock was protecting UDP sockets against packet floods,
but unfortunately was not protecting the host itself.

Under stress, many cpus could spin while acquiring the busylock,
and NIC had to drop packets. Or packets would be dropped
in cpu backlog if RPS/RFS were in place.

This patch replaces the busylock by intermediate
lockless queues. (One queue per NUMA node).

This means that fewer number of cpus have to acquire
the UDP receive queue lock.

Most of the cpus can either:
- immediately drop the packet.
- or queue it in their NUMA aware lockless queue.

Then one of the cpu is chosen to process this lockless queue
in a batch.

The batch only contains packets that were cooked on the same
NUMA node, thus with very limited latency impact.

Tested:

DDOS targeting a victim UDP socket, on a platform with 6 NUMA nodes
(Intel(R) Xeon(R) 6985P-C)

Before:

nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1004179            0.0
Udp6InErrors                    3117               0.0
Udp6RcvbufErrors                3117               0.0

After:
nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1116633            0.0
Udp6InErrors                    14197275           0.0
Udp6RcvbufErrors                14197275           0.0

We can see this host can now proces 14.2 M more packets per second
while under attack, and the victim socket can receive 11 % more
packets.

I used a small bpftrace program measuring time (in us) spent in
__udp_enqueue_schedule_skb().

Before:

@udp_enqueue_us[398]:
[0]                24901 |@@@                                                 |
[1]                63512 |@@@@@@@@@                                           |
[2, 4)            344827 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            244673 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)            54022 |@@@@@@@@                                            |
[16, 32)          222134 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                   |
[32, 64)          232042 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[64, 128)           4219 |                                                    |
[128, 256)           188 |                                                    |

After:

@udp_enqueue_us[398]:
[0]              5608855 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]              1111277 |@@@@@@@@@@                                          |
[2, 4)            501439 |@@@@                                                |
[4, 8)            102921 |                                                    |
[8, 16)            29895 |                                                    |
[16, 32)           43500 |                                                    |
[32, 64)           31552 |                                                    |
[64, 128)            979 |                                                    |
[128, 256)            13 |                                                    |

Note that the remaining bottleneck for this platform is in
udp_drops_inc() because we limited struct numa_drop_counters
to only two nodes so far.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20250922104240.2182559-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-09-23 16:38:39 -07:00
..
ila ipv6: adopt dst_dev() helper 2025-07-02 14:32:30 -07:00
netfilter netfilter: nf_reject: don't reply to icmp error messages 2025-09-11 15:40:55 +02:00
addrconf_core.c ipv6: Ensure natural alignment of const ipv6 loopback and router addresses 2024-01-30 12:43:18 +01:00
addrconf.c ipv6: Add sanity checks on ipv6_devconf.rpl_seg_enabled 2025-09-02 17:01:14 -07:00
addrlabel.c net: replace ADDRLABEL with dynamic debug 2025-07-08 15:04:05 +02:00
af_inet6.c ipv6: make ipv6_pinfo.daddr_cache a boolean 2025-09-18 10:17:09 +02:00
ah6.c net: fill in MODULE_DESCRIPTION()s for ipv6 modules 2024-02-09 14:12:01 -08:00
anycast.c ipv6: start using dst_dev_rcu() 2025-08-29 19:36:32 -07:00
calipso.c net: ipv6: Fix spelling mistake 2025-07-02 15:42:29 -07:00
datagram.c net: add sk_drops_read(), sk_drops_inc() and sk_drops_reset() helpers 2025-08-28 13:14:50 +02:00
esp6_offload.c xfrm: Add an inbound percpu state cache. 2024-10-29 11:56:18 +01:00
esp6.c tcp: Don't pass hashinfo to socket lookup helpers. 2025-08-25 17:53:35 -07:00
exthdrs_core.c ipv6: Fix out-of-bounds access in ipv6_find_tlv() 2023-05-24 08:43:39 +01:00
exthdrs_offload.c net: gso: add HBH extension header offload support 2024-01-05 08:11:49 -08:00
exthdrs.c ipv6: annotate data-races around devconf->rpl_seg_enabled 2025-09-02 17:01:06 -07:00
fib6_notifier.c net: do not acquire rtnl in fib_seq_sum() 2024-10-11 15:35:05 -07:00
fib6_rules.c ipv6: fib_rules: Add DSCP mask matching 2025-02-21 16:08:48 -08:00
fou6.c net: Add MODULE_DESCRIPTION entries to network modules 2020-06-20 21:33:57 -07:00
icmp.c ipv6: snmp: do not track per idev ICMP6_MIB_RATELIMITHOST 2025-09-08 18:06:20 -07:00
inet6_connection_sock.c ipv6: make ipv6_pinfo.daddr_cache a boolean 2025-09-18 10:17:09 +02:00
inet6_hashtables.c tcp: Remove inet6_hash(). 2025-09-22 11:38:43 -07:00
ioam6_iptunnel.c ipv6: adopt dst_dev() helper 2025-07-02 14:32:30 -07:00
ioam6.c ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
ip6_checksum.c
ip6_fib.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-07-26 11:49:45 -07:00
ip6_flowlabel.c treewide: Switch/rename to timer_delete[_sync]() 2025-04-05 10:30:12 +02:00
ip6_gre.c ipv6: ip6_gre: replace strcpy with strscpy for tunnel name 2025-08-19 18:06:24 -07:00
ip6_icmp.c icmp: fix icmp_ndo_send address translation for reply direction 2025-09-01 12:54:41 -07:00
ip6_input.c net: preserve MSG_ZEROCOPY with forwarding 2025-07-02 15:07:16 -07:00
ip6_offload.c ipv6: reject malicious packets in ipv6_gso_segment() 2025-08-01 14:40:53 -07:00
ip6_offload.h
ip6_output.c ipv6: make ipv6_pinfo.daddr_cache a boolean 2025-09-18 10:17:09 +02:00
ip6_tunnel.c ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
ip6_udp_tunnel.c ipv6: adopt dst_dev() helper 2025-07-02 14:32:30 -07:00
ip6_vti.c ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
ip6mr.c ipv6: ip6_mc_input() and ip6_mr_input() cleanups 2025-07-02 14:32:30 -07:00
ipcomp6.c xfrm: delete x->tunnel as we delete x 2025-07-08 13:28:27 +02:00
ipv6_sockglue.c net: psp: update the TCP MSS to reflect PSP packet overhead 2025-09-18 12:32:06 +02:00
Kconfig ipv6: sr: Use HMAC-SHA1 and HMAC-SHA256 library functions 2025-08-26 18:11:29 -07:00
Makefile net/tcp: Introduce TCP_AO setsockopt()s 2023-10-27 10:35:44 +01:00
mcast_snoop.c net: bridge: mcast: fix broken length + header check for MRDv6 Adv. 2021-04-27 14:02:06 -07:00
mcast.c ipv6: start using dst_dev_rcu() 2025-08-29 19:36:32 -07:00
mip6.c net: fill in MODULE_DESCRIPTION()s for ipv6 modules 2024-02-09 14:12:01 -08:00
ndisc.c ipv6: start using dst_dev_rcu() 2025-08-29 19:36:32 -07:00
netfilter.c netfilter: Switch to skb_dstref_steal to clear dst_entry 2025-08-19 17:54:19 -07:00
output_core.c ipv6: start using dst_dev_rcu() 2025-08-29 19:36:32 -07:00
ping.c inet: ping: remove ping_hash() 2025-09-01 13:15:14 -07:00
proc.c ipv6: snmp: do not track per idev ICMP6_MIB_RATELIMITHOST 2025-09-08 18:06:20 -07:00
protocol.c
raw.c ipv6: np->rxpmtu race annotation 2025-09-18 10:17:09 +02:00
reassembly.c ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
route.c ipv6: make ipv6_pinfo.daddr_cache a boolean 2025-09-18 10:17:09 +02:00
rpl_iptunnel.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-07-17 11:00:33 -07:00
rpl.c ipv6: rpl: Remove pskb(_may)?_pull() in ipv6_rpl_srh_rcv(). 2023-06-19 11:32:58 -07:00
seg6_hmac.c ipv6: sr: Prepare HMAC key ahead of time 2025-08-26 18:11:29 -07:00
seg6_iptunnel.c ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
seg6_local.c ipv6: adopt dst_dev() helper 2025-07-02 14:32:30 -07:00
seg6.c ipv6: sr: Use HMAC-SHA1 and HMAC-SHA256 library functions 2025-08-26 18:11:29 -07:00
sit.c ipv6: sit: Add ipip6_tunnel_dst_find() for cleanup 2025-09-04 10:03:59 +02:00
syncookies.c tcp: accecn: AccECN negotiation 2025-09-18 08:47:51 +02:00
sysctl_net_ipv6.c sysctl: treewide: constify the ctl_table argument of proc_handlers 2024-07-24 20:59:29 +02:00
tcp_ao.c net/tcp: Wire up l3index to TCP-AO 2023-10-27 10:35:46 +01:00
tcp_ipv6.c tcp: Remove inet6_hash(). 2025-09-22 11:38:43 -07:00
tcpv6_offload.c tcp: Don't pass hashinfo to socket lookup helpers. 2025-08-25 17:53:35 -07:00
tunnel6.c net: fill in MODULE_DESCRIPTION()s for ipv6 modules 2024-02-09 14:12:01 -08:00
udp_impl.h udp: move udp_memory_allocated into net_aligned_data 2025-07-02 14:22:02 -07:00
udp_offload.c udp_tunnel: create a fastpath GRO lookup. 2025-04-08 18:19:41 -07:00
udp.c udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
udplite.c udp: move udp_memory_allocated into net_aligned_data 2025-07-02 14:22:02 -07:00
xfrm6_input.c xfrm: Set transport header to fix UDP GRO handling 2025-07-02 09:19:56 +02:00
xfrm6_output.c ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
xfrm6_policy.c xfrm: respect ip protocols rules criteria when performing dst lookups 2024-09-23 07:02:07 +02:00
xfrm6_protocol.c xfrm: add support for UDPv6 encapsulation of ESP 2020-04-28 11:28:36 +02:00
xfrm6_state.c xfrm: remove output_finish indirection from xfrm_state_afinfo 2020-05-06 09:40:08 +02:00
xfrm6_tunnel.c xfrm: flush all states in xfrm_state_fini 2025-08-06 09:23:38 +02:00