linux/net/ipv4
Eric Dumazet b650bf0977 udp: remove busylock and add per NUMA queues
busylock was protecting UDP sockets against packet floods,
but unfortunately was not protecting the host itself.

Under stress, many cpus could spin while acquiring the busylock,
and NIC had to drop packets. Or packets would be dropped
in cpu backlog if RPS/RFS were in place.

This patch replaces the busylock by intermediate
lockless queues. (One queue per NUMA node).

This means that fewer number of cpus have to acquire
the UDP receive queue lock.

Most of the cpus can either:
- immediately drop the packet.
- or queue it in their NUMA aware lockless queue.

Then one of the cpu is chosen to process this lockless queue
in a batch.

The batch only contains packets that were cooked on the same
NUMA node, thus with very limited latency impact.

Tested:

DDOS targeting a victim UDP socket, on a platform with 6 NUMA nodes
(Intel(R) Xeon(R) 6985P-C)

Before:

nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1004179            0.0
Udp6InErrors                    3117               0.0
Udp6RcvbufErrors                3117               0.0

After:
nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1116633            0.0
Udp6InErrors                    14197275           0.0
Udp6RcvbufErrors                14197275           0.0

We can see this host can now proces 14.2 M more packets per second
while under attack, and the victim socket can receive 11 % more
packets.

I used a small bpftrace program measuring time (in us) spent in
__udp_enqueue_schedule_skb().

Before:

@udp_enqueue_us[398]:
[0]                24901 |@@@                                                 |
[1]                63512 |@@@@@@@@@                                           |
[2, 4)            344827 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            244673 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)            54022 |@@@@@@@@                                            |
[16, 32)          222134 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                   |
[32, 64)          232042 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[64, 128)           4219 |                                                    |
[128, 256)           188 |                                                    |

After:

@udp_enqueue_us[398]:
[0]              5608855 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]              1111277 |@@@@@@@@@@                                          |
[2, 4)            501439 |@@@@                                                |
[4, 8)            102921 |                                                    |
[8, 16)            29895 |                                                    |
[16, 32)           43500 |                                                    |
[32, 64)           31552 |                                                    |
[64, 128)            979 |                                                    |
[128, 256)            13 |                                                    |

Note that the remaining bottleneck for this platform is in
udp_drops_inc() because we limited struct numa_drop_counters
to only two nodes so far.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20250922104240.2182559-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-09-23 16:38:39 -07:00
..
netfilter netfilter: nf_reject: don't reply to icmp error messages 2025-09-11 15:40:55 +02:00
af_inet.c net: modify core data structures for PSP datapath support 2025-09-18 12:32:06 +02:00
ah4.c net: fill in MODULE_DESCRIPTION()s for ipv4 modules 2024-02-09 14:12:02 -08:00
arp.c net: set net.core.rmem_max and net.core.wmem_max to 4 MB 2025-08-20 19:35:00 -07:00
bpf_tcp_ca.c tcp: Pass flags to __tcp_send_ack 2025-03-17 13:56:38 +00:00
cipso_ipv4.c ipv4: icmp: Pass IPv4 control block structure as an argument to __icmp_send() 2025-09-11 12:22:38 +02:00
datagram.c net: dst: annotate data-races around dst->obsolete 2025-07-02 14:32:29 -07:00
devinet.c ipv4: Fix NULL vs error pointer check in inet_blackhole_dev_init() 2025-09-03 16:58:44 -07:00
esp4_offload.c xfrm: Add an inbound percpu state cache. 2024-10-29 11:56:18 +01:00
esp4.c tcp: Don't pass hashinfo to socket lookup helpers. 2025-08-25 17:53:35 -07:00
fib_frontend.c ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
fib_lookup.h
fib_notifier.c net: do not acquire rtnl in fib_seq_sum() 2024-10-11 15:35:05 -07:00
fib_rules.c ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
fib_semantics.c net: s/dev_get_flags/netif_get_flags/ 2025-07-18 17:27:47 -07:00
fib_trie.c ipv4: fib: Move fib_valid_key_len() to rtm_to_fib_config(). 2025-03-03 15:04:11 -08:00
fou_bpf.c ip_tunnel: convert __be16 tunnel flags to bitmaps 2024-04-01 10:49:28 +01:00
fou_core.c fou: fix initialization of grc 2024-09-09 17:21:47 -07:00
fou_nl.c netlink: specs: fou: change local-v6/peer-v6 check 2025-09-03 15:16:49 -07:00
fou_nl.h
gre_demux.c net: ip_gre: Fix spelling mistake "demultiplexor" -> "demultiplexer" 2025-04-24 18:20:40 -07:00
gre_offload.c net: gro: rename skb_gro_header_hard() 2024-03-05 13:30:11 +01:00
icmp.c ipv4: icmp: Fix source IP derivation in presence of VRFs 2025-09-11 12:22:38 +02:00
igmp_internal.h netlink: support dumping IPv4 multicast addresses 2025-02-11 11:26:53 +01:00
igmp.c ipv4: adopt dst_dev, skb_dst_dev and skb_dst_dev_net[_rcu] 2025-07-02 14:32:30 -07:00
inet_connection_sock.c tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet_diag.c inet_diag: avoid cache line misses in inet_diag_bc_sk() 2025-08-29 19:29:24 -07:00
inet_fragment.c net: replace use of system_wq with system_percpu_wq 2025-09-22 17:40:30 -07:00
inet_hashtables.c tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet_timewait_sock.c tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inetpeer.c inetpeer: use EXPORT_IPV6_MOD[_GPL]() 2025-02-14 13:09:39 -08:00
ip_forward.c
ip_fragment.c ipv4: start using dst_dev_rcu() 2025-08-29 19:36:32 -07:00
ip_gre.c ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
ip_input.c net: ipv4: convert ip_rcv_options to drop reasons 2025-09-19 17:35:52 -07:00
ip_options.c net: Switch to skb_dstref_steal/skb_dstref_restore for ip_route_input callers 2025-08-19 17:54:35 -07:00
ip_output.c tcp: add datapath logic for PSP with inline key exchange 2025-09-18 12:32:06 +02:00
ip_sockglue.c Networking changes for 6.14. 2025-01-22 08:28:57 -08:00
ip_tunnel_core.c tunnels: reset the GSO metadata before reusing the skb 2025-09-09 13:03:33 +02:00
ip_tunnel.c net: ipv4: Add a flags argument to iptunnel_xmit(), udp_tunnel_xmit_skb() 2025-06-17 18:18:44 -07:00
ip_vti.c ipv4: adopt dst_dev, skb_dst_dev and skb_dst_dev_net[_rcu] 2025-07-02 14:32:30 -07:00
ipcomp.c xfrm: delete x->tunnel as we delete x 2025-07-08 13:28:27 +02:00
ipconfig.c net: ipconfig: convert timeouts to secs_to_jiffies() 2025-07-09 19:25:01 -07:00
ipip.c ipv4: ip_tunnel: Convert ip_tunnel_delete_nets() callers to ->exit_rtnl(). 2025-04-14 17:08:42 -07:00
ipmr_base.c ipmr: do not call mr_mfc_uses_dev() for unres entries 2025-01-23 07:08:13 -08:00
ipmr.c ipv4: start using dst_dev_rcu() 2025-08-29 19:36:32 -07:00
Kconfig net: Retire DCCP socket. 2025-04-11 18:58:10 -07:00
Makefile bpfilter: remove bpfilter 2024-01-04 10:23:10 -08:00
metrics.c net: remove NULL-pointer net parameter in ip_metrics_convert 2024-06-05 10:06:00 +01:00
netfilter.c ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
netlink.c
nexthop.c net: When removing nexthops, don't call synchronize_net if it is not necessary 2025-08-19 17:50:33 -07:00
ping.c inet: ping: use EXPORT_IPV6_MOD[_GPL]() 2025-09-01 13:15:14 -07:00
proc.c ipv4: snmp: do not use SNMP_MIB_SENTINEL anymore 2025-09-08 18:06:20 -07:00
protocol.c
raw_diag.c inet_diag: change inet_diag_bc_sk() first argument 2025-08-29 19:29:24 -07:00
raw.c inet: raw: add drop_counters to raw sockets 2025-08-28 13:14:50 +02:00
route.c ipv4: icmp: Pass IPv4 control block structure as an argument to __icmp_send() 2025-09-11 12:22:38 +02:00
syncookies.c tcp: accecn: AccECN negotiation 2025-09-18 08:47:51 +02:00
sysctl_net_ipv4.c tcp: accecn: AccECN option send control 2025-09-18 08:47:52 +02:00
tcp_ao.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-18 11:26:06 -07:00
tcp_bbr.c tcp: Add new args for cong_control in tcp_congestion_ops 2024-05-02 16:26:56 -07:00
tcp_bic.c
tcp_bpf.c tcp_bpf: Call sk_msg_free() when tcp_bpf_send_verdict() fails to allocate psock->cork. 2025-09-10 06:53:56 -07:00
tcp_cdg.c tcp: cdg: remove redundant __GFP_NOWARN 2025-08-12 14:05:43 -07:00
tcp_cong.c tcp: only release congestion control if it has been initialized 2024-10-31 18:22:48 -07:00
tcp_cubic.c tcp_cubic: fix incorrect HyStart round start detection 2025-01-20 12:26:41 +00:00
tcp_dctcp.c tcp: helpers for ECN mode handling 2025-03-17 13:54:11 +00:00
tcp_dctcp.h tcp: Pass flags to __tcp_send_ack 2025-03-17 13:56:38 +00:00
tcp_diag.c inet_diag: change inet_diag_bc_sk() first argument 2025-08-29 19:29:24 -07:00
tcp_fastopen.c tcp: use dst_dev_rcu() in tcp_fastopen_active_disable_ofo_check() 2025-08-29 19:36:32 -07:00
tcp_highspeed.c
tcp_htcp.c tcp: Use clamp() in htcp_alpha_update() 2024-08-06 12:16:25 -07:00
tcp_hybla.c
tcp_illinois.c
tcp_input.c tcp: reclaim 8 bytes in struct request_sock_queue 2025-09-22 17:55:25 -07:00
tcp_ipv4.c net: psp: update the TCP MSS to reflect PSP packet overhead 2025-09-18 12:32:06 +02:00
tcp_lp.c
tcp_metrics.c tcp_metrics: use dst_dev_net_rcu() 2025-08-29 19:36:32 -07:00
tcp_minisocks.c tcp: add datapath logic for PSP with inline key exchange 2025-09-18 12:32:06 +02:00
tcp_nv.c
tcp_offload.c tcp: Don't pass hashinfo to socket lookup helpers. 2025-08-25 17:53:35 -07:00
tcp_output.c tcp: prefer sk_skb_reason_drop() 2025-09-19 17:00:23 -07:00
tcp_plb.c
tcp_rate.c
tcp_recovery.c tcp: update the outdated ref draft-ietf-tcpm-rack 2025-07-08 09:01:52 -07:00
tcp_scalable.c
tcp_sigpool.c net/tcp_sigpool: Use nested-BH locking for sigpool_scratch. 2024-06-24 16:41:22 -07:00
tcp_timer.c tcp: annotate data-races around icsk->icsk_probes_out 2025-08-25 16:20:59 -07:00
tcp_ulp.c
tcp_vegas.c
tcp_vegas.h
tcp_veno.c
tcp_westwood.c
tcp_yeah.c
tcp.c tcp: move tcp_clean_acked to tcp_sock_read_tx group 2025-09-22 17:55:25 -07:00
tunnel4.c net: fill in MODULE_DESCRIPTION()s for ipv4 modules 2024-02-09 14:12:02 -08:00
udp_bpf.c
udp_diag.c inet_diag: change inet_diag_bc_sk() first argument 2025-08-29 19:29:24 -07:00
udp_impl.h udp: move udp_memory_allocated into net_aligned_data 2025-07-02 14:22:02 -07:00
udp_offload.c udp: also consider secpath when evaluating ipsec use for checksumming 2025-08-07 08:07:15 +02:00
udp_tunnel_core.c ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
udp_tunnel_nic.c udp_tunnel: use netdev_warn() instead of netdev_WARN() 2025-09-11 19:09:48 -07:00
udp_tunnel_stub.c
udp.c udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
udplite.c udp: move udp_memory_allocated into net_aligned_data 2025-07-02 14:22:02 -07:00
xfrm4_input.c xfrm: Set transport header to fix UDP GRO handling 2025-07-02 09:19:56 +02:00
xfrm4_output.c ipv4: adopt dst_dev, skb_dst_dev and skb_dst_dev_net[_rcu] 2025-07-02 14:32:30 -07:00
xfrm4_policy.c ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
xfrm4_protocol.c ipv4: Convert ip_route_input_noref() to dscp_t. 2024-10-03 16:21:21 -07:00
xfrm4_state.c
xfrm4_tunnel.c net: fill in MODULE_DESCRIPTION()s for ipv4 modules 2024-02-09 14:12:02 -08:00