github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-08-04 22:38:46 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph
f770645860
linux/include/net
History
Eric Dumazet b650bf0977 udp: remove busylock and add per NUMA queues 
busylock was protecting UDP sockets against packet floods,
but unfortunately was not protecting the host itself.

Under stress, many cpus could spin while acquiring the busylock,
and NIC had to drop packets. Or packets would be dropped
in cpu backlog if RPS/RFS were in place.

This patch replaces the busylock by intermediate
lockless queues. (One queue per NUMA node).

This means that fewer number of cpus have to acquire
the UDP receive queue lock.

Most of the cpus can either:
- immediately drop the packet.
- or queue it in their NUMA aware lockless queue.

Then one of the cpu is chosen to process this lockless queue
in a batch.

The batch only contains packets that were cooked on the same
NUMA node, thus with very limited latency impact.

Tested:

DDOS targeting a victim UDP socket, on a platform with 6 NUMA nodes
(Intel(R) Xeon(R) 6985P-C)

Before:

nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1004179            0.0
Udp6InErrors                    3117               0.0
Udp6RcvbufErrors                3117               0.0

After:
nstat -n ; sleep 1 ; nstat | grep Udp
Udp6InDatagrams                 1116633            0.0
Udp6InErrors                    14197275           0.0
Udp6RcvbufErrors                14197275           0.0

We can see this host can now proces 14.2 M more packets per second
while under attack, and the victim socket can receive 11 % more
packets.

I used a small bpftrace program measuring time (in us) spent in
__udp_enqueue_schedule_skb().

Before:

@udp_enqueue_us[398]:
[0]                24901 |@@@                                                 |
[1]                63512 |@@@@@@@@@                                           |
[2, 4)            344827 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            244673 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)            54022 |@@@@@@@@                                            |
[16, 32)          222134 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                   |
[32, 64)          232042 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[64, 128)           4219 |                                                    |
[128, 256)           188 |                                                    |

After:

@udp_enqueue_us[398]:
[0]              5608855 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]              1111277 |@@@@@@@@@@                                          |
[2, 4)            501439 |@@@@                                                |
[4, 8)            102921 |                                                    |
[8, 16)            29895 |                                                    |
[16, 32)           43500 |                                                    |
[32, 64)           31552 |                                                    |
[64, 128)            979 |                                                    |
[128, 256)            13 |                                                    |

Note that the remaining bottleneck for this platform is in
udp_drops_inc() because we limited struct numa_drop_counters
to only two nodes so far.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20250922104240.2182559-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-09-23 16:38:39 -07:00
..
9p netfs, 9p: Implement helpers for new write code 2024-05-01 18:07:37 +01:00
bluetooth Bluetooth: hci_sync: fix set_local_name race condition 2025-08-22 13:57:31 -04:00
caif caif: Remove unused cfsrvl_getphyid 2024-10-08 15:33:49 -07:00
iucv net: reformat kdoc return statements 2024-12-09 14:44:59 -08:00
libeth xdp, libeth: make the xdp_init_buff() micro-optimization generic 2025-09-08 10:26:25 -07:00
mana net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency. 2025-08-19 14:42:44 +02:00
netfilter Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-09-11 17:40:13 -07:00
netns tcp: accecn: AccECN option send control 2025-09-18 08:47:52 +02:00
nfc net: nfc: nci: Increase NCI_DATA_TIMEOUT to 3000 ms 2025-09-03 17:02:12 -07:00
page_pool net: add helper to pre-check if PP for an Rx queue will be unreadable 2025-09-04 10:19:17 +02:00
phonet phonet: Convert phonet_routes.lock to spinlock_t. 2024-10-24 16:03:40 +02:00
psp psp: Fix typo in kdoc for struct psp_dev_caps.assoc_drv_spc. 2025-09-19 17:02:27 -07:00
sctp sctp: Convert cookie authentication to use HMAC-SHA256 2025-08-19 19:36:26 -07:00
tc_act net_sched: act_skbmod: use RCU in tcf_skbmod_dump() 2025-08-28 16:46:23 -07:00
6lowpan.h
act_api.h net_sched: act: remove tcfa_qstats 2025-09-02 15:52:24 -07:00
addrconf.h ipv6: Add __in6_dev_get_rtnl_net(). 2025-01-20 12:16:04 -08:00
af_ieee802154.h
af_rxrpc.h rxrpc: Remove deadcode 2025-04-24 17:03:45 -07:00
af_unix.h af_unix: Introduce SO_INQ. 2025-07-08 18:05:25 -07:00
af_vsock.h vsock: fix vsock_proto declaration 2025-07-07 16:55:54 -07:00
ah.h
aligned_data.h udp: move udp_memory_allocated into net_aligned_data 2025-07-02 14:22:02 -07:00
amt.h
arp.h
atmclip.h
ax25.h ax25: Remove broken autobind 2025-03-24 10:26:53 +00:00
ax88796.h
bareudp.h
bond_3ad.h bonding: support aggregator selection based on port priority 2025-09-09 10:56:02 +02:00
bond_alb.h bonding: Correct spelling in headers 2024-08-26 09:37:22 -07:00
bond_options.h bonding: add support for per-port LACP actor priority 2025-09-09 10:56:02 +02:00
bonding.h bonding: Remove support for use_carrier 2025-09-02 14:01:54 -07:00
bpf_sk_storage.h
busy_poll.h net: gro: decouple GRO from the NAPI layer 2025-02-27 14:03:14 +01:00
calipso.h move asm/unaligned.h to linux/unaligned.h 2024-10-02 17:23:23 -04:00
cfg80211-wext.h
cfg80211.h wifi: nl80211: allow drivers to support subset of NL80211_CMD_SET_BSS 2025-09-04 11:19:02 +02:00
cfg802154.h mac802154: fix llsec key resources release in mac802154_llsec_key_del 2024-03-06 21:01:26 +01:00
checksum.h net: Fix checksum update for ILA adj-transport 2025-05-30 19:53:51 -07:00
cipso_ipv4.h move asm/unaligned.h to linux/unaligned.h 2024-10-02 17:23:23 -04:00
cls_cgroup.h net/cls_cgroup: Fix task_get_classid() during qdisc run 2025-09-14 11:55:04 -07:00
codel_impl.h
codel_qdisc.h
codel.h codel: fix kernel-doc notation warnings 2023-07-14 20:39:29 -07:00
compat.h
datalink.h net: datalink: Remove unused declarations 2023-07-27 17:17:32 -07:00
dcbevent.h
dcbnl.h
devlink.h devlink: Add a 'num_doorbells' driverinit param 2025-09-17 18:30:51 -07:00
dropreason-core.h tcp: add datapath logic for PSP with inline key exchange 2025-09-18 12:32:06 +02:00
dropreason.h wifi: mac80211: Drop cooked monitor support 2025-02-11 11:58:17 +01:00
dsa_stubs.h net: dsa: Use conduit and user terms 2023-10-24 13:08:14 -07:00
dsa.h net: dsa: tag_brcm: add support for legacy FCS tags 2025-06-17 17:52:01 -07:00
dscp.h net: add IEEE 802.1q specific helpers 2024-05-08 10:35:09 +01:00
dsfield.h
dst_cache.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
dst_metadata.h net: dst_metadata: fix IP_DF bit not extracted from tunnel headers 2025-09-14 14:28:12 -07:00
dst_ops.h net: fix __dst_negative_advice() race 2024-05-29 17:34:49 -07:00
dst.h net: dst: introduce dst->dev_rcu 2025-08-29 19:36:31 -07:00
eee.h net: simplify eeecfg_mac_can_tx_lpi 2024-11-13 18:49:50 -08:00
erspan.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
esp.h
espintcp.h inet: preserve const qualifier in inet_csk() 2024-04-01 21:27:08 -07:00
ethoc.h
failover.h
fib_notifier.h net: do not acquire rtnl in fib_seq_sum() 2024-10-11 15:35:05 -07:00
fib_rules.h net: fib_rules: Fix iif / oif matching on L3 master device 2025-04-15 17:54:56 -07:00
firewire.h
flow_dissector.h flow_dissector: cleanup FLOW_DISSECTOR_KEY_ENC_FLAGS 2024-07-15 09:14:39 -07:00
flow_offload.h net: sched: propagate "skip_sw" flag to struct flow_cls_common_offload 2024-10-30 17:33:53 -07:00
flow.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
fou.h
fq_impl.h
fq.h net: fq: Remove unused typedef fq_flow_get_default_t 2023-08-08 15:58:23 -07:00
garp.h
gen_stats.h
genetlink.h genetlink: fix typo in comment 2025-09-03 15:16:49 -07:00
geneve.h
gre.h ip_tunnel: convert __be16 tunnel flags to bitmaps 2024-04-01 10:49:28 +01:00
gro_cells.h
gro.h net: Create separate gro_flush_normal function 2025-07-24 18:34:55 -07:00
gso.h
gtp.h gtp: properly parse extension headers 2024-05-07 01:35:55 +02:00
gue.h
handshake.h net/handshake: Add helpers for parsing incoming TLS Alerts 2023-07-28 14:07:59 -07:00
hotdata.h net: introduce per netns packet chains 2025-03-24 13:58:22 -07:00
hwbm.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
icmp.h ipv4: icmp: Pass IPv4 control block structure as an argument to __icmp_send() 2025-09-11 12:22:38 +02:00
ieee8021q.h net: add IEEE 802.1q specific helpers 2024-05-08 10:35:09 +01:00
ieee80211_radiotap.h wireless fixes for v6.12-rc5 2024-10-25 10:44:41 +01:00
ieee802154_netdev.h mac802154: Handle association requests from peers 2023-11-20 11:43:03 +01:00
if_inet6.h ipv6: anycast: complete RCU handling of struct ifacaddr6 2024-02-26 18:40:34 -08:00
ife.h
inet_common.h net: change proto and proto_ops accept type 2024-05-13 18:19:09 -06:00
inet_connection_sock.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet_dscp.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
inet_ecn.h
inet_frag.h inet: frags: change inet_frag_kill() to defer refcount updates 2025-03-18 13:18:36 +01:00
inet_hashtables.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet_sock.h sctp: Prepare sctp_v4_get_dst() to dscp_t conversion. 2025-01-06 13:49:38 -08:00
inet_timewait_sock.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
inet6_connection_sock.h tcp/dccp: Remove inet_connection_sock_af_ops.addr2sockaddr(). 2025-03-24 12:10:13 -07:00
inet6_hashtables.h tcp: Remove inet6_hash(). 2025-09-22 11:38:43 -07:00
inetpeer.h inetpeer: remove create argument of inet_getpeer() 2024-12-17 19:37:00 -08:00
ioam6.h net: ioam6: multicast event 2024-02-28 11:19:41 +00:00
ip_fib.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
ip_tunnels.h ipv4: Convert ->flowi4_tos to dscp_t. 2025-08-26 17:34:31 -07:00
ip_vs.h ipvs: Fix estimator kthreads preferred affinity 2025-08-13 08:34:33 +02:00
ip.h net: snmp: remove SNMP_MIB_SENTINEL 2025-09-08 18:06:21 -07:00
ip6_checksum.h
ip6_fib.h ipv6: Defer fib6_purge_rt() in fib6_add_rt2node() to fib6_add(). 2025-04-24 09:29:56 +02:00
ip6_route.h ipv6: make ipv6_pinfo.daddr_cache a boolean 2025-09-18 10:17:09 +02:00
ip6_tunnel.h ipv6: adopt skb_dst_dev() and skb_dst_dev_net[_rcu]() helpers 2025-07-02 14:32:30 -07:00
ipcomp.h xfrm: ipcomp: Use crypto_acomp interface 2025-03-21 17:36:49 +08:00
ipconfig.h
ipv6_frag.h inet: frags: change inet_frag_kill() to defer refcount updates 2025-03-18 13:18:36 +01:00
ipv6_stubs.h ipv6: udp: constify 'struct net' parameter of socket lookups 2024-08-05 16:27:26 -07:00
ipv6.h ipv6: fix _DEVADD() and _DEVUPD() macros 2025-03-25 07:31:24 -07:00
iw_handler.h Revert "wifi: cfg80211: unexport wireless_nlevent_flush()" 2024-10-09 08:53:01 +02:00
kcm.h net: kcm: Fix race condition in kcm_unattach() 2025-08-13 18:18:33 -07:00
l3mdev.h net: fib_rules: Fix iif / oif matching on L3 master device 2025-04-15 17:54:56 -07:00
lag.h
lapb.h net: lapb: increase LAPB_HEADER_LEN 2024-12-06 17:43:08 -08:00
llc_c_ac.h net: llc: Remove unused function declarations 2023-08-04 15:33:17 -07:00
llc_c_ev.h net: llc: Remove unused function declarations 2023-08-04 15:33:17 -07:00
llc_c_st.h llc: Constify struct llc_conn_state_trans 2024-07-15 08:51:01 -07:00
llc_conn.h llc: Check netns in llc_estab_match() and llc_listener_match(). 2023-07-20 10:46:28 +02:00
llc_if.h
llc_pdu.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
llc_s_ac.h
llc_s_ev.h
llc_s_st.h llc: Constify struct llc_sap_state_trans 2024-07-15 08:51:19 -07:00
llc_sap.h
llc.h
lwtunnel.h net: dst: annotate data-races around dst->output 2025-07-02 14:32:30 -07:00
mac80211.h wifi: mac80211: don't require cipher and keylen in gtk rekey 2025-07-22 10:43:19 +02:00
mac802154.h move asm/unaligned.h to linux/unaligned.h 2024-10-02 17:23:23 -04:00
macsec.h net: macsec: Add endianness annotations in salt struct 2025-01-20 12:20:42 +00:00
mctp.h net: mctp: Allow limiting binds to a peer address 2025-07-15 12:08:39 +02:00
mctpdevice.h net: mctp: Expose transport binding identifier via IFLA attribute 2024-11-09 09:04:54 -08:00
mip6.h
mld.h
mpls_iptunnel.h
mpls.h
mptcp.h mptcp: sched: remove mptcp_sched_data 2025-04-15 08:21:46 -07:00
mrp.h
ncsi.h
ndisc.h net: replace ND_PRINTK with dynamic debug 2025-07-10 15:27:32 -07:00
neighbour_tables.h neighbour: Create netdev->neighbour association 2024-11-09 13:22:57 -08:00
neighbour.h neighbour: Update pneigh_entry in pneigh_create(). 2025-07-17 16:25:22 -07:00
net_debug.h Kbuild updates for v6.13 2024-11-30 13:41:50 -08:00
net_failover.h
net_namespace.h net: Remove ->exit_batch_rtnl(). 2025-04-14 17:08:45 -07:00
net_ratelimit.h
net_shaper.h net-shapers: implement NL get operation 2024-10-10 08:30:22 -07:00
net_trackers.h
netdev_lock.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2025-05-22 09:42:41 -07:00
netdev_netlink.h net: add granular lock for the netdev netlink socket 2025-03-12 13:32:35 -07:00
netdev_queues.h net: add helper to pre-check if PP for an Rx queue will be unreadable 2025-09-04 10:19:17 +02:00
netdev_rx_queue.h net: designate XSK pool pointers in queues as "ops protected" 2025-04-09 17:01:51 -07:00
netevent.h
netkit.h bpf, netkit: Add indirect call wrapper for fetching peer dev 2023-11-20 10:15:16 -08:00
netlabel.h Networking changes for 6.13. 2024-11-21 08:28:08 -08:00
netlink.h netlink: introduce type-checking attribute iteration for nlmsg 2025-07-02 15:39:04 -07:00
netmem.h netmem, mlx4: access ->pp_ref_count through netmem_desc instead of page 2025-07-23 17:46:54 -07:00
netprio_cgroup.h
netrom.h
nexthop.h ipv6: Protect nh->f6i_list with spinlock and flag. 2025-04-24 09:29:56 +02:00
nl802154.h ieee802154: Correct spelling in nl802154.h 2024-08-30 22:30:55 +02:00
nsh.h net: NSH: fix kernel-doc notation warning 2023-07-14 20:39:29 -07:00
pfcp.h net: pfcp: fix typo in message_priority field name 2025-06-13 18:17:08 -07:00
pie.h pie: fix kernel-doc notation warning 2023-07-14 20:39:30 -07:00
ping.h inet: ping: remove ping_hash() 2025-09-01 13:15:14 -07:00
pkt_cls.h net: sched: refine software bypass handling in tc_run 2025-01-20 09:21:27 +00:00
pkt_sched.h net/sched: sch_qfq: Fix null-deref in agg_dequeue 2025-07-10 11:08:35 +02:00
pptp.h
proto_memory.h net-memcg: Pass struct sock to mem_cgroup_sk_under_memory_pressure(). 2025-08-19 19:20:59 -07:00
protocol.h ipv6: move tcp_ipv6_hash_secret and udp_ipv6_hash_secret to net_hotdata 2024-03-07 21:12:43 -08:00
psample.h net: psample: fix flag being set in wrong skb 2024-07-11 18:11:31 -07:00
psnap.h
psp.h psp: base PSP device support 2025-09-18 12:32:06 +02:00
raw.h net: use NUMA drop counters for softnet_data.dropped 2025-09-14 11:35:17 -07:00
rawv6.h
red.h net: sched: Correct spelling in headers 2024-08-26 09:37:23 -07:00
regulatory.h net: Correct spelling in headers 2024-08-26 09:37:23 -07:00
request_sock.h tcp: reclaim 8 bytes in struct request_sock_queue 2025-09-22 17:55:25 -07:00
rose.h net: rose: convert 'use' field to refcount_t 2025-08-27 07:43:08 -07:00
route.h net: use dst_dev_rcu() in sk_setup_caps() 2025-08-29 19:36:32 -07:00
rpl.h ipv6: rpl: Remove pskb(_may)?_pull() in ipv6_rpl_srh_rcv(). 2023-06-19 11:32:58 -07:00
rps.h net: Add rfs_needed() helper 2025-09-03 15:08:20 -07:00
rsi_91x.h rsi: remove kernel-doc comment marker 2023-07-14 20:39:30 -07:00
rstreason.h net: Retire DCCP socket. 2025-04-11 18:58:10 -07:00
rtnetlink.h rtnetlink: Remove "net" from newlink params 2025-02-21 15:28:03 -08:00
rtnh.h
sch_generic.h net/sched: Fix backlog accounting in qdisc_dequeue_internal 2025-08-14 17:52:29 -07:00
scm.h af_unix/scm: fix whitespace errors 2025-07-04 09:32:35 +02:00
secure_seq.h net: Retire DCCP socket. 2025-04-11 18:58:10 -07:00
seg6_hmac.h ipv6: sr: Prepare HMAC key ahead of time 2025-08-26 18:11:29 -07:00
seg6_local.h seg6: Use nested-BH locking for seg6_bpf_srh_states. 2024-06-24 16:41:23 -07:00
seg6.h ipv6: sr: restruct ifdefines 2024-05-30 18:29:38 -07:00
selftests.h
slhc_vj.h
smc.h dibs: Move event handling to dibs layer 2025-09-23 11:13:22 +02:00
snmp.h net: snmp: remove SNMP_MIB_SENTINEL 2025-09-08 18:06:21 -07:00
sock_reuseport.h net: core: annotate socks of struct sock_reuseport with __counted_by 2024-08-02 17:16:59 -07:00
sock.h tcp: Update bind bucket state on port release 2025-09-23 10:12:15 +02:00
Space.h net: appletalk: remove cops support 2023-10-04 11:49:20 -07:00
stp.h
strparser.h strparser: Remove unused __strp_unpause 2025-05-05 16:48:12 -07:00
switchdev.h net: bridge: switchdev: Skip MDB replays of deferred events on offload 2024-02-16 09:36:37 +00:00
tc_wrapper.h Merge branch 'x86/bugs' into x86/core, to pick up pending changes before dependent patches 2024-02-14 10:49:37 +01:00
tcp_ao.h tcp: Free TCP-AO/TCP-MD5 info/keys without RCU 2025-09-11 19:05:56 -07:00
tcp_ecn.h tcp: accecn: AccECN option failure handling 2025-09-18 08:47:52 +02:00
tcp_states.h tcp: Dump bound-only sockets in inet_diag. 2023-12-04 14:45:26 -08:00
tcp.h tcp: accecn: AccECN option ceb/cep and ACE field multi-wrap heuristics 2025-09-18 08:47:52 +02:00
tcx.h bpf: Remove location field in tcx_link 2025-07-11 11:00:57 -07:00
timewait_sock.h tcp: Remove timewait_sock_ops.twsk_destructor(). 2025-08-25 17:53:35 -07:00
tipc.h
tls_prot.h net/tls: Add TLS Alert definitions 2023-07-28 14:07:59 -07:00
tls_toe.h
tls.h tls: block decryption when a rekey is pending 2024-12-16 12:47:29 +00:00
transp_v6.h inet6: Remove unused function declaration udpv6_connect() 2023-08-01 15:06:27 -07:00
tso.h
tun_proto.h
udp_tunnel.h udp_tunnel: fix deadlock in udp_tunnel_nic_set_port_priv() 2025-06-24 16:31:36 -07:00
udp.h udp: remove busylock and add per NUMA queues 2025-09-23 16:38:39 -07:00
udplite.h udplite: fix various data-races 2023-09-14 16:16:36 +02:00
vsock_addr.h
vxlan.h vxlan: Support MC routing in the underlay 2025-06-17 18:18:46 -07:00
wext.h
x25.h net/x25: Remove unused x25_terminate_link() 2025-07-14 17:19:13 -07:00
x25device.h
xdp_priv.h
xdp_sock_drv.h xsk: Add launch time hardware offload support to XDP Tx metadata 2025-02-20 15:13:45 -08:00
xdp_sock.h net: xsk: introduce XDP_MAX_TX_SKB_BUDGET setsockopt 2025-07-10 14:48:29 +02:00
xdp.h net: xdp: handle frags with unreadable memory 2025-09-11 12:00:20 +02:00
xfrm.h Revert "xfrm: destroy xfrm_state synchronously on net exit path" 2025-07-08 13:28:29 +02:00
xsk_buff_pool.h xsk: Fix offset calculation in unaligned mode 2025-04-24 17:11:52 -07:00