linux/net/ipv4
Arjun Roy 18fb76ed53 net-zerocopy: Copy straggler unaligned data for TCP Rx. zerocopy.
When TCP receive zerocopy does not successfully map the entire
requested space, it outputs a 'hint' that the caller should recvmsg().

Augment zerocopy to accept a user buffer that it tries to copy this
hint into - if it is possible to copy the entire hint, it will do so.
This elides a recvmsg() call for received traffic that isn't exactly
page-aligned in size.

This was tested with RPC-style traffic of arbitrary sizes. Normally,
each received message required at least one getsockopt() call, and one
recvmsg() call for the remaining unaligned data.

With this change, almost all of the recvmsg() calls are eliminated,
leading to a savings of about 25%-50% in number of system calls
for RPC-style workloads.

Signed-off-by: Arjun Roy <arjunroy@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2020-12-04 13:40:52 -08:00
..
bpfilter net: Revert "net: optimize the sockptr_t for unified kernel/user address spaces" 2020-08-10 12:06:44 -07:00
netfilter Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-11-06 17:33:38 -08:00
af_inet.c bpf: Allow bpf_{s,g}etsockopt from cgroup bind{4,6} hooks 2020-12-02 13:25:11 -08:00
ah4.c inet: Use fallthrough; 2020-03-12 15:55:00 -07:00
arp.c net: Exempt multicast addresses from five-second neighbor lifetime 2020-11-13 14:24:39 -08:00
bpf_tcp_ca.c bpf: Remove hard-coded btf_vmlinux assumption from BPF verifier 2020-12-03 17:38:21 -08:00
cipso_ipv4.c cipso: fix 'audit_secid' kernel-doc warning in cipso_ipv4.c 2020-09-08 20:03:36 -07:00
datagram.c inet: stop leaking jiffies on the wire 2019-11-01 14:57:52 -07:00
devinet.c treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
esp4_offload.c net: Add MODULE_DESCRIPTION entries to network modules 2020-06-20 21:33:57 -07:00
esp4.c ESP: Export esp_output_fill_trailer function 2020-02-19 13:52:32 +01:00
fib_frontend.c ipv4: use IS_ENABLED instead of ifdef 2020-11-17 17:02:03 -08:00
fib_lookup.h net: add net available in build_state 2020-03-29 22:30:57 -07:00
fib_notifier.c net: fib_notifier: propagate extack down to the notifier block callback 2019-10-04 11:10:56 -07:00
fib_rules.c fib: use indirect call wrappers in the most common fib_rules_ops 2020-07-28 17:42:31 -07:00
fib_semantics.c treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
fib_trie.c nexthop: Remove in-kernel route notifications when nexthop changes 2020-11-06 11:28:50 -08:00
fou.c genetlink: move to smaller ops wherever possible 2020-10-02 19:11:11 -07:00
gre_demux.c gre: fix uninit-value in __iptunnel_pull_header 2020-03-08 21:25:37 -07:00
gre_offload.c net: gre: recompute gre csum for sctp over gre tunnels 2020-08-03 15:29:44 -07:00
icmp.c icmp: randomize the global rate limiter 2020-10-16 16:47:09 -07:00
igmp.c ip*_mc_gsfget(): lift copyout of struct group_filter into callers 2020-05-20 20:31:27 -04:00
inet_connection_sock.c tcp: fix race condition when creating child sockets from syncookies 2020-11-23 16:32:33 -08:00
inet_diag.c inet_diag: Fix error path to cancel the meseage in inet_req_diag_fill() 2020-11-17 16:08:36 -08:00
inet_fragment.c inet: frags: re-introduce skb coalescing for local delivery 2019-08-08 15:55:10 -07:00
inet_hashtables.c tcp: fix race condition when creating child sockets from syncookies 2020-11-23 16:32:33 -08:00
inet_timewait_sock.c treewide: Add SPDX license identifier for missed files 2019-05-21 10:50:45 +02:00
inetpeer.c inetpeer: fix data-race in inet_putpeer / inet_putpeer 2019-11-07 16:15:56 -08:00
ip_forward.c ipv4: Revert removal of rt_uses_gateway 2019-09-20 18:23:33 -07:00
ip_fragment.c inet: frags: re-introduce skb coalescing for local delivery 2019-08-08 15:55:10 -07:00
ip_gre.c ipv4/ipv6: switch to dev_get_tstats64 2020-11-09 17:50:28 -08:00
ip_input.c bpf: Add socket assign support 2020-03-30 13:45:04 -07:00
ip_options.c net: clean up codestyle for net/ipv4 2020-08-25 06:28:02 -07:00
ip_output.c networking changes for the 5.10 merge window 2020-10-15 18:42:13 -07:00
ip_sockglue.c net: Remove duplicated midx check against 0 2020-08-25 06:23:59 -07:00
ip_tunnel_core.c Merge https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-11-12 16:54:48 -08:00
ip_tunnel.c ip_tunnel: fix over-mtu packet send fail without TUNNEL_DONT_FRAGMENT flags 2020-10-31 17:19:02 -07:00
ip_vti.c vti: switch to dev_get_tstats64 2020-11-09 17:50:28 -08:00
ipcomp.c ipcomp: assign if_id to child tunnel from parent tunnel 2020-07-09 12:55:37 +02:00
ipconfig.c net: ipconfig: Avoid spurious blank lines in boot log 2020-11-12 14:53:33 -08:00
ipip.c ipv4/ipv6: switch to dev_get_tstats64 2020-11-09 17:50:28 -08:00
ipmr_base.c net: fib_notifier: propagate extack down to the notifier block callback 2019-10-04 11:10:56 -07:00
ipmr.c ipmr: Use full VIF ID in netlink cache reports 2020-09-10 12:25:51 -07:00
Kconfig net: ipv4: remove duplicate "the the" phrase in Kconfig text 2020-08-18 16:02:16 -07:00
Makefile udp_tunnel: add central NIC RX port offload infrastructure 2020-07-10 13:54:00 -07:00
metrics.c treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
netfilter.c netfilter: use actual socket sk rather than skb sk when routing harder 2020-10-30 12:57:39 +01:00
netlink.c treewide: Add SPDX license identifier for missed files 2019-05-21 10:50:45 +02:00
nexthop.c nexthop: Replay nexthops when registering a notifier 2020-11-06 11:28:50 -08:00
ping.c net: clean up codestyle 2020-08-31 12:33:34 -07:00
proc.c net: udp: introduce UDP_MIB_MEMERRORS for udp_mem 2020-11-09 15:34:44 -08:00
protocol.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
raw_diag.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-03-12 22:34:48 -07:00
raw.c networking changes for the 5.10 merge window 2020-10-15 18:42:13 -07:00
route.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-12-03 15:44:09 -08:00
syncookies.c net: Update window_clamp if SOCK_RCVBUF is set 2020-11-10 17:42:35 -08:00
sysctl_net_ipv4.c tcp: reflect tos value received in SYN to the socket 2020-09-10 13:15:40 -07:00
tcp_bbr.c tcp: only postpone PROBE_RTT if RTT is < current min_rtt estimate 2020-11-17 11:03:22 -08:00
tcp_bic.c tcp: fix stretch ACK bugs in BIC 2020-03-16 18:26:54 -07:00
tcp_bpf.c bpf, sockmap: Ensure SO_RCVBUF memory is observed on ingress redirect 2020-11-18 00:12:34 +01:00
tcp_cdg.c treewide: Add SPDX license identifier for more missed files 2019-05-21 10:50:45 +02:00
tcp_cong.c tcp: Set INET_ECN_xmit configuration in tcp_reinit_congestion_control 2020-11-20 18:09:47 -08:00
tcp_cubic.c tcp_cubic: fix spurious HYSTART_DELAY exit upon drop in min RTT 2020-06-25 16:08:47 -07:00
tcp_dctcp.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
tcp_dctcp.h tcp: refactor DCTCP ECN ACK handling 2018-10-10 22:26:00 -07:00
tcp_diag.c inet_diag: Move the INET_DIAG_REQ_BYTECODE nlattr to cb->data 2020-02-27 18:50:19 -08:00
tcp_fastopen.c bpf: tcp: Add bpf_skops_established() 2020-08-24 14:35:00 -07:00
tcp_highspeed.c Replace HTTP links with HTTPS ones: IPv* 2020-07-06 13:23:03 -07:00
tcp_htcp.c Replace HTTP links with HTTPS ones: IPv* 2020-07-06 13:23:03 -07:00
tcp_hybla.c treewide: Add SPDX license identifier for more missed files 2019-05-21 10:50:45 +02:00
tcp_illinois.c treewide: Add SPDX license identifier for more missed files 2019-05-21 10:50:45 +02:00
tcp_input.c tcp: merge 'init_req' and 'route_req' functions 2020-12-03 12:56:03 -08:00
tcp_ipv4.c tcp: merge 'init_req' and 'route_req' functions 2020-12-03 12:56:03 -08:00
tcp_lp.c net: ipv4: Fix some kerneldoc warnings in TCP Low Priority 2020-10-30 12:10:57 -07:00
tcp_metrics.c genetlink: move to smaller ops wherever possible 2020-10-02 19:11:11 -07:00
tcp_minisocks.c bpf: tcp: Do not limit cb_flags when creating child sk from listen sk 2020-10-02 11:34:48 -07:00
tcp_nv.c treewide: Add SPDX license identifier for more missed files 2019-05-21 10:50:45 +02:00
tcp_offload.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
tcp_output.c mptcp: track window announced to peer 2020-11-20 15:33:25 -08:00
tcp_rate.c treewide: Add SPDX license identifier for missed files 2019-05-21 10:50:45 +02:00
tcp_recovery.c tcp: avoid slow start during fast recovery on new losses 2020-11-02 17:17:40 -08:00
tcp_scalable.c net: ipv4: delete repeated words 2020-08-24 17:31:20 -07:00
tcp_timer.c inet: remove icsk_ack.blocked 2020-09-30 14:21:30 -07:00
tcp_ulp.c bpf: sockmap: Only check ULP for TCP sockets 2020-03-09 22:34:58 +01:00
tcp_vegas.c tcp: use semicolons rather than commas to separate statements 2020-10-13 17:11:52 -07:00
tcp_vegas.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
tcp_veno.c Replace HTTP links with HTTPS ones: IPv* 2020-07-06 13:23:03 -07:00
tcp_westwood.c treewide: Add SPDX license identifier for more missed files 2019-05-21 10:50:45 +02:00
tcp_yeah.c tcp: fix stretch ACK bugs in Yeah 2020-03-16 18:26:55 -07:00
tcp.c net-zerocopy: Copy straggler unaligned data for TCP Rx. zerocopy. 2020-12-04 13:40:52 -08:00
tunnel4.c tunnel4: add cb_handler to struct xfrm_tunnel 2020-07-09 12:51:36 +02:00
udp_bpf.c net: sk_msg: Simplify sk_psock initialization 2020-08-21 15:16:11 -07:00
udp_diag.c net: udp: remove redundant initialization in udp_dump_one 2020-11-09 16:42:49 -08:00
udp_impl.h net: pass a sockptr_t into ->setsockopt 2020-07-24 15:41:54 -07:00
udp_offload.c Merge https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-11-12 16:54:48 -08:00
udp_tunnel_core.c udp_tunnel: add central NIC RX port offload infrastructure 2020-07-10 13:54:00 -07:00
udp_tunnel_nic.c udp_tunnel: add the ability to share port tables 2020-09-28 12:50:12 -07:00
udp_tunnel_stub.c udp_tunnel: add central NIC RX port offload infrastructure 2020-07-10 13:54:00 -07:00
udp.c inet: unexport udp{4|6}_lib_lookup_skb() 2020-11-14 16:21:34 -08:00
udplite.c net/ipv4: remove compat_ip_{get,set}sockopt 2020-07-19 18:16:41 -07:00
xfrm4_input.c xfrm: state: remove extract_input indirection from xfrm_state_afinfo 2020-05-06 09:40:08 +02:00
xfrm4_output.c xfrm: fix unused variable warning if CONFIG_NETFILTER=n 2020-05-11 15:12:27 +02:00
xfrm4_policy.c net: add bool confirm_neigh parameter for dst_ops.update_pmtu 2019-12-24 22:28:54 -08:00
xfrm4_protocol.c xfrm: add route lookup to xfrm4_rcv_encap 2019-12-09 09:59:07 +01:00
xfrm4_state.c xfrm: remove output_finish indirection from xfrm_state_afinfo 2020-05-06 09:40:08 +02:00
xfrm4_tunnel.c xfrm: interface: fix the priorities for ipip and ipv6 tunnels 2020-10-09 12:29:48 +02:00