github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-06-04 20:46:48 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph
267463823a
linux/net/sched
History
Seth Forshee 267463823a net: sch: eliminate unnecessary RCU waits in mini_qdisc_pair_swap() 
Currently rcu_barrier() is used to ensure that no readers of the
inactive mini_Qdisc buffer remain before it is reused. This waits for
any pending RCU callbacks to complete, when all that is actually
required is to wait for one RCU grace period to elapse after the buffer
was made inactive. This means that using rcu_barrier() may result in
unnecessary waits.

To improve this, store the current RCU state when a buffer is made
inactive and use poll_state_synchronize_rcu() to check whether a full
grace period has elapsed before reusing it. If a full grace period has
not elapsed, wait for a grace period to elapse, and in the non-RT case
use synchronize_rcu_expedited() to hasten it.

Since this approach eliminates the RCU callback it is no longer
necessary to synchronize_rcu() in the tp_head==NULL case. However, the
RCU state should still be saved for the previously active buffer.

Before this change I would typically see mini_qdisc_pair_swap() take
tens of milliseconds to complete. After this change it typcially
finishes in less than 1 ms, and often it takes just a few microseconds.

Thanks to Paul for walking me through the options for improving this.

Cc: "Paul E. McKenney" <paulmck@kernel.org>
Signed-off-by: Seth Forshee <sforshee@digitalocean.com>
Link: https://lore.kernel.org/r/20211026130700.121189-1-seth@forshee.me
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2021-10-27 17:09:26 -07:00
..
act_api.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
act_bpf.c net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types 2021-10-18 12:54:41 +01:00
act_connmark.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_csum.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_ct.c net/sched: act_ct: Fix byte count on fragmented packets 2021-10-18 13:31:58 +01:00
act_ctinfo.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_gact.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_gate.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_ife.c net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types 2021-10-18 12:54:41 +01:00
act_ipt.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_meta_mark.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
act_meta_skbprio.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
act_meta_skbtcindex.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
act_mirred.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-08-13 06:41:22 -07:00
act_mpls.c net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types 2021-10-18 12:54:41 +01:00
act_nat.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_pedit.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_police.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
act_sample.c net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types 2021-10-18 12:54:41 +01:00
act_simple.c net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types 2021-10-18 12:54:41 +01:00
act_skbedit.c net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types 2021-10-18 12:54:41 +01:00
act_skbmod.c net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types 2021-10-18 12:54:41 +01:00
act_tunnel_key.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
act_vlan.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_api.c net: Fix offloading indirect devices dependency on qdisc order creation 2021-08-19 13:19:30 +01:00
cls_basic.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_bpf.c bpf: Refactor BPF_PROG_RUN into a function 2021-08-17 00:45:07 +02:00
cls_cgroup.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_flow.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_flower.c net: sched: flower: protect fl_walk() with rcu 2021-09-30 13:20:31 +01:00
cls_fw.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_matchall.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_route.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_rsvp.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
cls_rsvp.h net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_rsvp6.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
cls_tcindex.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
cls_u32.c net_sched: refactor TC action init API 2021-08-02 10:24:38 +01:00
em_canid.c net: sched: kerneldoc fixes 2020-07-13 17:20:40 -07:00
em_cmp.c net: sched: fix misspellings using misspell-fixer tool 2020-11-10 17:00:28 -08:00
em_ipset.c sched: consistently handle layer3 header accesses in the presence of VLANs 2020-07-03 14:34:53 -07:00
em_ipt.c sched: consistently handle layer3 header accesses in the presence of VLANs 2020-07-03 14:34:53 -07:00
em_meta.c sched: consistently handle layer3 header accesses in the presence of VLANs 2020-07-03 14:34:53 -07:00
em_nbyte.c net: sched: Return the correct errno code 2021-02-06 11:15:28 -08:00
em_text.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
em_u32.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
ematch.c net: sched: Fix spelling mistakes 2021-05-31 22:44:56 -07:00
Kconfig net: sched: incorrect Kconfig dependencies on Netfilter modules 2020-12-09 15:49:29 -08:00
Makefile net/sched: sch_frag: add generic packet fragment support. 2020-11-27 14:36:02 -08:00
sch_api.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_atm.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_blackhole.c Revert "net: sched: Pass root lock to Qdisc_ops.enqueue" 2020-07-16 16:48:34 -07:00
sch_cake.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-08-19 18:09:18 -07:00
sch_cbq.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_cbs.c net: don't include ethtool.h from netdevice.h 2020-11-23 17:27:04 -08:00
sch_choke.c net: sched: validate stab values 2021-03-10 15:47:52 -08:00
sch_codel.c Revert "net: sched: Pass root lock to Qdisc_ops.enqueue" 2020-07-16 16:48:34 -07:00
sch_drr.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_dsmark.c net/sched: store the last executed chain also for clsact egress 2021-07-29 22:17:37 +01:00
sch_etf.c Revert "net: sched: Pass root lock to Qdisc_ops.enqueue" 2020-07-16 16:48:34 -07:00
sch_ets.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_fifo.c net_sched: fix NULL deref in fifo_set_limit() 2021-10-01 14:59:10 -07:00
sch_fq_codel.c fq_codel: generalise ce_threshold marking for subset of traffic 2021-10-20 15:24:36 -07:00
sch_fq_pie.c net/sched: store the last executed chain also for clsact egress 2021-07-29 22:17:37 +01:00
sch_fq.c Revert "net: sched: Pass root lock to Qdisc_ops.enqueue" 2020-07-16 16:48:34 -07:00
sch_frag.c net/sched: sch_frag: fix stack OOB read while fragmenting IPv4 packets 2021-04-29 15:31:53 -07:00
sch_generic.c net: sch: eliminate unnecessary RCU waits in mini_qdisc_pair_swap() 2021-10-27 17:09:26 -07:00
sch_gred.c net: sched: gred: dynamically allocate tc_gred_qopt_offload 2021-10-27 12:06:52 -07:00
sch_hfsc.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_hhf.c Revert "net: sched: Pass root lock to Qdisc_ops.enqueue" 2020-07-16 16:48:34 -07:00
sch_htb.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_ingress.c net: sched: Pass ingress block to tcf_classify_ingress 2020-02-19 17:49:48 -08:00
sch_mq.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_mqprio.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_multiq.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_netem.c net: sched: Use struct_size() helper in kvmalloc() 2021-09-30 17:27:03 -07:00
sch_pie.c net: sched: fix misspellings using misspell-fixer tool 2020-11-10 17:00:28 -08:00
sch_plug.c Revert "net: sched: Pass root lock to Qdisc_ops.enqueue" 2020-07-16 16:48:34 -07:00
sch_prio.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_qfq.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_red.c net: sched: validate stab values 2021-03-10 15:47:52 -08:00
sch_sfb.c net/sched: store the last executed chain also for clsact egress 2021-07-29 22:17:37 +01:00
sch_sfq.c net/sched: store the last executed chain also for clsact egress 2021-07-29 22:17:37 +01:00
sch_skbprio.c Revert "net: sched: Pass root lock to Qdisc_ops.enqueue" 2020-07-16 16:48:34 -07:00
sch_taprio.c net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
sch_tbf.c net: sch_tbf: Add a graft command 2021-10-19 12:24:51 +01:00
sch_teql.c net: sched: sch_teql: fix null-pointer dereference 2021-04-08 14:14:42 -07:00