github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-09-13 15:40:03 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

rcu: Advance callbacks for expedited GP completion in rcu_core()

Browse Source 
Even when rcu_pending() triggers rcu_core(), the normal callback
advancement path through note_gp_changes() -> __note_gp_changes() bails
out when rdp->gp_seq == rnp->gp_seq (no normal GP change). Since
expedited GPs do not update rnp->gp_seq, rcu_advance_cbs() is never
called and callbacks remain stuck in RCU_WAIT_TAIL.

Add a direct callback advancement block in rcu_core() that checks for GP
completion via rcu_segcblist_nextgp() combined with
poll_state_synchronize_rcu_full(). When detected, trylock rnp and call
rcu_advance_cbs() to move completed callbacks to RCU_DONE_TAIL. Wake the
GP kthread if rcu_advance_cbs() requests a new grace period.

Uses trylock to avoid adding contention on rnp->lock. If the lock is
contended, callbacks will be advanced on the next tick.

Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit is contained in:
Puranjay Mohan 2026-06-24 06:23:52 -07:00 committed by Paul E. McKenney
parent fde2b18817
commit ec5ee5417b
1 changed files with 17 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
kernel/rcu/tree.c
View File

@ -2885,6 +2885,23 @@ static __latent_entropy void rcu_core(void)
/* Update RCU state based on any recent quiescent states. */
rcu_check_quiescent_state(rdp);
/* Advance callbacks if an expedited GP has completed. */
if (!rcu_rdp_is_offloaded(rdp) && rcu_segcblist_is_enabled(&rdp->cblist)) {
struct rcu_gp_seq gp_state;
if (rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
poll_state_synchronize_rcu_full(&gp_state)) {
guard(irqsave)();
if (raw_spin_trylock_rcu_node(rnp)) {
bool needwake = rcu_advance_cbs(rnp, rdp);
raw_spin_unlock_rcu_node(rnp);
if (needwake)
rcu_gp_kthread_wake();
}
}
}
/* No grace period and unregistered callbacks? */
if (!rcu_gp_in_progress() &&
rcu_segcblist_is_enabled(&rdp->cblist) && !rcu_rdp_is_offloaded(rdp)) {