github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-07-27 17:47:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

block: invalidate cached plug timestamp after task switch

Browse Source 
blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime
and marks the task with PF_BLOCK_TS. That cache is only valid while the
task keeps running; if the task is switched out, wall-clock time
advances and the cached value must not be reused when the task runs again.

The existing invalidation covers explicit plug flushes through
__blk_flush_plug(), and the schedule() / rtmutex paths through
sched_update_worker(). It does not cover in-kernel preemption paths such
as preempt_schedule(), preempt_schedule_notrace(), and
preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and
return without calling sched_update_worker().

As a result, a task preempted while holding a plug with PF_BLOCK_TS set
can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost
then consumes that stale timestamp through ioc_now(), producing stale vnow
values for throttle decisions, and through ioc_rqos_done(), inflating
on-queue time and feeding false missed-QoS samples into vrate
adjustment.

Move the schedule-side invalidation to finish_task_switch(), which runs
for the scheduled-in task after every actual context switch regardless
of which schedule entry point was used. Keep __blk_flush_plug() as the
explicit flush/finish-plug invalidation path, and remove only the
PF_BLOCK_TS handling from sched_update_worker().

Fixes: 06b23f92af ("block: update cached timestamp post schedule/preemption")
Cc: stable@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/20260616141604.328820-3-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
This commit is contained in:
Usama Arif 2026-06-16 07:15:18 -07:00 committed by Jens Axboe
parent fd38b75c4b
commit fad156c2af
2 changed files with 14 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
include/linux/blkdev.h
View File

@ -1222,16 +1222,12 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
__blk_flush_plug(plug, async); __blk_flush_plug(plug, async);
} }
/* static __always_inline void blk_plug_invalidate_ts(void)
* tsk == current here
*/
static inline void blk_plug_invalidate_ts(struct task_struct *tsk)
{ {
struct blk_plug *plug = tsk->plug; if (unlikely(current->flags & PF_BLOCK_TS)) {
current->plug->cur_ktime = 0;
if (plug) current->flags &= ~PF_BLOCK_TS;
plug->cur_ktime = 0; }
current->flags &= ~PF_BLOCK_TS;
} }
int blkdev_issue_flush(struct block_device *bdev); int blkdev_issue_flush(struct block_device *bdev);
@ -1257,7 +1253,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
{ {
} }
static inline void blk_plug_invalidate_ts(struct task_struct *tsk) static inline void blk_plug_invalidate_ts(void)
{ {
} }

12
kernel/sched/core.c
View File

@ -5368,6 +5368,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)
*/ */
kmap_local_sched_in(); kmap_local_sched_in();
/*
* Any cached block-layer timestamp (plug->cur_ktime) is stale now,
* invalidate it.
*/
blk_plug_invalidate_ts();
fire_sched_in_preempt_notifiers(current); fire_sched_in_preempt_notifiers(current);
/* /*
* When switching through a kernel thread, the loop in * When switching through a kernel thread, the loop in
@ -7290,12 +7296,10 @@ static inline void sched_submit_work(struct task_struct *tsk)
static void sched_update_worker(struct task_struct *tsk) static void sched_update_worker(struct task_struct *tsk)
{ {
if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER | PF_BLOCK_TS)) { if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER)) {
if (tsk->flags & PF_BLOCK_TS)
blk_plug_invalidate_ts(tsk);
if (tsk->flags & PF_WQ_WORKER) if (tsk->flags & PF_WQ_WORKER)
wq_worker_running(tsk); wq_worker_running(tsk);
else if (tsk->flags & PF_IO_WORKER) else
io_wq_worker_running(tsk); io_wq_worker_running(tsk);
} }
} }