github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-06 21:58:01 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

io_uring: wake ring pollers for local task_work

Browse Source 
With IORING_SETUP_DEFER_TASKRUN completions are queued as local work, and
only become CQEs when the submitter enters the kernel to wait for events.
For an application that finds out via poll() or epoll on the ring fd when
it is time to do that, io_uring_poll() reports EPOLLIN if local work is
pending.

That only helps if the work is there at the time of the call. When
DEFER_TASKRUN was added, io_req_local_work_add() did io_cqring_wake(),
which took care of tasks sleeping in poll() as well. Then polling got a
wait queue of its own, ->poll_wq, and the wake up in
io_req_local_work_add() was replaced by waking the submitter directly.
Since then ->poll_wq is only woken when CQEs are committed, which for
such a ring is done by the very task that is waiting to be told to:

  submitter                           completion
  poll(ring_fd)
    io_uring_poll()  // nothing yet
    schedule()
                                      io_req_local_work_add()
                                        // ->cq_wait_nr: nobody waits
  // sleeps on

Wake ->poll_wq when the first entry is put on the work list. Like for the
eventfd, that must not be done right away if the work is queued from a
wake up handler, as the chain of wake ups could get back to the wait
queue whose lock is held. Use an RCU callback in that case, with a
reference to the ring.

Fixes: 130bd686d9 ("io_uring: waitqueue-less cq waiting")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-19-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
This commit is contained in:
Josef Bacik 2026-10-15 19:30:43 +00:00 committed by Jens Axboe
parent fe6ca13c98
commit b88902d874
2 changed files with 37 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
include/linux/io_uring_types.h
View File

@ -519,6 +519,9 @@ struct io_ring_ctx {
u32 iowq_limits[2];
struct callback_head poll_wq_task_work;
/* deferred ->poll_wq wake up for local task_work */
struct rcu_head poll_wq_rcu;
atomic_t poll_wq_rcu_pending;
struct list_head defer_list;
unsigned nr_drained;

34
io_uring/tw.c
View File

@ -148,6 +148,37 @@ static void io_ctx_mark_taskrun(struct io_ring_ctx *ctx)
}
}
static void io_poll_wq_wake_rcu(struct rcu_head *rcu)
{
struct io_ring_ctx *ctx = container_of(rcu, struct io_ring_ctx,
poll_wq_rcu);
atomic_set(&ctx->poll_wq_rcu_pending, 0);
io_poll_wq_wake(ctx);
percpu_ref_put(&ctx->refs);
}
/*
* io_uring_poll() reports pending local work as EPOLLIN, as it takes a trip to
* the kernel to turn it into CQEs. Let those know who are asleep already.
*/
static void io_local_work_wake_poll(struct io_ring_ctx *ctx, unsigned int flags)
{
if (!(flags & IOU_F_TWQ_IN_WAKE)) {
io_poll_wq_wake(ctx);
return;
}
/*
* Called from a wake up handler with some wait queue lock held, this
* one could lead back to it. Do it from a clean context.
*/
if (!wq_has_sleeper(&ctx->poll_wq) ||
atomic_xchg(&ctx->poll_wq_rcu_pending, 1))
return;
percpu_ref_get(&ctx->refs);
call_rcu_hurry(&ctx->poll_wq_rcu, io_poll_wq_wake_rcu);
}
void io_req_local_work_add(struct io_kiocb *req, unsigned flags)
{
struct io_ring_ctx *ctx = req->ctx;
@ -174,6 +205,9 @@ void io_req_local_work_add(struct io_kiocb *req, unsigned flags)
io_ctx_mark_taskrun(ctx);
if (data_race(ctx->int_flags) & IO_RING_F_HAS_EVFD)
io_eventfd_signal(ctx, false, flags & IOU_F_TWQ_IN_WAKE);
/* whoever sets the flag wakes ->poll_wq afterwards */
if (data_race(ctx->int_flags) & IO_RING_F_POLL_ACTIVATED)
io_local_work_wake_poll(ctx, flags);
}
/*