-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmrLihgQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpqHYD/0WmtUynMfrRLjrod20cJV39pzOSBLVgAgu
9q4y6/o5UP6AFlenvvejA09AzAW14TpIx1xw+wz9W7kOIEfBFdYr+8ZlLdJCVY7+
COMDryOgoSZqd77CTUZ54V7d+z4dEOEYtkfpDBxq/0bbUrZpDB3UGHcQm1GBAj7y
FGPIAqmjJ6BH5ymNlCwjmBhgkSSsscKtlJXHUgpDNrwPwndmcg38on3QWmNJP97u
tnOO2j8Xy5TYDoGJ8liK37y56YKtR+vYiIRd1rlCvdtvtmaMBKkj3NNFMUGMEC25
1qapshKstL+20ZUmXTITGMxOfsrsZSj3nT93BV3W8jt/POMbhzX5W7zO8fS3gDKW
9XNYQXL2zMfp6iF3zIU3nsdcggK/T1uBtDXQureM8zl8QQa7XgJOCW8b8eNJZAo2
/W7Hz5TKshnrkXGEIvKWCb/UcrijHMLbtQ+eGcj/TeteMvO69bl+fQn0tzlOud0t
lqMmFsuZMa6mkYn2XV86Ie2tf0AOS5LSGKsWwvkmAg5Pkz1l5qIDTyuLFEDfWQpg
N3uLC/uo4+ap1s3ccUX6Pb8feZ1GVtjS7Wsqe04DFxFCESllbYiwkUMAGPFST7l8
SFZuafe/eJbW/MFnB/AKk5jBWMcwXP+usBWq/0w1/ETA4DhWXLQQnIefI1FoZpZa
OjOfFuKDYA==
=GaD2
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.3-20261010' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Don't spin on local work the exiting task can't run.
For a ring still R_DISABLED when the task got its tctx node, the
submitter is set later and may be another task, so DEFER_TASKRUN
local work stays pending forever and the exiting task spins in
do_exit().
- Don't charge the SQ/CQ rings or provided buffer rings to
RLIMIT_MEMLOCK. The region API conversion started charging them,
which the rings had been exempt from.
- Don't exempt disabled rings from task restrictions on register
- Populate the CONNECT filter fields for a 24-byte sockaddr_in6
* tag 'io_uring-7.3-20261010' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/cancel: don't spin on local work the task can't run
io_uring/net: populate CONNECT filter fields for 24-byte sockaddr_in6
io_uring/register: don't exempt disabled rings from task restrictions
io_uring/kbuf: don't charge provided buffer rings to RLIMIT_MEMLOCK
io_uring: do not charge the SQ/CQ rings to RLIMIT_MEMLOCK
io_uring_cancel_generic() refuses to sleep while any ring on the exiting
task's tctx has local work pending. It assumes the task is that ring's
submitter and that the next io_uring_try_cancel_requests() pass will run
the work. This does not hold for a ring that was still
IORING_SETUP_R_DISABLED when the task got its tctx node from
io_uring_create() or io_ringfd_register(). The submitter is set later by
IORING_REGISTER_ENABLE_RINGS, possibly from another task, and only the
submitter may run local work on an IORING_SETUP_DEFER_TASKRUN ring. To
the exiting task the work stays pending forever, the loop never reaches
schedule(), and the task trips the WARN_ON_ONCE() once and then spins in
do_exit().
An unprivileged user can set this up at will by creating a disabled
ring, having a child enable it and stop, posting to the ring with
IORING_OP_MSG_RING and exiting with a request in flight that only the
child can complete. The exiting thread burns a CPU until the child runs
or dies, and with panic_on_warn set the WARN panics the kernel.
Fix by skipping the sleep only when io_allowed_defer_tw_run() allows the
task to run the work. This is reasonable since work it cannot run cannot
complete any of its requests either, so nothing is lost by sleeping.
Drop the WARN_ON_ONCE() since handing a disabled ring to another thread
is a supported scenario, not a bug state.
Fixes: 360cd42c4e ("io_uring: optimise io_req_local_work_add")
Reported-by: AutonomousCodeSecurity@microsoft.com
Signed-off-by: Cen Zhang (Microsoft Security FORGE Labs) <cenzhang@linux.microsoft.com>
Link: https://patch.msgid.link/20261008191602.45101-1-cenzhang@linux.microsoft.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_connect_bpf_populate() fills connect.port and connect.v6_addr for
AF_INET6 only when addr_len >= sizeof(struct sockaddr_in6), i.e. 28
bytes.
The IPv6 connect paths accept SIN6_LEN_RFC2133 (24 bytes), which
already holds sin6_port and sin6_addr. io_connect() passes addr_len to
the socket layer unchanged.
At addr_len 24 the filter sees family AF_INET6 with port and address
zero, while the kernel connects to the real destination. A filter that
denies an IPv6 port or address is bypassed. Denying connect.port == P
rejects [::1]:P with -EACCES at addr_len 28 but connects at addr_len 24.
Use SIN6_LEN_RFC2133 as the threshold. sin6_scope_id stays zero when
absent.
The threshold mismatch was found by an AI coding assistant (Claude Code)
comparing this hook against the net/ipv6 length checks on my
instructions; it also drafted the reproducer and this patch. I verified
the bypass and the fix in a VM on 7.3-rc6 and that 7.2 has the same
code.
Fixes: 899bea8248 ("io_uring/net: allow filtering on IORING_OP_CONNECT")
Cc: stable@vger.kernel.org # 7.2
Signed-off-by: Natasha Klaus <natalie.klaus@runtimeverification.com>
Link: https://patch.msgid.link/20261009133807.730724-1-natalie.klaus@runtimeverification.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
A ring created with IORING_SETUP_R_DISABLED is exempt from its register
restrictions until it is enabled. This lets the creator register the
restrictions and then enable the ring with IORING_REGISTER_ENABLE_RINGS,
whether or not that opcode is in the allowed set.
Since commit ed82f35b92 ("io_uring: allow registration of per-task
restrictions"), a ring also inherits restrictions from the task that
creates it, and the same exemption applies to those. A task whose
register opcodes are restricted can therefore create a disabled ring,
issue any register opcode on it, and then enable it.
Mark rings whose restrictions were inherited from the task, and keep
checking register opcodes for them while the ring is disabled. Enabling
such a ring now requires IORING_REGISTER_ENABLE_RINGS to be in the
task's allowed set. Per-ring restrictions registered on a disabled ring
work as before.
Fixes: ed82f35b92 ("io_uring: allow registration of per-task restrictions")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Eugene Gvozdetsky <gvozdet@gmail.com>
Link: https://patch.msgid.link/20261009-iouring-task-restrict-fix-v1-1-a49daf55a12c@gmail.com
[axboe: amend with inheritance flag fixup]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Just like with the SQ and CQ rings, the provided buffer rings were also
not charged against the memlock limit prior to the below mentioned
commit. Restore that behavior.
Fixes: ef62de3c4a ("io_uring/kbuf: use region api for pbuf rings")
Reported-by: Hengyu Liang <hengyul@cs.unc.edu>
Cc: stable@vger.kernel.org
Tested-by: Hengyu Liang <hengyul@cs.unc.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Commit 8078486e1d ("io_uring: use region api for SQ") and commit
81a4058e0c ("io_uring: use region api for CQ") made io_uring_setup()
allocate the rings with io_create_region().
However, io_create_region() charges the memory to RLIMIT_MEMLOCK, and
the rings had been exempt from that limit since commit 26bfa89e25
("io_uring: place ring SQ/CQ arrays under memcg memory limits"). As of
now, a user without CAP_IPC_LOCK gets ENOMEM from io_uring_setup() when
their rings exceed the limit, which is 8 MiB by default. PostgreSQL
developers have already hit this in their io_uring tests [1].
The issue can be reproduced with a simple liburing program, run as an
unprivileged user:
#include <liburing.h>
#include <stdio.h>
int main(void)
{
static struct io_uring ring[64];
int i;
for (i = 0; i < 64; i++)
if (io_uring_queue_init(4096, &ring[i], 0) < 0)
break;
printf("%d rings\n", i);
return 0;
}
Before those commits (v6.13), it prints "64 rings". After those commits
(v6.14), it prints "21 rings".
This patch makes io_create_region() take the user to charge, and passes
no user for the SQ/CQ rings.
Link: https://www.postgresql.org/message-id/flat/667f7dcf-404b-4898-a426-57ca6e0617f6@gmail.com [1]
Fixes: 8078486e1d ("io_uring: use region api for SQ")
Fixes: 81a4058e0c ("io_uring: use region api for CQ")
Cc: stable@vger.kernel.org
Signed-off-by: Hengyu Liang <hengyul@cs.unc.edu>
Link: https://lore.kernel.org/io-uring/20261006125732.3425762-1-hengyul@cs.unc.edu/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmq/wk0QHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpkD/D/4mOJ3xw+h4BAzHa6SyquNgxXls8MXeJOrv
DfTDjclyBMMY60ALlZENTfoM1lIY22eFyl/e0guc9WnDqFEG2MwOAmLh1+YLDEdP
lX2c0ICo1Q3/JKUV2y6JPJeERalq+yP1HMl7ZhH2nIXX0vxW2ALGzWtmr1G5DfyI
KakNq7pMSQC554p5dZe3+X9HcxYFX8/EslygmNVquiRo6hm1Ys4srG9kJw/iOsw4
7ryI6hRAMjgmI39h/RVBnUVvGSgo+MU2OzPCYUqIN6lK+fvPiGaPfWib5NTOdpSJ
09PNhtB4gijCW+bnYAvXEQohq2ikPOGPmRYN320vUF6rryWWRJLF86IbE4t+/j3n
KV1i0UZK1Y7cL3HMRr1SKfedsNRjpoTdyBlMxC+hWbjcIFQNV1oLo6gEu5r/8M3b
oMiaQl18KHKcWhw5bbLGr9dSjgN2XHCZ0j38QwzGbbjKC+TRS0mYrjXCgNzEfdIC
tsnLWQE5CKPhWw2u3R/lMRFtSWs/ZWKOamONNGgxgGt53kbKOx1dY7UvhEx/cvxu
nA84L6sgE6KQzkQ5PshF3OFXw7ckm3v7hh0aiUtjczGz7FUK6db/h3sudxfiXazg
P7veXjlqnw4ny8MzfnSXkhjXM/zD3s8c33dNjEcgvVXVLCAkWWrTsiO9NTUA3fMI
7IC5n7RoSg==
=Sg1l
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.3-20261002' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a task_work add use-after-free with SQPOLL.
The sqpoll thread could pop and complete the last request while
io_req_normal_work_add() was still looking at them after the mpscq
push.
Use the same approach as DEFER_TASKRUN to protect from that, holding
an RCU read lock across the add, and have exit wait for an RCU grace
period for SQPOLL rings as well.
- CQE32 ring fixes: correct the free entry check for 32b CQEs, zero the
big_cqe for aux CQEs, and only post the dummy skip CQE on CQE_MIXED
rings
- Mark the source filter table as COW when cloning bpf filters, so
registering another filter on the source doesn't modify the shared
table in place
- Initialize the task context before running the BPF loop
- Requeue zcrx multishot receives stopped by a local resource
- End a TX_TIMESTAMP multishot cmd when the CQ is full (lollipopkit)
* tag 'io_uring-7.3-20261002' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring: fix task_work add use-after-free with SQPOLL
io_uring/cmd_net: end TX_TIMESTAMP multishot when the CQ is full
io_uring/zcrx: requeue multishot receives stopped by a local resource
io_uring: initialize task context before running the BPF loop
io_uring: zero big_cqe for aux CQEs on CQE32 rings
io_uring: fix free entry check for 32b CQEs on CQE32 rings
io_uring: only post the dummy skip CQE on CQE_MIXED rings
io_uring/bpf_filter: mark source as COW when cloning filters
For SQPOLL, the sqpoll thread runs task_work off its own list and
doesn't need to be told about it, so it can pop and complete a request
as soon as mpscq_push() has published it. If that was the last request
holding the ring alive, io_ring_exit_work() then frees the ctx, stops
the thread, and with it the tctx, while io_req_normal_work_add() is
still looking at both after the push:
BUG: KASAN: slab-use-after-free in io_req_normal_work_add+0x439/0x510
Read of size 4 at addr ff11000000ca0000 by task repro/55
(...)
BUG: KASAN: slab-use-after-free in queue_work_on+0x25/0x70
Write of size 8 at addr ff11000000c3bd00 by task repro/55
Hold an RCU read lock across the add, like io_req_local_work_add() already
does, and have io_ring_exit_work() wait for a grace period for SQPOLL rings
as well before freeing the ctx.
Reported-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://lore.kernel.org/io-uring/20260924205056.3759980-2-Jeremy.Jean@oss.cyber.gouv.fr/
Fixes: af5d68f889 ("io_uring/sqpoll: manage task_work privately")
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
SOCKET_URING_OP_TX_TIMESTAMP arms an EPOLLERR apoll multishot and, on
each error queue edge, posts one 32 byte aux CQE per timestamp skb. Aux
CQEs have no overflow backing, so io_uring_cmd_post_mshot_cqe32() fails
once the CQ ring is full. The loop then stops, splices the unprocessed
skbs back onto sk_error_queue and returns -EAGAIN.
-EAGAIN is IOU_RETRY, so the request goes idle until the next poll
event. io_arm_apoll() forces EPOLLET and draining the CQ does not re-run
the command, so the spliced-back timestamps stay queued until an
unrelated later timestamp produces a new edge.
End the multishot when a CQE cannot be posted, as multishot poll and
recv already do. There is no partial result to report, so complete with
-ENOBUFS: no CQ space is left for the timestamp CQEs. This command does
not use provided buffers, so the value cannot be confused with a buffer
ring running empty. The terminal CQE still reaches userspace because
request completions can go to the overflow list. The application sees a
CQE without IORING_CQE_F_MORE, which it already has to handle, and the
first issue of the re-armed request delivers the queued timestamps.
A timestamp that cannot be extracted keeps its current behaviour.
This was found with LLM assistance while reviewing io_uring multishot
handlers for the pattern behind a RECV_ZC stall reported earlier (see
Link), and confirmed with a reproducer on v7.3-rc2 and an A/B run of
this patch.
Fixes: 9e4ed359b8 ("io_uring/netcmd: add tx timestamping cmd support")
Link: https://lore.kernel.org/all/010001a0cd914529-23df2f94-bbe0-49cd-ae5d-05922356fa12-000000@email.amazonses.com/
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-6-sol
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: lollipopkit <a@lolli.tech>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/20260927074206.65427-1-a@lolli.tech
Signed-off-by: Jens Axboe <axboe@kernel.dk>
A multishot RECV_ZC can go idle with unread TCP data when a
receiver-local resource runs out. An empty copy-fallback niov freelist
returns -ENOMEM; a full CQ returns -ENOSPC. After partial progress the
stop reason is lost: io_zcrx_copy_chunk() and io_zcrx_recv_skb() report
the copied bytes, and tcp_read_sock() drops an error from a later call
once earlier data was consumed. The edge-triggered request is then not
requeued, so it cannot consume the remaining data until another socket
event arrives.
Record in io_zcrx_recv_skb() when a walk stops before consuming the
length it was offered, and requeue if the pass still returned data. The
failed chunk is first on the next pass. The resource is usually still
exhausted by then, so that pass fails at once and ends the request with
an error CQE, which the application must handle by re-arming. The
request does not spin, and the stall becomes a visible error. Keep the
existing skb limit and SOCK_DONE handling.
Fixes: 11ed914bbf ("io_uring/zcrx: add io_recvzc request")
Fixes: bc57c7d36c ("io_uring/zcrx: add copy fallback")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-6-sol
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Junyuan Feng <a@lollipopkit.com>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/010001a0cd914529-23df2f94-bbe0-49cd-ae5d-05922356fa12-000000@email.amazonses.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Submitting SQEs through an io_uring BPF loop can trigger a NULL pointer
dereference in io_submit_sqes(), as shown by the following arm64 report:
[ 2049.380301] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 2049.388788] pc : io_submit_sqes+0x48/0x6c4
[ 2049.389153] lr : bpf_io_uring_submit_sqes+0x10/0x1c
[ 2049.396310] Call trace:
[ 2049.396531] io_submit_sqes+0x48/0x6c4 (P)
[ 2049.396893] bpf_io_uring_submit_sqes+0x10/0x1c
[ 2049.397293] bpf_prog_1a2a95cb199202b5+0x3c/0x68
[ 2049.397699] bpf__io_uring_bpf_ops_loop_step+0x5c/0x94
[ 2049.398149] io_run_loop+0x70/0x2e4
[ 2049.398461] __arm64_sys_io_uring_enter+0xf0/0x710
[ 2049.398882] invoke_syscall+0x54/0x10c
[ 2049.399414] el0_svc_common.constprop.0+0x40/0xe0
[ 2049.399829] do_el0_svc+0x1c/0x28
[ 2049.400125] el0_svc+0x38/0x1d0
[ 2049.400409] el0t_64_sync_handler+0xa0/0xe4
[ 2049.400779] el0t_64_sync+0x198/0x19c
io_uring_enter() dispatches to io_run_loop() before reaching the normal
submission path's io_uring_add_tctx_node() call. The loop only checks
whether the caller is allowed to run task work; it does not initialize
current->io_uring or associate the task with the ring. A BPF call to
bpf_io_uring_submit_sqes() then reaches io_get_task_refs(), which assumes
that current->io_uring is valid.
A ring created with IORING_SETUP_R_DISABLED, IORING_SETUP_SINGLE_ISSUER
and IORING_SETUP_DEFER_TASKRUN can be enabled by a different task that
has never used io_uring. Enabling the ring makes that task the submitter
without allocating its io_uring task context. Its first BPF-driven
submission of a pending SQE can therefore cause a kernel Oops.
Call io_uring_add_tctx_node() in io_run_loop() before invoking the loop
and propagate any initialization error. Do this before acquiring
uring_lock, since task context setup may acquire the same mutex. This
also establishes the task-to-ring association used for cancellation.
Fixes: 033af2b3eb ("io_uring: introduce callback driven main loop")
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Link: https://patch.msgid.link/20260923014310.591292-1-yaokai34@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_fill_cqe_aux() only clears big_cqe if cqe32 is set, which is only the
case for CQE_MIXED. On a CQE32 ring every CQE is 32b, and the extra 16b
are left holding whatever was in that slot before.
Clear it for CQE32 rings as well, like it was before CQE_MIXED support
was added.
Fixes: e26dca67fd ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
On CQE32 rings, len is in units of 32b CQEs when io_cqe_cache_refill()
checks for free entries, so requiring cqe32 + 1 of them means insisting
on 2 free CQEs where 1 is enough, and a ring with a single free entry
needlessly goes to overflow.
Check for a free CQE directly in that case, and keep the cqe32 + 1 check
for CQE_MIXED where the units are 16b.
Fixes: e26dca67fd ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_cqe_cache_refill() posts a dummy skip CQE when a 32b CQE would
straddle the end of the ring, so the real one can start at offset 0. That
only makes sense on CQE_MIXED rings, where slots are 16b and a 32b CQE
needs two of them back to back. On a CQE32 ring every slot is already
32b, but io_fill_cqe_aux32() still passes cqe32 == true, and the dummy
then gets written at cqes[off] rather than cqes[off << 1], inside another
CQE.
Limit the dummy to CQE_MIXED rings.
Fixes: e26dca67fd ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_bpf_filter_clone() shares the filter table between source and
destination and marks the destination as needing a copy before it can be
modified, but leaves the source as is. Registering another filter on the
source then modifies the shared table in place, and the change shows up
in everything that cloned it.
Mark the source as COW as well.
Fixes: ed82f35b92 ("io_uring: allow registration of per-task restrictions")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919112523.3872581-1-benquike@gmail.com
[axboe: edited patch and commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmqj5fsQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpmGsD/0W7eNXJd/tFNPOdDwTr1/3yciqYXw5QO5Q
eb2SLyr34sAODOItTYoBrIZ5b1Mbgm0yhT1wJ3YrVKLNLVSvbDPNtx5qtnAGcKvm
09XsPmvlDcW8JLRsjd5dcsP+EDmgTNLuNGSeVycuTG91f5VD63vPHTKUhgKDabtC
InM1LbIs2rz0tojdV6n4MOOF2AwZAOHqFtac+cQUCwuJhNtwUFClSUHxqYoP+JzE
97ores3f+ArAtmOoVeH2wSCZTmwdB7MxOB8jmt8fdNSprW8nxTO3oT/eQzYXvlM9
7LRJV/6SGVsWjuoEHAfW8G0p+JOzGGFvFIXGWEBenjtNmlyXJqfxYhkcLMw826ob
pct3Rc5nheZbxKdCkHnY1AmEibS/c1kTZbDvSBXk9GLBKyQuI7B3yo163zMs3JjO
YSAlLllDjdzOR6gCPg5HHZlywBWtXveC/xVBzDq7b1q07ysHFNO1yQplMM3U8sC0
Eic4RcFU4ld+/L2GmbdgRZ9ataZsju+RFojSogA0NzR18HqVNQuZiVbY764GSop9
HVW28lzoYamgvg5fLWJPrzJmdFrj/p8pi+cw1dK8uMk9VnFwdA740Nfhz1NWFS0Y
TpvJtxc0f2ofrI1ilaa95vHC+wMyds1zxq9Bf4i9cgIhT17WqvE4SD0JMXmvza4k
Yi72jzTTfQ==
=zpCb
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.3-20260911' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a deadlock in the write path with superblock freezing
- Fix an issue where a provided buffer ring would overconsume when
using MSG_TRUNC
- Keep the CQE flags on iopoll requests when adding kbuf flags
* tag 'io_uring-7.3-20260911' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/rw: keep CQE flags on iopoll requests when adding kbuf flags
io_uring/net: don't overconsume buffers when using MSG_TRUNC
io_uring/net: let io_recv_buf_select return the length of the buffer region
io_uring/rw: end write accounting from ->ki_complete
io_do_iopoll() assigns the result of io_put_kbuf() to the request's CQE
flags upon completion. This overwrites any CQE flags that may have been
set by the opcode-specific layer. (For example, if __io_uring_cmd_done()
had set IORING_CQE_F_32, it would be cleared.) Switch the = to an |= so
the kbuf flags are added to the existing CQE flags rather than replacing
them. io_req_rw_complete() does the same with the io_put_kbuf() result.
Fixes: e26dca67fd ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Reported-by: sashiko-bot@kernel.org
Link: https://sashiko.dev/#/message/20260827191705.D53C91F000E9%40smtp.kernel.org
Signed-off-by: Caleb Sander Mateos <csander@purestorage.com>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://patch.msgid.link/20260902210200.2336720-1-csander@purestorage.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
When a recv/recvmsg is issued with MSG_TRUNC and the incoming packet is
larger than the provided buffer, the net layer returns the full length
of the packet rather than the number of bytes actually copied into the
buffer. As a result, io_uring advances more of the provided buffer ring
than was actually filled. Use the actual filled region size to consume
the buffer, but still return the full size to preserve MSG_TRUNC
semantics.
Take care with multishot, because that seems to already truncate the
consumption based on the available payload size.
This was reported in https://github.com/axboe/liburing/issues/1619.
Fixes: ae98dbf43d ("io_uring/kbuf: add support for incremental buffer consumption")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260728191454.1850326-1-krisman@suse.de
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260902230041.1320658-3-krisman@suse.de
[axboe: fold in size_t unsigned fix]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
In preparation to using this field as an upper limit to truncation,
return the size of the allocated region.
Fixes: ae98dbf43d ("io_uring/kbuf: add support for incremental buffer consumption")
Cc: stable@vger.kernel.org
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260902230041.1320658-2-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Commit b000145e99 moved both the fsnotify calls and the write
accounting out of the kiocb completion handler and into the
io_req_rw_complete() task_work. However, only the fsnotify part actually
needed to move as it may sleep. Ending the write accounting is just a
percpu_up_read() on the superblock writers sem.
Deferring it is a problem, because it makes dropping SB_FREEZE_WRITE
protection depend on the ring owner getting to running task_work. But
the task may be blocked in freeze_super(), causing it to never get to
that:
task io-wq worker
--------------------------------------------------------------
io_write()
io_kiocb_start_write() (takes sb_writers, hidden from
lockdep by __sb_writers_release)
write_iter() -> -EIOCBQUEUED
ioctl(FS_IOC_SHUTDOWN)
bdev_freeze()
freeze_super()
percpu_down_write() <- waits for the reader above
io_write()
kiocb_start_write()
percpu_down_read() <- queued
behind the
writer
<bio completes>
io_complete_rw()
queues io_req_rw_complete() <- never runs, task is in D state
End the write from io_complete_rw() instead, and leave only the fsnotify
calls in task_work.
Reported-by: syzbot+2eb3d983669d3e49d4fa@syzkaller.appspotmail.com
Cc: stable@vger.kernel.org
Fixes: b000145e99 ("io_uring/rw: defer fsnotify calls to task context")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmqR/ZwQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpr6GD/9uEDS/3lBXkP5r6qO4E5v/6eN7Sz1F/gZ4
vkH4reHh3U3VSjwOaWJHLXCkJV06JmlJAggCn0dq+P8cXWKH9VpwppzpedY/Ua05
oiDXgyoaAzhgYzE/seCALdFR0zOyvhSlluDiucrwWCcx0h4HOEnysZ3Q4RQ8gXXx
2mQ+M/n2WdXPPUHPEGezPL9PvtitFjUFr8WBeWksCtFQ+Ycyk6+NDPSanxeOjJCI
FhgQDxiUIWSEItEniuwq4lUQFtImkRyUvBPPbPb51ahzkCc1QSjr8WVtfO0RJtON
uiUU9hkTcKis/ufR+esikrjV2nwxo3fv2WkknrBfJ5jNFiLp/CR+FPZRwH3A/AoH
Onoi3qcpcHrXzQD4j/UhyUcaIM7sntwour4PBg8cKUh/lwZkt94+orp6lniiV/Dd
aS3IKmCymQjpf5N3b0QnQSi2lOGk6bUKyQD8YUnA/N3rw3olfkzSGBDz6Y4gQxAY
XpvmuPDXndhfn17JSGpsCjohWkWEUuSQh1fx5JLsV9XkaPaeA89/epUzu2+b1Kxw
c+/uaCkQsGGLpHa6xWhJLGocyYZViDUYeIPz8C7gqTDXTMU70mtZvILl+aZjVrjF
iU2aSWDuS9oD0g16yT7FY8pBQvB8RnuNCUmS92QIr+VMqnp7793DPD5gwN88k+8L
4x4gInB+ag==
=I3ma
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.3-20260828' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
"A few smaller fixes for io_uring that should go into the 7.3-rc1
kernel, all three headed to stable as well. This contains:
- A few fixes around cancellation and teardown for waitid
- Cap the user size for the query interface copy-out"
* tag 'io_uring-7.3-20260828' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/waitid: avoid siginfo copy during ring teardown
io_uring/waitid: honor task_work cancellation
io_uring/query: cap user size passed to copy_struct_to_user
During ring teardown, io_ring_exit_work() cancels outstanding requests
from a kworker with a NULL tctx. The waitid cancellation path eventually
reaches io_waitid_finish(), which copies the stored siginfo to the
userspace pointer supplied with the request.
Ring-wide teardown does not run in the task context that submitted the
request, so it must not access that task's userspace pointer. Depending
on the address and mm state, the copy may fail with -EFAULT, but the
uaccess itself is inappropriate from the teardown kworker.
Use a no-copy cancellation callback when io_waitid_remove_all() is
called without an owning task context. Complete the request with
-ECANCELED while releasing the waitid state without touching siginfo.
Keep the existing siginfo handling for explicit async cancellation and
task-scoped cancellation.
Fixes: f31ecf671d ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org
Signed-off-by: Hui Su <sh_def@163.com>
Link: https://patch.msgid.link/20260818103336.1922818-3-sh_def@163.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_waitid_cb() may run through the fallback task_work path when
task_work_add() can no longer queue work to the originating task. The
fallback runs from a kworker and io_uring marks such task work as
canceled through tw.cancel.
io_waitid_cb() currently ignores tw.cancel and calls __do_wait().
waitid is task-context dependent: __do_wait() performs child lookup
relative to current, and the retry path also uses
current->signal->wait_chldexit. If the callback runs from the fallback
kworker, current is therefore not the task that submitted the request.
Honor tw.cancel before entering __do_wait(). Complete the request with
-ECANCELED and skip the siginfo copy, since canceled task work may run
without the submitting task's userspace execution context.
Keep the existing siginfo handling for normal waitid completion and
explicit cancellation.
Fixes: f31ecf671d ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org
Signed-off-by: Hui Su <sh_def@163.com>
Link: https://patch.msgid.link/20260818103336.1922818-2-sh_def@163.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_handle_query_entry() clamps hdr.size for the inbound copy_from_user()
but keeps the original user value as usize. copy_struct_to_user() uses
that usize and, when it is larger than the kernel result, clear_user()s
the trailing bytes.
As hdr.size is a __u32, a query can request nearly 4 GiB of zeroing,
including on the error path where res_size stays 0. The interface is
reachable without a ring via IORING_REGISTER_QUERY.
Reject sizes larger than PAGE_SIZE, as recommended for copy_struct_*
interfaces.
Fixes: c265ae75f9 ("io_uring: introduce io_uring querying")
Cc: stable@vger.kernel.org # 6.18+
Signed-off-by: Laxman Acharya Padhya <acharyalaxman8848@gmail.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260821103317.91437-1-acharyalaxman8848@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQSQHSd0lITzzeNWNm3h3BK/laaZPAUCao1RzQAKCRDh3BK/laaZ
PJ2EAP9dfslni4sFYqtXv/43Wk2iVwwdGRRSy5Tfoq8nCeNXpQEAygIv9BuJYpl6
DL861AOn/NfDBXpeU0vX4WQpTgtFIgc=
=Av0t
-----END PGP SIGNATURE-----
Merge tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
Pull fuse updates from Miklos Szeredi:
- Improve performance of the io-uring transport by introducing buffer
pools and zero-copy (Joanne)
- Fix lots of bugs (Baokun Li)
- Fix io-uring initialization issues (Joanne, Bernd)
- More prep work for large folios (Joanne)
- Don't limit buffered read to 128k (Jim Harris)
- Fix zeroing of page end (dirtied with mmap) on file size extension
(Jimmy Zuber)
- Improve performance in certain cases with wake_up_sync() when queuing
request (Xuewen Yan)
- Misc fixes and cleanups (Xuewen Yan)
* tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse: (35 commits)
fuse: zero the partial EOF page when extending a file
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
fuse: Fix the condition to enable over-io-uring
fuse: invalidate the correct range after O_APPEND direct write
selftests/fuse: test post-EOF page zeroing when a file is extended
fuse: wake one waiter per freed slot when raising max_background
fuse: use min_not_zero() in fuse_init_server_timeout()
fuse: copy request headers via a stack buffer for io-uring
fuse: give wakeup hints to the scheduler for synchronous requests
fuse: check for NULL root inode in fuse_fill_super_submount
fuse: reject a duplicate fd= mount option
cuse: wait for pending RCU callbacks on module exit
fuse: fix invalidate lock leak on open O_TRUNC DAX failure
fuse: fix invalidate lock leak on setattr writeback failure
fuse: wait for FR_FINISHED on abort_on_kill to prevent use-after-free
fuse: make dentry_tree_work static
docs: fuse: document io-uring buffer pool and zero-copy uapi
fuse: add zero-copy over io-uring
fuse: support registered buffer pools in io-uring
fuse: add io-uring buffer pools
...
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmqGD2AQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpomLD/9rr3SIQvdQRTnSCJlw1cFypSrnRhGAx1g8
v1F7UXuf4Abt4GIerbdXmBTx9zZRctpcOC41ReQk3YKkYuukuvna+Z6tBGf5I5+e
3SR3Zkrj2k4zyNC49g0/QkUJkTTS4tlsBFhNJtY3Q5BBIRMRd/sPvZ+cLnD7g/wH
3zM9IaEMk9xu9mF1nGI7LRl1qY0IOr4ynQpqohl5sEL3TQWgIa0r2B0jhGacBAvC
GGDwIwGDrM6oGwCn6mLN78O5odhLyTufIMNy3EnVw7BcDpLApcnJzYYjGzv+ElCM
WLBJwWYj+PYeoenuJuFhbRihbOPg5gZT3Mn8CUhLxUYzwZvIdFGWOjcGffD9sVu/
JO4nB6A+2AG/wgsb2YVKefhN86FoD0F/2hhdEdgZPJE6a+7+xeyyp9IzyQEWd0cg
Seo06fxE4pwF+dczXT3jzU3LXSZb/Jbh9hzu163VMSWbGYDCpDzfipqZGeN6DWE7
nG2NssAQtjtZrgdggYbZXtwecLZhu3cMj3jbhS4wSUzhQFW42yceJstCx4wpRhW9
m9EXjHQnRFMAb2TjUC3FzEncjbxL+uACjcKhTetd1hLbTI5I8KArZXjam1OdJOHV
urdtsru79LanQLAgRwWYPeKCZGNhX7I2o3OsmVDLQgHIDaZguTcPUVQs3wVui7Dk
IDmmrtwzpg==
=G1l4
-----END PGP SIGNATURE-----
Merge tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe updates via Keith:
- Enable Clang context analysis for the nvme host driver, adding
context annotations across core, fabrics, rdma, tcp and pci
- nvmet reservation state exposed through a new namespace-level
debugfs directory, plus ABI documentation for the host sysfs and
target configfs interfaces
- nvme-tcp host memory disclosure fixes on the read path: reject a
read that transferred too few bytes, don't accept C2HData based
on blk_rq_payload_bytes() alone, and fix the R2T case for a read
command
- Parallelize nvme-rdma I/O queue allocation and startup (Surabhi)
- Apple nvme fixes and quirks: page aligned admin queue buffers,
destroy the admin queue on removal, and various DMA/NVMMU
correctness fixes
- A large pile of nvmet and host fixes for out-of-bounds reads,
refcount/resource leaks, and NULL derefs across auth, zns,
passthru, pci-epf, rdma and configfs
- Various other fixes and cleanups
- MD updates via Yu Kuai:
- llbitmap reshape support, the large series wiring exact bitmap
mapping and reshape lifecycle through raid5 and raid10, growing
the page cache in place, and remapping checkpointed bits as
reshape progresses
- raid5 fixes for lockless max_nr_stripes and recovery_offset
accesses, a reshape deadlock with more failed devices than max
degraded, and bitmap batch counter consistency
- Atomic write handling for raid1/raid10, and removal of the
REQ_NOWAIT support from raid1/10/456
- raid5-ppl use-after-free fix in ppl_do_flush()
- A batch of smaller fixes across md core and the bitmap code
- s390/dasd ESE full-track write support and the surrounding
infrastructure, plus enabling CONTEXT_ANALYSIS for s390/block
- RWF_DONTCACHE support for block devices, built on new task-context
bio completion infrastructure, and wiring it up for the iomap and
buffer dropbehind writeback paths
- Async io_uring zone reset all, plus zone management command cleanups
allowing REQ_NOWAIT and tightening conventional zone rejection
- Block integrity refactoring: lift BIP_CHECK_FLAGS to the shared
header, handle nogenerate/noverify properly in fs-integrity, and drop
the blk-integrity.h include from bdev.c
- Split out a new blk_plug.h header
- ublk improvements: add UBLK_F_IO_DESC_SIZE, split request validation
from io_desc init, reject non-power-of-2 zone sizes in SET_PARAMS,
and a series of hardening fixes around map/unmap and auto buf reg
- null_blk cleanups and configfs serialization fixes
- nbd queue freeze removal on the setup paths, and a new
pre_defined_connections module parameter for pre-created devices
- blk-cgroup fixes for the race between policy activation and blkg
destruction, and accounting per-cpu stats over possible CPUs across
blk-stat, iolatency, iocost and kyber
- Various dio fixes: leak on metadata mapping error, validate user
space vectors during extraction, and set dma_alignment from the
backing file for loop and zloop direct I/O
- bio cleanups
- Various other fixes and cleanups all over
* tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (241 commits)
nbd: add pre_defined_connections module parameter for pre-created devices
nbd: remove queue freeze for newly created nbd from netlink path
nbd: factor out a nbd_genl_foreach_sock
nbd: skip queue freeze when setting size at device startup
nbd: remove queue freeze in nbd_add_socket
nbd: clear queue limits on disconnect
nbd: disallow NBD_SET_SOCK on an active device
nbd: simplify find_fallback() by removing redundant logic
blk-mq: add missing call to srcu_barrier() in blk_mq_free_tag_set()
block: mtip32xx: synchronize ioctls with device removal
ublk: avoid teardown retry loop on xarray allocation failure
null_blk: fix UBSAN shift-out-of-bounds when zone_size is 0 or overflows
block: don't include blk-integrity.h in bdev.c
xfs: avoid double deferrals for RWF_DONTCACHE writes
loop: Fix recently introduced lock inversion
block: set QUEUE_FLAG_DYING unconditionally in blk_mark_disk_dead()
swim3: Add missing MODULE_DESCRIPTION
selftests: ublk: add SET_PARAMS validation test
selftests: ublk: add helper for SET_PARAMS
ublk: reject non-power-of-2 zone sizes in SET_PARAMS
...
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmqGD3YQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgple3D/4wIQyeuf0U2NfM78K4sAX8FjEPzNg/WDzW
LRozg+Sldz2PNpVIbsPqDT9oW+K46zK62ptfF2gTl++dAjP2ER+HD44aXnwLh2JF
Cz54n2PNzR6TY4PeMLwR/fJ7SsAf5pQSRJqsV5aQPCqfEEgviLXczXQP0RcLQdB4
ksqay2xN+MaRGbgTqYYxc53ghMRSxiUI2Hm8boLrKAY9xTIJiZHyjVZmdlv5qp0s
SUCHAnlkTN/zC0zrAi1N+uK+kCTXuQRYQSMbW7LKkKHseqC/4ZwT1HEL20s74cV4
NLGrl2qdAU4euygUADILXKPwz4T0qfZyuIPd4OOkBtoND/uspGQ93r77XJsb8i9J
4G0cNK4y5lppuhp5Qo51a6zLfVVjOoSXSS8hI869ELc2w2iVDmNWccnxxySywBkd
XwKJ/BuRtB+xj9PvxuFgIxBCZTD17Auk8YmY+ifVainrAdNC78/BOiESXR49Gz2i
2Jvhkj2msf7SeAMPuHNox0jTHm3dO2XAktlxNqlaPSYRb5dniZpjVm/0zvvKSKr9
BXULpzU5M+ka2bT7htzSDXOOK43Ospazc8wQwYR54DOiZAcfKizps7VwggWKRRN9
kAOEWIQaMdUMdJ2u/9KZjjMqhdlyarx/Q9R6qeI/Mu7UTzxCi0KbawdqSzl9X/mb
6K6Ks/pmYg==
=N+68
-----END PGP SIGNATURE-----
Merge tag 'for-7.3/io_uring-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring update from Jens Axboe:
"On top of the usual cleanups and fixes, the bigger items in here are:
- zcrx work, most of it centered around adding dynamic area
provisioning, plus a bunch of prep and cleanups leading up to it:
scale refilling with large pages, coalesce same-niov RQEs on
refill, separate the RQ head/tail cache lines and cache the RQ
tail, and rework the area creation locking.
- Fix the futex inflight accounting so that only private futex waits
are marked inflight, and don't mark wake requests as inflight at
all.
- Drop the custom iov copy in the buffer select prep and msg header
copy paths, using the generic helpers instead.
- Fix a folio size overflow in io_vec_fill_bvec(), and account the
pages a compound region really uses in the memmap path.
- Fix an iovec leak in uring_cmd when the async cmd isn't recycled,
skip the blocking task work for io_uring_cmd_issue_blocking(), and
don't skip completion for a synchronous multishot cmd
- Defer eventfd signaling when queued from a wakeup handler
- Fix io-wq worker accounting when canceling creation callbacks
- Annotate remote tasks for kcoverage"
* tag 'for-7.3/io_uring-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (30 commits)
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
io_uring/uring_cmd: don't skip completion for a synchronous multishot cmd
io_uring/memmap: account the pages a compound region really uses
io_uring/zcrx: add dynamic area provisioning
io_uring/zcrx: lock area creation with pp_lock
io_uring/zcrx: keep array of areas
io_uring/zcrx: move freelist lock to struct zcrx
io_uring/zcrx: unmap under netdev lock
io_uring/zcrx: split dmabuf unmap and release
io_uring/zcrx: don't pass ifq_reg to area creation
io_uring/zcrx: add helper for deriving area token
io_uring/zcrx: don't reload skb_shinfo
io_urint/zcrx: narrow var scope in io_zcrx_recv_skb()
io_uring/zcrx: constify area_reg on import
io_uring/zcrx: coalesce same-niov RQEs on refill
io_uring/zcrx: cache RQ tail
io_uring/zcrx: add RQ iterator
io_uring/zcrx: move RQ head/tail to separate cache lines
io_uring/zcrx: scale refilling with large pages
io_uring/io-wq: fix worker accounting when canceling creation callbacks
...
Rename IO_IMU_DEST and IO_IMU_SOURCE to IO_BUF_DEST and IO_BUF_SOURCE
and export it so subsystems may use it.
This is needed by the io_buffer_register_bvec() path for callers who may
need the buffer to be both readable and writable.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260612184840.4058966-5-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Split the main initialization logic in io_buffer_register_request() into
a helper function.
This is a preparatory patch for supporting kernel-populated buffers in
fuse io-uring, which will be reusing this logic.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-3-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Currently, io_buffer_register_bvec() takes in a request. In preparation
for supporting kernel-populated buffers in fuse io-uring (which will
need to register bvecs directly, not through a struct request), rename
this to io_buffer_register_request().
A subsequent patch will commandeer the "io_buffer_register_bvec()"
function name to support registering bvecs directly.
Rename io_buffer_unregister_bvec() to a more generic name,
io_buffer_unregister(), as both io_buffer_register_request() and
io_buffer_register_bvec() callers will use it for unregistration.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-2-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
If IORING_URING_CMD_MULTISHOT is set, io_uring_cmd() treats any
non-negative return from ->uring_cmd() as the driver having taken
ownership of the request and returns IOU_ISSUE_SKIP_COMPLETE. But
nothing guarantees that the driver did so, and any handler that just
completes the command inline and returns 0 or a positive result then
leaves the request orphaned, leaking the io_kiocb, the async data,
and the file reference.
The special case isn't needed. ublk returns -EIOCBQUEUED for the
multishot fetch command, which is passed through as-is, and the poll
driven socket timestamp command returns -EAGAIN. Kill it, a multishot
handler that wants to hang on to the request must return -EIOCBQUEUED
or -EAGAIN like any other command.
Fixes: 620a50c927 ("io_uring: uring_cmd: add multishot support")
Cc: stable@vger.kernel.org
Reported-by: syzbot+a4ccdd7ebf452e4d4701@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a7a0194.b50370da.49fe0.0056.GAE@google.com/
Link: https://lore.kernel.org/all/20260811115125.1831170-1-vasilisalmpanis@gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_mem_alloc_compound() allocates get_order(size) pages, which rounds a
region size that is not a power of two up to the next order. The pages
past the region are part of the same allocation and cannot be used for
anything else, but io_create_region() accounts reg->size >> PAGE_SHIFT,
so they are never charged against RLIMIT_MEMLOCK. For a ring with 4096
SQ entries and the default CQ size the region is 37 pages while the
allocation is 64.
Account the tail pages together with the region, and fall back to the
exact sized bulk allocation when they do not fit the limit, so a user
close to their limit still gets the region rather than an error.
io_free_region() gives the same amount back, the compound case being the
one that set IO_REGION_F_SINGLE_REF.
Counting how many 4096 entry rings an unprivileged user can create under
a given RLIMIT_MEMLOCK, before and after:
limit (pages) before after
256 2 2
300 2 2
350 3 2
400 3 3
512 5 4
Five rings under a 512 page limit really pin 640 pages.
Fixes: dfbbfbf191 ("io_uring: introduce concept of memory regions")
Link: https://lore.kernel.org/all/87ik5ncj8d.fsf@mailhost.krisman.be/
Signed-off-by: Ali Ahmet Memis <ali@iusegentoo.com>
Link: https://patch.msgid.link/20260806180044.275543-1-ali@iusegentoo.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Currently, we have only a one area per zcrx instance, and struct
io_zcrx_ifq stores a single pointer. To prepare for adding more areas,
replace it with an array of areas.
We'll be creating them at runtime, and the array is protected by 3
locks: ->pp_lock, ->alloc_lock and ->rq.lock. It takes all of them when
switching arrays, and readers should hold either of them.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/d186adfa0ea26f651804cffef77d477f257eb404.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Until now unmapping and destroying dmabuf were the same thing. To keep
it consistent with non-dmabuf, split it into two separate helpers. Unmap
destroys mappings and attachements as it should, and release only
putting down the dmabuf fd reference.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/bd8c1fd8dac85ce0cf839b389ecd6050761965ca.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
We might want to create an area without having an instance of struct
io_uring_zcrx_ifq_reg. Extract a helper that doesn't have the ifq
registration structure as an argument but takes the buf length
explicitly.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/76751ee4747abb4893058fcaaa36895dfe24a297.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The RQ tail is updated by the user space. Cache it to reduce cache line
bouncing. Refilling now tries to exhaust the previous batch of rqes, but
since it could be too low, the iterator is allowed to recalculate the
rqes to process once after synching the tail value.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/b42d2ed8b0e697110b646573b5da2a5f8e85f92e.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
RQ head and tail are currently put into the same cache line, which can
cause false sharing problems when refill is run on another CPU. Put them
into separate cache lines.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Link: https://patch.msgid.link/f769116a3f5267f06cb9a9a819d482ddea2e0852.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_zcrx_ring_refill() caps the loop by mixing the max number of
allocated netmems and the number of available RQEs together, which
caps the number of entries to process the pp cache size. As a result,
when niovs are heavily fragmented, the refilling logic allocates only a
small number of niovs per call on average and sometimes even none.
Keep a separate counter for the number of processed RQ entries, which is
capped by a roughly calculated from the page size value to keep the
cache full. And separately break if it allocates enough niovs.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/143cf439299728759eb0a840c866363fe99293f0.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
