create_worker_cb() reserves an io-wq worker slot only after its
task-work callback runs. If the callback is canceled before then,
io_worker_cancel_cb() still decrements acct->nr_workers. When an
existing worker retires with its creation callback pending, that
worker has already decremented the same account's worker count.
The resulting undercount permits worker creation beyond the account's
configured limit. On an AST2600 OpenBMC system, an unchanged sensor
daemon reached 4,291 threads with the original kernel. With an
equivalent downstream fix, 25 passive samples under its normal
workload showed 6-9 threads.
Decrement nr_workers only when the canceled callback is not
create_worker_cb(). Continuation callbacks still release their reserved
slot, and both callback types retain the existing running-count,
reference-count, and create-state cleanup.
Fixes: 1d5f5ea7cb ("io-wq: remove worker to owner tw dependency")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Vishnu Razdan <vrazdan@openai.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260811-vrazdan-io-wq-b4-submit-v1-1-719ced16c921@openai.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
ddir was only ever forwarded to io_compat_msg_copy_hdr, which
never looked at it. Drop it.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-4-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Similar to commit f4eaf8eda8 ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid the custom copy of a single iovec and just
rely on the iovec api. This lets the compat and native paths share the
buffer-select length lookup
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-3-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Similar to commit f4eaf8eda8 ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid custom copy and just rely on the iovec api.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-2-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_req_local_work_add() signals the CQ ring eventfd inline when it is the
one to push the first entry onto ->work_list. For DEFER_TASKRUN rings that
add is frequently done from a waitqueue wakeup handler, where an
arbitrary waitqueue lock is held.
eventfd_signal_mask() only refuses to recurse when current->in_eventfd
is set, but that bit is set by eventfd_signal_mask() itself. If the wake
chain starts somewhere else, signal goes out inline and can feed back
into epoll.
Add IOU_F_TWQ_IN_WAKE, set it on the task_work add done from the three
waitqueue callbacks, and use it to force io_eventfd_signal() down the
existing call_rcu_hurry() deferral instead of signaling inline.
Fixes: 21a091b970 ("io_uring: signal registered eventfd to process deferred task work")
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20260813133843.2933127-1-4ncienth@gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_import_reg_vec() converts the estimated number of bio_vec entries
into iovec-sized storage when struct bio_vec is larger than struct
iovec. The conversion still multiplies nr_segs by sizeof(struct
bio_vec) in size_t without checking for overflow.
On 32-bit kernels, a registered buffer large enough to make
io_estimate_bvec_size() return 357913942 segments wraps the byte count
from 0x100000008 to 8. io_vec_realloc() then reserves only the input
iovecs plus one extra slot while io_vec_fill_bvec() writes the full
bio_vec array.
Check both the multiplication and the rounding addition before
deriving the replacement iovec count.
Fixes: b4e41050b2 ("io_uring/rsrc: raise registered buffer 1GB limit")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://patch.msgid.link/20260812203042.720348-1-Jeremy.Jean@oss.cyber.gouv.fr
Signed-off-by: Jens Axboe <axboe@kernel.dk>
struct bvec_iter::bi_bvec_done is used an offset in the current bvec,
let's rename it accordingly for better clarity. I also plan to use it
for non-bvec based iteration in the future like dma-buf, so drop the
"bvec" part.
Suggested-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/4e4c21858705a200bd8848ffe4080522e3eb5c1c.1786018753.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
blkdev.h gets included in various places outside the block layer just
for struct blk_plug and related plugging functions.
Split blk_plug into a separate helper to reduce the amount of code
that needs to get rebuilt when blkdev.h changes and to slightly
reduce compile times.
In io_uring this requires pulling in a few other headers explicitly that
previously were implicitly included through blkdev.h.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christian Brauner (Amutable) <brauner@kernel.org>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125524.740996-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_vec_fill_bvec() computes the folio size with a plain int 1:
unsigned long folio_size = 1 << imu->folio_shift;
imu->folio_shift is unsigned int and comes from folio_shift() of the
folio backing the registered buffer, so it can be 32 or more on a 64 bit
kernel. Shifting int 1 that far is undefined, and on x86 and arm64 the
count is taken modulo 32, so a shift of 34 yields 4 rather than 16G.
Every other folio_shift shift in this file already uses 1UL.
The result is that the segment estimate and the fill loop disagree.
io_estimate_bvec_size() sizes the bvec array with the real shift:
max_segs += (iov[i].iov_len >> shift) + 2;
so a 1M iovec on a 16G folio is charged 2 segments, while
io_vec_fill_bvec() then walks the same iovec in folio_size chunks of 4
bytes and writes res_bvec[bvec_idx] a quarter of a million times, past
the end of the array it was given. src_bvec is advanced once per
iteration as well, so imu->bvec is read past its end at the same time.
validate_fixed_range() only checks that the range is inside the
registered buffer and does not bound the segment count.
Reaching it needs a folio with a shift of at least 32, which means a
gigantic hugetlb page: 16G on arm64 with 64K pages, where
CONT_PMD_SHIFT is 34 and hugetlb_add_hstate(CONT_PMD_SHIFT - PAGE_SHIFT)
registers that size, and likewise on powerpc. x86_64 tops out at 1G, so
a shift of 30, which still fits in int and is unaffected.
Use 1UL, as the rest of the file does.
Fixes: 9ef4cbbcb4 ("io_uring: add infra for importing vectored reg buffers")
Cc: stable@vger.kernel.org
Signed-off-by: Ali Ahmet Memis <ali@iusegentoo.com>
Link: https://patch.msgid.link/20260802163030.51005-1-ali@iusegentoo.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
An io_async_cmd carries an iovec array in ->vec.iovec, allocated when the
vec has to grow and kept across recycling through ctx->cmd_cache. On two
paths nothing frees it and io_clean_op()'s kfree(req->async_data) drops
the io_async_cmd without it.
io_req_uring_cleanup() clears the async data flags only when
io_alloc_cache_put() succeeds, and the cache holds IO_ALLOC_CACHE_MAX ==
128 entries, so once it is full the put fails and the vec is left behind.
An NVMe passthrough workload gets there without doing anything unusual:
nvme_uring_cmd_io() returns -EIOCBQUEUED, so the io_async_cmd stays
attached for the lifetime of the command and the live object count tracks
the queue depth. Above 128 the puts start failing.
->cleanup is the last chance to free an inherited vec, since
io_req_uring_cleanup() returns early for an io-wq issued command and is
not called at all for one completed without ever being issued. But
io_clean_op() calls ->cleanup only if REQ_F_NEED_CLEANUP is set, and for
uring_cmd that happens only where the vec has to grow, so a command
reusing a large enough cached vec never sets it. io_rw_alloc_async() and
io_msg_alloc_async() flag an inherited vec for exactly this reason;
io_uring_cmd_prep() does not.
Flag an inherited vec in io_uring_cmd_prep(), and free the vec when the
cache put fails, as io_req_rw_cleanup() does.
The leak is invisible under KASAN, where io_alloc_cache_vec_kasan() frees
the vec unconditionally.
Fixes: 3a4689ac10 ("io_uring/cmd: add iovec cache for commands")
Cc: stable@vger.kernel.org
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Link: https://patch.msgid.link/20260802073518.419-1-worapat.kd2@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Per-task restrictions apply to all rings created by a task. Once
installed, they should not be dropped across exec.
For a task that has used io_uring, the exec cancellation path calls
__io_uring_free(). This frees both the task context and the per-task
restriction, so a ring created after exec is unrestricted.
Split task context cleanup into io_uring_free_tctx(), and use it from
the exec cancellation path. Keep __io_uring_free() for final task
cleanup, where both the context and restriction are released.
Fixes: ed82f35b92 ("io_uring: allow registration of per-task restrictions")
Cc: stable@vger.kernel.org # 7.1+
Signed-off-by: Kyumin Lee <fyonglkm@gmail.com>
Link: https://patch.msgid.link/20260730192734.459247-1-fyonglkm@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring_cmd_issue_blocking() is only called from blk_cmd_complete(),
which is already a task work callback. However, it queues another task
work item, to call io_queue_iowq(). Just call io_queue_iowq() directly
to skip the CPU cost and latency of the redundant task work proxying.
Signed-off-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260702184847.1709378-1-csander@purestorage.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
import_zcrx() attaches an existing ifq to another ring. It never calls
zcrx_set_ring_ctx() and so never takes the ->master_ctx reference, but
its error path still passes @ctx to zcrx_unregister(), which clears
->master_ctx and drops its percpu_ref whenever ifq->master_ctx == ctx.
That condition is reachable. A ring that registers an ifq with a
non-zero event type_mask gets ->master_ctx pointed at itself, and
nothing stops it from exporting that ifq with ZCRX_CTRL_EXPORT and
importing the resulting fd back into the same ring. Failing the import
after the refcount bumps -- an argument page mapped PROT_READ makes the
copy_to_user() in import_zcrx() return -EFAULT -- then clears the
->master_ctx owned by the original registration, which is still live.
Refcounts stay balanced and nothing is freed early, so there is no
splat. The ring silently stops receiving ZCRX_EVENT_ALLOC_FAIL and
ZCRX_EVENT_COPY: zcrx_send_notif() returns early on a NULL
->master_ctx, and ->master_ctx is only ever set on a freshly allocated
ifq, so it cannot be restored without tearing the ring down.
Pass NULL instead, matching zcrx_box_release() and the zcrx_export()
error path. io_register_zcrx() only gets away with passing @ctx
because zcrx_set_ring_ctx() runs after its last goto err.
Fixes: 00d9148127 ("io_uring/zcrx: share an ifq between rings")
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Link: https://patch.msgid.link/20260730162741.1125-1-worapat.kd2@gmail.com
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
[axboe: add pavel edit]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Turns out that my naive assumption that surely nobody is using io_uring
to manage nested epoll instances was wrong - libuv is in fact doing that
very thing. Since we have users in the wild using that feature, we
cannot remove it from upstream. Revert the commit that introduced that
limitation.
This reverts commit cfa1539b24.
Link: https://lore.kernel.org/io-uring/605939add4dfb674bd7ab39ef8dbd3fd22e754e4.camel@xry111.site/
Reported-by: Xi Ruoyao <xry111@xry111.site>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Inflight tracking of futex wait requests exists to ensure that do_exit()
-> io_uring_files_cancel() cancels them before the mm goes away, as a
private futex wait depends on the mm private futex hash staying alive
for the duration of the request.
Shared futexes have no such dependency. A FLAGS_SHARED request always
resolves to either an inode based key or an mm-shared key, both of which
fail futex_key_is_private() and hence always hash into the global futex
hash, whose lifetime isn't tied to the mm.
Only mark vectored futex waits as inflight if the futex is private.
Cc: stable@vger.kernel.org
Fixes: 079afb081c ("io_uring/futex: mark wait requests as inflight")
Link: https://lore.kernel.org/io-uring/CANGjgdn=R_qyUdE=j9za+vkmqcxacbP-84OHXF4nZ4ho9qRyVg@mail.gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Commit 079afb081c ("io_uring/futex: mark wait requests as inflight")
added inflight tracking to ensure that do_exit() ->
io_uring_files_cancel() finds and cancels pending futex waits before the
mm goes away, as a private futex wait depends on the mm private futex
hash staying alive for the duration of the request. However, as
io_futex_prep() is shared between FUTEX_WAIT and FUTEX_WAKE, wake
requests got marked as inflight as well.
A futex wake executes fully inline at issue time and never depends on
the mm staying alive after completion, hence there's no need to track
it. Kill it.
Cc: stable@vger.kernel.org
Fixes: 079afb081c ("io_uring/futex: mark wait requests as inflight")
Reported-by: Chengfeng Lin <lin2530632123@gmail.com>
Link: https://lore.kernel.org/io-uring/CANGjgdn=R_qyUdE=j9za+vkmqcxacbP-84OHXF4nZ4ho9qRyVg@mail.gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_ring_buffers_peek() builds an iovec array from provided buffers, and
that in turn can be handed off to a lower level provider. Be prudent and
cap the total size to MAX_RW_COUNT, which is the Linux default for how
much IO do to in a single call.
No bugs here, but it's a good preventative measure to avoid truncation
issues.
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Commit:
6a8afb9fff ("io_uring/net: allow multishot receive per-invocation cap")
changed how io_mshot_prep_retry() set sr->len, and added the same
initialization in io_mshot_prep_retry(). But it neglected to touch the
send path, which may also uses the mshot retry path. Ensure that
sr->mshot_len always gets initialized correctly.
Fixes: 6a8afb9fff ("io_uring/net: allow multishot receive per-invocation cap")
Cc: stable@vger.kernel.org
Reported-by: Sung Keum <kambodi127@gmail.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmpkDfYQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpgJgEACAoCs3hC/reUovTYQB8+68AyyBmF5xeJMI
uxYKcVEZGjOo6gTjSm2GYEBa6P85kQtbWEKEfuxXxV2gT1S/3yulV+uy8NfI/lO7
NsKYRHszaa/AFr2RbeZeYLU9TN/qFbZinfxukpW1zWrfwg83IhGGe6QN53G1XB2v
+pO0bqSZ/qKkJ4IXGubLxSxUnDWg6bav4XUeEkjSlE6mzp1+BgpesLLOsZNvvLST
5QTGt5EfWFBgo9LdYHz/ky7MnJjkioFDY+c10sLp2+6ghtyMTZcACB1W4gSSyJG+
QsLYJ6Z25f7zo7WIci42pCDDDhYWQgDFc7r9+uJ1ISFKXLGnEyONnBaxmk2LTWdX
g9V03+pI17vTOQNU0XlkpajUykOmMCkhZWA8jwlOAFJXtz+pdV1vSKeSZt9tEB4j
aEd6IjoBUrPLsv7WIlomf/h9quHfdjMogA6eyaXc4227XzKWo63XlGe4HxBC6yQQ
4Lq8QH8dta6BOm+5APQXNQ5g/EB0OVLDkgh8EzSTDChgoJd4hVwoHnGbGidZSAcR
mgSl4SG/XdXMIwhqAH5iiI+7YZ71ALnoRkLujED90c+lGHrrY0T7H99XaHprp6xa
NOUbz5CAZf+bWPfXbLpJG+QTh/evh9zxfJ30VprHByl99EJFxjK7rW3BJvOAokjs
1O635JJo7A==
=pQDU
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.2-20260724' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a missing ERESTARTSYS conversion in the read paths, which got
messed up back when some code consolidation was done for read
multishot support
- zcrx UAPI rename, dropping the abbreviated "notif" naming in favor of
"event" for consistency and to be less ambiguous for users. This was
added for 7.2, so let's rename it while we still can. No functional
or code changes, just a strict rename
* tag 'io_uring-7.2-20260724' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/zcrx: rename notif to event
io_uring/zcrx: rename ZCRX_NOTIF_NO_BUFFERS
io_uring/zcrx: drop "notif" from stats struct names
io_uring/rw: fix missing ERESTARTSYS conversion in read paths
"Notification" is too long and the abbreviated version is used in
several places, which is inconsistent and more ambiguous for users.
Rename it to event, which is easier to keep consistent. To keep the
change small, only change uapi/ + do necessary fix ups, and the rest of
internals can be adjusted in the next release.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/f95ca6717da3c8d3649a1a7f0d883a563f545052.1784726895.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Both read and write may receive internal restart error codes from
the filesystem layer and should be converted to -EINTR. However,
when multishot read support was added, the error code normalization
was lost for both io_read() and io_read_mshot().
Extract the conversion into io_fixup_restart_res() and apply it
in all three locations: io_rw_done(), io_read(), and io_read_mshot().
Fixes: a08d195b58 ("io_uring/rw: split io_read() into a helper")
Cc: stable@vger.kernel.org
Signed-off-by: Yitang Yang <yi1tang.yang@gmail.com>
Link: https://patch.msgid.link/20260722124551.130563-1-yi1tang.yang@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmpbBbsQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpgpFD/9NnN5A9eMm+4ZeC41yeynykN1QOl5wCqoT
zi7CIizuwtsCQAjgpFUahMosvkLg3P77S/MJxqzlGVPhaSvp/QISxr7oH9WqyoYf
jEfi3Dx8XfdxK+5GG0dC7s4iiWuxZha7iGt4W2DZmjXWWu0ofCKB0FTVK7J/AkC0
7DA4hMmts643cE9pwmD1Bbuj7SYmqjneDhgwe/JaSuCiuuclN5L359E1rnxzc+dS
9McCmwPjwF0f+Vs9IWyHb1KNMBBmC2T0n2zvzKoJKL72mQ0v3IaOlGkUEDgp7EpT
TIK5rEKPl/Wpw970BOlzdYgD01vDiWeLuQQQlGfAaLheUUImLoFPo1rf/q8nCgFN
CVEWDSZZWRnwH6FjFCa5JwOS0B9YcamYzhkTck1O+g7KcNB6ZCU1zMX8+pCMEVgr
Ju5J2uoazT2txEZEgCSSUYLuRRSpz5P5s/VmFEXNxeRAhndlGhyNq16kVnRqyLfA
OzVS5uYeqSOlOYbY8pL4QkGaxBUr0CSBOnhCT7q9d8KYXnh9CHReFQcy9wPclSfd
D+F5v4CPqWN0tg2rrYBgo3Zpt4Mwb1JEiyOAaIapAKQT2T9tNtp1fubq4PA9c1ub
VkSRutPGeUWNC1fEjs67I9V3V0bzbR8j+AMC9AAFFqaqpn2B5NGHqvlcYdaNuuje
Dr7yXvROWw==
=Ue52
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.2-20260717' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a use-after-free in the bpf-ops struct_ops path, where the same
io_uring_bpf_ops map could be registered more than once.
- Fix the deferred iovec free for the provided-buffer grow path, which
could leave the caller with a dangling iovec and result in repeated
frees. Follow-up to the earlier fix in this series.
- Zero-check the unused addr3/pad2 SQE fields for unlinkat
* tag 'io_uring-7.2-20260717' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/bpf-ops: reject re-registration of an already-bound ops
io_uring/fs: check unused sqe fields for unlinkat
io_uring/kbuf: free the replaced iovec after a successful grow
io_install_bpf() only rejects a second registration on the ctx side
(ctx->bpf_ops) and sets the per-map back-pointer ops->priv
unconditionally. The struct_ops link path never advances a map past
BPF_STRUCT_OPS_STATE_READY, so the same io_uring_bpf_ops map can be
registered more than once, and bpf_io_reg() re-resolves the target ring
via fget(ops->ring_fd) on every call. A caller can therefore point the
same ring_fd at a different io_ring_ctx between two BPF_LINK_CREATE
calls.
The second registration passes the ctx->bpf_ops check (the new ctx has
none) and overwrites ops->priv, orphaning the first ctx. Teardown
(io_eject_bpf()/bpf_io_unreg()) only reaches a ctx through ops->priv, so
the orphaned ctx is never torn down: its ctx->loop_step keeps pointing
into the struct_ops trampoline, which is freed once the map is gone. A
later io_uring_enter() on the orphaned ring then calls the dangling
ctx->loop_step from io_run_loop() -- a use-after-free of freed
executable memory, reachable by a task with CAP_BPF + CAP_PERFMON.
Reject registration when ops->priv is already set, as hid_bpf_reg()
does for its struct_ops.
Cc: stable@vger.kernel.org
Fixes: 98f37634b1 ("io_uring/bpf-ops: implement bpf ops registration")
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/20260717154537.129736-1-worapat.kd2@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Zero check unused SQE fields addr3 and pad2 for unlinkat. They're
not needed now, but could be used sometime in the future.
Signed-off-by: Yi Xie <xieyi@kylinos.cn>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260714030306.64820-1-xieyi@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The provided-buffer validation fix deferred freeing a cached iovec
until validation completed. However, the deferred free uses arg->iovs.
After a grow, that points to the newly allocated array. Without a grow,
it points to the cached array that remains in use.
This leaves the caller with a dangling iovec in both cases and can
result in repeated frees. Only free org_iovs when arg->iovs actually
replaced it.
Fixes: cd053d788c ("io_uring: fix dangling iovec after provided-buffer bundle grow failure")
Assisted-by: Codex:gpt-5.3-codex-spark
Signed-off-by: Jaeyeong Lee <iostreampy@proton.me>
Link: https://patch.msgid.link/20260712142612.188695595-iostreampy@proton.me
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmpRch8QHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpqORD/9xYHpNwkeIXfG+6giRGCF03u0eGlreUsIl
uCGdqfNKjKZqWOw6q7uzNHll7/2SmwOniB32mdo6jpUgXeE3/RJ4U37kZ8ypIyO8
1iOSM5zz5ZNQQuHa/HZMqMRy9mqBa5WIJqRO4CpRk9uaqq6flU2CEq4/y3SdOc0S
9u+UxzVhmoYuL7iQhaLg6xQSrNxtwynVXqIru/bN1Ft01sCo3Yf3w0SmNnmsrr34
jG1tkn0J7QtwkaN81poX8WTqm0D7McFc5xcq8wWM+BDnpvllPxQ+J+J3+zMfvAbN
tbb3FiXkc5SeYTOrBu8FWwPMn8reNpUBgEGjrcZWyRIkHdzxrDhwvC9tyQs85l4w
j93PSsJnlprVPI4LWGfglCaNBZ85hFFKpm83rAFJgJjLuyzRFSF7+/EKrHFP58sK
v8oYq/iXpQunu1YLfBIAjW4OnVkqSYM95Xax4l5yFSgeCS+hNd/naRgvqzrM4puq
tspIH9zjZCyRftRuVebW4V/cxfceSp3kg5cPyW/PRonO9iGZ1Q2scK0bovNO0QKC
gDjax1qancwp/GlT65jgdLp9lAx1TIofPz2gP22lGlZe732bi3gvX5rGPo1bT98v
1NJXrEIqLHOuId308+pzpvl4j4YyJAC3llThVg4L2TZb6jy+S5qYnbBnRkQoNPVi
iMP+XrWPjw==
=vHWI
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.2-20260710' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Restore full RCU read section in io_req_local_work_add(), which was
mistakenly dropped with the DEFER_TASKRUN rework in this merge
window. Revert the commit that grabbed the RCU read lock in
io_ctx_mark_taskrun(), as that's no longer required with the previous
fix.
- Fix a dangling iovec after a provided-buffer bundle grow failure,
also an issue introduced in this merge window.
- Reject IORING_CQE_F_32 flag pass-through in MSG_RING to rings that
weren't setup with CQE32 or CQE_MIXED.
- Return -EINVAL rather than -ENOMEM from get_unmapped_area() when mmap
validation fails, matching io_uring_mmap().
* tag 'io_uring-7.2-20260710' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
Revert "io_uring: grab RCU read lock marking task run"
io_uring: restore RCU read section in io_req_local_work_add()
io_uring: fix dangling iovec after provided-buffer bundle grow failure
io_uring/uring_cmd: fix uring_cmd.c comments
io_uring/msg_ring: reject CQE32 flag pass-through to normal rings
io_uring/memmap: return -EINVAL from get_unmapped_area() on bad mmap
This reverts commit ed64f5c546.
Since commit:
648790e095 ("io_uring: restore RCU read section in io_req_local_work_add()")
io_ctx_mark_taskrun() is only ever called with the RCU read lock
already held, like previously. Hence's there's no need for this commit
anymore, which grabbed the RCU read lock inside io_ctx_mark_taskrun().
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The task-work refactor that moved io_req_local_work_add() out of
io_uring.c into the new io_uring/tw.c dropped the whole-body
guard(rcu)() that used to cover the function body.
For DEFER_TASKRUN rings the ring teardown still relies on that RCU read
section pairing with its grace period:
/* pairs with RCU read section in io_req_local_work_add() */
if (ctx->flags & IORING_SETUP_DEFER_TASKRUN)
synchronize_rcu();
io_ring_ctx_free(ctx);
io_req_local_work_add() keeps dereferencing ctx after mpscq_push() has
published the request to the work list (ctx->cq_wait_nr, and
ctx->submitter_task in the final wake_up_state()), without holding a ctx
reference across that window. The RCU read section was the only thing
guaranteeing an in-flight adder had finished touching ctx before
io_ring_ctx_free() ran; synchronize_rcu() only waits for readers that
are actually inside an RCU read-side critical section. With the guard
gone the grace period no longer pairs with anything on the add side, so
ctx can be freed and reused while io_req_local_work_add() is still using
it.
Fixes: d46ab2c98a ("io_uring: switch local task_work to a mpscq")
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Link: https://patch.msgid.link/20260709035100.2269-1-worapat.kd2@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
When growing a provided-buffer bundle, the old cached iovec is freed
before the new buffers have all been validated. If validation fails, the
request still points at the freed iovec, which can be freed again during
completion cleanup.
Fix this by deferring the free of the old cached iovec until validation
has succeeded. On failure, free the newly allocated iovec and leave the
request pointing at the original one.
Fixes: 46800585ae ("io_uring/kbuf: validate ring provided buffer addresses with access_ok()")
Signed-off-by: Hao-Yu Yang <naup96721@gmail.com>
Link: https://patch.msgid.link/20260706183304.919275-1-naup96721@gmail.com
Suggested-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fix "concelable" -> "cancelable" in the comment above
io_uring_cmd_mark_cancelable(), and fix the memory ordering comment
in __io_uring_cmd_done() to reference io_do_iopoll() and
->iopoll_completed.
Signed-off-by: Yang Xiuwei <yangxiuwei@kylinos.cn>
Link: https://patch.msgid.link/20260702082937.3707134-3-yangxiuwei@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
IORING_OP_MSG_RING with IORING_MSG_RING_FLAGS_PASS allows a sender to
pass completion flags through sqe->file_index. If the sender sets
IORING_CQE_F_32 in file_index, the target-side completion path treats it
as a 32b CQE and writes big_cqe[0] and big_cqe[1] into the CQ ring
regardless of whether the target ring was created with
IORING_SETUP_CQE32 or IORING_SETUP_CQE_MIXED.
On a normal 16b CQE ring, this writes 16 extra bytes (two u64 big_cqe
fields) into the next CQE slot in the ring buffer. As the receiving ring
doesn't understand 32b CQEs, this is incorrect and they should be
rejected.
Fixes: cbeb47a7b5 ("io_uring/msg_ring: Pass custom flags to the cqe")
Signed-off-by: Melbin K Mathew <mlbnkm1@gmail.com>
Link: https://patch.msgid.link/20260701081145.196730-1-mlbnkm1@gmail.com
[axboe: edit commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
get_unmapped_area() returns -ENOMEM when io_uring_validate_mmap_request()
fails, but validation errors are -EINVAL. Propagate that errno to
userspace, like io_uring_mmap() already does.
Signed-off-by: Yi Xie <xieyi@kylinos.cn>
Link: https://patch.msgid.link/20260630091206.126206-1-xieyi@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fuzzers use coverage information to guide generation of test cases
towards new or interesting code paths. Syzkaller, specifically, makes
use kcoverage (CONFIG_KCOV). Coverage information is not collected for
kernel tasks unless annotated by kcov_remote_start and kcov_remote_stop.
This patch annotates io-uring's work queue and sqpoll tasks.
Depends-On: 20260430-kcov-refactor-common-handle-v1-1-23a0c7a0ba38@google.com
Signed-off-by: Robert Femmer <robert@fmmr.tech>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmo9GAwQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpoQ5D/9yn5dVqZLzqshu/sXJ3uw8usm+u+wvinc1
l6Lp2Q5sfAtmN+nGIZJvwRGqpglw4RRKxiZVHm2yGJ79g3HJIAIgcmlBLtI7HwZ0
foaKhaYB2hvEr6E/hpc0gL3OcRuc3cYjtWu9TsA3OOU4yf9vt9JehHoqlll0SOju
a+0d7EJJ0HaEdi52qhMPhgiWaLfWfD5UGKkVZZTn8JfLyli8Mvn18zcmdSJ82WVp
1CSaquc02meIKUEGruB2T0pQe1Nnk20trP3rosGzc54h501W81hB2vZIVRi7lq5S
m4jgDM+xp/H31JXoelz2VXXXWEg+wxuB2u8WsVuD2yjJc5HFD/S5NKWOJ4oKbae/
A/5xujtkOLPrVeEZmoP/pvAZbEbjpRxw7hwhKWQeed5ir82TgV0MRoiBMsYBpKW8
xfLAn/aJiU0O6F0xcftOsMibWLR40raIk1+OdoWm+JWFELzcF03bQ/875zqbFu9h
/7EFhYWgMW8mvn+/gH5IiN04YxdWu13RsWHi93didxxeCRN8YGgJgH8ysyIhE5O6
Tbcdu0ye0NY4O0JuWVrxowHOno5MewUO7SWUbBOEOsrn8Xc4Bl3oFRbIYuwph5Jz
rhABOf5Rkyt7ypOV8PWDwroTj3IAUq02HivsesRMxbcEEM/cgvIcse19qAIo+dGW
LkIJCI+H3Q==
=cmtx
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.2-20260625' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a file reference leak in the nop opcode when used with
IOSQE_FIXED_FILE
- Preserve the SQ array entries when resizing the ring via the register
path
- Preserve the partial result for an iopoll request rather than
overwriting it
- Don't audit log IORING_OP_RECV_ZC
- Bound io_pin_pages() by the page array byte size in the memmap path
- Follow-up cleanup to the task_work mpscq conversion, getting rid of
the now-unnecessary tw_pending tracking for the !DEFER_TASKRUN path
- Switch a system_unbound_wq user over to system_dfl_wq
* tag 'io_uring-7.2-20260625' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/memmap: bound io_pin_pages() by page array byte size
io_uring: Use system_dfl_wq instead of system_unbound_wq
io_uring/register: preserve SQ array entries on resize
io_uring, audit: don't log IORING_OP_RECV_ZC
io_uring: get rid of tw_pending for !DEFER task work
io_uring/rw: preserve partial result for iopoll
io_uring/nop: fix file reference leak with IOSQE_FIXED_FILE
io_pin_pages() checks that nr_pages does not exceed INT_MAX, then
allocates a struct page * array of nr_pages entries. kvmalloc() limits
allocations to INT_MAX bytes, but the check counts pages, not bytes.
On 64-bit each entry is 8 bytes, so the array hits the INT_MAX byte
limit at INT_MAX / sizeof(struct page *) pages, well before the page
count check fires.
Since commit b4e41050b2 ("io_uring/rsrc: raise registered buffer 1GB
limit") raised the per-buffer cap to 1TB, a buffer near that cap maps
~2^28 pages, making the array allocation exceed INT_MAX bytes. This
passes the page count check, reaches kvmalloc(), and triggers the
WARN_ON_ONCE() for oversized allocations in __kvmalloc_node_noprof().
Check nr_pages against INT_MAX / sizeof(struct page *) so the buffer is
rejected with -EOVERFLOW before the allocation is attempted.
Reported-by: syzbot+f99b00a963915b6b52c6@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=f99b00a963915b6b52c6
Fixes: b4e41050b2 ("io_uring/rsrc: raise registered buffer 1GB limit")
Tested-by: syzbot+f99b00a963915b6b52c6@syzkaller.appspotmail.com
Signed-off-by: Deepanshu Kartikey <kartikey406@gmail.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260621012933.50571-1-kartikey406@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmoxQDoQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgphm7EADQ7UmRhQGFQCbykRQ2joI0T4FGBgiRQaEa
Rkpcob8xeqzkX16G00TzXeiM4K76QVaIHG6bDpXNwpHeL/2flsMLSpD0XuQONzAC
g3OZQsaPyZqXwUZ+3mLKEgd830LgTeIC0iiVm6hnxNhTo29O1v1ueU/K0LgaE0AG
40q2NSpzxMbnFM/c3UVpepLsZ9XwGwE10xzZJTgnX5VhVxXwlNlT0scmEaHbfeJ3
hh9fn6KqW0LqnrK7Pq3UBKNxxoiVM5sruFDWPevXgw+LyEBykPiNR3+zzRoXFurx
cIwavXR9tVnTFFdfSpZ/YWA11TuOlIyELxb50R3cN4ks8y7nVTNbv6vF7fVCJDIO
iHylAsAHEvkLaCt8X4qZsZQwO/10ZeaUA4gNPnT15gs8jxvk5SWEQOX74tZHps4S
eBPB9zyxfRXs6KZWRfONEHpYQnkfQH5/jIYGjycenWlyZ2jHqwZYcOWQcn9lqHzF
UWWblfp8JfRegmp5i5vwJNUSRq3FUVDiifVSYkSrYpeVztIbSz8E36pes8GKVSZe
gKKw1qgEcqio72dmprZFVAuG3YhAD6ebXAI5KWQ3XZcwJCKnaw16H4ObjBXrqOgH
69jzB4xlwvHlbyFnBQpYFdtfge0ziD0DbMVYT701TCUwsA7CItBEiOe5ZQ6N8u2/
QME4m1sg8A==
=yyaO
-----END PGP SIGNATURE-----
Merge tag 'for-7.2/io_uring-epoll-20260616' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring epoll update from Jens Axboe:
"As discussed a few months ago, this pull request gets rid of allowing
nested epoll notification contexts via io_uring.
Nested contexts have been a source of issues on the epoll side, and
there should not be a need to support them from io_uring. The epoll
io_uring side exists mainly to facilitate a gradual migration from a
notification based epoll setup to an io_uring ditto"
* tag 'for-7.2/io_uring-epoll-20260616' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/epoll: disallow adding an epoll file to an epoll context
io_uring/epoll: switch to using do_epoll_ctl_file() interface
Core & protocols
----------------
- Work on removing rtnl_lock protection throughout the stack continues.
In this chapter:
- don't use rtnl_lock for IPv6 multicast routing configuration
- don't take rtnl_lock in ethtool for modern drivers
- prepare Qdisc dump callbacks for rtnl_lock removal
- Support dumping just ifindex + name of all interfaces, under RCU.
It's a common operation for Netlink CLI tools (when translating
names to ifindexes) and previously required full rtnl_lock.
- Support dumping qdiscs and page pools for a specific netdev. Even tho
user space wants a dump of all netdevs, most of the time, the OOO
programming model results in repeating the dump for each netdev.
Which, in absence of a cache, leads to a O(n^2) behavior.
- Flush nexthops once on multi-nexthop removal (e.g. when device goes
down), another O(n^2) -> O(n) improvement.
- Rehash locally generated traffic to a different nexthop on retransmit
timeout.
- Honor oif when choosing nexthop for locally generated IPv6 traffic.
- Convert TCP Auth Option to crypto library, and drop non-RFC algos.
- Increase subflow limits in MPTCP to 64 and endpoint limit to 256.
- Support MPTCP signaling of IPv6 address + port (ADD_ADDR). We need
to selectively skip reporting of the standard TCP Timestamp option,
because they won't fit into the header space together (12 + 30 > 40).
- Support using bridge neighbor suppression, Duplicate Address
Detection, Gratuitous ARP and unsolicited NA forwarding - in EVPN
deployments, e.g. VXLAN fabrics (IPv4 and IPv6).
- Improve link state reporting for upper netdevs (e.g. macvlan) over
tunnel devices (again, mostly for EVPN deployments).
- Support binding GENEVE tunnels to a local address.
- Speed up UDP tunnel destruction (remove one synchronize_rcu()).
- Support exponential field encoding in multicast (IGMPv3 and MLDv2).
- Support attaching PSP crypto offload to containers (veth, netkit).
- Add a new IPSec Netlink message XFRM_MSG_MIGRATE_STATE that allows
migrating individual IPsec SAs independently of their policies.
The existing XFRM_MSG_MIGRATE is tightly coupled to policy+SA
migration, lacks SPI for unique SA identification, and cannot express
reqid changes or migrate Transport mode selectors. The new interface
identifies the SA via SPI and mark, supports reqid changes, address
family changes, encap removal, and uses an atomic create+install
flow under x->lock to prevent SN/IV reuse during AEAD SA migration.
- Implement GRO/GSO support for PPPoE.
- Convert sockopt callbacks in a number of protocols to iov_iter.
Cross-tree stuff
----------------
- Remove support for Crypto TFM cloning (unblocked after the TCP Auth
Option rework). This feature regressed performance for all crypto API
users, since it changed crypto transformation objects into reference-
-counted objects.
- Add FCrypt-PCBC implementation to rxrpc and remove it from the global
crypto API as obsolete and insecure.
Wireless
--------
- Major rework of station bandwidth handling, fixing issues with lower
capability than AP.
- Cleanups for EMLSR spec issues (drafts differed).
- More Neighbor Awareness Networking (Wi-Fi Aware) work (multicast,
schedule improvements, multi-station etc.)
- Some Ultra High Reliability (UHR) / IEEE 802.11bn (D1.4) work
(e.g. non-primary channel access, UHR DBE support).
- Fine Timing Measurement ranging (i.e. distance measurement) APIs.
Netfilter
---------
- Use per-rule hash initval in nf_conncount. This avoids unnecessary
lock contention with short keys (e.g. conntrack zones) in different
namespaces.
- Various safety improvements, both in packet parsing and object
lifetimes. Notably add refcounts to conntrack timeout policy.
Deletions
---------
- Remove TLS + sockmap integration. TLS wants to pin user pages
to avoid a copy, and sockmap wants to write to the input stream.
More work on this integration is clearly needed, and we can't find
any users (original author admitted that they never deployed it).
- Remove support for TLS offload with TCP Offload Engine (the far
more common opportunistic offload is retained). The locking looks
unfixable (driver sleeps under TCP spin locks) and people from
the vendor that added this are AWOL.
- Remove more ATM code, trying to leave behind only what PPPoATM needs,
AAL5 and br2684 with permanent circuits.
- Remove AppleTalk. Let it join hamradio in our out of tree protocol
graveyard, I mean, repository.
- Disable 32-bit x_tables compatibility (32bit binaries on 64bit kernel)
interface in user namespaces. To be deleted completely, soon.
- Remove 5/10 MHz support from cfg80211/mac80211.
Drivers
-------
- Software:
- Support DEVMEM/DMABUF Tx over NETMEM_TX_NO_DMA devices (netkit).
- bonding: add knob to strictly follow 802.3ad for link state.
- New drivers:
- Alibaba Elastic Ethernet Adaptor (cloud vNIC).
- NXP NETC switch within i.MX94.
- DPLL:
- Add operational state to pins (implement in zl3073x).
- Add generic DPLL type, for daisy-chaining DPLLs (implement in ice).
- Ethernet high-speed NICs:
- Huawei (hinic3):
- enhance tc flow offload support with queue selection, tunnels
- nVidia/Mellanox:
- avoid over-copying payload to the skb's linear part (up to 60%
win for LRO on slow CPUs like ARM64 V2)
- expose more per-queue stats over the standard API
- support additional, unprivileged PFs in the DPU configuration
- support Socket Direct (multi-PF) with switchdev offloads
- add a pool / frag allocator for DMA mapped buffers for control
objects, save memory on systems with 64kB page size
- take advantage of the ability to dynamically change RSS table
size, even when table is configured by the user
- increase the max RSS table size for even traffic distribution
- Ethernet NICs:
- Marvell/Aquantia:
- AQC113 PTP support
- Realtek USB (r8152):
- support 10Gbit Link Speeds and Energy-Efficient Ethernet (EEE)
- support firmware loaded (for RTL8157/RTL8159)
- support for the RTL8159
- Intel (ixgbe):
- support Energy-Efficient Ethernet (EEE) on E610 devices
- Ethernet switches:
- Airoha:
- support multiple netdevs on a single GDM block / port
- Marvell (mv88e6xxx):
- support SERDES of mv88e6321
- Microchip (ksz8/9):
- rework the driver callbacks to remove one indirection layer
- Motorcomm (yt921x):
- support port rate policing
- support TBF qdisc offload
- support ACL/flower offload
- nVidia/Mellanox:
- expose per-PG rx_discards
- Realtek:
- rtl8365mb: bridge offloading and VLAN support
- Ethernet PHYs:
- Airoha:
- support Airoha AN8801R Gigabit PHYs.
- Micrel:
- implement 3 low-loss cable tunables
- Realtek:
- support MDI swapping for RTL8226-CG
- support MDIO for RTL931x
- Qualcomm:
- at803x: Rx and Tx clock management for IPQ5018 PHY
- Motorcomm:
- support YT8522 100M RMII PHY
- set drive strength in YT8531s RGMII
- TI:
- dp83822: add optional external PHY clock
- Bluetooth:
- hci_sync: add support for HCI_LE_Set_Host_Feature [v2]
- SMP: use AES-CMAC library API
- Intel:
- support Product level reset
- support smart trigger dump
- Mediatek:
- add event filter to filter specific event
- Realtek:
- fix RTL8761B/BU broken LE extended scan
- WiFi:
- Broadcom (b43):
- new support for a 11n device
- MediaTek (mt76):
- support mt7927
- mt792x: broken usb transport detection
- mt7921: regulatory improvements
- Qualcomm (ath9k):
- GPIO interface improvements
- Qualcomm (ath12k):
- WDS support
- replace dynamic memory allocation in WMI Rx path
- thermal throttling/cooling device support
- 6 GHz incumbent interference detection
- channel 177 in 5 GHz
- Realtek (rt89):
- RTL8922AU support
- USB 3 mode switch for performance
- better monitor radiotap support
- RTL8922DE preparations
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE6jPA+I1ugmIBA4hXMUZtbf5SIrsFAmox1O4ACgkQMUZtbf5S
Irve7hAAgvCYRFvQ/1dvL+PORg2KqVW4rWXMvNrEWaMVAEXJIIPJoxen6zn6a/72
aRs2Ord/wVM/Uh291TOrARwtv1U0oOvOvnbrVM2u06o/EqjtxRAZSF9quTlPdm6Q
RvlkR5X9XXM2MwbDc1zYNTITfhVv80YDCFmYGI7O8dyhLuyspX2eU0SiW33Twl4f
y6RLcn12gyUbDpa6vt+tari6cFCO/X2zot/uwaDY38b+kwsMHunrj3/PSfJzeYIU
5mD7jG+xsPxQlEskMYGOjVT5VeMdRquhb9UGwoostZpuEPYZfzTdFtphC1qZqIPk
8O8ngAW1yYNIY9TTMPF3Y3LMpOiyZq6/P4KY+tksbeDw8vvTYJzEgw3VpwurZQLM
vfHKDj1kM6VLNekeCI+2cip06flVH3jtIkN0KOERwzTSS8NUbs6PuoEP0HLdjafj
cr/pgSSTQPUsQkpS5R5Ld3bwpnKea56CQ36p2E8JvU2akwK5RnvqiZiTAfz2z2Tg
M0NVlTsJ07ucNBxsO4l7osDtRTnzUqo+ZeTo+zR5l0cGRXPd9nJByk257z4IAlcQ
2MHrJmCZ+ZFKhmkUNUPBdFBATbRgOZHzA/iiiThBrT2hDvWQ2kygO9+IkJBS/WYx
eQz7G9hb6Xn1ifWVvPVD0oDL3DKKTotQIN90PZCzvObQFWMc1Vk=
=oNfE
-----END PGP SIGNATURE-----
Merge tag 'net-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
Pull networking updates from Jakub Kicinski:
"Core & protocols:
- Work on removing rtnl_lock protection throughout the stack
continues. In this chapter:
- don't use rtnl_lock for IPv6 multicast routing configuration
- don't take rtnl_lock in ethtool for modern drivers
- prepare Qdisc dump callbacks for rtnl_lock removal
- Support dumping just ifindex + name of all interfaces, under RCU.
It's a common operation for Netlink CLI tools (when translating
names to ifindexes) and previously required full rtnl_lock.
- Support dumping qdiscs and page pools for a specific netdev. Even
tho user space wants a dump of all netdevs, most of the time, the
OOO programming model results in repeating the dump for each
netdev. Which, in absence of a cache, leads to a O(n^2) behavior.
- Flush nexthops once on multi-nexthop removal (e.g. when device goes
down), another O(n^2) -> O(n) improvement.
- Rehash locally generated traffic to a different nexthop on
retransmit timeout.
- Honor oif when choosing nexthop for locally generated IPv6 traffic.
- Convert TCP Auth Option to crypto library, and drop non-RFC algos.
- Increase subflow limits in MPTCP to 64 and endpoint limit to 256.
- Support MPTCP signaling of IPv6 address + port (ADD_ADDR). We need
to selectively skip reporting of the standard TCP Timestamp option,
because they won't fit into the header space together (12 + 30 >
40).
- Support using bridge neighbor suppression, Duplicate Address
Detection, Gratuitous ARP and unsolicited NA forwarding - in EVPN
deployments, e.g. VXLAN fabrics (IPv4 and IPv6).
- Improve link state reporting for upper netdevs (e.g. macvlan) over
tunnel devices (again, mostly for EVPN deployments).
- Support binding GENEVE tunnels to a local address.
- Speed up UDP tunnel destruction (remove one synchronize_rcu()).
- Support exponential field encoding in multicast (IGMPv3 and MLDv2).
- Support attaching PSP crypto offload to containers (veth, netkit).
- Add a new IPSec Netlink message XFRM_MSG_MIGRATE_STATE that allows
migrating individual IPsec SAs independently of their policies.
The existing XFRM_MSG_MIGRATE is tightly coupled to policy+SA
migration, lacks SPI for unique SA identification, and cannot
express reqid changes or migrate Transport mode selectors.
The new interface identifies the SA via SPI and mark, supports
reqid changes, address family changes, encap removal, and uses an
atomic create+install flow under x->lock to prevent SN/IV reuse
during AEAD SA migration.
- Implement GRO/GSO support for PPPoE.
- Convert sockopt callbacks in a number of protocols to iov_iter.
Cross-tree stuff:
- Remove support for Crypto TFM cloning (unblocked after the TCP Auth
Option rework). This feature regressed performance for all crypto
API users, since it changed crypto transformation objects into
reference-counted objects.
- Add FCrypt-PCBC implementation to rxrpc and remove it from the
global crypto API as obsolete and insecure.
Wireless:
- Major rework of station bandwidth handling, fixing issues with
lower capability than AP.
- Cleanups for EMLSR spec issues (drafts differed).
- More Neighbor Awareness Networking (Wi-Fi Aware) work (multicast,
schedule improvements, multi-station etc.)
- Some Ultra High Reliability (UHR) / IEEE 802.11bn (D1.4) work
(e.g. non-primary channel access, UHR DBE support).
- Fine Timing Measurement ranging (i.e. distance measurement) APIs.
Netfilter:
- Use per-rule hash initval in nf_conncount. This avoids unnecessary
lock contention with short keys (e.g. conntrack zones) in different
namespaces.
- Various safety improvements, both in packet parsing and object
lifetimes. Notably add refcounts to conntrack timeout policy.
Deletions:
- Remove TLS + sockmap integration. TLS wants to pin user pages to
avoid a copy, and sockmap wants to write to the input stream. More
work on this integration is clearly needed, and we can't find any
users (original author admitted that they never deployed it).
- Remove support for TLS offload with TCP Offload Engine (the far
more common opportunistic offload is retained). The locking looks
unfixable (driver sleeps under TCP spin locks) and people from the
vendor that added this are AWOL.
- Remove more ATM code, trying to leave behind only what PPPoATM
needs, AAL5 and br2684 with permanent circuits.
- Remove AppleTalk. Let it join hamradio in our out of tree protocol
graveyard, I mean, repository.
- Disable 32-bit x_tables compatibility (32bit binaries on 64bit
kernel) interface in user namespaces. To be deleted completely,
soon.
- Remove 5/10 MHz support from cfg80211/mac80211.
Drivers:
- Software:
- Support DEVMEM/DMABUF Tx over NETMEM_TX_NO_DMA devices (netkit)
- bonding: add knob to strictly follow 802.3ad for link state
- New drivers:
- Alibaba Elastic Ethernet Adaptor (cloud vNIC).
- NXP NETC switch within i.MX94.
- DPLL:
- Add operational state to pins (implement in zl3073x).
- Add generic DPLL type, for daisy-chaining DPLLs (implement in ice).
- Ethernet high-speed NICs:
- Huawei (hinic3):
- enhance tc flow offload support with queue selection,
tunnels
- nVidia/Mellanox:
- avoid over-copying payload to the skb's linear part (up to
60% win for LRO on slow CPUs like ARM64 V2)
- expose more per-queue stats over the standard API
- support additional, unprivileged PFs in the DPU
configuration
- support Socket Direct (multi-PF) with switchdev offloads
- add a pool / frag allocator for DMA mapped buffers for
control objects, save memory on systems with 64kB page size
- take advantage of the ability to dynamically change RSS
table size, even when table is configured by the user
- increase the max RSS table size for even traffic
distribution
- Ethernet NICs:
- Marvell/Aquantia:
- AQC113 PTP support
- Realtek USB (r8152):
- support 10Gbit Link Speeds and Energy-Efficient Ethernet
(EEE)
- support firmware loaded (for RTL8157/RTL8159)
- support for the RTL8159
- Intel (ixgbe):
- support Energy-Efficient Ethernet (EEE) on E610 devices
- Ethernet switches:
- Airoha:
- support multiple netdevs on a single GDM block / port
- Marvell (mv88e6xxx):
- support SERDES of mv88e6321
- Microchip (ksz8/9):
- rework the driver callbacks to remove one indirection layer
- Motorcomm (yt921x):
- support port rate policing
- support TBF qdisc offload
- support ACL/flower offload
- nVidia/Mellanox:
- expose per-PG rx_discards
- Realtek:
- rtl8365mb: bridge offloading and VLAN support
- Ethernet PHYs:
- Airoha:
- support Airoha AN8801R Gigabit PHYs.
- Micrel:
- implement 3 low-loss cable tunables
- Realtek:
- support MDI swapping for RTL8226-CG
- support MDIO for RTL931x
- Qualcomm:
- at803x: Rx and Tx clock management for IPQ5018 PHY
- Motorcomm:
- support YT8522 100M RMII PHY
- set drive strength in YT8531s RGMII
- TI:
- dp83822: add optional external PHY clock
- Bluetooth:
- hci_sync: add support for HCI_LE_Set_Host_Feature [v2]
- SMP: use AES-CMAC library API
- Intel:
- support Product level reset
- support smart trigger dump
- Mediatek:
- add event filter to filter specific event
- Realtek:
- fix RTL8761B/BU broken LE extended scan
- WiFi:
- Broadcom (b43):
- new support for a 11n device
- MediaTek (mt76):
- support mt7927
- mt792x: broken usb transport detection
- mt7921: regulatory improvements
- Qualcomm (ath9k):
- GPIO interface improvements
- Qualcomm (ath12k):
- WDS support
- replace dynamic memory allocation in WMI Rx path
- thermal throttling/cooling device support
- 6 GHz incumbent interference detection
- channel 177 in 5 GHz
- Realtek (rt89):
- RTL8922AU support
- USB 3 mode switch for performance
- better monitor radiotap support
- RTL8922DE preparations"
* tag 'net-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (1778 commits)
ipv4: fib_rule: Move fib4_rules_exit() to ->exit().
net: serialize netif_running() check in enqueue_to_backlog()
net: skmsg: preserve sg.copy across SG transforms
appletalk: move the protocol out of tree
appletalk: stop storing per-interface state in struct net_device
selftests/bpf: test that TLS crypto is rejected on a sockmap socket
selftests/bpf: drop the unused kTLS program from test_sockmap
selftests/bpf: remove sockmap + ktls tests
tls: remove dead sockmap (psock) handling from the SW path
tls: reject the combination of TLS and sockmap
atm: remove orphaned uAPI for deleted drivers, protocols and SVCs
atm: remove unused ATM PHY operations
atm: remove the unused pre_send and send_bh device operations
atm: remove the unused change_qos device operation
atm: remove SVC socket support and the signaling daemon interface
atm: remove the local ATM (NSAP) address registry
atm: remove dead SONET PHY ioctls
atm: remove the unused send_oam / push_oam callbacks
atm: remove AAL3/4 transport support
net: dsa: sja1105: fix lastused timestamp in flower stats
...
Commit de7341ffe4 ("io_uring: switch normal task_work to a mpscq")
added a use of system_unbound_wq, which is deprecated in favor of
system_dfl_wq added by commit 128ea9f6cc ("workqueue: Add
system_percpu_wq and system_dfl_wq"). An upcoming warning in the
workqueue tree flags this with:
workqueue: work func io_tctx_fallback_work enqueued on deprecated workqueue. Use system_{percpu|dfl}_wq instead.
Switch to system_dfl_wq to clear up the warning.
Fixes: de7341ffe4 ("io_uring: switch normal task_work to a mpscq")
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Link: https://patch.msgid.link/20260616-io_uring-fix-wq-warning-v1-1-cfc9d934eedb@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Ring resizing copies pending SQEs from the old SQE array into the new
one so submissions queued before the resize can still be consumed
afterwards.
That copy currently walks the SQ head/tail range directly. This is only
correct when there is no SQ array indirection. With a regular SQ array,
each pending SQ entry contains an index into the SQE array. After resize,
ctx->sq_array is repointed at the newly allocated array, so pending
entries lose their old logical-to-physical mapping and may submit the
wrong SQE.
Remember the old and new SQ arrays while migrating pending SQ entries. For
each pending entry, copy the SQE selected by the old array into the new
destination slot and rebuild the new array entry to point at the copied
SQE. Keep invalid user-provided entries invalid so the normal submission
path still drops them after resize.
Fixes: 79cfe9e59c ("io_uring/register: add IORING_REGISTER_RESIZE_RINGS")
Signed-off-by: guzebing <guzebing1612@gmail.com>
Link: https://patch.msgid.link/20260608133316.3656440-1-guzebing1612@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
IORING_OP_RECV_ZC is a read operation. Audit only tracks file/socket
creation, not subsequent reads. Set audit_skip to align with
audit-userspace uringop_table.h.
Fixes: 11ed914bbf ("io_uring/zcrx: add io_recvzc request")
Suggested-by: Steve Grubb <sgrubb@redhat.com>
Signed-off-by: Ricardo Robaina <rrobaina@redhat.com>
Acked-by: Paul Moore <paul@paul-moore.com>
Link: https://patch.msgid.link/20260616123632.3209545-1-rrobaina@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The normal task_work path used a tw_pending bit to ensure the callback
was only added once: the mpscq drains incrementally, so a single
tctx_task_work() run can take the queue through empty -> non-empty
several times, and each transition would otherwise re-add the already
pending callback_head. This corrupts the task_work list, and is what
tw_pending protects again.
This can go away, if we stop running the task_work as soon as the queue
empties.
Suggested-by: Caleb Sander Mateos <csander@purestorage.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
A partial read will store the completed byte count in io->bytes_done.
The regular completion path applies io_fixup_rw_res() so that, when the
following operation reaches EOF, the number of bytes already read is
returned.
The iopoll completion path does not apply this fixup to the return value
and can return zero instead.
Use the fixup result when updating the CQE, and the raw result for the
reissue check.
Cc: stable@vger.kernel.org
Fixes: 4d9cb92ca4 ("io_uring/rw: fix short rw error handling")
Signed-off-by: Michael Wigham <michael@wigham.net>
Link: https://patch.msgid.link/20260613225240.34032-1-michael@wigham.net
Signed-off-by: Jens Axboe <axboe@kernel.dk>
NOP file-acquisition support choses between a fixed (registered) file and
a normal fget()'d file based on its own IORING_NOP_FIXED_FILE flag in
sqe->nop_flags. However, a request's REQ_F_FIXED_FILE is set
independently from the generic IOSQE_FIXED_FILE sqe flag during request
init, before the issue handler runs.
If a NOP is submitted with IOSQE_FIXED_FILE set (so REQ_F_FIXED_FILE is
set) but without IORING_NOP_FIXED_FILE, io_nop() takes the normal path
and grabs a real reference via io_file_get_normal(). On completion,
io_put_file() only drops the reference when REQ_F_FIXED_FILE is clear,
so the fget()'d file is never released and leaks:
BUG: memory leak
unreferenced object 0xffff88800f42c240 (size 176):
kmem_cache_alloc_noprof+0x358/0x440
alloc_empty_file+0x57/0x180
path_openat+0x44/0x1e50
do_file_open+0x121/0x200
do_sys_openat2+0xa7/0x150
__x64_sys_openat+0x82/0xf0
Decide between fixed and normal file acquisition from REQ_F_FIXED_FILE,
the same way io_assign_file() does for every other opcode, and fold
IORING_NOP_FIXED_FILE into REQ_F_FIXED_FILE at prep time.
Cc: stable@vger.kernel.org
Fixes: a85f31052b ("io_uring/nop: add support for testing registered files and buffers")
Reported-by: syzbot+2cd473471e77bda12b0e@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?id=879092631b98f73a28ea405adacfa5bb34a14a25
Signed-off-by: Vasileios Almpanis <vasilisalmpanis@gmail.com>
Link: https://patch.msgid.link/20260615144619.482749-1-vasilisalmpanis@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmowCYEQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpjUvD/0YcIgniTL/BDQkEQk4+FBVOqLiyPFPhdEY
z/zxMfvRDcUx48E5t4uDHMFwytugTtymRpmkpLANUuUHHpigLPWhT1kCjzIc8OOU
534VJ2Ksay5f3i9gVBhpORmdcBokevdmt5KSqvc+it0k+DNUy1o8DJGcQqPSnRtA
C5Kk6e9QNHpOR30pNygaxtaAsx2rlyrBYw0zWUy7yeUQqI9X+0yZDLC8FUsVDVXW
fptHqQN4///4zm8DdNvTAGhGPGCAcyciXQbrJKBrH9DlZuD5XzrDBFieuhZ96H6M
jvr6eL6XlNx3qd/3x9PYsV4zyArTQZw++XLHakNMoP7Gd3ddSVutfKPLN4TZf5ij
TA4chrhMoIhrubU8LYmDHhTFFCXoRd5NCG5f6JKu7usEzBQVKNO4IJ4fJDA/efeG
iy3W1mdyAAefLS/DfH+i5foWUJbl5VXe4H/d1o8uIdgU2y8rM9d6NdSFWIHNbvc/
Dhq47AHExyWONtIpIPnRjgL9lMVqyG0KjmnPLq8hFhPRUzu5XxvxufyhEfCbWBEr
x6gWvnQlY35DmZYCLzn0FOJRIpYXy49hcJw+FS6fkMaBELhj+OQ7nbaKvCWTw91H
utTltXKvYjJwisMlzfH6/V2Gv7bE4ytRwRTU5GKQQjYIn572RUe4DPbPd3SGxm6T
mt9q2CMGbw==
=upvV
-----END PGP SIGNATURE-----
Merge tag 'for-7.2/block-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe pull request via Keith:
- Per-controller admin and IO timeout sysfs attributes, and
letting the block layer set request timeouts (Maurizio,
Maximilian)
- Multipath passthrough iostats, and PCI P2PDMA enablement for
multipath devices (Keith, Kiran)
- A new diag sysfs attribute group exporting per-controller
counters (retries, multipath failover, error counters, requeue
and failure counts, reset and reconnect events) (Nilay)
- FDP configuration validation and bounds check fixes (liuxixin)
- Various nvmet fixes, including a pre-auth out-of-bounds read in
the Discovery Get Log Page handler, auth payload bounds
validation, and tcp error-path leak fixes (Bryam, Tianchu,
Geliang)
- nvme-tcp lockdep and workqueue fixes (Shin'ichiro, Kuniyuki,
Eric)
- Assorted other fixes and cleanups (John, Yao, Chao, Mateusz,
Achkinazi, Wentao)
- MD pull request via Yu Kuai:
- raid1/raid10 fixes for a deadlock in the read error recovery
path, error-path detection and bio accounting with cloned bios,
and an nr_pending leak in the REQ_ATOMIC bad-block error path
(Abd-Alrhman)
- PCI P2PDMA propagation from member devices to the RAID device
(Kiran)
- dm-raid bio requeue fix, and various smaller fixes and cleanups
(Benjamin, Chen, Li, Thorsten)
- Enable Clang lock context analysis for the block layer, with the
accompanying annotations across queue limits, the blk_holder_ops
callbacks, crypto, cgroup, iocost, kyber and mq-deadline (Bart)
- Block status code infrastructure work: a tagged status table, a
str_to_blk_op() helper, a bio_endio_status() helper, and on top of
that a new configurable block-layer error injection facility
(Christoph)
- DRBD netlink rework, replacing the genl_magic machinery with explicit
netlink serialization and moving the DRBD UAPI headers to
include/uapi/linux/ (Christoph Böhmwalder)
- bvec improvements: a bvec_folio() helper and making the bvec_iter
helpers proper inline functions (Willy, Christoph)
- ublk cleanups and a canceling-flag fix for the disk-not-allocated
case (Caleb, Ming)
- Partition handling fixes: bound the AIX pp_count scan, fix an of_node
refcount leak, and replace __get_free_page() with kmalloc() (Bryam,
Wentao, Mike)
- Convert numa_node to int in blk_mq_hw_ctx and ->init_request, and add
WQ_PERCPU to the block workqueue users (Mateusz, Marco)
- Block statistics and tracing: propagate in-flight to the whole disk
on partition IO, export passthrough stats, and a new
block_rq_tag_wait tracepoint (Tang, Keith, Aaron)
- A round of removals, unexports and cleanups across bio, direct-io and
the bvec helpers (Christoph)
- Various driver fixes (mtip32xx use-after-free, rbd snap_count
validation and strscpy conversion, nbd socket lockdep reclassify,
virtio-blk zone report clamp, floppy) and a batch of MAINTAINERS
email/list updates (Coly, Li, Yu, Christoph Böhmwalder)
- Other little fixes and cleanups all over
* tag 'for-7.2/block-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (117 commits)
MAINTAINERS: Update Coly Li's email address
block: check bio split for unaligned bvec
nbd: Reclassify sockets to avoid lockdep circular dependency
block: add configurable error injection
block: add a str_to_blk_op helper
block: add a "tag" for block status codes
block: add a macro to initialize the status table
floppy: Drop unused pnp driver data
block: propagate in_flight to whole disk on partition I/O
virtio-blk: clamp zone report to the report buffer capacity
block: optimize I/O merge hot path with unlikely() hints
drivers/block/rbd: Use strscpy() to copy strings into arrays
partitions: aix: bound the pp_count scan to the ppe array
block: Enable lock context analysis
block/mq-deadline: Make the lock context annotations compatible with Clang
block/Kyber: Make the lock context annotations compatible with Clang
block/blk-mq-debugfs: Improve lock context annotations
block/blk-iocost: Inline iocg_lock() and iocg_unlock()
block/blk-iocost: Split ioc_rqos_throttle()
block/crypto: Annotate the crypto functions
...
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmowCCYQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgprbuEACf4MxZZvzd1Be5Fneo4ghy0IjJ0s8j67yo
hERtfO0TPCFPvFi+cWmSQo/YTIhi62Z1nFro3Cr9opTF38NypbN2kB642KOaQPse
f1LVQDg1p5MIj2bMGgBQsYeLlGipRwm/vLN7qX+Dfv6jL4Yq7KuIEtxkBr2UJefY
FUEhW5xm45DgRriyZSn+Gn39yLB4XGL8pEntqrSMlrTw3fI8ZzyIw5w6e8Kgn/ES
upVo8oxnrUSlPrGrfH2Eu42sFixWko9QULwASNgXsIT3yBRZbYYyBjXmWAQGER6k
D3Nn5Zt/UogyMg6BLb/S2dgCnFn2VPKPD8l//4IPQrAAhPdtWpxaTjQUXq0c2w4M
IaGD1EQDYw/VBc6jjRPAuqwEmKZmTXZirfXamvF4Y2MtSz0b84LulaeqqwGvmtUT
Z2pOaIZBffJ1E//B4FpiG1xuwHpchc3F6esP7FyhyRlWGP93WR3ADo2em7/jKxEi
IOFjXHAOO70C1wpdOgAxToRggc4GV16RAx+v4/GFRUI1m6okuxiwTxle6Ig+H4eR
zFXBvroz7PDDaiqbyGk5kIQfOVhwmPnp9EZUUJIf4h9lIPgwJAjb4yLwcF8ySXyg
x8C+Vp4DP1r0xccZLkkElIzOmmeR4bUaBeyFY33pT83WchnU1NxusBpEVpRL1FPf
YR8Icz8F1A==
=pkmg
-----END PGP SIGNATURE-----
Merge tag 'for-7.2/io_uring-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring updates from Jens Axboe:
- Rework the task_work infrastructure.
Both the local (DEFER_TASKRUN) and the normal (tctx) task_work lists
were llist based, which is LIFO ordered, and hence each run had to do
an O(n) list reversal pass first to restore queue order.
Additionally, to cap the amount of task_work run, each method needed
a retry list as well.
Add a lockless MPCS FIFO queue (based on Dmitry Vyukov's intrusive
MPSC algorithm) and switch both task_work lists to it. It performs
better than llists and we can then also ditch the retry lists as well
as entries are popped one-at-the-time.
On top of those changes, run the tctx fallback task_work directly and
remove the now-unused per-ctx fallback machinery entirely.
- zcrx user notifications.
Add a mechanism for zcrx to communicate conditions back to userspace
via a dedicated CQE, with the initial users being notification on
running out of buffers and on a frag copy fallback, plus
shared-memory notification statistics.
Alongside that, a series of zcrx reliability and cleanup fixes: more
reliable scrubbing, poisoning pointers on unregistration, dropping an
extra ifq close, adding a ctx back-pointer, reordering fd allocation
in the export path, and killing a dead 'sock' member.
- Allow using io_uring registered buffers for plain SEND and RECV, not
just for the zero-copy send path.
This enables targets like ublk's NBD backend to push/pull IO data
directly to/from a registered buffer over a plain send/recv on a TCP
socket.
- Registered buffer improvements: account huge pages correctly, bump
the io_mapped_ubuf length field to size_t, and raise the previous 1GB
registered buffer size limit.
- Restrict the ctx access exposed to io_uring BPF struct_ops programs
by handing them an opaque type rather than the full io_ring_ctx, and
add a separate MAINTAINERS entry for the bpf-ops code.
- Allow opcode filtering on IORING_OP_CONNECT.
- Validate ring-provided buffer addresses with access_ok(), and align
the legacy buffer add limit with MAX_BIDS_PER_BGID.
- Various other cleanups and minor fixes, including avoiding msghdr
async data on connect/bind, dropping async_size for OP_LISTEN, making
the POLL_FIRST receive side checks consistent, re-checking
IO_WQ_BIT_EXIT for each linked work item, and using
trace_call__##name() at guarded tracepoint call sites.
* tag 'for-7.2/io_uring-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (31 commits)
io_uring/bpf-ops: add a separate maintainer entry
io_uring/net: make POLL_FIRST receive side checks consistent
io_uring: remove the per-ctx fallback task_work machinery
io_uring: run the tctx task_work fallback directly
io_uring: switch normal task_work to a mpscq
io_uring: switch local task_work to a mpscq
io_uring/mpscq: add lockless multi-producer, single-consumer FIFO queue
io_uring: grab RCU read lock marking task run
io_uring/zcrx: kill dead 'sock' member in struct io_zcrx_args
io_uring/kbuf: validate ring provided buffer addresses with access_ok()
io_uring/net: support registered buffer for plain send and recv
io_uring/nop: Drop a wrong comment in struct io_nop
io_uring/net: Remove async_size for OP_LISTEN
io_uring/net: Avoid msghdr on op_connect/op_bind async data
io_uring/bpf-ops: restrict ctx access to BPF
io_uring/io-wq: re-check IO_WQ_BIT_EXIT for each linked work item
io_uring/kbuf: align legacy buffer add limit with MAX_BIDS_PER_BGID
io_uring/zcrx: add shared-memory notification statistics
io_uring/zcrx: notify user on frag copy fallback
io_uring/zcrx: notify user when out of buffers
...
API:
- Drop support for off-CPU cryptography in af_alg.
- Document that af_alg is *always* slower.
- Document the deprecation of af_alg.
- Remove zero-copy support from skcipher and aead in af_alg.
- Cap AEAD AD length to 0x80000000 in af_alg.
- Free default RNG on module exit.
Algorithms:
- Fix vli multiplication carry overflow in ecc.
- Drop unused cipher_null crypto_alg.
- Remove unused variants of drbg.
- Use lib/crypto in drbg.
- Use memcpy_from/to_sglist in authencesn.
- Allow authenc(hmac(sha{256,384}),cts(cbc(aes))) in FIPS mode.
- Disallow RSA PKCS#1 SHA-1 sig algs in FIPS mode.
- Filter out async aead implementations at alloc in krb5.
- Fix non-parallel fallback by rstoring callback in pcrypt.
- Validate poly1305 template argument in chacha20poly1305.
Drivers:
- Add sysfs PCI reset support to qat.
- Add KPT support for GEN6 devices to qat.
- Remove unused character device and ioctls from qat.
- Add support for hw access via SMCC to mtk.
- Remove prng support from crypto4xx.
- Remove prng support from hisi-trng.
- Remove prng support from sun4i-ss.
- Remove prng support from xilinx-trng.
- Remove loongson-rng.
- Remove exynos-rng.
Others:
- Remove support for AIO on sockets.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEn51F/lCuNhUwmDeSxycdCkmxi6cFAmovcvMACgkQxycdCkmx
i6emMA/8DfjbQLm7kGXKhlHaAZ9MgSxVUDs+yz+I1XbFL0sbEbLmZlwMiEta2Rg0
GteR+ZGxkzBbDwL3CHQ+ZJHycCgPxokeRJJ9rWT1j+tGEkJoj/00SRtzMqVXXcY5
0LGVcw8Ru15oUH7+PlMqnIKYhxrmIPXYGsSwP8FBdRF5CN2q1KNmfW5rtrc8KhPH
g3SRgH6sE2Bv1cQ7C6GKlgqE89WX41nUil5F5jyQwMlylqZV7SErPjFtc6aqgcoV
CQP0C2rC0LzZhfTrrnzXLT1h6RUNwulnQAkSICo40F9XZYO+o5EeZvRP0ESNvrda
fNYCqFOWZaQDjtfnxNT1ZrjT0R00H/Am/Hz5rNFng2sB/XArAqzvHTHtvbuV8t0O
Kew+KrMEWHSaGnsr+W4lYTw4wZKS1mxjE1P/FcVQN1WLeI5cNywyCVeD3qmhgcws
GACmM4ghFErBTajBfcHa8dMCshidMQVA6fqHEdOmNn+CxOCZhEdlLmfnJO1KCnEr
Aw9dnwhtuEdwha1kjNlzN0HC0T/zoumtINqI3Els4/23L7LW8doBAuyGLDd3VmII
Vr3gu3Oj9pv4Tl0RYJKRObFWM7MwVU6RpVxz2eVkEuM+ZM2je5kZo56tEa8vELrp
aM/L3yMeuYohssBcP9mfLpR7JCGB2186h47IUGGP6Jz0XjSy7Ns=
=4BGT
-----END PGP SIGNATURE-----
Merge tag 'v7.2-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
Pull crypto updates from Herbert Xu:
"API:
- Drop support for off-CPU cryptography in af_alg
- Document that af_alg is *always* slower
- Document the deprecation of af_alg
- Remove zero-copy support from skcipher and aead in af_alg
- Cap AEAD AD length to 0x80000000 in af_alg
- Free default RNG on module exit
Algorithms:
- Fix vli multiplication carry overflow in ecc
- Drop unused cipher_null crypto_alg
- Remove unused variants of drbg
- Use lib/crypto in drbg
- Use memcpy_from/to_sglist in authencesn
- Allow authenc(hmac(sha{256,384}),cts(cbc(aes))) in FIPS mode
- Disallow RSA PKCS#1 SHA-1 sig algs in FIPS mode
- Filter out async aead implementations at alloc in krb5
- Fix non-parallel fallback by rstoring callback in pcrypt
- Validate poly1305 template argument in chacha20poly1305
Drivers:
- Add sysfs PCI reset support to qat
- Add KPT support for GEN6 devices to qat
- Remove unused character device and ioctls from qat
- Add support for hw access via SMCC to mtk
- Remove prng support from crypto4xx
- Remove prng support from hisi-trng
- Remove prng support from sun4i-ss
- Remove prng support from xilinx-trng
- Remove loongson-rng
- Remove exynos-rng
Others:
- Remove support for AIO on sockets"
* tag 'v7.2-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (196 commits)
crypto: tegra - fix refcount leak in tegra_se_host1x_submit()
crypto: rng - Free default RNG on module exit
crypto: testmgr - allow authenc(hmac(sha{256,384}),cts(cbc(aes))) in FIPS mode
hwrng: jh7110 - fix refcount leak in starfive_trng_read()
crypto: atmel-ecc - drop dead code in atmel_ecdh_max_size
crypto: cavium/cpt - fix DMA cleanup using wrong loop index
crypto: marvell/octeontx - fix DMA cleanup using wrong loop index
MAINTAINERS: make myself the maintainer of the Qualcomm QCE driver
crypto: amcc - convert irq_of_parse_and_map to platform_get_irq
crypto: sun4i-ss - Remove insecure and unused rng_alg
hwrng: xilinx - Move xilinx-rng into drivers/char/hw_random/
crypto: xilinx-trng - Replace crypto_drbg_ctr_df() with HMAC-SHA512
crypto: xilinx-trng - Fix return value of xtrng_hwrng_trng_read()
crypto: xilinx-trng - Remove crypto_rng interface
crypto: exynos-rng - Remove exynos-rng driver
hwrng: hisi-trng - Move hisi-trng into drivers/char/hw_random/
crypto: hisi-trng - Remove crypto_rng interface
crypto: loongson - Remove broken and unused loongson-rng
crypto: crypto4xx - Remove insecure and unused rng_alg
crypto: qat - validate RSA CRT component lengths
...
-----BEGIN PGP SIGNATURE-----
iQFPBAABCAA5FiEEe7vIQRWZI0iWSE3xu+CwddJFiJoFAmovtcQbFIAAAAAABAAO
bWFudTIsMi41KzEuMTIsMiwyAAoJELvgsHXSRYia17cH/ROle2R7pTZbk60nupo3
JHWvvOB6j2xh5s4bxbcoBIb8ZdS5L3YeAE8u/1NUGFHckIJ/7/rB3njxeDtoJyIn
K1ER5LD2wd0jf0Z6P2WOB4MmoF0LSi+KcT0xRbMlmcd9KM9xQIFy7aLxYoWXlqqM
T+kDmLuMtUawZGlN+foSFMp2ewO9Iz31kiQACreLa246zTIkCAhqzTBBCFRPkCMX
cEES6J8Xb3YcfO66PUXqUC63pS0ikovP8P3yt92jWgaXTrntGDQYG/zxm2dkdwQl
UklmXA0KT1mkaTA8vUDPdBk9LFWVCdhK2rkMWyeirQZqIsHYGfj8vB1HoR839m6a
vTk=
=YSns
-----END PGP SIGNATURE-----
Merge tag 'slab-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab
Pull slab updates from Vlastimil Babka:
- Support for "allocation tokens" (currently available in Clang 22+)
for smarter partitioning of kmalloc caches based on the allocated
object type, which can be enabled instead of the "random"
per-caller-address-hash partitioning.
It should be able to deterministically separate types containing a
pointer from those that do not (Marco Elver)
- Improvements and simplification of the kmem_cache_alloc_bulk() and
mempool_alloc_bulk() API. This includes adaptation of callers
(Christoph Hellwig)
- Performance improvements and cleanups related mostly to sheaves
refill (Hao Li, Shengming Hu, Vlastimil Babka)
- Several fixups for the slabinfo tool (Xuewen Wang)
* tag 'slab-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab:
mm/slab: do not limit zeroing to orig_size when only red zoning is enabled
mm/slub: preserve original size in _kmalloc_nolock_noprof retry path
mm: simplify the mempool_alloc_bulk API
mm/slab: improve kmem_cache_alloc_bulk
mm/slub: detach and reattach partial slabs in batch
mm/slub: introduce helpers for node partial slab state
mm/slub: use empty sheaf helpers for oversized sheaves
tools/mm/slabinfo: remove redundant slab->partial assignment
tools/mm/slabinfo: remove dead assignment in get_obj_and_str()
tools/mm/slabinfo: Fix trace disable logic inversion
MAINTAINERS: add slab-related scripts and tools to SLAB ALLOCATOR
mm/slub: fix typo in sheaves comment
mm, slab: simplify returning slab in __refill_objects_node()
mm, slab: add an optimistic __slab_try_return_freelist()
slab: fix kernel-docs for mm-api
slab: improve KMALLOC_PARTITION_RANDOM randomness
slab: support for compiler-assisted type-based slab cache partitioning
mm/slub: defer freelist construction until after bulk allocation from a new slab
Futex updates:
- Optimize futex hash bucket access patterns (Peter Zijlstra)
- Large series to address the robust futex unlock race for real,
by Thomas Gleixner:
"The robust futex unlock mechanism is racy in respect
to the clearing of the robust_list_head::list_op_pending
pointer because unlock and clearing the pointer are not
atomic. The race window is between the unlock and clearing the
pending op pointer. If the task is forced to exit in this
window, exit will access a potentially invalid pending op
pointer when cleaning up the robust list. That happens if
another task manages to unmap the object containing the lock
before the cleanup, which results in an UAF. In the worst case
this UAF can lead to memory corruption when unrelated content
has been mapped to the same address by the time the access
happens.
User space can't solve this problem without help from the kernel.
This series provides the kernel side infrastructure to help it
along:
1) Combined unlock, pointer clearing, wake-up for the
contended case
2) VDSO based unlock and pointer clearing helpers with a
fix-up function in the kernel when user space was interrupted
within the critical section.
... with help by André Almeida:
- Add a note about robust list race condition (André Almeida)
- Add self-tests for robust release operations (André Almeida)
Context analysis updates:
- Implement context analysis for 'struct rt_mutex'.
(Bart Van Assche)
- Bump required Clang version to 23 (Marco Elver)
Guard infrastructure updates:
- Series to remove NULL check from unconditional guards
(Dmitry Ilvokhin)
Lockdep updates:
- Restore self-test migrate_disable() and sched_rt_mutex
state on PREEMPT_RT (Karl Mehltretter)
Membarriers updates:
- Use per-CPU mutexes for targeted commands (Aniket Gattani)
- Modernize membarrier_global_expedited with cleanup guards
(Aniket Gattani)
- Add rseq stress test for CFS throttle interactions (Aniket Gattani)
percpu-rwsems updates:
- Extract __percpu_up_read() to optimize inlining overhead
(Dmitry Ilvokhin)
Seqlocks updates:
- Allow UBSAN_ALIGNMENT to fail optimizing (Heiko Carstens)
Lock tracing:
- Add contended_release tracepoint to sleepable locks
such as mutexes, percpu-rwsems, rtmutexes, rwsems
and semaphores. (Dmitry Ilvokhin)
MAINTAINERS updates:
- MAINTAINERS: Add RUST [SYNC] entry (Boqun Feng)
Misc updates and fixes by Randy Dunlap, YE WEI-HONG,
Fabricio Parra, Dmitry Ilvokhin and Peter Zijlstra.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmoujYURHG1pbmdvQGtl
cm5lbC5vcmcACgkQEnMQ0APhK1jQFw//SNo1zxqGJuVudIWTpk5hUbWO6La9dc2C
2RW4lMNM551AaGFK6V01u1hD7dBZkWBqER+f3+CMtkqi734pAevTfkYJOitnGOOX
YZicGqTufV+Q+hcEiHAY3ADF6384SiyvAGNHL5wk3eLqC+Raug0f0cNt4opzFvr7
I6vRcW02JQMRKBe50+Aoa5zg7ymmbqaH0KkiPvfY5UFl38MKABhkY1izA9o+/Q/S
1/joNo5iQ3Ciqf6tn76vTg4Xiz7napikHB5LfWxorgmQo7FEdUY54GGMaa8aBOPi
EfgIV9p+qOJhmWn4H1HgVSYlnUn3tJICYnVVCruXWXD+pzITS+F5qSQ3p25F3VIt
c3jh3NaVW71pR/Sbj4zGvf8QguJL340RGJs/e5qcj91pjOSCH5B75ruNgFgYK9Lv
iRi6dnHBBHxaN57YHTOrhphvjyfc0L7i6cGyJgHJWwyi0y3XBYlNuP/VLPuwquKl
anK86hlkl9W4kO0OOukD1xhfuRv7066EdJc3iWWlp9yz+yI5xDGuV4FhvEHRoy38
R9VY2bwzgZrW/NkodHIb95+W30qEmeq2z1W4wiBu+mf1JnyW2g/n8jKkc6AINMpI
VGcom3MdinxjUtmybZT9Fx0SzBYCc+aRPkh7lPV7eXdZAWufxUO/6eVKiHBQ2HUO
RFDkOZU0r+c=
=gpHw
-----END PGP SIGNATURE-----
Merge tag 'locking-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull locking updates from Ingo Molnar:
"Futex updates:
- Optimize futex hash bucket access patterns (Peter Zijlstra)
- Large series to address the robust futex unlock race for real, by
Thomas Gleixner:
"The robust futex unlock mechanism is racy in respect to the
clearing of the robust_list_head::list_op_pending pointer because
unlock and clearing the pointer are not atomic.
The race window is between the unlock and clearing the pending op
pointer. If the task is forced to exit in this window, exit will
access a potentially invalid pending op pointer when cleaning up
the robust list.
That happens if another task manages to unmap the object
containing the lock before the cleanup, which results in an UAF.
In the worst case this UAF can lead to memory corruption when
unrelated content has been mapped to the same address by the time
the access happens.
User space can't solve this problem without help from the kernel.
This series provides the kernel side infrastructure to help it
along:
1) Combined unlock, pointer clearing, wake-up for the
contended case
2) VDSO based unlock and pointer clearing helpers with a
fix-up function in the kernel when user space was interrupted
within the critical section.
... with help by André Almeida:
- Add a note about robust list race condition (André Almeida)
- Add self-tests for robust release operations (André Almeida)
Context analysis updates:
- Implement context analysis for 'struct rt_mutex'. (Bart Van Assche)
- Bump required Clang version to 23 (Marco Elver)
Guard infrastructure updates:
- Series to remove NULL check from unconditional guards (Dmitry
Ilvokhin)
Lockdep updates:
- Restore self-test migrate_disable() and sched_rt_mutex state on
PREEMPT_RT (Karl Mehltretter)
Membarriers updates:
- Use per-CPU mutexes for targeted commands (Aniket Gattani)
- Modernize membarrier_global_expedited with cleanup guards (Aniket
Gattani)
- Add rseq stress test for CFS throttle interactions (Aniket Gattani)
percpu-rwsems updates:
- Extract __percpu_up_read() to optimize inlining overhead (Dmitry
Ilvokhin)
Seqlocks updates:
- Allow UBSAN_ALIGNMENT to fail optimizing (Heiko Carstens)
Lock tracing:
- Add contended_release tracepoint to sleepable locks such as
mutexes, percpu-rwsems, rtmutexes, rwsems and semaphores (Dmitry
Ilvokhin)
MAINTAINERS updates:
- MAINTAINERS: Add RUST [SYNC] entry (Boqun Feng)
Misc updates and fixes by Randy Dunlap, YE WEI-HONG, Fabricio Parra,
Dmitry Ilvokhin and Peter Zijlstra"
* tag 'locking-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (36 commits)
locking: Add contended_release tracepoint to sleepable locks
locking/percpu-rwsem: Extract __percpu_up_read()
tracing/lock: Remove unnecessary linux/sched.h include
futex: Optimize futex hash bucket access patterns
rust: sync: completion: Mark inline complete_all and wait_for_completion
MAINTAINERS: Add RUST [SYNC] entry
cleanup: Specify nonnull argument index
selftests: futex: Add tests for robust release operations
Documentation: futex: Add a note about robust list race condition
x86/vdso: Implement __vdso_futex_robust_try_unlock()
x86/vdso: Prepare for robust futex unlock support
futex: Provide infrastructure to plug the non contended robust futex unlock race
futex: Add robust futex unlock IP range
futex: Add support for unlocking robust futexes
futex: Cleanup UAPI defines
x86: Select ARCH_MEMORY_ORDER_TSO
uaccess: Provide unsafe_atomic_store_release_user()
futex: Provide UABI defines for robust list entry modifiers
futex: Move futex related mm_struct data into a struct
futex: Make futex_mm_init() void
...
io_recv() and io_recvzc() are the odd ones out, as they checks for
whether POLL_FIRST should be honored before checking if the file is a
socket. It doesn't really matter, but might as well make it consistent
across all receive and send types.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
With the tctx fallback running its entries directly, the per-ctx
fallback work has a single user left: moving local (DEFER_TASKRUN)
task_work entries out of a ring that is going away. Both of its call
sites are process context and don't hold ->uring_lock, the same
conditions the deferred fallback work itself ran under - so run the
entries in cancel mode right there instead, and rename the helper to
io_cancel_local_task_work() to match what it now does.
With that, ->fallback_llist, ->fallback_work, io_fallback_req_func()
and __io_fallback_tw() can all go away, along with the fallback work
flushing in the ring exit and cancel paths. Requests that get
orphaned by an exiting task now run via the tctx fallback work, which
the ring exit side implicitly waits on through the ctx refs those
requests hold.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The fallback work drains the tctx queue only to redistribute the entries
into the per-ctx fallback lists, bouncing them through a second
(per-ctx) work item before they finally run. That made sense when the
producer side did the draining and could be in any context, but the
fallback work is a regular process context kworker: it can just run the
entries itself. Reuse the normal run loop - if run from the fallback
kernel thread, ts.cancel will get set, and the work terminated.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Like the local task_work list, the normal (tctx) task_work list is an
llist, and hence needs the O(n) llist_reverse_order() pass before
running entries in queue order. On top of that, capped runs - sqpoll
processing IORING_TW_CAP_ENTRIES_VALUE entries at a time - need the
claimed-but-unprocessed leftovers carried in a separate retry_list,
as they can't be pushed back to the shared list.
Switch tctx->task_list to a mpscq, like what was done for the
DEFER_TASKRUN paths as well.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The local (DEFER_TASKRUN) task_work list is an llist, which is LIFO
ordered, and hence __io_run_local_work() has to restore the right
running order with an O(n) llist_reverse_order() pass first. On top of
that, a batch that gets capped by max_events needs the leftover entries
parked on a separate ->retry_llist, as they can't be pushed back to the
shared list.
Switch it to the FIFO mpscq. Adds are wait-free instead of a cmpxchg
retry loop, entries are popped in queue order with no reversal pass,
capping a run simply leaves the remainder on the queue, and
->retry_llist goes away entirely. The consumer cursor, ->work_head,
lives with the rest of the ->uring_lock protected state rather than
next to the queue, so that popping entries doesn't dirty the producer
side cacheline.
For low amounts of task_work, this ends up being a bit more efficient
than the existing scheme. As an example of that, doing multishot
receives for 8 clients has the following task_work overhead:
1.02% sock-test [kernel.kallsyms] [k] io_req_local_work_add
0.88% sock-test [kernel.kallsyms] [k] __io_run_local_work_loop
0.60% sock-test [kernel.kallsyms] [k] llist_reverse_order
0.14% sock-test [kernel.kallsyms] [k] __io_run_local_work
2.64% at ~46Gb/sec
and after this change:
1.08% sock-test [kernel.kallsyms] [k] io_req_local_work_add
1.03% sock-test [kernel.kallsyms] [k] __io_run_local_work
2.11% at ~53Gb/sec
which has less overhead even though that test run was faster. For a case
of having 1024 clients on a single ring:
2.22% sock-test [kernel.kallsyms] [k] llist_reverse_order
0.84% sock-test [kernel.kallsyms] [k] __io_run_local_work_loop
0.42% sock-test [kernel.kallsyms] [k] io_req_local_work_add
0.02% sock-test [kernel.kallsyms] [k] __io_run_local_work
3.50% at ~24Gb/sec
we start to see the llist reversing taking a considerable amount of
time, and the total add+run task_work overhead is around 3.5%. After
the change:
0.90% sock-test [kernel.kallsyms] [k] __io_run_local_work
0.42% sock-test [kernel.kallsyms] [k] io_req_local_work_add
1.32% at ~26Gb/sec
most of that overhead is gone, and performance is better as well.
Caleb Sander Mateos <csander@purestorage.com> reports that it improves
the performance of a ublk 4kb workload by 4% [1], while testing v1 of
this patchset.
[1] https://lore.kernel.org/io-uring/CADUfDZr-MMYBaP-e+y9+xuRhuiunO2sBTUCmwZyd7AgT8sVtiQ@mail.gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Local task_work is currently using llists for managing the work,
but that's a LIFO type of list. This means that running this task_work
needs to reverse the list first, to ensure fairness in running the
queued items.
Add a lockless FIFO queued, based on Dmitry Vyukov's intrusive MPSC
node-based queue algorithm, modified with an externally held consumer
cursor and conditional stub reinsertion. See comments in the header.
Producers are wait-free: a push is a single xchg() on the queue tail,
which serializes concurrent producers and defines the FIFO order, plus
a store linking the node to its predecessor. There are no cmpxchg retry
loops, and pushing is safe from any context, including hardirq.
The cost of linked list FIFO ordering is that a push publishes the node
in two steps - the xchg() makes it visible as the new tail before the
subsequent store links it into the chain that is reachable from the
head. A consumer hitting that window gets a NULL from mpscq_pop() while
mpscq_empty() reports false, and must retry later rather than treat the
queue as empty. The window is two instructions wide, but a producer can
get preempted inside it, so the consumer must not busy wait on it.
The consumer side supports a single consumer at a time, with callers
providing their own serialization. A stub node, which also defines the
empty state (tail == stub), allows the consumer to detach the final
node without racing against producer link stores: that node is only
handed out once the stub has been cmpxchg'ed back in as the tail. This
also guarantees that the previous tail returned by mpscq_push() cannot
get freed before that push has linked it, making it always valid for
comparisons.
The consumer cursor is deliberately not part of the queue struct - the
caller owns it and passes it to mpscq_pop(). This is done to separate
the consumer and producers cacheline. The cursor is written for every
popped entry, and keeping it on the same cacheline as ->tail would have
the consumer invalidating the line that producers need for every push.
Keeping it external lets the caller place it with its own consumer side
data instead.
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Not required right now, as io_req_local_work_add() already calls this
helper with the RCU read lock held. But in preparation for that not
being the case, grab it locally.
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmorP5QQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpvLwEAC2LpbRdv9oclx2l1zhM8RvnRcSMcMrcNzB
HpDXBe8ljMg/9Omv90FFTjtfmrUvu9McnkxCsvd7j2xLBz1WUOadD1M9qDeDtzSC
u7BqrL8XnpCrEds5N6nMyXaFnT50HXI5WO2LVGW7Hk3gVO6L6po2aBlXgOJ6K2a+
BLQ2DN05rNbD4dTbO73wUObSCqEwVrCZp7jnPWL/Z7ff/04HATPOtaEPVtjCyVW9
v+gFb9DDXzAGVp4MKTeT3rUQQI3wKj/REmI/4hTMHWofcif6rvuSDQg9ktMu/3SW
Weg7fmbjubeYiKLSsjx+abJ2hefyHOZrXZoaPtx0Moxou6K3RnxFUSCgod+QsBUF
u1MoDAWhXJx9lDoPjYOjaL8gEv/rwNZoS15OCY3eUWLWtPG6XhYSUQNaIV6Y66eM
7QzajFfZ50QMPNXSN8jyedOeiXIMu8sjcFdmGiL7ALtGfqdZApVyYZIxXOJ31q6D
R+ucSQgUXeDF1VsUmLIGkFsbRvPPpoxn+tPZUSVt2me4V2aWWtzdN+ciXaknrKj/
z1lmEL8eiHBacVIlSnJcUMEChmC59K6M6+YFPJUUQbzLeiYyw5qxdk093OLrfeC/
ZMw0OpSbjI7antbqHA8UvDlrUGCg89+96sbt5fDfed0dk0j/kjOOfTQEvQO4dOcn
qlksyQnguQ==
=cim5
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.1-20260611' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Tweak for an off-by-one in the CQ ring accounting for the min wait
support.
- Don't truncate end buffer length for a bundle, as the transfer might
not happen. It's not required in the first place, as the completion
side handles this condition already.
* tag 'io_uring-7.1-20260611' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/wait: fix min_timeout behavior
io_uring/kbuf: don't truncate end buffer for bundles
Commit:
809b997a5c ("x86-64/arm64/powerpc: clean up and rename __copy_from_user_flushcache")
sanitized that any provided copy helper should separately validate
destination and source addresses, but we should also ensure that
anything that is retrieved from a buffer is validated upfront. For ring
provided buffers, always include an access_ok() when grabbing a new
buffer.
Fixes: c7fb19428d ("io_uring: add support for ring mapped supplied buffers")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
So far IORING_RECVSEND_FIXED_BUF is only honoured on the SEND_ZC path,
even though the import wiring is already present for plain send and
completely absent for recv. Targets such as ublk's NBD backend want to
push/pull I/O data directly to/from an io_uring registered buffer over a
plain send/recv on a TCP socket.
Wire IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and
IORING_OP_RECV paths:
- Accept the flag in SENDMSG_FLAGS / RECVMSG_FLAGS and, at prep time,
restrict it to the non-vectorized IORING_OP_SEND / IORING_OP_RECV
opcodes. It is mutually exclusive with buffer select, bundles and
(for recv) multishot, and records sqe->buf_index.
- For recv, set REQ_F_IMPORT_BUFFER in setup so the registered buffer
is imported lazily at issue time, mirroring the send path.
- In io_send()/io_recv(), import the registered buffer via
io_import_reg_buf() (ITER_SOURCE for send, ITER_DEST for recv) and
clear REQ_F_IMPORT_BUFFER. The resulting bvec iter persists in
async_data, so MSG_WAITALL partial send/recv retries resume at the
right offset.
Signed-off-by: Ming Lei <tom.leiming@gmail.com>
Link: https://patch.msgid.link/20260608142511.659240-2-ming.lei@redhat.com
[axboe: combine flags checks]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The wakeup condition if a min timeout is present and has expired is that
at least _one_ CQE was posted. Thus set the cq_tail target to
->cq_min_tail + 1. Without this commit a spurious wakeup can result in a
premature wakeup because io_should_wake() will return true even if _no_
CQE was posted at all.
Cc: Tip ten Brink <tip@tenbrinkmeijs.com>
Fixes: e15cb2200b ("io_uring: fix min_wait wakeups for SQPOLL")
Cc: stable@vger.kernel.org
Signed-off-by: Christian A. Ehrhardt <lk@c--e.de>
Link: https://patch.msgid.link/20260606201120.1441447-1-lk@c--e.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
If buffers have been peeked for a bundle receive, the kernel will
truncate the end buffer, if the available length is shorter than the
buffer itself. This is unnecessary, as applications iterating bundle
receives must always use the minimum size of the buffer length and the
remaining number of bytes in the bundle. The examples in liburing do
that as well, eg examples/proxy.c.
If the kernel does truncate this buffer AND the current transfer fails,
then the buffer will be left with a smaller size than what is otherwise
available.
Just remove the buffer truncation, as it's not necessary in the first
place.
Link: https://lore.kernel.org/io-uring/CAAEr8jbY60noGj1fw_k91UJRBkyiRVoS6=nLhZ7Svwidjn4CAA@mail.gmail.com/
Reported-by: Federico Brasili <federico.brasili@gmail.com>
Cc: stable@vger.kernel.org
Fixes: 35c8711c8f ("io_uring/kbuf: add helpers for getting/peeking multiple buffers")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmojJQ0QHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgppGsD/0d9CxOz1SLooS0ufvmb0+Jm1d8tHe+NeNv
ALINBbvytrEppGNh4M+u25ZcWzevKHZTCIM9++nBNJOlTHLvP+E8dIdUuZvct2GU
y1jK28y/K9iM11MMAN2ey7OI8J20ABQGoCfXXwmBbqw3eM27gap74GpLwQ6vDG7E
xCnzS8M9R7MUxO2Y2ZfQkfgMUuViNMe3V9r4zz0Dgjn1Ao8cWduJ4BsZBN3sRMtA
D9EqU2kd4SscnwYSb8guPlshf3u0oY1GBTFozgUiqPw9bz6EAPk90ikr8gUgsWyk
UxgEWQLgIcwkpM+ym5kBGYcy4htA3bYM0XkYN5CaTzFg47roaQW9XVe+w14IAtKR
iSVpVGjDv/zZRJjDadN95GEXtVy5ZYZrIt2+2qwns9lTqGwAzHfVlnjNx2tqq4VS
LVhqgM6UzXHXun0CpBJVZ66j6kKjDyme8j0RjgZ/Gt2QhnrFIBfUiXjac5UgiZjp
+4s9VWdNViomho24lgxgtpGwRPXfdZjbl8LPS2O3tywOQOikUGBpVM7Nja82N/0b
mV2szmMl6nCu0wHfqQ54nP7maZa0TtLU6K1ozSd5spOrY5ZDsrleLiciRSCbNR2R
jRKEUsReZe7CZ/4h/atZws2/eVCpYh2PMnSFMb1w8v5gnQk94ryPd9KKYbK8dsGk
zU0OdJxJ2A==
=OdYU
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.1-20260605' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fix from Jens Axboe:
"A single fix for a missing flag mask when multishot is used with
an incrementally consumed buffer ring, potentially leading to
application confusion because of lack of IORING_CQE_F_BUF_MORE
consistency"
* tag 'io_uring-7.1-20260605' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/net: inherit IORING_CQE_F_BUF_MORE across bundle recv retries
When a bundle recv retries inside io_recv_finish(), the merge logic OR
the saved cflags from the previous iteration with the cflags returned by
the new iteration:
cflags = req->cqe.flags | (cflags & CQE_F_MASK);
Bits listed in CQE_F_MASK are inherited from the new iteration, and all
other bits (notably IORING_CQE_F_BUFFER and the buffer ID) come from the
saved cflags. Before this change CQE_F_MASK covered only
IORING_CQE_F_SOCK_NONEMPTY and IORING_CQE_F_MORE.
When using provided buffer rings (IOU_PBUF_RING_INC) with incremental
mode, and bundle recv, io_kbuf_inc_commit() can leave the head ring
entry partially consumed, __io_put_kbufs() then sets
IORING_CQE_F_BUF_MORE on the returned cflags so userspace knows the
buffer ID will be reused for subsequent completions.
Because IORING_CQE_F_BUF_MORE was not in CQE_F_MASK, the merge above
silently dropped it whenever the final retry iteration partially
consumed the buffer, and the subsequent req->cqe.flags = cflags &
~CQE_F_MASK save would have left a stale IORING_CQE_F_BUF_MORE in the
carried-over cflags had one been present. Userspace would then
wrongfully advance it ring head past an entry the kernel still uses.
Add IORING_CQE_F_BUF_MORE to CQE_F_MASK so it is both inherited from the
new iteration into the user-visible CQE and stripped from the saved
cflags between iterations.
Cc: stable@vger.kernel.org
Signed-off-by: Clément Léger <cleger@meta.com>
Assisted-by: Claude:claude-opus-4.6
Fixes: ae98dbf43d ("io_uring/kbuf: add support for incremental buffer consumption")
Link: https://patch.msgid.link/20260604160715.2482972-1-cleger@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The kmem_cache_alloc_bulk return value is weird. It returns the number
of allocated objects, but that must always be 0 or the requested number
based on the implementations and the handling in the callers, but that
assumption is not actually documented anywhere, which confuses automated
review tools.
Fix this by returning a bool if the allocation succeeded and adding a
kerneldoc comment explaining the API.
[rob.clark@oss.qualcomm.com: fixups in
msm_iommu_pagetable_prealloc_allocate() ]
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com> # skbuff
Link: https://patch.msgid.link/20260528093437.2519248-2-hch@lst.de
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Unlocking robust non-PI futexes happens in user space with the following
sequence:
1) robust_list_set_op_pending(mutex);
2) robust_list_remove(mutex);
lval = 0;
3) lval = atomic_xchg(lock, lval);
4) if (lval & WAITERS)
5) sys_futex(WAKE,....);
6) robust_list_clear_op_pending();
That opens a window between #3 and #6 where the mutex could be acquired by
some other task which observes that it is the last user and:
A) unmaps the mutex memory
B) maps a different file, which ends up covering the same address
When the original task exits before reaching #6 then the kernel robust list
handling observes the pending op entry and tries to fix up user space.
In case that the newly mapped data contains the TID of the exiting thread
at the address of the mutex/futex the kernel will set the owner died bit in
that memory and therefore corrupting unrelated data.
PI futexes have a similar problem both for the non-contented user space
unlock and the in kernel unlock:
1) robust_list_set_op_pending(mutex);
2) robust_list_remove(mutex);
lval = gettid();
3) if (!atomic_try_cmpxchg(lock, lval, 0))
4) sys_futex(UNLOCK_PI,....);
5) robust_list_clear_op_pending();
Address the first part of the problem where the futexes have waiters and
need to enter the kernel anyway. Add a new FUTEX_ROBUST_UNLOCK flag, which
is valid for the sys_futex() FUTEX_UNLOCK_PI, FUTEX_WAKE, FUTEX_WAKE_BITSET
operations.
This deliberately omits FUTEX_WAKE_OP from this treatment as it's unclear
whether this is needed and there is no usage of it in glibc either to
investigate.
For the futex2 syscall family this needs to be implemented with a new
syscall.
The sys_futex() case [ab]uses the @uaddr2 argument to hand the pointer to
robust_list_head::list_pending_op into the kernel. This argument is only
evaluated when the FUTEX_ROBUST_UNLOCK bit is set and is therefore backward
compatible.
This is an explicit argument to avoid the lookup of the robust list pointer
and retrieving the pending op pointer from there. User space has the
pointer already available so it can just put it into the @uaddr2
argument. Aside of that this allows the usage of multiple robust lists in
the future without any changes to the internal functions as they just operate
on the provided pointer.
This requires a second flag FUTEX_ROBUST_LIST32 which indicates that the
robust list pointer points to an u32 and not to an u64. This is required
for two reasons:
1) sys_futex() has no compat variant
2) The gaming emulators use both both 64-bit and compat 32-bit robust
lists in the same 64-bit application
As a consequence 32-bit applications have to set this flag unconditionally
so they can run on a 64-bit kernel in compat mode unmodified. 32-bit
kernels return an error code when the flag is not set. 64-bit kernels will
happily clear the full 64 bits if user space fails to set it.
In case of FUTEX_UNLOCK_PI this clears the robust list pending op when the
unlock succeeded. In case of errors, the user space value is still locked
by the caller and therefore the above cannot happen.
In case of FUTEX_WAKE* this does the unlock of the futex in the kernel and
clears the robust list pending op when the unlock was successful. If not,
the user space value is still locked and user space has to deal with the
returned error. That means that the unlocking of non-PI robust futexes has
to use the same try_cmpxchg() unlock scheme as PI futexes.
If the clearing of the pending list op fails (fault) then the kernel clears
the registered robust list pointer if it matches to prevent that exit()
will try to handle invalid data. That's a valid paranoid decision because
the robust list head sits usually in the TLS and if the TLS is not longer
accessible then the chance for fixing up the resulting mess is very close
to zero.
The problem of non-contended unlocks still exists and will be addressed
separately.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: André Almeida <andrealmeid@igalia.com>
Link: https://patch.msgid.link/20260602090535.670514505@kernel.org
This was copy-pasted from io_rw, where the comment actually makes sense.
Here, we don't have struct kiocb. Drop the comment.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260602215327.1885109-4-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Both IORING_OP_CONNECT and IORING_OP_BIND reuse the msghdr object just
to store the sockaddr. Beyond allocating a much larger object than
needed, msghdr can also wrap an iovec, which will be recycled
unnecessarily. This uses the sockaddr directly.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260602215327.1885109-2-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
BPF programs should have no need in looking into struct io_ring_ctx, if
anything, most of such cases would be anti patterns like looking up ring
indices directly via the context.
Replace it with a new empty structure, which is just an alias to struct
io_ring_ctx. It'll create a new BTF type and fail verification if a BPF
program tries to access it (beyond the first byte). It'll also give more
flexibility for the future, and otherwise it can be made aligned with
io_ring_ctx as before with struct groups if ever needed or extended in a
different way.
Fixes: d0e437b76b ("io_uring/bpf-ops: implement loop_step with BPF struct_ops")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/5f6ca3649e9e0bae8667db4357e28dd00cd07901.1780394491.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmoZxb4QHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpgDQEADbTyZ2P1pOKJwr7Gt5HETXXuvsocb7Z+CY
qfdfVgTz3aUcIg1ui+2QbpNU6l5zQKlp8IA7igjLHMhiif9nNzZTUPHayBrlhTo2
gS/RNOLtwGEv5oz+7i+2JAl+SyNRLQzK+UH0kmePIZp0bQPr1ciS13Ndb33Eus/K
FmtfeRJEsT6fZ+Q+QooxiN5n5jRdcB/WKSX5cwoIOnKJxKCyms/C8hUsG8Tz+0L/
2Ys8W06as3tE5eWjwpOXo5sb2d0zhr6tiqPUNpwEVbaBzLiBqI2wQW9gAy2ZXa1Z
vgCSceKOtt1I9/tf0y3xOy/V6LgUe+fp8X5V4yNPiDOwDTVrTYDOfT0qeqWJpwF+
4W+ZaIjHCNY2q/lbdeEuHYOOstvc5hShH4AJFQqbhUCXyB/qivO0rwQtZ75pGZRF
RhYJ8VdBdC740OyAdHPyTTUxWHAHr3+9lA10DsEw2dMxeDDmqdNTUrnw5b4/OVFl
jk2/cC1MmxtnPWPFK4NQGj2BMWpTqNfTykD7WQWOHUwvtCj7ZQHH6g4qfovuu2VR
Jt0vQE2uAab440knpQZCbURkgRrlZLLNYImhtbgTUyN/Jsk7vrV5/bevFwGWsA4y
CiwHsdztx62J2OsAbNBGw/a4qvMn7X3cLeylEeJiWyQxtkpb8JgdmxDjBGDavwLB
H5rtGBToWg==
=tYXi
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.1-20260529' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fix from Jens Axboe:
"Just a single fix for a regression introduced in this cycle, where
we should ensure the node is visible before the entry is added to
the tctx list"
* tag 'io_uring-7.1-20260529' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/tctx: set ->io_uring before publishing the tctx node
This is a simple helper which replaces page_folio(bvec->bv_page).
Minor improvement in readability, but the real motivation is to reduce
the number of references to bvec->bv_page so that it can be changed
with less work.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Leon Romanovsky <leon@kernel.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: William Kucharski <william.kucharski@linux.dev>
Link: https://patch.msgid.link/20260528175905.1102280-2-willy@infradead.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The only user of msg->msg_iocb was AF_ALG, but that's deprecated.
It can be removed entirely at the cost of only supporting synchronous
operations. This doesn't break userspace, which will silently block
(for a bounded amount of time) in io_submit instead of operating
asynchronously.
This also makes struct msghdr smaller, helping every other caller of
sendmsg().
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
Acked-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
commit 10dc959398 ("io_uring/io-wq: check IO_WQ_BIT_EXIT inside work
run loop") fixed the obvious case where io_worker_handle_work() took one
exit-bit snapshot before draining pending work, but the fix stops one
level too early.
io_worker_handle_work() now re-checks IO_WQ_BIT_EXIT in its outer work
run loop, yet it still snapshots that bit once before processing a whole
dependent linked-work chain. If io_wq_exit_start() sets IO_WQ_BIT_EXIT
after the first linked item has started, the remaining linked items can
still reuse stale do_kill = false, skip IO_WQ_WORK_CANCEL, and continue
running after exit has begun.
Move the check further inside, so it covers linked items too. Note: this
is a syzbot special as it loves setting up tons of slow linked work on
weird devices like msr that take forever to read, and immediately close
the ring. Exit then takes a long time.
Fixes: 10dc959398 ("io_uring/io-wq: check IO_WQ_BIT_EXIT inside work run loop")
Cc: stable@vger.kernel.org
Signed-off-by: Runyu Xiao <runyu.xiao@seu.edu.cn>
Link: https://patch.msgid.link/20260527172203.2043962-1-runyu.xiao@seu.edu.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_provide_buffers_prep() accepts nbufs up to MAX_BIDS_PER_BGID, but
io_add_buffers() stops when bl->nbufs reaches USHRT_MAX. This makes the
effective add limit one lower than the validated limit.
Use MAX_BIDS_PER_BGID in the add-side boundary check so validation and
execution use the same limit, and update the comment to refer to the
actual limit constant.
Signed-off-by: liyouhong <liyouhong@kylinos.cn>
Link: https://patch.msgid.link/20260528024936.3672659-1-dayou5941@163.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Add support for an optional stats struct embedded in the refill queue
region, allowing userspace to monitor copy-fallback in real-time.
Userspace queries the stats struct size and alignment via
IO_URING_QUERY_ZCRX_NOTIF (notif_stats_size / notif_stats_alignment),
then provides a stats_offset in zcrx_notification_desc pointing to a
location within the refill queue region.
The kernel updates the stats counters in-place on every copy-fallback
event.
Signed-off-by: Clément Léger <cleger@meta.com>
[pavel: rename io_uring_zcrx_notif_stats]
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/f6af5a21015efea4b733b9d77aba22c637788fe4.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Add a ZCRX_NOTIF_COPY notification type to signal userspace when a
received fragment could not be delivered using zero-copy and was
instead copied into a buffer.
Signed-off-by: Clément Léger <cleger@meta.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/3d54bcd8bf10b3a1e88beb0cd39c40c3937bea4f.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
There are currently no easy ways for the user to know if zcrx is out of
buffers and page pool fails to allocate. Add uapi for zcrx to communicate
it back.
It's implemented as a separate CQE, which for now is posted to the creator
ctx. To use it, on registration the user space needs to pass an instance
of struct zcrx_notification_desc, which tells the kernel the user_data
for resulting CQEs and which event types are expected / allowed.
When an allowed event happens, zcrx will post a CQE containing the
specified user_data, and lower bits of cqe->res will be set to the event
mask. Before the kernel could post another notification of the given
type, the user needs to acknowledge that it processed the previous one
by issuing IORING_REGISTER_ZCRX_CTRL with ZCRX_CTRL_ARM_NOTIFICATION.
The only notification type the patch implements is
ZCRX_NOTIF_NO_BUFFERS, but we'll need more of them in the future.
Co-developed-by: Vishwanath Seshagiri <vishs@meta.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Vishwanath Seshagiri <vishs@meta.com>
Link: https://patch.msgid.link/35cd307a03a43583838a2e151fc641c69abd786f.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
zcrx will need to have a pointer to an owning ctx to communicate
different events. Reference the ctx while it's attached to zcrx, and
rely on zcrx termination to drop the ctx to avoid circular ref deps.
Co-developed-by: Vishwanath Seshagiri <vishs@meta.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Vishwanath Seshagiri <vishs@meta.com>
Link: https://patch.msgid.link/b60514b3d1bd92f571e3bd91751166f8c3599256.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Currently, zcrx_export() allocates a file descriptor and copies the
control structure to userspace before the backing file is created.
While the operation returns an error on failure, it is cleaner to
follow the standard kernel pattern of performing the copy_to_user()
and fd_install() only after all resource allocations (like the
anon_inode) have succeeded. This aligns the code with other
fd-publishing paths in the VFS.
Signed-off-by: Bertie Tryner <Bertie.Tryner@warwick.ac.uk>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/1513a3f4ae7161692ca6e991b9f01278a6bc60e4.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Currently, scrubbing is done once before killing all recvzc requests.
It's fine as those are cancelled and don't return buffers afterwards,
but it'll be more reliable not to rely that much on cancellations.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/c4ea127023494cbbedebd21a2b7ae5ff0448eb95.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_register_iowq_max_workers() walks ctx->tctx_list under ctx->tctx_lock
and dereferences each node's task->io_uring without a NULL check:
list_for_each_entry(node, &ctx->tctx_list, ctx_node) {
tctx = node->task->io_uring;
if (WARN_ON_ONCE(!tctx->io_wq))
continue;
...
}
__io_uring_add_tctx_node() installs the node into ctx->tctx_list (via
io_tctx_install_node(), which does the list_add() under tctx_lock) and
only assigns current->io_uring = tctx afterwards. A task doing its first
io_uring operation on a shared ring therefore has a window in which its
node is already visible on ctx->tctx_list while node->task->io_uring is
still NULL. A concurrent IORING_REGISTER_IOWQ_MAX_WORKERS on the same
ring reads that NULL and dereferences tctx->io_wq:
KASAN: null-ptr-deref in range [0x0000000000000018-0x000000000000001f]
RIP: io_register_iowq_max_workers io_uring/register.c:423
Publish current->io_uring = tctx before installing the node, so any node
visible on ctx->tctx_list always has a valid task->io_uring.
Fixes: 7880174e1e ("io_uring/tctx: clean up __io_uring_add_tctx_node() error handling")
Signed-off-by: Lim HyeonJun <shja0831@gmail.com>
Link: https://patch.msgid.link/20260524110853.115634-1-shja0831@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmoQefMQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgppAuD/9zWiRNad+8O/tr5C2RuS0JFTKdnRV1tnnZ
IxsYqi2CM68ETERx9Rs7Orp0a8HjOav9F9Z88gNDXQpX+XdzmL4gvTix/DwbaM8f
01UgNpDDIZSdFDmFG2U/gN0430j10wXnPaWglFsKVydSGEoE42RAQ4OEdcwOmqDJ
Xt46mHOnAHN1BRi3g38JnWhFCQfTv0CxYVF1sAbOZFQH2xJG1NdkdVoaVM4kg1a4
s1Wc+CSYQsyuQq2+LbAeY33ZsdukDvow8EAYIW+zpSivxIqgF6iD4uYTwHp4hTEr
Yq3nPAoZax8VhuE+WODrpCdP1NUzdgjlrb3wu/Lqc/KaoP8iX/TrmPdFmirGD5Lo
2PYKdMu+aXxJUpGAPtwHGLgNP9ZV1P+BLph6oHol+I8BwCl35XfcGVDOjKEnVJjk
6FISG9WLUSWQHLFY9+FX7smX0zbhPJJCK4LH1mScZxkOf8D+MrwwU//gTJrcCMK+
56f0IjHJhR2goic3dDzu2MQxCl2xSKI3y+yHiH9Qy21WigZcZBmDfh0XoLC51m/a
ewlL/xCNK0MvX3iAm5dv+fUGqdRVAwxO7cRd2FXkD1lL6fJYzLn8+pGA4iRTux2Z
3MyO/yEORHShwf89phKSmGlVPY/i5aUAb8O6CXVicNpV2YMGrUw1u5XOT+u6R834
qccogH2jYQ==
=RUiY
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.1-20260522' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix for an issue with IORING_OP_NOP and using injection results
- Fix for an issue in IORING_OP_WAITID, where the info state was
assumed cleared by the lower level syscall handler, but for some
cases it is not. Just clear the data upfront, so that non-initialized
data isn't copied back to userspace
- Fix for a lockdep reported issue, where IORING_OP_BIND enters file
create and hence hits mnt_want_write(), which creates a three part
lockdep cycle between the super lock, io_uring's uring_lock, and the
cred mutex
- Fix a regression introduced in this cycle with how linked timeouts
are deleted
- Ensure that the ->opcode nospec indexing on the opcode issue side
covers all the cases
* tag 'io_uring-7.1-20260522' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/nop: pass all errors to userspace
io_uring/timeout: splice timed out link in timeout handler
io_uring: propagate array_index_nospec opcode into req->opcode
io_uring/waitid: clear waitid info before copying it to userspace
io_uring/net: punt IORING_OP_BIND async if it needs file create
This fixes an inconsistency where io_nop() called req_set_fail()
based on ret, but passed just nop->result to userspace.
Originally, ret is a even copy of nop->result, but is set to an error
when such happens subsequently. Now that's also passed to userspace.
Fixes: a85f31052b ("io_uring/nop: add support for testing registered files and buffers")
Signed-off-by: Alexander A. Klimov <grandmaster@al2klimov.de>
Link: https://patch.msgid.link/20260520180045.538533-1-grandmaster@al2klimov.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
A previous commit deferred this to the task_work part of it, so it could
be protected by ->uring_lock. But that's actually not necessary here,
and in fact the head clearing is not enough to make that safe. For those
two reasons, just re-instate the local splicing.
Fixes: 49ae66eb8c ("io_uring: defer linked-timeout chain splice out of hrtimer context")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Commit 1e988c3fe1 ("io_uring: prevent opcode speculation") added
array_index_nospec() to io_init_req(), but applied it only to a local
opcode variable. req->opcode is initialized from sqe->opcode before the
bounds check and remains the raw value.
Keep req->opcode as the canonical opcode in io_init_req(): reject
out-of-range values architecturally, then write the array_index_nospec()
result back to req->opcode before any table lookup. This keeps downstream
users of req->opcode from observing the raw user byte on a mispredicted
path.
No functional change: array_index_nospec() is a no-op for opcodes in
[0, IORING_OP_LAST), and out-of-range opcodes are still rejected at the
bounds check above the assignment.
Fixes: 1e988c3fe1 ("io_uring: prevent opcode speculation")
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Link: https://patch.msgid.link/20260517213010.696135-1-michael.bommarito@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
IORING_OP_WAITID stores its result fields in struct io_waitid::info and
later copies them to userspace siginfo. The prep path initializes the
request arguments, but it does not initialize info itself.
If the wait operation completes without reporting a child event, the common
wait code can return without writing wo_info. In that case io_waitid_finish()
still copies iw->info to userspace, exposing stale bytes from the reused
io_kiocb command storage.
Clear the result storage during prep so the io_uring path matches the
regular waitid syscall, which uses a zero-initialized struct waitid_info.
Fixes: f31ecf671d ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org # 6.7+
Signed-off-by: Heechan Kang <gganji11@naver.com>
Link: https://patch.msgid.link/20260516184709.852814-1-gganji11@naver.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmoHJXkQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgptflEADCMbEFFKJnBzlk20pd0xJ9YS5DM03GyWyF
oX2kY2tas51sxB0f/uRw6dPhYZZzYgwUGfhtAGN0661KBGlwd1akWgcS4HecgTAe
B8RQdoHUetIYYnXsHfInQixN5Tv/hVVabZlRoQyBGTRU+6fOWYvHBEB4mIceXjdy
dMm94vOiOzV9tTgp1mI+Ywb1pzZtNMjhjVb2xwtS7EFgy5Y9lk8IP0/FJLU+fB90
FSCQo19J8fuRzqjv37qjBqOscn8CBOq16GwlhnTOqR2pFKGAPhm8wfTegHlAKdpD
GDRvMoiLvfuR1/w82v7MpUemL07kvasr8hHpoHgeMWYsaritnNClNXmW5kxQARbG
Wd+mNFvu37lTxd/umBJC+Ci+NMAq95koK1EZfsG4PI9PzZeQkVl4GJ3oNXxg3e8Z
Z7ZrG0a5b0CW6qwEw7/KsICgrdr/uGVqDZIhynpjtAQr5D4lrEkqfPVJ1y/hpJYE
aPr1RrvJZ+lUqX9h/uDkOGfV4psWeEqU95tuGahJ29tHzfUkOmhBx/ruiqBWM7c8
qHctYdE6zq9DY3uimNTgqvnJlQWw0dEx4JfdY8k4Oea2qJxHFqzG3b8qkht3tly5
dJNlfutd/nvr16gHV3lrRfkoiSVIVW+9AX17Climlgmk9LOl7KwgUdQWmthQa69q
1h+O/GWAOA==
=9orn
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.1-20260515' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Small series sanitizing the locking done for either modifying or
reading a chain of requests
- If the application has a pid namespace, ensure that the sqthread pid
is correctly printed in fdinfo
- Fix for a hashing issue in the io-wq thread pool, which could lead to
a use-after-free
- Kill dead argument from io_prep_rw_pi()
- Fix for a missed validation of the CQ ring head, affecting CQE refill
* tag 'io_uring-7.1-20260515' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring: validate user-controlled cq.head in io_cqe_cache_refill()
io-wq: check that the predecessor is hashed in io_wq_remove_pending()
io_uring/rw: drop unused attr_type_mask from io_prep_rw_pi()
io_uring: hold uring_lock across io_kill_timeouts() in cancel path
io_uring: defer linked-timeout chain splice out of hrtimer context
io_uring: hold uring_lock when walking link chain in io_wq_free_work()
io_uring/fdinfo: translate SqThread PID through caller's pid_ns
For two reasons:
1) An opcode cannot block inside io_uring_enter() doing submissions, as
it'll stall the submission side pipeline.
2) Ending up in sb_start_write() -> __sb_start_write() ->
percpu_down_read_freezable() introduces a new lockdep edge, which it
correctly complains about.
Check if the socket type is AF_UNIX and has a non-empty pathname. If it
does, mark it REQ_F_FORCE_ASYNC to punt the submission to io-wq rather
than attempt to do it inline.
Fixes: 7481fd93fa ("io_uring: Introduce IORING_OP_BIND")
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
One of the nastier things about epoll is how it allows adding epoll
files to epoll contexts. This leads to all sorts of loop detection
code, and has been a source of issues in the past.
Arguably adding IORING_EPOLL_CTL is a historical mistake on the
io_uring side, but we're kind of stuck with it now as it does seem
to be in use according to code searches. But we can at least minimize
the damage a bit and just disallow this part of epoll, where nesting
issues can arise.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Replace trace_foo() with the new trace_call__foo() at sites already
guarded by trace_foo_enabled(), avoiding a redundant
static_branch_unlikely() re-evaluation inside the tracepoint.
trace_call__foo() calls the tracepoint callbacks directly without
utilizing the static branch again.
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Vineeth Pillai (Google) <vineeth@bitbyteword.org>
Assisted-by: Claude:claude-sonnet-4-6
Link: https://patch.msgid.link/20260515135903.2238731-1-vineeth@bitbyteword.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
There's no real reason to have a limit, as the memory is accounted by
the lockmem limits anyway, if any exist. io_pin_pages() will still
restrict the maximum allowed limit per buffer, which is INT_MAX
number of pages. Cap it a bit lower than that, at 1TB for a 64-bit
system. Surely that should be enough for everyone. For now.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
In preparation for supporting bigger individual buffers, bump the length
field to a full 8-bytes with size_t rather than an unsigned int.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Track huge page references in a per-ring xarray to prevent double
accounting when the same huge page is used by multiple registered
buffers, either within the same ring or across cloned rings.
When registering buffers backed by huge pages, we need to account for
RLIMIT_MEMLOCK. But if multiple buffers share the same huge page (common
with cloned buffers), we must not account for the same page multiple
times. Similarly, we must only unaccount when the last reference to a
huge page is released.
Maintain a per-ring xarray (hpage_acct) that tracks reference counts for
each huge page. When registering a buffer, for each unique huge page,
increment its accounting reference count, and only account pages that
are newly added.
When unregistering a buffer, for each unique huge page, decrement its
refcount. Once the refcount hits zero, the page is unaccounted.
Note: any account is done against the ctx->user that was assigned when
the ring was setup. As before, if root is running the operation, no
accounting is done.
With these changes, any use of imu->acct_pages is also dead, hence kill
it from struct io_mapped_ubuf. This shrinks it from 56b to 48b on a
64-bit arch. Additionally, hpage_already_acct() is gone, which was an
O(M*M) scan over current + previous registrations.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
This adds custom filtering for IORING_OP_CONNECT, where the target
family is always exposed, and (for AF_INET / AF_INET6) port and
address are exposed. port and v4_addr are in network byte order so
filter authors can compare against on-wire constants.
Skip population unless addr_len covers the populated fields, to
avoid leaking stale io_async_msghdr data on short connects.
Signed-off-by: Shouvik Kar <auxcorelabs@gmail.com>
Link: https://patch.msgid.link/20260512110242.26219-1-auxcorelabs@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
