linux/fs/squashfs
Usama Arif f6f47a9ca8 squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy.  Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable.  A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning.  One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.

On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry.  This was causing significant spikes
in CPU usage.

Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes.  Two things follow.

A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed.  So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.

And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice.  So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one.  That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.

Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key.  Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.

Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:

  elbencho, 64 threads
    metadata stat        700 ->  1320 files/s    1.9x
    small-file read       40 ->    60 MiB/s      1.5x

  filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
    throughput        11,314 -> 25,186 ops/s     2.2x
    sched:sched_wakeup  27.0 ->   4.55 per op    5.9x fewer
    context switches    37.2 ->   7.64 per op    4.9x fewer

Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work.  Workloads which never queue for a cache
entry gain no wakeups.

Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-19 19:55:06 -07:00
..
block.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
cache.c squashfs: avoid thundering-herd cache wakeups 2026-08-19 19:55:06 -07:00
decompressor_multi_percpu.c squashfs: fix percpu address space issues in decompressor_multi_percpu.c 2024-09-09 16:47:41 -07:00
decompressor_multi.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
decompressor_single.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
decompressor.c squashfs: don't include buffer_head.h 2023-06-09 17:44:14 -07:00
decompressor.h squashfs: extend "page actor" to handle missing pages 2022-06-16 19:58:21 -07:00
dir.c squashfs: add setlease file operation 2026-01-12 10:55:47 +01:00
export.c exportfs: make ->encode_fh() a mandatory method for NFS export 2023-10-28 16:15:15 +02:00
file_cache.c squashfs; convert squashfs_copy_cache() to take a folio 2025-01-24 22:47:22 -08:00
file_direct.c squashfs: convert squashfs_readpage_block() to take a folio 2025-01-24 22:47:22 -08:00
file.c Convert remaining multi-line kmalloc_obj/flex GFP_KERNEL uses 2026-02-22 08:26:33 -08:00
fragment.c
id.c squashfs: fix xattr id and id lookup sanity checks 2021-03-25 09:22:55 -07:00
inode.c Coccinelle-based conversion to use ->i_state accessors 2025-10-20 20:22:26 +02:00
Kconfig squashfs: add optional full compressed block caching 2025-05-27 19:40:33 -07:00
lz4_wrapper.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
lzo_wrapper.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
Makefile squashfs: always build "file direct" version of page actor 2022-07-29 18:12:34 -07:00
namei.c Squashfs: remove deprecated strncpy by not copying the string 2024-04-25 21:07:05 -07:00
page_actor.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
page_actor.h Squashfs: Ensure all readahead pages have been used 2024-08-23 13:11:36 +02:00
squashfs_fs_i.h Squashfs: fix uninit-value in squashfs_get_parent 2025-09-28 11:36:12 -07:00
squashfs_fs_sb.h squashfs: avoid thundering-herd cache wakeups 2026-08-19 19:55:06 -07:00
squashfs_fs.h Squashfs: add SEEK_DATA/SEEK_HOLE support 2025-09-28 11:36:14 -07:00
squashfs.h Squashfs: add SEEK_DATA/SEEK_HOLE support 2025-09-28 11:36:14 -07:00
super.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
symlink.c squashfs: convert squashfs_symlink_read_folio to use folio APIs 2024-05-08 08:41:28 -07:00
xattr_id.c revert "squashfs: harden sanity check in squashfs_read_xattr_id_table" 2023-02-03 17:52:25 -08:00
xattr.c squashfs: move squashfs_xattr_handlers to .rodata 2023-10-10 13:49:20 +02:00
xattr.h Squashfs: fix handling and sanity checking of xattr_ids count 2023-01-31 16:44:10 -08:00
xz_wrapper.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
zlib_wrapper.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
zstd_wrapper.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00