lib/plist: fix plist_requeue() corrupting order in the last bucket

plist_requeue() is meant to move a node to the end of its own priority
run.  When the node heads the *last* priority bucket it is instead placed
at the head of the whole list, leaving the plist unsorted:

  built:      A(prio 0) B(prio 1) C(prio 1)
  requeue(B): B(prio 1) A(prio 0) C(prio 1)
  expected:   A(prio 0) C(prio 1) B(prio 1)

prio_list is a *headless* circular ring of the nodes that lead each
priority bucket.  The shortcut added by commit 95d4b3450e ("lib/plist.c:
add shortcut for plist_requeue()") takes

	iter = list_entry(iter->prio_list.next, struct plist_node,
			  prio_list);
	node_next = &iter->node_list;

which from the last bucket wraps round to the *first* bucket, so node_next
ends up pointing at the head of the list rather than at its end.  The
plist_for_each_continue() loop immediately below it computes the correct
answer (&head->node_list) for that case.

With any bucket after it the shortcut is correct, which is why this went
unnoticed: the benchmark in that commit measured elapsed time and never
checked the resulting order.

Keep the shortcut -- it is a real win -- but exclude the case where iter's
bucket is the last one, which is exactly when its ring successor is the
first bucket again.

Reachable from mm/swapfile.c, which rotates swap_avail_heads[] with
plist_requeue().  It takes three or more swap devices: at least two
distinct priorities, so that a later bucket exists for the ring to wrap
round from, and two or more devices sharing the lowest priority, so that
plist_requeue() does not return early.  One device per priority returns
early at the node->prio != iter->prio test.  A single priority is also
safe, but for a different reason worth stating: with one bucket no node is
ever linked onto prio_list at all -- plist_add() skips it for the first
node and for every node whose predecessor shares its priority -- so
list_empty(&iter->prio_list) holds and the shortcut is never entered.

Tested by driving three implementations -- the pre-95d4b3450ebe code,
current mainline, and this patch -- through 1,084,492 identical random
add/del/requeue operations over 24 nodes and 1..5 distinct priorities,
comparing the resulting node_list node for node after every operation:

  variant             differs from pre-95d4b3450ebe  left list unsorted
  pre-95d4b3450ebe    -- (reference)                                  0
  mainline                                  289,297             276,660
  this patch                                      0                   0

Link: https://lore.kernel.org/20260903222456.1881786-1-handyhandyman.adam@gmail.com
Fixes: 95d4b3450e ("lib/plist.c: add shortcut for plist_requeue()")
Signed-off-by: Adam Harshbarger <handyhandyman.adam@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Assisted-by: Claude:claude-opus-5
Cc: I Hsin Cheng <richard120310@gmail.com>
Cc: <stable@vger.kernel.org> # v6.15+
This commit is contained in:
Adam Harshbarger 2026-09-03 17:24:56 -05:00 committed by Andrew Morton
parent 6a3ed6b290
commit 43c91b1522

View File

@ -174,8 +174,15 @@ void plist_requeue(struct plist_node *node, struct plist_head *head)
/*
* After plist_del(), iter is the replacement of the node. If the node
* was on prio_list, take shortcut to find node_next instead of looping.
*
* prio_list is a headless ring, so from the LAST bucket ->next wraps
* round to the first one; in that case node_next is the list head.
*/
if (!list_empty(&iter->prio_list)) {
struct plist_node *first = plist_first(head);
if (iter->prio_list.next == &first->prio_list)
goto queue;
iter = list_entry(iter->prio_list.next, struct plist_node,
prio_list);
node_next = &iter->node_list;