From df81d444dc740778f7c7a2d4c3500136851375aa Mon Sep 17 00:00:00 2001 From: sunyi <279644543@qq.com> Date: Mon, 6 Jul 2026 17:29:34 +0800 Subject: [PATCH] lib: bitmap: optimize bitmap_find_next_zero_area_off() Finding a contiguous free region in a highly fragmented bitmap is not easy and may require many repeated attempts. Therefore, find_next_bit(map, end, index) is not the optimal choice. This is because there may be multiple scattered free regions within the range [index, end) and none of them will meet the length requirement of @nr. Instead, it's sufficient to directly find the last bit within the range [index, end), thus reducing unnecessary repeated calls. An example of a bitmap: Bits 0-3: cleared(4 bits) Bits 4-5: set (2 bits) Bits 6-8: cleared(4 bits) Bits 9-10: set (2 bits) Bits 11-20: cleared(10 bits) The goal is to find a 10-bit free region. The old code logic is as follows: find_next_zero_bit(start = 0, find bit 0) -> find_next_bit(find bit 4) -> next loop -> find_next_zero_bit(start = 5, find bit 6) -> find_next_bit(find bit 9) -> next loop -> find_next_zero_bit(start = 10, find bit 11) -> success The new code logic is as follows: find_next_zero_bit(start = 0, find bit 0) -> find_last_bit(find bit 9) -> next loop -> find_next_zero_bit(start = 10, find bit 11) -> success Performance test results on my hardware(use lib/find_bit_benchmark.c): before after change p-value dense 1211 688 -43.2% 8.3e-11 sparse 13.3 13.4 0.8% 0.27 Yury: The less micro-benchmark kselftest/dmabuf-heaps/dmabuf-heap gives even better numbers: Metric Before After Change Trace span 194.0 ms 87.1 ms -55.1% Total CMA alloc time 48.46 ms 16.11 ms -66.8% Avg alloc latency 184.94 us 61.49 us -66.8% Median alloc latency 73.72 us 20.59 us -72.1% p90 alloc latency 329.76 us 55.63 us -83.1% p99 alloc latency 1866.76 us 859.83 us -53.9% Max alloc latency 4821.91 us 2324.41 us -51.8% By request size: Request Before Avg After Avg Change 1 page 79.68 us 34.47 us -56.7% 256 pages 285.50 us 87.30 us -69.4% Co-developed-by: Yury Norov Signed-off-by: Yury Norov Signed-off-by: sunyi <279644543@qq.com> Signed-off-by: Yury Norov --- lib/bitmap.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/lib/bitmap.c b/lib/bitmap.c index b9bfa157e095..b464d843f4eb 100644 --- a/lib/bitmap.c +++ b/lib/bitmap.c @@ -432,22 +432,28 @@ unsigned long bitmap_find_next_zero_area_off(unsigned long *map, unsigned long align_mask, unsigned long align_offset) { - unsigned long index, end, i; -again: - index = find_next_zero_bit(map, size, start); + unsigned long end, i, off; - /* Align allocation */ - index = __ALIGN_MASK(index + align_offset, align_mask) - align_offset; + for_each_clear_bit_from(start, map, size) { + start = __ALIGN_MASK(start + align_offset, align_mask) - align_offset; + end = start + nr; + if (end > size) + break; - end = index + nr; - if (end > size) - return end; - i = find_next_bit(map, end, index); - if (i < end) { - start = i + 1; - goto again; + off = round_down(start, BITS_PER_LONG); + i = find_last_bit(map + start / BITS_PER_LONG, end - off) + off; + if (i >= end || i < start) + return start; + + start = i; } - return index; + + /* + * Here, returning size + 1 is to maintain consistency + * with the old version, where the return value is always + * greater than size when no zero areas are found. + */ + return size + 1; } EXPORT_SYMBOL(bitmap_find_next_zero_area_off);