github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-09-23 05:04:02 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

lib: bitmap: optimize bitmap_find_next_zero_area_off()

Browse Source 
Finding a contiguous free region in a highly fragmented
bitmap is not easy and may require many repeated attempts.
Therefore, find_next_bit(map, end, index) is not the optimal choice.
This is because there may be multiple scattered free regions
within the range [index, end) and none of them will meet the length
requirement of @nr.
Instead, it's sufficient to directly find the last bit within
the range [index, end), thus reducing unnecessary repeated calls.

An example of a bitmap:
Bits 0-3:   cleared(4 bits)
Bits 4-5:   set    (2 bits)
Bits 6-8:   cleared(4 bits)
Bits 9-10:  set    (2 bits)
Bits 11-20: cleared(10 bits)

The goal is to find a 10-bit free region.

The old code logic is as follows:
find_next_zero_bit(start = 0, find bit 0) -> find_next_bit(find bit 4) ->
next loop ->
find_next_zero_bit(start = 5, find bit 6) -> find_next_bit(find bit 9) ->
next loop ->
find_next_zero_bit(start = 10, find bit 11) -> success

The new code logic is as follows:
find_next_zero_bit(start = 0, find bit 0) -> find_last_bit(find bit 9) ->
next loop ->
find_next_zero_bit(start = 10, find bit 11) -> success

Performance test results on my hardware(use lib/find_bit_benchmark.c):

		before	after	change	p-value
dense		1211	688	-43.2%	8.3e-11
sparse		13.3	13.4	0.8%	0.27

Yury:

The less micro-benchmark kselftest/dmabuf-heaps/dmabuf-heap gives
even better numbers:

Metric                      Before         After          Change
Trace span                194.0 ms       87.1 ms          -55.1%
Total CMA alloc time      48.46 ms      16.11 ms          -66.8%
Avg alloc latency        184.94 us      61.49 us          -66.8%
Median alloc latency      73.72 us      20.59 us          -72.1%
p90 alloc latency        329.76 us      55.63 us          -83.1%
p99 alloc latency       1866.76 us     859.83 us          -53.9%
Max alloc latency       4821.91 us    2324.41 us          -51.8%

By request size:

Request      Before Avg    After Avg          Change
1 page         79.68 us     34.47 us          -56.7%
256 pages     285.50 us     87.30 us          -69.4%

Co-developed-by: Yury Norov <yury.norov@gmail.com>
Signed-off-by: Yury Norov <yury.norov@gmail.com>
Signed-off-by: sunyi <279644543@qq.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
This commit is contained in:
sunyi 2026-07-06 17:29:34 +08:00 committed by Yury Norov
parent 199dc60543
commit df81d444dc
1 changed files with 19 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
lib/bitmap.c
View File

@ -432,22 +432,28 @@ unsigned long bitmap_find_next_zero_area_off(unsigned long *map,
unsigned long align_mask,
unsigned long align_offset)
{
unsigned long index, end, i;
again:
index = find_next_zero_bit(map, size, start);
unsigned long end, i, off;
/* Align allocation */
index = __ALIGN_MASK(index + align_offset, align_mask) - align_offset;
for_each_clear_bit_from(start, map, size) {
start = __ALIGN_MASK(start + align_offset, align_mask) - align_offset;
end = start + nr;
if (end > size)
break;
end = index + nr;
if (end > size)
return end;
i = find_next_bit(map, end, index);
if (i < end) {
start = i + 1;
goto again;
off = round_down(start, BITS_PER_LONG);
i = find_last_bit(map + start / BITS_PER_LONG, end - off) + off;
if (i >= end || i < start)
return start;
start = i;
}
return index;
/*
* Here, returning size + 1 is to maintain consistency
* with the old version, where the return value is always
* greater than size when no zero areas are found.
*/
return size + 1;
}
EXPORT_SYMBOL(bitmap_find_next_zero_area_off);