mirror of
https://github.com/torvalds/linux.git
synced 2026-06-10 15:42:19 +02:00
7479f23c2a
do_blockdev_direct_IO() increments and decrements the inode ->i_dio_count for each IO operation. It does this to protect against truncate of a file. Block devices don't need this sort of protection. For a capable multiqueue setup, this atomic int is the only shared state between applications accessing the device for O_DIRECT, and it presents a scaling wall for that. In my testing, as much as 30% of system time is spent incrementing and decrementing this value. A mixed read/write workload improved from ~2.5M IOPS to ~9.6M IOPS, with better latencies too. Before: clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 80], | 99.00th=[ 98], 99.50th=[ 151], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 165] After: clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 129], 20.00th=[ 149], | 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 171], | 70.00th=[ 177], 80.00th=[ 185], 90.00th=[ 201], 95.00th=[ 270], | 99.00th=[ 390], 99.50th=[ 398], 99.90th=[ 418], 99.95th=[ 422], | 99.99th=[ 438] In other setups, Robert Elliott reported seeing good performance improvements: https://lkml.org/lkml/2015/4/3/557 The more applications accessing the device, the worse it gets. Add a new direct-io flags, DIO_SKIP_DIO_COUNT, which tells do_blockdev_direct_IO() that it need not worry about incrementing or decrementing the inode i_dio_count for this caller. Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Christoph Hellwig <hch@lst.de> Cc: Theodore Ts'o <tytso@mit.edu> Cc: Elliott, Robert (Server Storage) <elliott@hp.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Jens Axboe <axboe@fb.com> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com> Tested-and-Reviewed-by: Shawn Lin <shawn.lin@rock-chips.com> |
||
|---|---|---|
| .. | ||
| acl.c | ||
| acl.h | ||
| balloc.c | ||
| bitmap.c | ||
| block_validity.c | ||
| dir.c | ||
| ext4_extents.h | ||
| ext4_jbd2.c | ||
| ext4_jbd2.h | ||
| ext4.h | ||
| extents_status.c | ||
| extents_status.h | ||
| extents.c | ||
| file.c | ||
| fsync.c | ||
| hash.c | ||
| ialloc.c | ||
| indirect.c | ||
| inline.c | ||
| inode.c | ||
| ioctl.c | ||
| Kconfig | ||
| Makefile | ||
| mballoc.c | ||
| mballoc.h | ||
| migrate.c | ||
| mmp.c | ||
| move_extent.c | ||
| namei.c | ||
| page-io.c | ||
| resize.c | ||
| super.c | ||
| symlink.c | ||
| truncate.h | ||
| xattr_security.c | ||
| xattr_trusted.c | ||
| xattr_user.c | ||
| xattr.c | ||
| xattr.h | ||