Marco Elver d5dd3b4448 kfence: default to dynamic branch instead of static keys mode ... commit 4f612ed3f7 upstream. We have observed that on very large machines with newer CPUs, the static key/branch switching delay is on the order of milliseconds. This is due to the required broadcast IPIs, which simply does not scale well to hundreds of CPUs (cores). If done too frequently, this can adversely affect tail latencies of various workloads. One workaround is to increase the sample interval to several seconds, while decreasing sampled allocation coverage, but the problem still exists and could still increase tail latencies. As already noted in the Kconfig help text, there are trade-offs: at lower sample intervals the dynamic branch results in better performance; however, at very large sample intervals, the static keys mode can result in better performance -- careful benchmarking is recommended. Our initial benchmarking showed that with large enough sample intervals and workloads stressing the allocator, the static keys mode was slightly better. Evaluating and observing the possible system-wide side-effects of the static-key-switching induced broadcast IPIs, however, was a blind spot (in particular on large machines with 100s of cores). Therefore, a major downside of the static keys mode is, unfortunately, that it is hard to predict performance on new system architectures and topologies, but also making conclusions about performance of new workloads based on a limited set of benchmarks. Most distributions will simply select the defaults, while targeting a large variety of different workloads and system architectures. As such, the better default is CONFIG_KFENCE_STATIC_KEYS=n, and re-enabling it is only recommended after careful evaluation. For reference, on x86-64 the condition in kfence_alloc() generates exactly 2 instructions in the kmem_cache_alloc() fast-path: | ... | cmpl $0x0,0x1a8021c(%rip) # ffffffff82d560d0 <kfence_allocation_gate> | je ffffffff812d6003 <kmem_cache_alloc+0x243> | ... which, given kfence_allocation_gate is infrequently modified, should be well predicted by most CPUs. Link: https://lkml.kernel.org/r/20211019102524.2807208-2-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>		2021-11-12 15:05:49 +01:00
..
842
crypto	crypto: arm64/sm4-ce - Make dependent on sm4 library instead of sm4-generic	2021-07-30 10:58:30 +08:00
dim
fonts	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
kunit	kunit: fix reference count leak in kfree_at_end	2021-10-01 13:49:38 -06:00
livepatch
lz4	lz4_decompress: declare LZ4_decompress_safe_withPrefix64k static	2021-07-01 11:06:05 -07:00
lzo
math	math: make RATIONAL tristate	2021-09-08 11:50:26 -07:00
mpi	lib/mpi: use kcalloc in mpi_resize	2021-08-12 19:17:00 +08:00
pldmfw	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
raid6
reed_solomon	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
vdso
xz	lib/decompressors: fix spelling mistakes	2021-07-01 11:06:05 -07:00
zlib_deflate
zlib_dfltcc
zlib_inflate	lib/zlib_inflate/inffast: check config in C to avoid unused function warning	2021-09-24 16:13:35 -07:00
zstd	lib/decompressors: fix spelling mistakes	2021-07-01 11:06:05 -07:00
.gitignore	.gitignore: prefix local generated files with a slash	2021-05-02 00:43:35 +09:00
argv_split.c
ashldi3.c
ashrdi3.c
asn1_decoder.c
asn1_encoder.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
assoc_array.c
atomic64_test.c
atomic64.c	locking/atomic: atomic64: support ARCH_ATOMIC	2021-05-26 13:20:50 +02:00
audit.c
bcd.c
bch.c	lib/bch.c: fix a typo in the file bch.c	2021-05-06 19:24:12 -07:00
bitfield_kunit.c
bitmap.c	bitmap: extend comment to bitmap_print_bitmask/list_to_buf	2021-08-13 10:27:50 +02:00
bitrev.c
bootconfig.c	memblock: introduce saner 'memblock_free_ptr()' interface	2021-09-14 13:23:22 -07:00
bsearch.c
btree.c
bucket_locks.c
bug.c
build_OID_registry
buildid.c	kdump: use vmlinux_build_id to simplify	2021-07-08 11:48:22 -07:00
bust_spinlocks.c
check_signature.c
checksum.c
clz_ctz.c
clz_tab.c
cmdline_kunit.c	lib/cmdline_kunit: Remove a cast which are no-longer required	2021-06-23 16:41:41 -06:00
cmdline.c	lib/cmdline: Export next_arg() for being used in modules	2021-05-05 16:07:40 +02:00
cmpdi2.c
compat_audit.c
cpu_rmap.c
cpumask.c
crc-ccitt.c
crc-itu-t.c
crc-t10dif.c
crc4.c
crc7.c
crc8.c	lib: crc8: pointer to data block should be const	2021-05-06 19:24:12 -07:00
crc16.c
crc32.c
crc32defs.h
crc32test.c
crc64.c	lib: crc64: fix kernel-doc warning	2021-06-05 08:58:12 -07:00
ctype.c
debug_info.c	isystem: ship and use stdarg.h	2021-08-19 09:02:55 +09:00
debug_locks.c	locking/lockdep: Improve noinstr vs errors	2021-06-22 13:56:43 +02:00
debugobjects.c	debugobjects: Make them PREEMPT_RT aware	2021-08-13 10:07:44 +02:00
dec_and_lock.c
decompress_bunzip2.c	lib/decompressors: fix spelling mistakes	2021-07-01 11:06:05 -07:00
decompress_inflate.c
decompress_unlz4.c	lib/decompress_unlz4.c: correctly handle zero-padding around initrds.	2021-07-01 11:06:06 -07:00
decompress_unlzma.c	lib: fix inconsistent indenting in process_bit1()	2021-05-06 19:24:12 -07:00
decompress_unlzo.c	lib/decompressors: remove set but not used variabled 'level'	2021-07-01 11:06:06 -07:00
decompress_unxz.c	lib/decompressors: fix spelling mistakes	2021-07-01 11:06:05 -07:00
decompress_unzstd.c	lib/decompressors: fix spelling mistakes	2021-07-01 11:06:05 -07:00
decompress.c
devmem_is_allowed.c	lib: use PFN_PHYS() in devmem_is_allowed()	2021-08-13 14:09:32 -10:00
devres.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
digsig.c
dump_stack.c	lib/dump_stack: correct kernel-doc notation	2021-09-08 11:50:26 -07:00
dynamic_debug.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
dynamic_queue_limits.c
earlycpio.c
errname.c
error-inject.c
errseq.c
extable.c
fault-inject-usercopy.c
fault-inject.c
fdt_addresses.c
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
fdt.c
find_bit_benchmark.c
find_bit.c	lib: add fast path for find_first_*_bit() and find_last_bit()	2021-05-06 19:24:12 -07:00
flex_proportions.c
gen_crc32table.c
gen_crc64table.c
genalloc.c	lib/genalloc: add parameter description to fix doc compile warning	2021-05-06 19:24:12 -07:00
generic-radix-tree.c
glob.c
globtest.c
hexdump.c
hweight.c
idr.c
inflate.c
interval_tree_test.c
interval_tree.c
iomap_copy.c
iomap.c
iommu-helper.c
iov_iter.c	iov_iter: add helper to save iov_iter state	2021-09-14 08:12:18 -06:00
irq_poll.c
irq_regs.c
is_single_threaded.c
kasprintf.c	isystem: ship and use stdarg.h	2021-08-19 09:02:55 +09:00
Kconfig	asm-generic changes for 5.15	2021-09-01 15:13:02 -07:00
Kconfig.debug	xtensa: increase size of gcc stack frame check	2021-09-24 16:13:34 -07:00
Kconfig.kasan	kasan: fix Kconfig check of CC_HAS_WORKING_NOSANITIZE_ADDRESS	2021-09-24 16:13:34 -07:00
Kconfig.kcsan	kcsan: Make strict mode imply interruptible watchers	2021-07-20 13:49:44 -07:00
Kconfig.kfence	kfence: default to dynamic branch instead of static keys mode	2021-11-12 15:05:49 +01:00
Kconfig.kgdb
Kconfig.ubsan
kfifo.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
klist.c
kobject_uevent.c
kobject.c
kstrtox.c	Merge branch 'akpm' (patches from Andrew)	2021-07-02 12:08:10 -07:00
kstrtox.h	lib: vsprintf: Fix handling of number field widths in vsscanf	2021-05-19 15:05:11 +02:00
libcrc32c.c
linear_ranges.c	lib: add linear range get selector within	2021-08-13 18:37:38 +02:00
list_debug.c
list_sort.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
list-test.c
llist.c
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-rtmutex.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c	lockdep/selftest: Remove wait-type RCU_CALLBACK tests	2021-06-22 16:42:08 +02:00
lockref.c
logic_iomem.c	lib/logic_iomem: fix sparse warnings	2021-08-26 22:28:02 +02:00
logic_pio.c
lru_cache.c
lshrdi3.c
Makefile	bitfield: build kunit tests without structleak plugin	2021-10-06 17:53:54 -06:00
memcat_p.c
memory-notifier-error-inject.c
memregion.c
memweight.c
muldi3.c
net_utils.c
netdev-notifier-error-inject.c
nlattr.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
nmi_backtrace.c	lib/nmi_backtrace: Serialize even messages about idle CPUs	2021-07-30 09:25:34 +02:00
nodemask.c
notifier-error-inject.c
notifier-error-inject.h
objagg.c
of-reconfig-notifier-error-inject.c
oid_registry.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
once.c	once: Fix panic when module unload	2021-08-08 13:00:20 +01:00
packing.c	net: update NXP copyright text	2021-09-17 13:52:17 +01:00
parman.c
parser.c	kernel.h: split out kstrtox() and simple_strtox() to a separate header	2021-07-01 11:06:05 -07:00
pci_iomap.c	pci_iounmap'2: Electric Boogaloo: try to make sense of it all	2021-09-19 17:13:35 -07:00
percpu_counter.c	lib/percpu_counter: tame kernel-doc compile warning	2021-05-06 19:24:12 -07:00
percpu_test.c
percpu-refcount.c	percpu_ref: Don't opencode percpu_ref_is_dying	2021-05-13 03:27:38 +00:00
plist.c
pm-notifier-error-inject.c
radix-tree.c
random32.c
ratelimit.c
rbtree_test.c
rbtree.c
refcount.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
rhashtable.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
sbitmap.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
scatterlist.c	Merge branch 'akpm' (patches from Andrew)	2021-09-03 10:08:28 -07:00
seq_buf.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
sg_pool.c	lib/scatterlist: Fix wrong update of orig_nents	2021-08-24 19:52:40 -03:00
sg_split.c
sha1.c
show_mem.c
siphash.c
slub_kunit.c	mm/slub, kunit: add a KUnit test for SLUB debugging functionality	2021-06-29 10:53:46 -07:00
smp_processor_id.c	lib/smp_processor_id: Use is_percpu_thread() instead of nr_cpus_allowed	2021-05-19 10:51:40 +02:00
sort.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
stackdepot.c	lib: fix spelling mistakes	2021-07-08 11:48:20 -07:00
stmp_device.c
string_helpers.c	string_helpers: Escape double quotes in escape_special	2021-07-19 11:39:28 +02:00
string.c	string: improve default out-of-line memcmp() implementation	2021-08-30 07:50:56 -07:00
strncpy_from_user.c
strnlen_user.c
syscall.c	sched: Change task_struct::state	2021-06-18 11:43:09 +02:00
test_bitmap.c	lib: test_bitmap: add bitmap_print_bitmask/list_to_buf test cases	2021-08-13 10:27:49 +02:00
test_bitops.c	lib/test: fix spelling mistakes	2021-07-08 11:48:20 -07:00
test_bits.c
test_blackhole_dev.c
test_bpf.c	bpf: Refactor BPF_PROG_RUN into a function	2021-08-17 00:45:07 +02:00
test_debug_virtual.c
test_firmware.c	firmware: replace HOTPLUG with UEVENT in FW_ACTION defines	2021-05-13 16:14:45 +02:00
test_fpu.c
test_free_pages.c
test_hash.c
test_hexdump.c
test_hmm_uapi.h	mm: selftests for exclusive device memory	2021-07-01 11:06:03 -07:00
test_hmm.c	lib/test_hmm: remove set but unused page variable	2021-07-15 10:13:49 -07:00
test_ida.c
test_kasan_module.c	kasan: test: avoid corrupting memory in kasan_rcu_uaf	2021-09-03 09:58:15 -07:00
test_kasan.c	Merge branch 'akpm' (patches from Andrew)	2021-09-03 10:08:28 -07:00
test_kmod.c	lib/test: fix spelling mistakes	2021-07-08 11:48:20 -07:00
test_linear_ranges.c
test_list_sort.c	lib/test: convert lib/test_list_sort.c to use KUnit	2021-06-25 11:31:03 -06:00
test_lockup.c	lib/test_lockup: Adapt to changed variables	2021-08-17 19:05:33 +02:00
test_memcat_p.c
test_meminit.c
test_min_heap.c
test_module.c
test_objagg.c
test_overflow.c
test_parman.c
test_printf.c	Merge branch 'akpm' (patches from Andrew)	2021-09-08 12:55:35 -07:00
test_rhashtable.c
test_scanf.c	lib/test_scanf: split up number parsing test routines	2021-09-06 11:04:03 -07:00
test_siphash.c
test_sort.c	lib/test: convert test_sort.c to use KUnit	2021-09-08 11:50:26 -07:00
test_stackinit.c	lib/test_stackinit: Add assigned initializers	2021-08-22 00:21:36 -07:00
test_static_key_base.c
test_static_keys.c
test_string.c	lib/test_string.c: allow module removal	2021-07-01 11:06:05 -07:00
test_strscpy.c
test_sysctl.c
test_ubsan.c
test_user_copy.c
test_uuid.c
test_vmalloc.c	lib/test_vmalloc.c: add a new 'nr_pages' parameter	2021-09-03 09:58:14 -07:00
test_xarray.c
test-kstrtox.c
test-string_helpers.c	string_helpers: Escape double quotes in escape_special	2021-07-19 11:39:28 +02:00
textsearch.c
timerqueue.c
ts_bm.c
ts_fsm.c
ts_kmp.c
ubsan.c	kunit: ubsan integration	2021-08-13 13:19:06 -06:00
ubsan.h
ucmpdi2.c
ucs2_string.c
usercopy.c
uuid.c
vsprintf.c	Merge branch 'akpm' (patches from Andrew)	2021-09-08 12:55:35 -07:00
win_minmax.c
xarray.c
xxhash.c