Marco Elver b9bb7f3153 FROMGIT: kfence: await for allocation using wait_event ... Patch series "kfence: optimize timer scheduling", v2. We have observed that mostly-idle systems with KFENCE enabled wake up otherwise idle CPUs, preventing such to enter a lower power state. Debugging revealed that KFENCE spends too much active time in toggle_allocation_gate(). While the first version of KFENCE was using all the right bits to be scheduling optimal, and thus power efficient, by simply using wait_event() + wake_up(), that code was unfortunately removed. As KFENCE was exposed to various different configs and tests, the scheduling optimal code slowly disappeared. First because of hung task warnings, and finally because of deadlocks when an allocation is made by timer code with debug objects enabled. Clearly, the "fixes" were not too friendly for devices that want to be power efficient. Therefore, let's try a little harder to fix the hung task and deadlock problems that we have with wait_event() + wake_up(), while remaining as scheduling friendly and power efficient as possible. Crucially, we need to defer the wake_up() to an irq_work, avoiding any potential for deadlock. The result with this series is that on the devices where we observed a power regression, power usage returns back to baseline levels. This patch (of 3): On mostly-idle systems, we have observed that toggle_allocation_gate() is a cause of frequent wake-ups, preventing an otherwise idle CPU to go into a lower power state. A late change in KFENCE's development, due to a potential deadlock [1], required changing the scheduling-friendly wait_event_timeout() and wake_up() to an open-coded wait-loop using schedule_timeout(). [1] https://lkml.kernel.org/r/000000000000c0645805b7f982e4@google.com To avoid unnecessary wake-ups, switch to using wait_event_timeout(). Unfortunately, we still cannot use a version with direct wake_up() in __kfence_alloc() due to the same potential for deadlock as in [1]. Instead, add a level of indirection via an irq_work that is scheduled if we determine that the kfence_timer requires a wake_up(). Link: https://lkml.kernel.org/r/20210421105132.3965998-1-elver@google.com Link: https://lkml.kernel.org/r/20210421105132.3965998-2-elver@google.com Fixes: 0ce20dd840 ("mm: add Kernel Electric-Fence infrastructure") Signed-off-by: Marco Elver <elver@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jann Horn <jannh@google.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Hillf Danton <hdanton@sina.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Bug: 172317151 Bug: 185280916 Test: power team confirmed there's no regression (cherry picked from commit 0ac66fbbadde5547db190e9d87ff2e77d245f9a2 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git akpm) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Idba39683f0b76eb3f70df1113e43d94845fab5bf		2021-04-29 08:13:58 +02:00
..
842	kbuild: trace functions in subdirectories of lib/	2020-08-10 01:32:59 +09:00
crypto	UPSTREAM: crypto: blake2s - move update and final logic to internal/blake2s.h	2021-02-23 08:06:19 +01:00
dim	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
fonts	Fonts: Replace discarded const qualifier	2020-11-03 10:51:34 +01:00
kunit	linux-kselftest-kunit-5.10-rc1	2020-10-18 14:45:59 -07:00
livepatch	Kbuild updates for v5.9	2020-08-09 14:10:26 -07:00
lz4	lz4: fix kernel decompression speed	2020-08-14 19:56:56 -07:00
lzo	kbuild: trace functions in subdirectories of lib/	2020-08-10 01:32:59 +09:00
math	math: Export mul_u64_u64_div_u64	2021-04-10 13:36:10 +02:00
mpi	lib/mpi/mpi-bit.c: fix spello of "functions"	2020-10-16 11:11:20 -07:00
pldmfw	Add pldmfw library for PLDM firmware update	2020-07-28 17:07:06 -07:00
raid6	lib/raid6: Let $(UNROLL) rules work with macOS userland	2021-01-19 18:27:25 +01:00
reed_solomon	kbuild: trace functions in subdirectories of lib/	2020-08-10 01:32:59 +09:00
vdso	vdso/treewide: Add vdso_data pointer argument to __arch_get_hw_counter()	2020-08-06 10:57:30 +02:00
xz	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
zlib_deflate	kbuild: trace functions in subdirectories of lib/	2020-08-10 01:32:59 +09:00
zlib_dfltcc	zlib: move EXPORT_SYMBOL() and MODULE_LICENSE() out of dfltcc_syms.c	2021-01-06 14:56:51 +01:00
zlib_inflate	kbuild: trace functions in subdirectories of lib/	2020-08-10 01:32:59 +09:00
zstd	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
.gitignore	.gitignore: add SPDX License Identifier	2020-03-25 11:50:48 +01:00
argv_split.c
ashldi3.c
ashrdi3.c
asn1_decoder.c	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
assoc_array.c	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
atomic64_test.c
atomic64.c
audit.c
bcd.c
bch.c	lib/bch: Allow easy bit swapping	2020-05-24 20:48:11 +02:00
bitfield_kunit.c	lib: kunit: Fix compilation test when using TEST_BIT_FIELD_COMPILE	2020-10-16 13:25:14 -06:00
bitmap.c	lib: bitmap: delete duplicated words	2020-10-16 11:11:19 -07:00
bitrev.c
bootconfig.c	lib/bootconfig: Fix to remove tailing spaces after value	2020-09-21 21:50:09 -04:00
bsearch.c	lib/bsearch: Provide __always_inline variant	2020-06-11 15:14:53 +02:00
btree.c
bucket_locks.c
bug.c	ANDROID: bug: add vendor hook for bug trap	2021-01-21 12:04:44 +00:00
build_OID_registry
bust_spinlocks.c
check_signature.c
checksum.c	unify generic instances of csum_partial_copy_nocheck()	2020-08-20 15:45:14 -04:00
clz_ctz.c
clz_tab.c
cmdline.c	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
cmpdi2.c
compat_audit.c
cpu_rmap.c
cpumask.c	Revert "lib: Restrict cpumask_local_spread to houskeeping CPUs"	2021-02-17 11:02:24 +01:00
crc-ccitt.c
crc-itu-t.c
crc-t10dif.c	crc-t10dif: clean up some more things	2020-06-18 17:26:43 +10:00
crc4.c
crc7.c
crc8.c
crc16.c
crc32.c	lib/crc32.c: fix trivial typo in preprocessor condition	2020-10-16 11:11:20 -07:00
crc32defs.h
crc32test.c	lib/crc32test: remove extra local_irq_disable/enable	2020-11-02 12:14:19 -08:00
crc64.c	lib/: replace HTTP links with HTTPS ones	2020-08-12 10:58:00 -07:00
ctype.c
debug_info.c
debug_locks.c	lockdep: Prepare for noinstr sections	2020-05-19 15:47:21 +02:00
debugobjects.c	debugobjects: Free per CPU pool after CPU unplug	2020-10-01 16:13:54 +02:00
dec_and_lock.c
decompress_bunzip2.c	lib: decompress_bunzip2: delete duplicated words	2020-10-16 11:11:19 -07:00
decompress_inflate.c
decompress_unlz4.c
decompress_unlzma.c	lib/: replace HTTP links with HTTPS ones	2020-08-12 10:58:00 -07:00
decompress_unlzo.c
decompress_unxz.c
decompress_unzstd.c	lib: decompress_unzstd: Limit output size	2020-09-03 10:13:09 +02:00
decompress.c	lib: Add zstd support to decompress	2020-07-31 11:49:08 +02:00
devres.c	driver core: platform: Document return type of more functions	2020-09-10 18:30:01 +02:00
digsig.c
dump_stack.c	kernel: rename show_stack_loglvl() => show_stack()	2020-06-09 09:39:13 -07:00
dynamic_debug.c	dyndbg: fix use before null check	2020-12-30 11:54:11 +01:00
dynamic_queue_limits.c	lib: dynamic_queue_limits: delete duplicated words + fix typo	2020-10-16 11:11:20 -07:00
earlycpio.c	lib: earlycpio: delete duplicated words	2020-10-16 11:11:20 -07:00
errname.c
error-inject.c
errseq.c
extable.c
fault-inject-usercopy.c	lib, include/linux: add usercopy failure capability	2020-10-16 11:11:22 -07:00
fault-inject.c	fault_inject: Don't rely on "return value" from WRITE_ONCE()	2020-04-15 21:36:41 +01:00
fdt_addresses.c
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
fdt.c
find_bit_benchmark.c
find_bit.c	kernel.h: split out min()/max() et al. helpers	2020-10-16 11:11:19 -07:00
flex_proportions.c	lib/flex_proportions.c: cleanup __fprop_inc_percpu_max	2020-06-04 19:06:25 -07:00
gen_crc32table.c
gen_crc64table.c
genalloc.c	lib/genalloc: fix the overflow when size is too big	2021-01-12 20:18:16 +01:00
generic-radix-tree.c
glob.c	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
globtest.c
hexdump.c	kernel.h: split out min()/max() et al. helpers	2020-10-16 11:11:19 -07:00
hweight.c
idr.c	XArray updates for 5.9	2020-10-20 14:39:37 -07:00
inflate.c
interval_tree_test.c
interval_tree.c
iomap_copy.c
iomap.c	iomap: constify ioreadX() iomem argument (as in generic implementation)	2020-08-14 19:56:57 -07:00
iommu-helper.c
iov_iter.c	udp: fix skb_copy_and_csum_datagram with odd segment sizes	2021-02-17 11:02:28 +01:00
irq_poll.c
irq_regs.c
is_single_threaded.c
kasprintf.c
Kconfig	FROMGIT: lib: stackdepot: add support to configure STACK_HASH_SIZE	2021-02-15 15:11:44 -08:00
Kconfig.debug	This is the 5.10.32 stable release	2021-04-22 11:12:08 +02:00
Kconfig.kasan	BACKPORT: kasan: remove redundant config option	2021-03-24 15:09:15 -07:00
Kconfig.kcsan	kcsan: Test support for compound instrumentation	2020-08-24 15:09:58 -07:00
Kconfig.kfence	FROMGIT: kfence: await for allocation using wait_event	2021-04-29 08:13:58 +02:00
Kconfig.kgdb	kgdb: Honour the kprobe blocklist when setting breakpoints	2020-09-28 12:14:08 +01:00
Kconfig.ubsan	ubsan: introduce CONFIG_UBSAN_LOCAL_BOUNDS for Clang	2020-10-16 11:11:22 -07:00
kfifo.c
klist.c
kobject_uevent.c
kobject.c	kobject: Drop unneeded conditional in __kobject_del()	2020-09-07 11:24:17 +02:00
kstrtox.c	kstrto*: do not describe simple_strto*() as obsolete/replaced	2020-08-12 10:58:00 -07:00
kstrtox.h
libcrc32c.c	lib: libcrc32c: delete duplicated words	2020-10-16 11:11:19 -07:00
linear_ranges.c	lib: linear_ranges: Add missing MODULE_LICENSE()	2020-05-11 11:55:28 +01:00
list_debug.c
list_sort.c	FROMLIST: lib/list_sort: fix function type mismatches	2021-01-14 16:32:31 +00:00
list-test.c	Fix linked-list KUnit test when run multiple times	2020-03-25 16:38:39 -06:00
llist.c
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-rtmutex.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c	lockdep/selftest: Introduce recursion3	2020-08-26 12:42:08 +02:00
lockref.c
logic_pio.c	PCI: Fix pci_register_io_range() memory leak	2021-03-17 17:06:25 +01:00
lru_cache.c
lshrdi3.c
Makefile	FROMGIT: kasan: rename CONFIG_TEST_KASAN_MODULE	2021-02-07 13:41:41 -08:00
memcat_p.c
memory-notifier-error-inject.c
memregion.c	lib/memregion.c: include memregion.h	2020-09-26 10:33:57 -07:00
memweight.c
muldi3.c
net_utils.c
netdev-notifier-error-inject.c
nlattr.c	netlink: export policy in extended ACK	2020-10-09 20:22:32 -07:00
nmi_backtrace.c	lib: Add backtrace_idle parameter to force backtrace of idle CPUs	2020-08-24 14:24:25 -07:00
nodemask.c
notifier-error-inject.c
notifier-error-inject.h
objagg.c
of-reconfig-notifier-error-inject.c
oid_registry.c
once.c
packing.c	lib: packing: add documentation for pbuflen argument	2020-06-28 20:45:27 -07:00
parman.c
parser.c
pci_iomap.c
percpu_counter.c	lib/percpu_counter.c: use helper macro abs()	2020-10-16 11:11:20 -07:00
percpu_test.c
percpu-refcount.c	percpu_ref: don't refer to ref->data if it isn't allocated	2020-10-09 12:32:06 -06:00
plist.c	ANDROID: lib/plist.c: Export plist-related APIs	2021-03-16 01:19:21 +00:00
pm-notifier-error-inject.c
radix-tree.c	XArray updates for 5.9	2020-10-20 14:39:37 -07:00
random32.c	random32: add a selftest for the prandom32 code	2020-10-24 20:21:57 +02:00
ratelimit.c
rbtree_test.c
rbtree.c	lib/: replace HTTP links with HTTPS ones	2020-08-12 10:58:00 -07:00
refcount.c
rhashtable.c	rhashtable: Restore RCU marking on rhash_lock_head	2020-07-28 17:09:49 -07:00
sbitmap.c	sbitmap: Consider cleared bits in sbitmap_bitmap_show()	2020-07-01 10:53:00 -06:00
scatterlist.c	lib/scatterlist: use consistent sg_copy_buffer() return type	2020-10-29 08:55:45 -06:00
seq_buf.c	seq_buf: Export seq_buf_printf	2020-06-15 18:22:43 -07:00
sg_pool.c
sg_split.c
sha1.c	crypto: lib/sha1 - fold linux/cryptohash.h into crypto/sha.h	2020-05-08 15:32:17 +10:00
show_mem.c	ANDROID: vendor_hooks: Add hooks for memory when debug	2021-03-19 04:54:07 +00:00
siphash.c	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
smp_processor_id.c	lib/smp_processor_id: Move it into noinstr section	2020-06-11 15:14:36 +02:00
sort.c
stackdepot.c	FROMGIT: lib: stackdepot: fix ignoring return value warning	2021-02-15 15:11:44 -08:00
stmp_device.c
string_helpers.c	lib: string_helpers: provide kfree_strarray()	2020-09-30 10:50:30 +02:00
string.c	lib/string.c: implement stpcpy	2020-09-26 10:33:57 -07:00
strncpy_from_user.c	lib/strncpy_from_user.c: Mask out bytes after NUL terminator.	2020-11-19 11:56:16 -08:00
strnlen_user.c	uaccess: Selectively open read or write user access	2020-05-01 12:35:21 +10:00
syscall.c	lib/syscall: fix syscall registers retrieval on 32-bit platforms	2020-12-03 09:52:44 -08:00
test_bitmap.c	UPSTREAM: lib: use KSTM_MODULE_GLOBALS macro in kselftest drivers	2021-02-25 21:36:45 +00:00
test_bitops.c	lib/test_bitops: do the full test during module init	2020-08-12 10:57:59 -07:00
test_bits.c	lib/test_bits.c: add tests of GENMASK	2020-08-12 10:58:00 -07:00
test_blackhole_dev.c
test_bpf.c	bpf: revert "test_bpf: Flag tests that cannot be jited on s390"	2020-07-16 20:52:43 +02:00
test_debug_virtual.c
test_firmware.c	test_firmware: Test partial read support	2020-10-05 13:37:04 +02:00
test_fpu.c	selftests/fpu: Add an FPU selftest	2020-06-29 10:02:23 +02:00
test_free_pages.c	mm/page_alloc.c: fix freeing non-compound pages	2020-10-13 18:38:33 -07:00
test_hash.c
test_hexdump.c
test_hmm_uapi.h	mm/hmm: add tests for hmm_pfn_to_map_order()	2020-07-10 16:24:28 -03:00
test_hmm.c	lib/test_hmm.c: fix an error code in dmirror_allocate_chunk()	2020-10-16 11:11:20 -07:00
test_ida.c
test_kasan_module.c	FROMGIT: kasan: clean up comments in tests	2021-02-07 13:41:41 -08:00
test_kasan.c	FROMGIT: kasan, arm64: tests supports for HW_TAGS async mode	2021-04-27 10:40:13 +02:00
test_kmod.c	test_kmod: avoid potential double free in trigger_config_run_type()	2020-08-12 10:58:01 -07:00
test_linear_ranges.c	lib/test_linear_ranges: add a test for the 'linear_ranges'	2020-05-08 18:18:12 +01:00
test_list_sort.c
test_lockup.c	lib/test_lockup.c: fix return value of test_lockup_init()	2020-08-12 10:58:00 -07:00
test_memcat_p.c
test_meminit.c
test_min_heap.c
test_module.c
test_objagg.c	test_objagg: Fix potential memory leak in error handling	2020-06-15 13:32:11 -07:00
test_overflow.c
test_parman.c
test_printf.c	UPSTREAM: lib/vsprintf: no_hash_pointers prints all addresses as unhashed	2021-02-25 21:37:05 +00:00
test_rhashtable.c	rhashtable: fix indentation of a continue statement	2020-09-20 14:10:06 -07:00
test_siphash.c
test_sort.c
test_stackinit.c	lib: test_stackinit.c: XFAIL switch variable init tests	2020-04-07 10:43:43 -07:00
test_static_key_base.c
test_static_keys.c
test_string.c
test_strscpy.c
test_sysctl.c	lib: test_sysctl: delete duplicated words	2020-10-16 11:11:20 -07:00
test_ubsan.c
test_user_copy.c
test_uuid.c
test_vmalloc.c	lib/test_vmalloc.c: Add test cases for kvfree_rcu()	2020-06-29 11:59:26 -07:00
test_xarray.c	XArray: Fix splitting to non-zero orders	2021-04-16 11:43:21 +02:00
test-kstrtox.c
test-string_helpers.c	lib/test-string_helpers.c: Add string_upper() and string_lower() tests	2020-07-15 12:45:06 +03:00
textsearch.c
timerqueue.c
ts_bm.c	lib/: replace HTTP links with HTTPS ones	2020-08-12 10:58:00 -07:00
ts_fsm.c	lib: Revert use of fallthrough pseudo-keyword in lib/	2020-08-24 14:17:44 -07:00
ts_kmp.c	lib/ts_kmp.c: replace zero-length array with flexible-array member	2020-04-07 10:43:43 -07:00
ubsan.c	ubsan: implement __ubsan_handle_alignment_assumption	2021-02-17 11:02:24 +01:00
ubsan.h	ubsan: implement __ubsan_handle_alignment_assumption	2021-02-17 11:02:24 +01:00
ucmpdi2.c
ucs2_string.c
usercopy.c	lib, uaccess: add failure injection to usercopy functions	2020-10-16 11:11:22 -07:00
uuid.c	uuid: Provide a GUID generator for raw buffer	2020-03-23 17:01:47 +01:00
vsprintf.c	UPSTREAM: lib/vsprintf: no_hash_pointers prints all addresses as unhashed	2021-02-25 21:37:05 +00:00
win_minmax.c
xarray.c	XArray: Fix splitting to non-zero orders	2021-04-16 11:43:21 +02:00
xxhash.c	lib/: replace HTTP links with HTTPS ones	2020-08-12 10:58:00 -07:00