linux/mm
Collin Fijalkovich 28b4b1588e FROMLIST: mm, thp: Relax the VM_DENYWRITE constraint on file-backed THPs
Transparent huge pages are supported for read-only non-shmem files,
but are only used for vmas with VM_DENYWRITE. This condition ensures that
file THPs are protected from writes while an application is running
(ETXTBSY).  Any existing file THPs are then dropped from the page cache
when a file is opened for write in do_dentry_open(). Since sys_mmap
ignores MAP_DENYWRITE, this constrains the use of file THPs to vmas
produced by execve().

Systems that make heavy use of shared libraries (e.g. Android) are unable
to apply VM_DENYWRITE through the dynamic linker, preventing them from
benefiting from the resultant reduced contention on the TLB.

This patch reduces the constraint on file THPs allowing use with any
executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to
ensure that files opened for write will never be backed by file THPs.

Restricting the use of THPs to executable mappings eliminates the risk that
a read-only file later opened for write would encounter significant
latencies due to page cache truncation.

The ld linker flag '-z max-page-size=(hugepage size)' can be used to
produce executables with the necessary layout. The dynamic linker must
map these file's segments at a hugepage size aligned vma for the mapping to
be backed with THPs.

Comparison of the performance characteristics of 4KB and 2MB-backed
libraries follows; the Android dex2oat tool was used to AOT compile an
example application on a single ARM core.

4KB Pages:
==========

count              event_name            # count / runtime
598,995,035,942    cpu-cycles            # 1.800861 GHz
 81,195,620,851    raw-stall-frontend    # 244.112 M/sec
347,754,466,597    iTLB-loads            # 1.046 G/sec
  2,970,248,900    iTLB-load-misses      # 0.854122% miss rate

Total test time: 332.854998 seconds.

2MB Pages:
==========

count              event_name            # count / runtime
592,872,663,047    cpu-cycles            # 1.800358 GHz
 76,485,624,143    raw-stall-frontend    # 232.261 M/sec
350,478,413,710    iTLB-loads            # 1.064 G/sec
    803,233,322    iTLB-load-misses      # 0.229182% miss rate

Total test time: 329.826087 seconds

A check of /proc/$(pidof dex2oat64)/smaps shows THPs in use:

/apex/com.android.art/lib64/libart.so
FilePmdMapped:      4096 kB

/apex/com.android.art/lib64/libart-compiler.so
FilePmdMapped:      2048 kB

Bug: 158135888
Link: https://lore.kernel.org/patchwork/patch/1408266/

Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Acked-by: Hugh Dickins <hughd@google.com>
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Collin Fijalkovich <cfijalkovich@google.com>
Change-Id: I75c693a4b4e7526d374ef2c010bde3094233eef2
2021-04-28 18:41:35 +00:00
..
kasan FROMGIT: kasan, arm64: tests supports for HW_TAGS async mode 2021-04-27 10:40:13 +02:00
kfence UPSTREAM: kfence: report sensitive information based on no_hash_pointers 2021-03-01 18:11:05 +01:00
backing-dev.c bdi: replace BDI_CAP_NO_{WRITEBACK,ACCT_DIRTY} with a single flag 2020-09-24 13:43:39 -06:00
balloon_compaction.c
cleancache.c
cma_debug.c FROMLIST: mm: cma: introduce gfp flag in cma_alloc instead of no_warn 2021-01-25 12:21:02 -08:00
cma_sysfs.c ANDROID: make cma_sysfs experimental 2021-03-25 19:20:18 +00:00
cma.c FROMGIT: mm: cma: add the CMA instance name to cma trace events 2021-04-23 22:41:43 +00:00
cma.h FROMLIST: mm: cma: support sysfs 2021-03-25 19:20:09 +00:00
compaction.c ANDROID: mm: compaction: fix isolate_and_split_free_page() redefinition 2021-04-23 21:54:18 +00:00
debug_page_ref.c
debug_vm_pgtable.c mm/debug_vm_pgtable: avoid doing memory allocation with pgtable_t mapped. 2020-10-16 11:11:14 -07:00
debug.c mm, dump_page: rename head_mapcount() --> head_compound_mapcount() 2020-10-13 18:38:29 -07:00
dmapool.c mm/dmapool.c: replace hard coded function name with __func__ 2020-10-13 18:38:32 -07:00
early_ioremap.c mm/early_ioremap.c: use %pa to print resource_size_t variables 2020-01-31 10:30:38 -08:00
fadvise.c mm, fadvise: improve the expensive remote LRU cache draining after FADV_DONTNEED 2020-10-13 18:38:29 -07:00
failslab.c
filemap.c ANDROID: mm: allow vmas with vm_ops to be speculatively handled 2021-04-23 18:42:39 -07:00
frame_vector.c mmap locking API: convert mmap_sem comments 2020-06-09 09:39:14 -07:00
frontswap.c mm/frontswap: mark various intentional data races 2020-08-14 19:56:56 -07:00
gup_benchmark.c mm/gup_benchmark: take the mmap lock around GUP 2020-10-18 09:27:09 -07:00
gup.c mm/gup: combine put_compound_head() and unpin_user_page() 2020-12-30 11:53:54 +01:00
highmem.c mm/highmem.c: clean up endif comments 2020-10-16 11:11:18 -07:00
hmm.c mm: do page fault accounting in handle_mm_fault 2020-08-12 10:58:02 -07:00
huge_memory.c Merge 5.10.27 into android12-5.10 2021-04-02 15:25:50 +02:00
hugetlb_cgroup.c hugetlb_cgroup: fix imbalanced css_get and css_put pair for shared mappings 2021-03-30 14:31:54 +02:00
hugetlb.c FROMLIST: userfaultfd/hugetlbfs: Fix minor fault page leak 2021-04-09 15:36:01 -07:00
hwpoison-inject.c mm,hwpoison-inject: don't pin for hwpoison_filter 2020-10-16 11:11:16 -07:00
init-mm.c FROMLIST: mm: protect mm_rb tree with a rwlock 2021-01-22 18:00:57 +00:00
internal.h FROMLIST: mm: provide speculative fault infrastructure 2021-01-22 18:01:16 +00:00
interval_tree.c
ioremap.c mm: move p?d_alloc_track to separate header file 2020-08-07 11:33:26 -07:00
Kconfig FROMLIST: mm: cma: support sysfs 2021-03-25 19:20:09 +00:00
Kconfig.debug FROMGIT: mm, page_poison: remove CONFIG_PAGE_POISONING_ZERO 2021-03-24 15:09:17 -07:00
khugepaged.c FROMLIST: mm, thp: Relax the VM_DENYWRITE constraint on file-backed THPs 2021-04-28 18:41:35 +00:00
kmemleak.c mm/kmemleak: rely on rcu for task stack scanning 2020-10-13 18:38:27 -07:00
ksm.c docs: get rid of :c:type explicit declarations for structs 2020-10-15 07:49:40 +02:00
list_lru.c mm: list_lru: set shrinker map bit when child nr_items is not zero 2020-12-06 10:19:07 -08:00
maccess.c uaccess: add force_uaccess_{begin,end} helpers 2020-08-12 10:57:59 -07:00
madvise.c This is the 5.10.24 stable release 2021-03-19 09:42:56 +01:00
Makefile FROMLIST: mm: cma: support sysfs 2021-03-25 19:20:09 +00:00
mapping_dirty_helpers.c mm/mapping_dirty_helpers: update huge page-table entry callbacks 2020-04-02 09:35:29 -07:00
memblock.c This is the 5.10.15 stable release 2021-02-10 09:53:50 +01:00
memcontrol.c mm/memcg: fix 5.10 backport of splitting page memcg 2021-03-30 14:32:07 +02:00
memfd.c
memory_hotplug.c ANDROID: mm/memory_hotplug: fix minor printk format warnings 2021-04-14 17:36:16 +00:00
memory-failure.c mm: fix memory_failure() handling of dax-namespace metadata 2021-03-04 11:38:21 +01:00
memory.c ANDROID: mm: allow vmas with vm_ops to be speculatively handled 2021-04-23 18:42:39 -07:00
mempolicy.c FROMLIST: mm: replace migrate_[prep|finish] with lru_cache_[disable|enable] 2021-03-23 04:05:24 +00:00
mempool.c FROMGIT: kasan, mm: integrate page_alloc init with HW_TAGS 2021-03-24 15:09:18 -07:00
memremap.c mm: fix memory_failure() handling of dax-namespace metadata 2021-03-04 11:38:21 +01:00
memtest.c
migrate.c FROMGIT: mm: cma: add trace events for CMA alloc perf testing 2021-04-23 22:41:28 +00:00
mincore.c mm: factor find_get_incore_page out of mincore_page 2020-10-13 18:38:29 -07:00
mlock.c FROMLIST: mm: protect VMA modifications using VMA sequence count 2021-01-22 17:59:47 +00:00
mm_init.c mm: adjust vm_committed_as_batch according to vm overcommit policy 2020-08-07 11:33:26 -07:00
mmap.c ANDROID: mm: use raw seqcount variants in vm_write_* 2021-01-22 18:02:43 +00:00
mmu_gather.c mmap locking API: convert mmap_sem comments 2020-06-09 09:39:14 -07:00
mmu_notifier.c mm/mmu_notifiers: ensure range_end() is paired with range_start() 2021-03-30 14:32:06 +02:00
mmzone.c ANDROID: mm: export zone_watermark_ok 2021-02-25 19:36:38 +00:00
mprotect.c FROMLIST: mm: protect VMA modifications using VMA sequence count 2021-01-22 17:59:47 +00:00
mremap.c FROMLIST: mm: Extend MREMAP_DONTUNMAP to non-anonymous mappings 2021-03-25 22:44:44 +00:00
msync.c mmap locking API: use coccinelle to convert mmap_sem rwsem call sites 2020-06-09 09:39:14 -07:00
nommu.c ANDROID: mm: allow vmas with vm_ops to be speculatively handled 2021-04-23 18:42:39 -07:00
oom_kill.c ANDROID: mm, oom: Prevent extra put_task when oom_evaluate_task is aborted 2021-02-24 22:21:44 +00:00
OWNERS ANDROID: Add OWNERS files referring to the respective android-mainline OWNERS 2021-04-03 14:11:30 +00:00
page_alloc.c ANDROID: implement wrapper for reverse migration 2021-04-20 21:26:32 +00:00
page_counter.c mm/page_counter: correct the obsolete func name in the comment of page_counter_try_charge() 2020-10-13 18:38:30 -07:00
page_ext.c ANDROID: mm: Export lookup_page_ext 2021-01-06 16:16:41 +00:00
page_idle.c mm/page_idle.c: skip offline pages 2020-06-08 11:05:55 -07:00
page_io.c UPSTREAM: mm/page_io: use pr_alert_ratelimited for swap read/write errors 2021-03-30 18:44:11 +00:00
page_isolation.c mm: rename page_order() to buddy_order() 2020-10-16 11:11:19 -07:00
page_owner.c ANDROID: mm: Make page_owner_enabled global 2021-04-01 00:09:00 +00:00
page_poison.c FROMGIT: mm, page_poison: remove CONFIG_PAGE_POISONING_NO_SANITY 2021-03-24 15:09:17 -07:00
page_reporting.c mm: rename page_order() to buddy_order() 2020-10-16 11:11:19 -07:00
page_reporting.h mm: introduce include/linux/pgtable.h 2020-06-09 09:39:13 -07:00
page_vma_mapped.c mm: replace hpage_nr_pages with thp_nr_pages 2020-08-14 19:56:56 -07:00
page-writeback.c mm: make wait_on_page_writeback() wait for multiple pending writebacks 2021-01-12 20:18:22 +01:00
pagewalk.c mmap locking API: convert mmap_sem comments 2020-06-09 09:39:14 -07:00
percpu-internal.h percpu: make pcpu_nr_empty_pop_pages per chunk type 2021-04-14 08:42:03 +02:00
percpu-km.c mm: memcg/percpu: account percpu memory to memory cgroups 2020-08-12 10:57:55 -07:00
percpu-stats.c percpu: make pcpu_nr_empty_pop_pages per chunk type 2021-04-14 08:42:03 +02:00
percpu-vm.c mm: memcg/percpu: account percpu memory to memory cgroups 2020-08-12 10:57:55 -07:00
percpu.c Merge 5.10.30 into android12-5.10 2021-04-15 14:23:41 +02:00
pgalloc-track.h mm: move p?d_alloc_track to separate header file 2020-08-07 11:33:26 -07:00
pgtable-generic.c mm: introduce include/linux/pgtable.h 2020-06-09 09:39:13 -07:00
process_vm_access.c mm/process_vm_access.c: include compat.h 2021-01-19 18:27:21 +01:00
ptdump.c This is the 5.10.32 stable release 2021-04-22 11:12:08 +02:00
readahead.c ANDROID: mm: Create vendor hooks to control ZONE_MOVABLE allocations 2020-12-01 18:07:54 +00:00
rmap.c FROMLIST: mm: introduce __page_add_new_anon_rmap() 2021-01-22 18:00:48 +00:00
rodata_test.c mm/rodata_test.c: fix missing function declaration 2020-08-21 09:52:53 -07:00
shmem.c ANDROID: mm: allow vmas with vm_ops to be speculatively handled 2021-04-23 18:42:39 -07:00
shuffle.c mm: rename page_order() to buddy_order() 2020-10-16 11:11:19 -07:00
shuffle.h mm/shuffle: remove dynamic reconfiguration 2020-08-07 11:33:29 -07:00
slab_common.c ANDROID: vendor_hooks: Add hooks for memory when debug 2021-03-19 04:54:07 +00:00
slab.c FROMGIT: kasan, mm: integrate slab init_on_free with HW_TAGS 2021-03-24 15:09:18 -07:00
slab.h ANDROID: mm: Make slub_debug global 2021-04-01 00:09:35 +00:00
slob.c mm: memcg: convert vmstat slab counters to bytes 2020-08-07 11:33:24 -07:00
slub.c ANDROID: mm: Make slub_debug global 2021-04-01 00:09:35 +00:00
sparse-vmemmap.c mm/sparse: only sub-section aligned range would be populated 2020-08-07 11:33:27 -07:00
sparse.c mm/memory_hotplug: guard more declarations by CONFIG_MEMORY_HOTPLUG 2020-10-16 11:11:18 -07:00
swap_cgroup.c mm: memcontrol: make swap tracking an integral part of memory control 2020-06-03 20:09:48 -07:00
swap_slots.c mm/swap_slots.c: remove always zero and unused return value of enable_swap_slots_cache() 2020-10-13 18:38:30 -07:00
swap_state.c FROMLIST: mm: protect VMA modifications using VMA sequence count 2021-01-22 17:59:47 +00:00
swap.c FROMLIST: mm: fs: Invalidate BH LRU during page migration 2021-03-23 04:05:31 +00:00
swapfile.c This is the 5.10.21 stable release 2021-03-07 12:53:30 +01:00
truncate.c mm/truncate.c: make __invalidate_mapping_pages() static 2020-11-02 12:14:19 -08:00
usercopy.c mm/usercopy.c: delete duplicated word 2020-08-12 10:57:58 -07:00
userfaultfd.c BACKPORT: FROMGIT: userfaultfd: support minor fault handling for shmem 2021-04-09 15:36:00 -07:00
util.c mm/util.c: update the kerneldoc for kstrdup_const() 2020-10-16 11:11:17 -07:00
vmacache.c kernel: better document the use_mm/unuse_mm API contract 2020-06-10 19:14:18 -07:00
vmalloc.c This is the 5.10.9 stable release 2021-01-19 18:49:54 +01:00
vmpressure.c mm: vmpressure: use mem_cgroup_is_root API 2020-04-02 09:35:31 -07:00
vmscan.c ANDROID: vendor_hooks: tune reclaim swappiness or scan type 2021-04-15 23:08:29 +00:00
vmstat.c ANDROID: mm: allow vmas with vm_ops to be speculatively handled 2021-04-23 18:42:39 -07:00
workingset.c XArray updates for 5.9 2020-10-20 14:39:37 -07:00
z3fold.c z3fold: prevent reclaim/free race for headless pages 2021-03-30 14:31:54 +02:00
zbud.c mm/zbud: remove redundant initialization 2020-10-13 18:38:34 -07:00
zpool.c mm/zpool.c: delete duplicated word and fix grammar 2020-08-12 10:57:58 -07:00
zsmalloc.c This is the 5.10.21 stable release 2021-03-07 12:53:30 +01:00
zswap.c mm/zswap: allow setting default status, compressor and allocator in Kconfig 2020-04-07 10:43:41 -07:00