github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-07 06:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge branch 'bpf-fix-arena-memory-incoherence'

Browse Source 
Emil Tsalapatis says:

====================
bpf: Fix arena memory incoherence

Arena memory is tracked in two places: the range tree, which records
which addresses are allocated, and the kernel and user page tables. The
two are updated under different locks, and that leaves two races:

- Since b8467290ed ("bpf: arena: make arena kfuncs any context safe"),
  a free releases its range before the kernel TLB flush and the user zap
  complete. A concurrent allocation can then write through stale
  translations into pages that are about to be freed.

- A user fault takes a reference to the page and installs the user PTE
  after dropping the arena lock. A free that runs in between leaves user
  space mapping a page that the arena has already released.

In both cases BPF programs and user space can see different pages at the
same arena address.

- A free keeps its range allocated until the TLB flush and the user zap
  are complete, and only then releases it with range_tree_set(). User
  faults on the range get SIGBUS until then. Frees serialize on
  zap_mutex.

- A user fault pins the page, locks its folio, and revalidates the
  kernel mapping before installing the user PTE, the way filemap_fault()
  synchronizes with truncate. The free waits for the folio lock of every
  page it removed before it zaps user mappings.

Patches 1 and 2 fix the two races. Patch 3 adds selftests.

Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Cc: Puranjay Mohan <puranjay@kernel.org>

v5 -> v6 (https://lore.kernel.org/bpf/20261016193703.9203-1-emil@etsalapatis.com/)

- Return the locked page from arena_vm_fault() with VM_FAULT_LOCKED
  instead of installing the user PTE there with finish_fault(). The
  fault checks PageHWPoison() before it returns the page. (Alexei)
- Give faults on a range that is being freed the same SIGBUS as faults
  on a reserved range. Drop the record of the range being freed and the
  fault-side wait. (Alexei)
- Drop the trylock and deferral scheme. Frees serialize on zap_mutex and
  zap user mappings with zap_pages(), which takes mmap_read_lock() of
  each mapping process as in mainline. Keep the single-page zap
  shortcut. Sleepable frees are deferred only when the arena spinlock
  cannot be taken, and the worker retries such a free in place instead
  of requeueing itself. (Alexei)
- Drop v5 patch 1. The worker no longer requeues itself, so the
  flush_work() in arena_map_free() waits for all of its work. (Alexei)
- Drop v5 patch 4, which retried faults when the arena lock timed out.
  (Alexei)
- Drop the immediate release of ranges without kernel PTEs; they go
  through the regular free path.
- Selftests: map the page into user space before the paused free, and
  let the fault_free_realloc reader skip reads that raise SIGBUS.

v4 -> v5 (https://lore.kernel.org/bpf/20261014233442.8392-1-emil@etsalapatis.com/)

- Drop the third range-tree state and the is_range_tree_set() change
  (v4 patches 4 and 5). A free keeps its range allocated until the TLB
  flush and the user zap complete, then releases it with
  range_tree_set(). range_tree_clear() only allocates when the whole
  range is free, and then there is nothing to free. (Alexei)
- Drop the per-arena mapping, page->mapping, the mapping-tree zap and
  the new bpf_map_ops callback (v4 patches 1 and 2). (Alexei)
- Replace opref (v4 patch 7) with the folio lock. A fault rechecks the
  kernel PTE with the folio locked and installs the user PTE itself with
  finish_fault(), and a free waits for the folio lock of each page it
  removed before zapping (new patch 2). (Alexei)
- Give faults on a reserved range the scratch_page error. Faults on the
  range being freed sleep until the free releases it, instead of
  spinning on VM_FAULT_RETRY. (Alexei)
- Trylock arena->lock and the mmap read lock of every process that maps
  the arena before retiring a range, and defer the free if one is
  contended. The worker waits for the contended lock instead of
  requeueing itself. The deferred span, allocated with
  __GFP_RETRY_MAYFAIL when the caller can sleep, is the only allocation
  before the ptes are cleared. (Alexei)
- Release a range without kernel ptes, such as a reserved one, right
  away without arena->lock or mmap_lock, so freeing and reallocating a
  reserved range is never deferred in sleepable context.
- Retry faults when the arena lock times out, keeping SIGBUS for
  -EDEADLK (new patch 4).
- Selftests: rename the subtests, and pause frees in zap_vma_range(),
  which arm64 can trace, instead of flush_tlb_kernel_range().

v3 -> v4 (https://lore.kernel.org/bpf/20260923191125.5311-1-emil@etsalapatis.com/)

- Add per-arena anonymous mappings and associate pages with them
- Return VM_FAULT_LOCKED and let generic MM install user PTEs; wait
  for locked fault pages before TLB flushing and zapping (Alexei).
- Add mutual exclusion between arena free and fault operations to
  prevent incoherency.
- Return VM_FAULT_NOPAGE for non-retryable GUP faults when progress is
  impossible.
- Stop and drain self-requeueing deferred frees before map teardown.
- Track ownership of unavailable ranges independently of free progress
  and retry worker reservations on -ENOMEM without starting teardown.
- Make arena free span state machine explicit.

v2 -> v3 (https://lore.kernel.org/bpf/20260923061425.7045-1-emil@etsalapatis.com/)

- Rebase on bpf-next and resolve conflicts
- Fold in the atomic mark_available range tree operation into initial
  fix patches

v1 -> v2 (https://lore.kernel.org/bpf/20260902070239.16968-1-emil@etsalapatis.com/)

- Use __range_iter_next() in is_range_tree_set() (bot-ci)
- Move -EAGAIN handling to the patch where the errno is introduced (bot-ci)
- Add refactoring patch for (Puranjay)
- Avoid leaking unavailable range tree nodes with atomic mark_available
  operation()
- Avoid retries on res spinlock locking for -EDEADLK (Puranjay)
====================

Link: https://patch.msgid.link/20261019201108.4552-1-emil@etsalapatis.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit is contained in:
Alexei Starovoitov 2026-10-20 00:18:32 +00:00
commit 1907a7d0a1
No known key found for this signature in database
3 changed files with 575 additions and 88 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

177
kernel/bpf/arena.c
View File

@ -543,6 +543,12 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
goto out_err_locked;
}
if (is_range_tree_set(&arena->rt, vmf->pgoff, 1)) {
/* Allocated without a page: reserved or being freed */
fault_ret = VM_FAULT_SIGBUS;
goto out_err_locked;
}
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
if (!new_page) {
@ -584,8 +590,22 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
if (new_page)
free_pages_nolock(new_page, 0);
vmf->page = page;
return 0;
/*
* A free clears the kernel pte and then takes the page lock before it
* zaps user vma-s. Either it waits for this fault and zaps its pte, or
* the page is gone from here.
*/
lock_page(page);
if (vmalloc_to_page((void *)kaddr) == page && !PageHWPoison(page)) {
vmf->page = page;
return VM_FAULT_LOCKED;
}
/* The page was freed meanwhile. Let the fault start over. */
fault_ret = PageHWPoison(page) ? VM_FAULT_HWPOISON : VM_FAULT_NOPAGE;
unlock_page(page);
put_page(page);
return fault_ret;
out_err_locked_memcg:
bpf_map_memcg_exit(old_memcg, new_memcg);
@ -854,7 +874,7 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
* otherwise call B's mark would make call A skip a vml that A has
* not yet zapped for A's uaddr range.
*/
mutex_lock(&arena->zap_mutex);
lockdep_assert_held(&arena->zap_mutex);
mutex_lock(&arena->lock);
my_gen = ++arena->zap_gen;
for (;;) {
@ -888,49 +908,41 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
mutex_lock(&arena->lock);
}
mutex_unlock(&arena->lock);
mutex_unlock(&arena->zap_mutex);
}
static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)
/*
* The range stays allocated in the range tree until the pages are unmapped
* from vmalloc area and from all user space vma-s, so that nobody can
* populate it while stale TLB entries or user ptes are still around.
*/
static int arena_free_range(struct bpf_arena *arena, long uaddr, long page_cnt)
{
struct mem_cgroup *new_memcg, *old_memcg;
u64 full_uaddr, uaddr_end;
long kaddr, pgoff;
struct page *page;
struct llist_head free_pages;
struct llist_node *pos, *t;
struct arena_free_span *s;
u64 full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
long kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr;
long pgoff = compute_pgoff(arena, uaddr);
struct clear_range_data cdata;
struct llist_node *pos, *t;
LLIST_HEAD(free_pages);
unsigned long flags;
int ret = 0;
struct page *page;
int ret;
/* only aligned lower 32-bit are relevant */
uaddr = (u32)uaddr;
uaddr &= PAGE_MASK;
kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr;
full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
if (full_uaddr < arena->user_vm_start)
return;
uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT));
if (full_uaddr >= uaddr_end)
return;
page_cnt = (uaddr_end - full_uaddr) >> PAGE_SHIFT;
pgoff = compute_pgoff(arena, uaddr);
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
if (!sleepable)
goto defer;
/* An overlapping free must not release the range before this one is done */
guard(mutex)(&arena->zap_mutex);
ret = raw_res_spin_lock_irqsave(&arena->spinlock, flags);
/* Can't proceed without holding the spinlock so defer the free */
if (ret)
goto defer;
return ret;
range_tree_set(&arena->rt, pgoff, page_cnt);
/*
* Take the free pages of the range too. It allocates, and can fail,
* only when the whole range is free. Then there is nothing to do.
*/
if (range_tree_clear(&arena->rt, pgoff, page_cnt)) {
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
return 0;
}
init_llist_head(&free_pages);
cdata.arena = arena;
cdata.free_pages = &free_pages;
/* clear ptes and collect struct pages */
@ -940,6 +952,12 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
/* drop the lock to do the tlb flush and zap pages */
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
/* wait for arena_vm_fault() that found the page before its pte was cleared */
llist_for_each_entry(page, free_pages.first, pcp_llist) {
lock_page(page);
unlock_page(page);
}
/* ensure no stale TLB entries */
flush_tlb_kernel_range(kaddr, kaddr + (page_cnt * PAGE_SIZE));
@ -958,8 +976,39 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
zap_pages(arena, full_uaddr, 1);
__free_page(page);
}
bpf_map_memcg_exit(old_memcg, new_memcg);
/* The callers hold no other rqspinlock, so this one can only time out */
while (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
cond_resched();
range_tree_set(&arena->rt, pgoff, page_cnt);
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
return 0;
}
static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)
{
struct mem_cgroup *new_memcg, *old_memcg;
u64 full_uaddr, uaddr_end;
struct arena_free_span *s;
/* only aligned lower 32-bit are relevant */
uaddr = (u32)uaddr;
uaddr &= PAGE_MASK;
full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
if (full_uaddr < arena->user_vm_start)
return;
uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT));
if (full_uaddr >= uaddr_end)
return;
page_cnt = (uaddr_end - full_uaddr) >> PAGE_SHIFT;
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
/* Can't proceed without holding the spinlock so defer the free */
if (!sleepable || arena_free_range(arena, uaddr, page_cnt))
goto defer;
bpf_map_memcg_exit(old_memcg, new_memcg);
return;
defer:
@ -1021,66 +1070,18 @@ static void arena_free_worker(struct work_struct *work)
{
struct bpf_arena *arena = container_of(work, struct bpf_arena, free_work);
struct mem_cgroup *new_memcg, *old_memcg;
struct llist_node *list, *pos, *t;
struct llist_node *pos, *t;
struct arena_free_span *s;
u64 arena_vm_start, user_vm_start;
struct llist_head free_pages;
struct clear_range_data cdata;
struct page *page;
unsigned long full_uaddr;
long kaddr, page_cnt, pgoff;
unsigned long flags;
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) {
schedule_work(work);
return;
}
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
init_llist_head(&free_pages);
cdata.arena = arena;
cdata.free_pages = &free_pages;
arena_vm_start = bpf_arena_get_kern_vm_start(arena);
user_vm_start = bpf_arena_get_user_vm_start(arena);
list = llist_del_all(&arena->free_spans);
llist_for_each(pos, list) {
llist_for_each_safe(pos, t, llist_del_all(&arena->free_spans)) {
s = llist_entry(pos, struct arena_free_span, node);
page_cnt = s->page_cnt;
kaddr = arena_vm_start + s->uaddr;
pgoff = compute_pgoff(arena, s->uaddr);
/* clear ptes and collect pages in free_pages llist */
apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
apply_range_clear_cb, &cdata);
range_tree_set(&arena->rt, pgoff, page_cnt);
}
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
/* Iterate the list again without holding spinlock to do the tlb flush and zap_pages */
llist_for_each_safe(pos, t, list) {
s = llist_entry(pos, struct arena_free_span, node);
page_cnt = s->page_cnt;
full_uaddr = clear_lo32(user_vm_start) + s->uaddr;
kaddr = arena_vm_start + s->uaddr;
/* ensure no stale TLB entries */
flush_tlb_kernel_range(kaddr, kaddr + (page_cnt * PAGE_SIZE));
/* remove pages from user vmas */
zap_pages(arena, full_uaddr, page_cnt);
while (arena_free_range(arena, s->uaddr, s->page_cnt))
cond_resched();
kfree_nolock(s);
}
/* free all pages collected by apply_to_existing_page_range() in the first loop */
llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {
page = llist_entry(pos, struct page, pcp_llist);
__free_page(page);
}
bpf_map_memcg_exit(old_memcg, new_memcg);
}

320
tools/testing/selftests/bpf/prog_tests/arena_race.c Normal file
View File

@ -0,0 +1,320 @@
// SPDX-License-Identifier: GPL-2.0
#define _GNU_SOURCE
#include <sched.h>
#include <setjmp.h>
#include <signal.h>
#include <sys/syscall.h>
#include <test_progs.h>
#include "arena_race.skel.h"
struct free_thread_ctx {
struct arena_race *skel;
int err;
__u32 retval;
};
struct fault_thread_ctx {
__u64 *addr;
int stop;
};
static int run_prog(struct bpf_program *prog, const char *name)
{
LIBBPF_OPTS(bpf_test_run_opts, opts);
int err;
err = bpf_prog_test_run_opts(bpf_program__fd(prog), &opts);
return ASSERT_OK(err, name) && ASSERT_OK(opts.retval, name) ? 0 : -1;
}
/* Trigger the sleepable free page path. */
static void *run_free_thread(void *arg)
{
LIBBPF_OPTS(bpf_test_run_opts, opts);
struct free_thread_ctx *ctx = arg;
ctx->skel->bss->target_tid = sys_gettid();
ctx->err = bpf_prog_test_run_opts(
bpf_program__fd(ctx->skel->progs.free_page), &opts);
ctx->retval = opts.retval;
return NULL;
}
static __thread sigjmp_buf reader_env;
static __thread bool reader_armed;
/* Only the reader expects SIGBUS; anywhere else it keeps its default action. */
static void reader_sigbus(int sig)
{
if (!reader_armed) {
signal(sig, SIG_DFL);
raise(sig);
return;
}
siglongjmp(reader_env, 1);
}
/* Continuously fault in the address. */
static void *fault_reader_thread(void *arg)
{
struct fault_thread_ctx *ctx = arg;
__u64 *addr;
/* Touching a page while BPF frees it raises SIGBUS; skip that read. */
sigsetjmp(reader_env, 1);
reader_armed = true;
while (!READ_ONCE(ctx->stop)) {
addr = __atomic_load_n(&ctx->addr, __ATOMIC_ACQUIRE);
(void)READ_ONCE(*addr);
}
reader_armed = false;
return NULL;
}
static int wait_for(int *p)
{
__u64 deadline = get_time_ns() + 5ULL * 1000 * 1000 * 1000;
while (!READ_ONCE(*p)) {
if (get_time_ns() > deadline)
return -ETIMEDOUT;
}
return 0;
}
/* Skip the subtest, saying why, if @func cannot be traced with fentry. */
static bool require_fentry_target(const char *test, const char *func)
{
if (libbpf_find_vmlinux_btf_id(func, BPF_TRACE_FENTRY) > 0)
return true;
printf("%s:SKIP: %s is not an fentry target\n", test, func);
test__skip();
return false;
}
static struct arena_race *setup_arena(__u64 **addr, bool trace_zap)
{
struct arena_race *skel;
size_t arena_sz;
char *base;
int err;
skel = arena_race__open();
if (!ASSERT_OK_PTR(skel, "open"))
return NULL;
err = bpf_program__set_autoload(skel->progs.trace_zap, trace_zap);
if (!ASSERT_OK(err, "trace_zap_autoload"))
goto err_out;
err = arena_race__load(skel);
if (!ASSERT_OK(err, "load"))
goto err_out;
err = arena_race__attach(skel);
if (!ASSERT_OK(err, "attach"))
goto err_out;
if (run_prog(skel->progs.alloc_old, "alloc_old"))
goto err_out;
if (skel->bss->skip) {
printf("%s:SKIP: compiler doesn't support arena_cast\n", __func__);
test__skip();
goto err_out;
}
base = bpf_map__initial_value(skel->maps.arena, &arena_sz);
if (!ASSERT_OK_PTR(base, "arena_base"))
goto err_out;
*addr = (__u64 *)(base + getpagesize());
if (!ASSERT_EQ((unsigned long)skel->bss->ptr, (unsigned long)*addr,
"arena_ptr"))
goto err_out;
return skel;
err_out:
arena_race__destroy(skel);
return NULL;
}
static void test_realloc_during_free(bool deferred)
{
struct free_thread_ctx ctx = {};
struct arena_race *skel;
pthread_t thread;
__u64 *addr;
bool thread_created = false, zap_seen = false, completed = false;
int err;
if (!require_fentry_target(__func__, "zap_vma_range"))
return;
skel = setup_arena(&addr, true);
if (!skel)
return;
/* Map the page into user space, so that the free has to zap it. */
if (!ASSERT_EQ(READ_ONCE(*addr), skel->rodata->old_marker, "old_marker"))
goto out;
/* Pause the free after its TLB flush, before it releases the range. */
skel->bss->pause_on_zap = 1;
if (deferred) {
/*
* Test the nonsleepable free path that gets
* deferred to a worker in the kernel. We do
* so by triggering the arena operation from
* a nonsleepable tracepoint context.
*/
skel->bss->trigger_pid_tgid =
((__u64)getpid() << 32) | (__u32)sys_gettid();
skel->bss->trigger_syscall = SYS_getpgid;
skel->bss->deferred_free = 1;
if (!ASSERT_GE(syscall(SYS_getpgid, 0), 0, "deferred_free"))
goto release;
} else {
/*
* Test the sleepable arena free path through a
* syscall test prog.
*/
ctx.skel = skel;
err = pthread_create(&thread, NULL, run_free_thread, &ctx);
if (!ASSERT_OK(err, "pthread_create")) {
skel->bss->release = 1;
goto out;
}
thread_created = true;
}
/* Wait until the free reaches the user zap. */
err = wait_for(&skel->bss->zap_entered);
if (!ASSERT_OK(err, "zap_entered"))
goto release;
zap_seen = true;
/* The range must stay allocated until the free completes. */
run_prog(skel->progs.try_realloc, "realloc_during_free");
ASSERT_NULL(skel->bss->realloc_ptr, "realloc_during_free");
release:
skel->bss->release = 1;
if (thread_created) {
ASSERT_OK(pthread_join(thread, NULL), "pthread_join");
thread_created = false;
completed = ASSERT_OK(ctx.err, "free_run") &&
ASSERT_OK(ctx.retval, "free_retval");
} else if (skel->bss->target_tid) {
err = wait_for(&skel->bss->worker_exited);
completed = ASSERT_OK(err, "worker_exited");
}
ASSERT_FALSE(skel->bss->timed_out, "zap_timed_out");
if (zap_seen && completed &&
!run_prog(skel->progs.try_realloc, "realloc_after_free")) {
ASSERT_EQ((unsigned long)skel->bss->realloc_ptr,
(unsigned long)addr, "realloc_after_free");
ASSERT_EQ(*addr, skel->rodata->new_marker, "new_marker");
}
out:
arena_race__destroy(skel);
}
/*
* Force a race between a faulting thread in userspace and a
* free operation on the arena.
*/
static void test_fault_free_realloc(void)
{
struct sigaction sigbus = { .sa_handler = reader_sigbus }, old_sigbus;
struct fault_thread_ctx fault = {};
struct arena_race *skel;
pthread_t fault_thread;
bool fault_created = false;
__u64 *addr;
__u64 expected, value;
int err, i;
skel = setup_arena(&addr, false);
if (!skel)
return;
skel->bss->realloc_after_free = 1;
if (!ASSERT_OK(sigaction(SIGBUS, &sigbus, &old_sigbus), "sigaction_sigbus"))
goto out_destroy;
fault.addr = addr;
err = pthread_create(&fault_thread, NULL, fault_reader_thread, &fault);
if (!ASSERT_OK(err, "pthread_create_fault"))
goto out;
fault_created = true;
for (i = 0; i < 1000; i++) {
LIBBPF_OPTS(bpf_test_run_opts, opts);
/* Keep deferred frees from earlier iterations on separate slots. */
if (i) {
skel->bss->page_idx = i;
if (run_prog(skel->progs.alloc_old, "alloc_next"))
goto out;
addr = (__u64 *)(unsigned long)skel->bss->ptr;
__atomic_store_n(&fault.addr, addr, __ATOMIC_RELEASE);
}
err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.free_page),
&opts);
if (err) {
ASSERT_OK(err, "free_realloc");
break;
}
if (opts.retval) {
ASSERT_OK(opts.retval, "free_realloc");
break;
}
/* The reader may populate the slot first, or free may defer. */
if (!skel->bss->realloc_ptr)
continue;
value = *addr;
expected = skel->bss->current_marker;
if (value != expected) {
ASSERT_EQ(value, expected, "marker_after_realloc");
break;
}
}
WRITE_ONCE(fault.stop, 1);
if (fault_created) {
ASSERT_OK(pthread_join(fault_thread, NULL), "pthread_join_fault");
fault_created = false;
}
out:
WRITE_ONCE(fault.stop, 1);
if (fault_created)
pthread_join(fault_thread, NULL);
sigaction(SIGBUS, &old_sigbus, NULL);
out_destroy:
arena_race__destroy(skel);
}
void serial_test_arena_race(void)
{
cpu_set_t cpuset;
int err;
err = sched_getaffinity(0, sizeof(cpuset), &cpuset);
if (!ASSERT_OK(err, "sched_getaffinity"))
return;
if (CPU_COUNT(&cpuset) < 2) {
printf("%s:SKIP: at least two runnable CPUs are required\n", __func__);
test__skip();
return;
}
if (test__start_subtest("realloc_during_free"))
test_realloc_during_free(false);
if (test__start_subtest("realloc_during_deferred_free"))
test_realloc_during_free(true);
if (test__start_subtest("fault_free_realloc"))
test_fault_free_realloc();
}

166
tools/testing/selftests/bpf/progs/arena_race.c Normal file
View File

@ -0,0 +1,166 @@
// SPDX-License-Identifier: GPL-2.0
#define BPF_NO_KFUNC_PROTOTYPES
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "bpf_experimental.h"
#include <bpf_arena_common.h>
const volatile __u64 old_marker = 0x1111222233334444ULL;
const volatile __u64 new_marker = 0x5555666677778888ULL;
struct {
__uint(type, BPF_MAP_TYPE_ARENA);
__uint(map_flags, BPF_F_MMAPABLE);
__uint(max_entries, 1024);
#ifdef __TARGET_ARCH_arm64
__ulong(map_extra, 0x1ull << 32);
#else
__ulong(map_extra, 0x1ull << 44);
#endif
} arena SEC(".maps");
bool skip;
int page_idx;
void __arena *ptr;
void __arena *realloc_ptr;
bool realloc_after_free;
__u64 current_marker;
__u64 marker_seq;
int target_tid;
int pause_on_zap;
int zap_entered;
int release;
int timed_out;
__u64 trigger_pid_tgid;
long trigger_syscall;
int deferred_free;
int worker_armed;
int worker_exited;
static __always_inline void wait_for_release(void)
{
while (!*(volatile int *)&release && can_loop)
;
if (!*(volatile int *)&release)
timed_out = 1;
}
SEC("syscall")
int alloc_old(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) || defined(BPF_ARENA_FORCE_ASM)
__u64 __arena *p;
char __arena *base = arena_base(&arena);
realloc_ptr = NULL;
ptr = bpf_arena_alloc_pages(&arena, base + (page_idx + 1) * __PAGE_SIZE, 1,
NUMA_NO_NODE, 0);
if (!ptr)
return 1;
p = (__u64 __arena *)ptr;
*p = old_marker;
current_marker = old_marker;
#else
skip = true;
#endif
return 0;
}
SEC("syscall")
int free_page(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) || defined(BPF_ARENA_FORCE_ASM)
__u64 __arena *p;
__u64 marker;
if (!ptr)
return 1;
bpf_arena_free_pages(&arena, ptr, 1);
if (!realloc_after_free)
return 0;
marker = new_marker + ++marker_seq;
realloc_ptr = bpf_arena_alloc_pages(&arena, ptr, 1, NUMA_NO_NODE, 0);
if (!realloc_ptr)
return 0;
ptr = realloc_ptr;
p = (__u64 __arena *)ptr;
*p = marker;
current_marker = marker;
#endif
return 0;
}
SEC("syscall")
int try_realloc(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) || defined(BPF_ARENA_FORCE_ASM)
__u64 __arena *p;
realloc_ptr = bpf_arena_alloc_pages(&arena, ptr, 1, NUMA_NO_NODE, 0);
if (realloc_ptr) {
ptr = realloc_ptr;
p = (__u64 __arena *)realloc_ptr;
*p = new_marker;
current_marker = new_marker;
}
#endif
return 0;
}
SEC("tp_btf/sys_enter")
int BPF_PROG(deferred_free_prog, struct pt_regs *regs, long id)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) || defined(BPF_ARENA_FORCE_ASM)
if (!deferred_free || bpf_get_current_pid_tgid() != trigger_pid_tgid ||
id != trigger_syscall)
return 0;
deferred_free = 0;
/* The worker can run on another CPU before the kfunc returns. */
worker_armed = 1;
bpf_arena_free_pages(&arena, ptr, 1);
#endif
return 0;
}
SEC("fentry/arena_free_worker")
int BPF_PROG(trace_free_worker, struct work_struct *work)
{
if (worker_armed && !target_tid)
target_tid = (__u32)bpf_get_current_pid_tgid();
return 0;
}
SEC("fexit/arena_free_worker")
int BPF_PROG(trace_free_worker_ret, struct work_struct *work)
{
if ((__u32)bpf_get_current_pid_tgid() == target_tid)
worker_exited = 1;
return 0;
}
/*
* Pause a free while it zaps user mappings: after it cleared the kernel PTEs
* and flushed the TLB, but before it releases the range. zap_vma_range() is
* generic MM code, so it is an fentry target even where
* flush_tlb_kernel_range() is inline (arm64).
*/
SEC("?fentry/zap_vma_range")
int BPF_PROG(trace_zap, struct vm_area_struct *vma, unsigned long address,
unsigned long size)
{
if ((__u32)bpf_get_current_pid_tgid() != target_tid)
return 0;
zap_entered = 1;
if (pause_on_zap)
wait_for_release();
return 0;
}
char _license[] SEC("license") = "GPL";