linux/kernel/trace
Steven Rostedt (Google) 899ac41804 ring-buffer: Fix slowpath of interrupted event
commit b803d7c664 upstream.

To synchronize the timestamps with the ring buffer reservation, there are
two timestamps that are saved in the buffer meta data.

1. before_stamp
2. write_stamp

When the two are equal, the write_stamp is considered valid, as in, it may
be used to calculate the delta of the next event as the write_stamp is the
timestamp of the previous reserved event on the buffer.

This is done by the following:

 /*A*/	w = current position on the ring buffer
	before = before_stamp
	after = write_stamp
	ts = read current timestamp

	if (before != after) {
		write_stamp is not valid, force adding an absolute
		timestamp.
	}

 /*B*/	before_stamp = ts

 /*C*/	write = local_add_return(event length, position on ring buffer)

	if (w == write - event length) {
		/* Nothing interrupted between A and C */
 /*E*/		write_stamp = ts;
		delta = ts - after
		/*
		 * If nothing interrupted again,
		 * before_stamp == write_stamp and write_stamp
		 * can be used to calculate the delta for
		 * events that come in after this one.
		 */
	} else {

		/*
		 * The slow path!
		 * Was interrupted between A and C.
		 */

This is the place that there's a bug. We currently have:

		after = write_stamp
		ts = read current timestamp

 /*F*/		if (write == current position on the ring buffer &&
		    after < ts && cmpxchg(write_stamp, after, ts)) {

			delta = ts - after;

		} else {
			delta = 0;
		}

The assumption is that if the current position on the ring buffer hasn't
moved between C and F, then it also was not interrupted, and that the last
event written has a timestamp that matches the write_stamp. That is the
write_stamp is valid.

But this may not be the case:

If a task context event was interrupted by softirq between B and C.

And the softirq wrote an event that got interrupted by a hard irq between
C and E.

and the hard irq wrote an event (does not need to be interrupted)

We have:

 /*B*/ before_stamp = ts of normal context

   ---> interrupted by softirq

	/*B*/ before_stamp = ts of softirq context

	  ---> interrupted by hardirq

		/*B*/ before_stamp = ts of hard irq context
		/*E*/ write_stamp = ts of hard irq context

		/* matches and write_stamp valid */
	  <----

	/*E*/ write_stamp = ts of softirq context

	/* No longer matches before_stamp, write_stamp is not valid! */

   <---

 w != write - length, go to slow path

// Right now the order of events in the ring buffer is:
//
// |-- softirq event --|-- hard irq event --|-- normal context event --|
//

 after = write_stamp (this is the ts of softirq)
 ts = read current timestamp

 if (write == current position on the ring buffer [true] &&
     after < ts [true] && cmpxchg(write_stamp, after, ts) [true]) {

	delta = ts - after  [Wrong!]

The delta is to be between the hard irq event and the normal context
event, but the above logic made the delta between the softirq event and
the normal context event, where the hard irq event is between the two. This
will shift all the remaining event timestamps on the sub-buffer
incorrectly.

The write_stamp is only valid if it matches the before_stamp. The cmpxchg
does nothing to help this.

Instead, the following logic can be done to fix this:

	before = before_stamp
	ts = read current timestamp
	before_stamp = ts

	after = write_stamp

	if (write == current position on the ring buffer &&
	    after == before && after < ts) {

		delta = ts - after

	} else {
		delta = 0;
	}

The above will only use the write_stamp if it still matches before_stamp
and was tested to not have changed since C.

As a bonus, with this logic we do not need any 64-bit cmpxchg() at all!

This means the 32-bit rb_time_t workaround can finally be removed. But
that's for a later time.

Link: https://lore.kernel.org/linux-trace-kernel/20231218175229.58ec3daf@gandalf.local.home/
Link: https://lore.kernel.org/linux-trace-kernel/20231218230712.3a76b081@gandalf.local.home

Cc: stable@vger.kernel.org
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Fixes: dd93942570 ("ring-buffer: Do not try to put back write_stamp")
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-01-05 15:18:40 +01:00
..
rv
blktrace.c trace/blktrace: fix memory leak with using debugfs_lookup() 2023-03-10 09:33:46 +01:00
bpf_trace.c fprobe: Pass entry_data to handlers 2023-10-25 12:03:12 +02:00
bpf_trace.h
error_report-traces.c
fgraph.c
fprobe.c fprobe: Fix to ensure the number of active retprobes is not zero 2023-10-25 12:03:12 +02:00
ftrace_internal.h
ftrace.c ftrace: Fix possible warning on checking all pages used in ftrace_process_locs() 2023-07-23 13:49:44 +02:00
Kconfig
kprobe_event_gen_test.c tracing: Fix wrong return in kprobe_event_gen_test.c 2023-04-06 12:10:41 +02:00
Makefile
pid_list.c
pid_list.h
power-traces.c
preemptirq_delay_test.c
rethook.c rethook: Use __rcu pointer for rethook::handler 2023-12-13 18:39:17 +01:00
ring_buffer_benchmark.c
ring_buffer.c ring-buffer: Fix slowpath of interrupted event 2024-01-05 15:18:40 +01:00
rpm-traces.c
synth_event_gen_test.c tracing / synthetic: Disable events after testing in synth_event_gen_test_init() 2024-01-01 12:39:07 +00:00
trace_benchmark.c
trace_benchmark.h
trace_boot.c
trace_branch.c
trace_clock.c
trace_dynevent.c
trace_dynevent.h
trace_entries.h
trace_eprobe.c kernel/trace: Fix cleanup logic of enable_trace_eprobe 2023-07-23 13:49:24 +02:00
trace_event_perf.c
trace_events_filter_test.h
trace_events_filter.c tracing: Have trace_event_file have ref counters 2023-11-28 17:07:23 +00:00
trace_events_hist.c tracing: Allow synthetic events to pass around stacktraces 2023-08-03 10:23:55 +02:00
trace_events_inject.c tracing: Have event inject files inc the trace array ref count 2023-09-23 11:11:11 +02:00
trace_events_synth.c tracing: Have the user copy of synthetic event address use correct context 2023-11-28 17:07:13 +00:00
trace_events_trigger.c
trace_events_user.c tracing/user_events: Fix struct arg size match check 2023-07-23 13:49:47 +02:00
trace_events.c tracing: Have trace_event_file have ref counters 2023-11-28 17:07:23 +00:00
trace_export.c tracing: Fix TASK_COMM_LEN in trace event format file 2023-02-14 19:11:54 +01:00
trace_functions_graph.c
trace_functions.c
trace_hwlat.c tracing: Remove extra space at the end of hwlat_detector/mode 2023-09-13 09:42:57 +02:00
trace_irqsoff.c tracing: Fix memleak due to race between current_tracer and trace 2023-08-30 16:11:00 +02:00
trace_kdb.c
trace_kprobe_selftest.c
trace_kprobe_selftest.h
trace_kprobe.c tracing/kprobes: Fix the order of argument descriptions 2023-11-20 11:52:18 +01:00
trace_mmiotrace.c
trace_nop.c
trace_osnoise.c tracing/timerlat: Always wakeup the timerlat thread 2023-06-09 10:34:24 +02:00
trace_output.c
trace_output.h
trace_preemptirq.c
trace_printk.c
trace_probe_kernel.h tracing/probes: Fix to record 0-length data_loc in fetch_store_string*() if fails 2023-08-03 10:23:55 +02:00
trace_probe_tmpl.h tracing/probes: Fix to record 0-length data_loc in fetch_store_string*() if fails 2023-08-03 10:23:55 +02:00
trace_probe.c Revert "tracing: Add "(fault)" name injection to kernel probes" 2023-08-03 10:23:55 +02:00
trace_probe.h tracing/kprobes: Return EADDRNOTAVAIL when func matches several symbols 2023-10-25 12:03:16 +02:00
trace_recursion_record.c
trace_sched_switch.c
trace_sched_wakeup.c tracing: Fix memleak due to race between current_tracer and trace 2023-08-30 16:11:00 +02:00
trace_selftest_dynamic.c
trace_selftest.c
trace_seq.c
trace_stack.c
trace_stat.c
trace_stat.h
trace_synth.h tracing: Allow synthetic events to pass around stacktraces 2023-08-03 10:23:55 +02:00
trace_syscalls.c
trace_uprobe.c bpf: Clear the probe_addr for uprobe 2023-09-13 09:42:30 +02:00
trace.c tracing: Fix blocked reader of snapshot buffer 2024-01-05 15:18:39 +01:00
trace.h tracing: Have trace_event_file have ref counters 2023-11-28 17:07:23 +00:00
tracing_map.c
tracing_map.h