linux/kernel
Steven Rostedt 77271ce4b2 tracing: Add irq, preempt-count and need resched info to default trace output
People keep asking how to get the preempt count, irq, and need resched info
and we keep telling them to enable the latency format. Some developers think
that traces without this info is completely useless, and for a lot of tasks
it is useless.

The first option was to enable the latency trace as the default format, but
the header for the latency format is pretty useless for most tracers and
it also does the timestamp in straight microseconds from the time the trace
started. This is sometimes more difficult to read as the default trace is
seconds from the start of boot up.

Latency format:

 # tracer: nop
 #
 # nop latency trace v1.1.5 on 3.2.0-rc1-test+
 # --------------------------------------------------------------------
 # latency: 0 us, #159771/64234230, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
 #    -----------------
 #    | task: -0 (uid:0 nice:0 policy:0 rt_prio:0)
 #    -----------------
 #
 #                  _------=> CPU#
 #                 / _-----=> irqs-off
 #                | / _----=> need-resched
 #                || / _---=> hardirq/softirq
 #                ||| / _--=> preempt-depth
 #                |||| /     delay
 #  cmd     pid   ||||| time  |   caller
 #     \   /      |||||  \    |   /
 migratio-6       0...2 41778231us+: rcu_note_context_switch <-__schedule
 migratio-6       0...2 41778233us : trace_rcu_utilization <-rcu_note_context_switch
 migratio-6       0...2 41778235us+: rcu_sched_qs <-rcu_note_context_switch
 migratio-6       0d..2 41778236us+: rcu_preempt_qs <-rcu_note_context_switch
 migratio-6       0...2 41778238us : trace_rcu_utilization <-rcu_note_context_switch
 migratio-6       0...2 41778239us+: debug_lockdep_rcu_enabled <-__schedule

default format:

 # tracer: nop
 #
 #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
 #              | |       |          |         |
      migration/0-6     [000]    50.025810: rcu_note_context_switch <-__schedule
      migration/0-6     [000]    50.025812: trace_rcu_utilization <-rcu_note_context_switch
      migration/0-6     [000]    50.025813: rcu_sched_qs <-rcu_note_context_switch
      migration/0-6     [000]    50.025815: rcu_preempt_qs <-rcu_note_context_switch
      migration/0-6     [000]    50.025817: trace_rcu_utilization <-rcu_note_context_switch
      migration/0-6     [000]    50.025818: debug_lockdep_rcu_enabled <-__schedule
      migration/0-6     [000]    50.025820: debug_lockdep_rcu_enabled <-__schedule

The latency format header has latency information that is pretty meaningless
for most tracers. Although some of the header is useful, and we can add that
later to the default format as well.

What is really useful with the latency format is the irqs-off, need-resched
hard/softirq context and the preempt count.

This commit adds the option irq-info which is on by default that adds this
information:

 # tracer: nop
 #
 #                              _-----=> irqs-off
 #                             / _----=> need-resched
 #                            | / _---=> hardirq/softirq
 #                            || / _--=> preempt-depth
 #                            ||| /     delay
 #           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
 #              | |       |   ||||       |         |
           <idle>-0     [000] d..2    49.309305: cpuidle_get_driver <-cpuidle_idle_call
           <idle>-0     [000] d..2    49.309307: mwait_idle <-cpu_idle
           <idle>-0     [000] d..2    49.309309: need_resched <-mwait_idle
           <idle>-0     [000] d..2    49.309310: test_ti_thread_flag <-need_resched
           <idle>-0     [000] d..2    49.309312: trace_power_start.constprop.13 <-mwait_idle
           <idle>-0     [000] d..2    49.309313: trace_cpu_idle <-mwait_idle
           <idle>-0     [000] d..2    49.309315: need_resched <-mwait_idle

If a user wants the old format, they can disable the 'irq-info' option:

 # tracer: nop
 #
 #           TASK-PID   CPU#      TIMESTAMP  FUNCTION
 #              | |       |          |         |
           <idle>-0     [000]     49.309305: cpuidle_get_driver <-cpuidle_idle_call
           <idle>-0     [000]     49.309307: mwait_idle <-cpu_idle
           <idle>-0     [000]     49.309309: need_resched <-mwait_idle
           <idle>-0     [000]     49.309310: test_ti_thread_flag <-need_resched
           <idle>-0     [000]     49.309312: trace_power_start.constprop.13 <-mwait_idle
           <idle>-0     [000]     49.309313: trace_cpu_idle <-mwait_idle
           <idle>-0     [000]     49.309315: need_resched <-mwait_idle

Requested-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-11-17 09:58:48 -05:00
..
debug Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
events events: Don't divide events if it has field period 2011-11-14 13:31:28 +01:00
gcov gcov: disable CONSTRUCTORS for UML 2011-07-26 16:49:45 -07:00
irq Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
power PM / QoS: Set cpu_dma_pm_qos->name 2011-11-07 23:02:24 +01:00
time kernel: Fix files explicitly needing EXPORT_SYMBOL infrastructure 2011-10-31 19:30:05 -04:00
trace tracing: Add irq, preempt-count and need resched info to default trace output 2011-11-17 09:58:48 -05:00
.gitignore
acct.c
async.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
audit_tree.c audit_tree,rcu: Convert call_rcu(__put_tree) to kfree_rcu() 2011-07-20 14:10:11 -07:00
audit_watch.c kill path_lookup() 2011-03-14 09:15:23 -04:00
audit.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
audit.h
auditfilter.c netlink: kill loginuid/sessionid/sid members from struct netlink_skb_parms 2011-03-03 10:55:40 -08:00
auditsc.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
backtracetest.c
bounds.c memcg: remove direct page_cgroup-to-page pointer 2011-03-23 19:46:28 -07:00
capability.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
cgroup_freezer.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
cgroup.c memcg: replace ss->id_lock with a rwlock 2011-11-02 16:07:03 -07:00
compat.c kernel: Fix files explicitly needing EXPORT_SYMBOL infrastructure 2011-10-31 19:30:05 -04:00
configs.c kernel/configs.c: include MODULE_*() when CONFIG_IKCONFIG_PROC=n 2011-07-25 20:57:15 -07:00
cpu_pm.c cpu_pm: call notifiers during suspend 2011-09-23 12:05:29 +05:30
cpu.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
cpuset.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
crash_dump.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
cred.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
delayacct.c KVM: Steal time implementation 2011-07-14 12:59:14 +03:00
dma.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
elfcore.c
exec_domain.c
exit.c oom: remove oom_disable_count 2011-10-31 17:30:45 -07:00
extable.c extable, core_kernel_data(): Make sure all archs define _sdata 2011-05-20 08:56:56 +02:00
fork.c Merge branch 'writeback-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/wfg/linux 2011-11-06 19:02:23 -08:00
freezer.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
futex_compat.c userns: user namespaces: convert several capable() calls 2011-03-23 19:47:08 -07:00
futex.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
groups.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
hrtimer.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
hung_task.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
irq_work.c kernel: fix two implicit header assumptions in irq_work.c 2011-10-31 09:20:12 -04:00
itimer.c
jump_label.c Merge branch 'tip/perf/core' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace into perf/core 2011-11-11 08:19:37 +01:00
kallsyms.c Merge branch 'core-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip 2011-03-25 17:52:22 -07:00
Kconfig.freezer
Kconfig.hz
Kconfig.locks arch:Kconfig.locks Remove unused config option. 2011-04-10 17:01:05 +02:00
Kconfig.preempt sched: Isolate preempt counting in its own config option 2011-06-10 15:15:40 +02:00
kexec.c [S390] kdump: Add infrastructure for unmapping crashkernel memory 2011-10-30 15:16:42 +01:00
kfifo.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
kmod.c kmod: prevent kmod_loop_msg overflow in __request_module() 2011-10-26 13:10:39 +10:30
kprobes.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
ksysfs.c kernel: ksysfs.c is implicitly using stat.h 2011-10-31 09:20:13 -04:00
kthread.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
latencytop.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
lockdep_internals.h
lockdep_proc.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
lockdep_states.h
lockdep.c Merge branch 'tip/perf/core' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace into perf/core 2011-11-11 08:19:37 +01:00
Makefile Merge branch 'devel-stable' of http://ftp.arm.linux.org.uk/pub/linux/arm/kernel/git-cur/linux-2.6-arm 2011-10-28 12:02:27 -07:00
module.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
mutex-debug.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
mutex-debug.h mutex: Use p->on_cpu for the adaptive spin 2011-04-14 08:52:33 +02:00
mutex.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
mutex.h mutex: Use p->on_cpu for the adaptive spin 2011-04-14 08:52:33 +02:00
notifier.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
nsproxy.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
padata.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
panic.c module,bug: Add TAINT_OOT_MODULE flag for modules not built in-tree 2011-11-07 07:54:42 +10:30
params.c kernel: params.c needs module.h not moduleparam.h 2011-10-31 09:20:13 -04:00
pid_namespace.c pidns: call pid_ns_prepare_proc() from create_pid_namespace() 2011-03-23 19:46:58 -07:00
pid.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
posix-cpu-timers.c Merge branch 'core-locking-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2011-10-26 16:17:32 +02:00
posix-timers.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
printk.c printk: remove bounds checking for log_prefix 2011-10-31 17:30:53 -07:00
profile.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
ptrace.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
range.c range: fix bogus misuse of module.h to get printk() 2011-10-31 09:20:11 -04:00
rcu.h rcu: Add grace-period, quiescent-state, and call_rcu trace events 2011-09-28 21:38:21 -07:00
rcupdate.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
rcutiny_plugin.h kernel: fix up module header handling in rcutiny files 2011-10-31 09:20:13 -04:00
rcutiny.c kernel: fix up module header handling in rcutiny files 2011-10-31 09:20:13 -04:00
rcutorture.c rcu: Make rcu_torture_boost() exit loops at end of test 2011-09-28 21:38:46 -07:00
rcutree_plugin.h rcu: Remove rcu_needs_cpu_flush() to avoid false quiescent states 2011-09-28 21:38:48 -07:00
rcutree_trace.c rcu: Simplify quiescent-state accounting 2011-09-28 21:38:22 -07:00
rcutree.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
rcutree.h rcu: Remove rcu_needs_cpu_flush() to avoid false quiescent states 2011-09-28 21:38:48 -07:00
relay.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
res_counter.c memcg: res_counter_read_u64(): fix potential races on 32-bit machines 2011-03-23 19:46:22 -07:00
resource.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
rtmutex_common.h
rtmutex-debug.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
rtmutex-debug.h
rtmutex-tester.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
rtmutex.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
rtmutex.h
rwsem.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
sched_autogroup.c Fix common misspellings 2011-03-31 11:26:23 -03:00
sched_autogroup.h sched: Skip autogroup when looking for all rt sched groups 2011-07-01 10:39:08 +02:00
sched_clock.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
sched_cpupri.c sched/cpupri: Remove cpupri->pri_active 2011-08-14 12:01:11 +02:00
sched_cpupri.h sched/cpupri: Remove cpupri->pri_active 2011-08-14 12:01:11 +02:00
sched_debug.c sched: Get rid of lock_depth 2011-04-24 13:18:38 +02:00
sched_fair.c sched: Wrap scheduler p->cpus_allowed access 2011-10-06 12:46:56 +02:00
sched_features.h sched: Kill WAKEUP_PREEMPT 2011-08-14 12:00:41 +02:00
sched_idletask.c sched: Drop the rq argument to sched_class::select_task_rq() 2011-04-14 08:52:36 +02:00
sched_rt.c sched: Warn on rt throttling 2011-10-06 12:47:04 +02:00
sched_stats.h locking, sched: Annotate thread_group_cputimer as raw 2011-09-13 11:11:55 +02:00
sched_stoptask.c sched: Implement hierarchical task accounting for SCHED_OTHER 2011-08-14 12:01:13 +02:00
sched.c Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc 2011-11-06 17:12:03 -08:00
seccomp.c
semaphore.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
signal.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
smp.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
softirq.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
spinlock.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
srcu.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
stacktrace.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
stop_machine.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
sys_ni.c Cross Memory Attach 2011-10-31 17:30:44 -07:00
sys.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
sysctl_binary.c ipv4: NET_IPV4_ROUTE_GC_INTERVAL removal 2011-10-03 14:13:01 -04:00
sysctl_check.c xfs: remove subdirectories 2011-08-12 16:21:35 -05:00
sysctl.c Merge branch 'akpm' (Andrew's incoming) 2011-10-31 17:46:07 -07:00
taskstats.c Make TASKSTATS require root access 2011-09-19 17:04:37 -07:00
test_kprobes.c
time.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
timeconst.pl
timer.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
tracepoint.c Tracepoint: Dissociate from module mutex 2011-08-10 20:38:14 -04:00
tsacct.c Make taskstats round statistics down to nearest 1k bytes/events 2011-09-19 17:10:57 -07:00
uid16.c userns: user namespaces: convert several capable() calls 2011-03-23 19:47:08 -07:00
up.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
user_namespace.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
user-return-notifier.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
user.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
utsname_sysctl.c Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux 2011-11-06 19:44:47 -08:00
utsname.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
wait.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00
watchdog.c watchdog: move watchdog_*_all_cpus under CONFIG_SYSCTL 2011-10-31 17:30:53 -07:00
workqueue_sched.h
workqueue.c kernel: Map most files to use export.h instead of module.h 2011-10-31 09:20:12 -04:00