github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-10-04 18:29:02 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph
afd876bbdc
linux/tools/perf/tests/shell
History
Adrian Hunter e0c48bf9e8 perf scripts python: Add a script to run instances of 'perf script' in parallel 
Add a Python script to run a perf script command multiple times in
parallel, using perf script options --cpu and --time so that each job
processes a different chunk of the data.

Extend perf script tests to test also the new script.

The script supports the use of normal 'perf script' options like
--dlfilter and --script, so that the benefit of running parallel jobs
naturally extends to them also. In addition, a command can be provided
(refer --pipe-to option) to pipe standard output to a custom command.

Refer to the script's own help text at the end of the patch for more
details.

The script is useful for Intel PT traces, that can be efficiently
decoded by 'perf script' when split by CPU and/or time ranges. Running
jobs in parallel can decrease the overall decoding time.

Committer testing:

  Ian reported that shellcheck found some issues, I installed it as there
  are no warnings about it not being available, but when available it
  fails the build with:

    TEST    /tmp/build/perf-tools-next/tests/shell/script.sh.shellcheck_log
    CC      /tmp/build/perf-tools-next/util/header.o

  In tests/shell/script.sh line 20:
                  rm -rf "${temp_dir}/"*
                         ^-------------^ SC2115 (warning): Use "${var:?}" to ensure this never expands to /* .

  In tests/shell/script.sh line 83:
          output1_dir="${temp_dir}/output1"
          ^---------^ SC2034 (warning): output1_dir appears unused. Verify use (or export if used externally).

  In tests/shell/script.sh line 84:
          output2_dir="${temp_dir}/output2"
          ^---------^ SC2034 (warning): output2_dir appears unused. Verify use (or export if used externally).

  In tests/shell/script.sh line 86:
          python3 "${pp}" -o "${output_dir}" --jobs 4 --verbose -- perf script -i "${perf_data}"
                              ^-----------^ SC2154 (warning): output_dir is referenced but not assigned (did you mean 'output1_dir'?).

  For more information:
    https://www.shellcheck.net/wiki/SC2034 -- output1_dir appears unused. Verif...
    https://www.shellcheck.net/wiki/SC2115 -- Use "${var:?}" to ensure this nev...
    https://www.shellcheck.net/wiki/SC2154 -- output_dir is referenced but not ...

Did these fixes:

  -               rm -rf "${temp_dir}/"*
  +               rm -rf "${temp_dir:?}/"*

And:

   @@ -83,8 +83,8 @@ test_parallel_perf()
          output1_dir="${temp_dir}/output1"
          output2_dir="${temp_dir}/output2"
          perf record -o "${perf_data}" --sample-cpu uname
  -       python3 "${pp}" -o "${output_dir}" --jobs 4 --verbose -- perf script -i "${perf_data}"
  -       python3 "${pp}" -o "${output_dir}" --jobs 4 --verbose --per-cpu -- perf script -i "${perf_data}"
  +       python3 "${pp}" -o "${output1_dir}" --jobs 4 --verbose -- perf script -i "${perf_data}"
  +       python3 "${pp}" -o "${output2_dir}" --jobs 4 --verbose --per-cpu -- perf script -i "${perf_data}"

After that:

  root@number:~# perf test -vv "perf script tests"
   97: perf script tests:
  --- start ---
  test child forked, pid 4084139
  DB test
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.032 MB /tmp/perf-test-script.T4MJDr0L6J/perf.data (7 samples) ]
  <SNIP>
  DB test [Success]
  parallel-perf test
  Linux
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.034 MB /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data (7 samples) ]
  Starting: perf script --time=,91898.301878499 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --time=91898.301878500,91898.301905999 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --time=91898.301906000,91898.301933499 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --time=91898.301933500, -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --time=91898.301878500,91898.301905999 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --time=91898.301906000,91898.301933499 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 4 jobs: 2 completed, 2 running
  Finished: perf script --time=,91898.301878499 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --time=91898.301933500, -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 4 jobs: 4 completed, 0 running
  All jobs finished successfully
  parallel-perf.py done
  Starting: perf script --cpu=0 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=1 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=2 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=3 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=0 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=1 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=2 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=3 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 4 completed, 0 running
  Starting: perf script --cpu=4 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=5 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=6 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=7 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=4 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=5 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=6 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=7 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 8 completed, 0 running
  Starting: perf script --cpu=8 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=9 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=10 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=11 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=8 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=9 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=10 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=11 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 12 completed, 0 running
  Starting: perf script --cpu=12 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=13 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=14 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=15 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=12 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=13 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=14 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=15 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 16 completed, 0 running
  Starting: perf script --cpu=16 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=17 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=18 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=19 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=16 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=17 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=18 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=19 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 20 completed, 0 running
  Starting: perf script --cpu=20 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=21 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=22 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=23 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=20 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=21 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=22 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=23 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 24 completed, 0 running
  Starting: perf script --cpu=24 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=25 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=26 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Starting: perf script --cpu=27 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=25 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=26 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  Finished: perf script --cpu=27 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 27 completed, 1 running
  Finished: perf script --cpu=24 -i /tmp/perf-test-script.T4MJDr0L6J/pp-perf.data
  There are 28 jobs: 28 completed, 0 running
  All jobs finished successfully
  parallel-perf.py done
  parallel-perf test [Success]
  --- Cleaning up ---
  ---- end(0) ----
   97: perf script tests                                               : Ok
  root@number:~#

Reviewed-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/r/20240423133248.10206-1-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2024-04-26 22:07:19 -03:00
..
base_probe perf tests shell kprobes: Add missing description as used by 'perf test' output 2024-04-26 22:07:19 -03:00
common perf testsuite: Add common output checking helpers 2024-02-16 11:49:36 -08:00
coresight perf tests coresight: Remove unused variables 2023-11-27 11:35:43 -03:00
lib perf test: Stat output per thread of just the parent process 2024-03-21 13:54:39 -03:00
buildid.sh perf tests shell: Fixed shellcheck warnings 2023-06-13 23:40:34 -03:00
daemon.sh perf test shell daemon: Make signal test less racy 2024-01-26 10:51:49 -03:00
diff.sh perf test shell diff: Skip test if test_loop symbol is missing in the perf binary 2023-12-06 13:01:36 -03:00
list.sh perf test: Workaround debug output in list test 2024-01-26 10:51:49 -03:00
lock_contention.sh perf tests: Ignore shellcheck warning in lock_contention 2023-10-17 12:40:51 -07:00
perftool-testsuite_probe.sh perf testsuite: Add test case for perf probe 2024-02-16 11:49:22 -08:00
pipe_test.sh perf tests: Skip pipe test if noploop symbol is missing 2023-11-27 15:40:00 -03:00
probe_vfs_getname.sh tests/shell: Fix shellcheck SC1090 to handle the location of sourced files 2023-09-26 21:47:12 -07:00
record_bpf_filter.sh perf test shell record_bpf_filter: Skip 6.2 kernel 2023-08-29 14:16:14 -03:00
record_offcpu.sh perf tests offcpu: Adjust test case perf record offcpu profiling tests for s390 2023-11-09 13:49:33 -03:00
record_sideband.sh perf tests: Fix shellcheck warning in record_sideband.sh 2023-10-17 12:40:51 -07:00
record.sh perf tests: Skip record test if test_loop symbol is missing 2023-11-27 15:40:16 -03:00
record+probe_libc_inet_pton.sh perf test record+probe_libc_inet_pton: Fix call chain match on powerpc 2023-11-29 17:59:36 -03:00
record+script_probe_vfs_getname.sh tests/shell: Fix shellcheck SC1090 to handle the location of sourced files 2023-09-26 21:47:12 -07:00
record+zstd_comp_decomp.sh perf tests record+zstd_comp_decomp: Fix the shellcheck warnings about word splitting/quoting 2023-08-03 17:01:27 -03:00
script.sh perf scripts python: Add a script to run instances of 'perf script' in parallel 2024-04-26 22:07:19 -03:00
stat_all_metricgroups.sh perf tests: Fix shellcheck warning in stat_all_metricgroups 2023-10-17 12:40:51 -07:00
stat_all_metrics.sh perf tests stat_all_metrics: Fix shellcheck warning SC2076 2023-06-13 23:40:33 -03:00
stat_all_pfm.sh perf test: Add test of libpfm4 events 2023-06-12 15:57:53 -03:00
stat_all_pmu.sh perf tools: Fix spelling mistake "parametrized" -> "parameterized" 2023-11-09 13:49:32 -03:00
stat_bpf_counters_cgrp.sh perf test stat_bpf_counters_cgrp: Enhance perf stat cgroup BPF counter test 2023-08-29 14:16:15 -03:00
stat_bpf_counters.sh perf test bpf-counters: Add test for BPF event modifier 2024-04-18 22:22:51 -03:00
stat_metrics_values.sh perf test: Simplify metric value validation test final report 2024-02-01 22:16:37 -08:00
stat.sh perf test: Add cputype testing to perf stat 2023-05-15 09:18:43 -03:00
stat+csv_output.sh perf stat: Support per-cluster aggregation 2024-02-09 14:59:53 -08:00
stat+csv_summary.sh tests/shell: Fix shellcheck issues in tests/shell/stat+shadow_stat.sh tetscase 2023-09-26 21:47:12 -07:00
stat+json_output.sh perf test: Stat output per thread of just the parent process 2024-03-21 13:54:39 -03:00
stat+shadow_stat.sh tests/shell: Fix shellcheck issues in tests/shell/stat+shadow_stat.sh tetscase 2023-09-26 21:47:12 -07:00
stat+std_output.sh perf stat: Support per-cluster aggregation 2024-02-09 14:59:53 -08:00
test_arm_callgraph_fp.sh perf tests: Remove dependency on lscpu 2024-04-12 12:02:06 -03:00
test_arm_coresight.sh perf test shell arm_coresight: Increase buffer size for Coresight basic tests 2024-04-18 22:22:51 -03:00
test_arm_spe_fork.sh perf tests: Address signal case issues detected via shellcheck 2023-08-03 17:01:25 -03:00
test_arm_spe.sh perf tests test_arm_spe: Address shellcheck warnings about signal name case 2023-06-13 23:40:33 -03:00
test_brstack.sh perf tests: Skip branch stack sampling test if brstack_bench symbol is missing 2023-11-27 15:40:22 -03:00
test_data_symbol.sh perf tests: Skip data symbol test if buf1 symbol is missing 2023-11-27 15:40:28 -03:00
test_intel_pt.sh tests/shell: Fix shellcheck SC1090 to handle the location of sourced files 2023-09-26 21:47:12 -07:00
test_java_symbol.sh perf test java symbol: Remove needless debuginfod queries 2023-05-10 12:54:53 -03:00
test_perf_data_converter_json.sh perf test: Use common python setup library 2023-12-05 15:48:52 -03:00
test_task_analyzer.sh perf tests task_analyzer: Check perf build options for libtraceevent support 2023-08-03 17:01:25 -03:00
test_uprobe_from_different_cu.sh perf test uprobe_from_different_cu: Skip if there is no gcc 2023-07-28 15:31:21 -03:00
trace+probe_vfs_getname.sh tests/shell: Fix shellcheck SC1090 to handle the location of sourced files 2023-09-26 21:47:12 -07:00