Machines with a larger number of CPUs and under heavy load sometimes
loose PAI counter increments during recording using events
-e CRYPTO_ÂLL or -e NNPA_ALL. Counting is not affected.
This happens when several PAI crypto counters are incremented during
the same cryptographic operation.
During schedule out the functions
paiXXX_sched_task() (with XXX either crypt or ext)
+--> pai_have_samples()
+--> pai_have_sample()
+--> pai_copy()
+--> pai_push_sample()
are called to read out PAI counter values.
In pai_copy() the current values of PAI counters are read from the
PMU memory mapped page and compared to the values read during last
schedule out operation, which have been saved in a backup page
named PAI_SAVE_AREA(event). For each PAI counter a delta is calculated
and when the delta is positive, that PAI counter was incremented by
hardware. This positve delta is reported as raw data record attached
to a sample.
After all deltas have been calculated, the new PAI counter values
are saved in the backup page PAI_SAVE_AREA(event). However this is
done in pai_push_sample(), leaving a small window for missing hardware
triggered updates. Here is one scenario:
PAI counter idx: 0 1 2 3 4 5 6 7 .... N
+---+---+---+---+---+---+---+---+ +---+
PAI counter page:| | | X | | | | | |....| Y |
+---+---+---+---+---+---+---+---+ +---+
In pai_copy() each PAI counter value is read and compared
to its old value. This is done in a loop. When PAI counter indexed
N is read, the hardware might increment PAI counter indexed 2 again,
updating its value from X to X+1.
Later pai_push_sample() simply mem-copies the complete PAI counter
page to a backup page and the increment of X+1 is lost, because the
backup page now contains the new value.
Read each PAI counter and save this value in the backup page when
there is a positive delta. This omits any time window between read
and store. This also reduced the work load as only modified PAI
counters are saved.
Cc: stable@vger.kernel.org
Fixes: fe861b0c8d ("s390/pai: save PAI counter value page in event structure")
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Reviewed-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
The PAI crypto counter design allows for user space and kernel space
PAI counter increment recording. This is achieved by splitting the
recording page in half. The upper part of the 4KB page records user
space increments of PAI crypto counter and the lower half records
kernel space increments. The page itself looks like:
lowcore ptr ---> ++++++++++++++++++++++++
|user space area |
+----------------------+
|kernel space area |
++++++++++++++++++++++++
User space and kernel space entries are handled via a kernel_offset
value when wrting. For PAI crypto counters this offset is 2048 or
half of a page size.
For PAI NNPA counter design this distinction was not needed. There is
no user and kernel space part for the page pointed to by lowcore.
The set up is:
lowcore ptr ---> ++++++++++++++++++++++++
|user + kernel space |
|area |
| |
++++++++++++++++++++++++
There is always only one counter value recorded and saved.
Depending on number of CPUs and machine load, the number of PAI NNPA
counter increment differs between counting (perf stat) and recording
(perf record). The number reported by sampling was double the number
shown by counting.
This was caused by a double read of the PAI NNPA values in function
pai_copy(). The first part of that function reads the kernel space part.
The offset into the kernel page part must be larger than zero.
The second part of that function reads the user space part, which
begins of offset zero. This works fine for PAI crypto counters.
It fails for PAI NNPA counters because the PMU device driver does
not support that feature and has a kernel_offset value of 0x0.
Executing both user and kernel space read out might end up reading
user space value twice.
For the PAI NNPA PMU prohibit the kernel space part read out.
Cc: stable@vger.kernel.org
Fixes: f124735413 ("s390/pai_crypto: Rename paicrypt_copy() to pai_copy()")
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Reviewed-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
This was done entirely with mindless brute force, using
git grep -l '\<k[vmz]*alloc_objs*(.*, GFP_KERNEL)' |
xargs sed -i 's/\(alloc_objs*(.*\), GFP_KERNEL)/\1)/'
to convert the new alloc_obj() users that had a simple GFP_KERNEL
argument to just drop that argument.
Note that due to the extreme simplicity of the scripting, any slightly
more complex cases spread over multiple lines would not be triggered:
they definitely exist, but this covers the vast bulk of the cases, and
the resulting diff is also then easier to check automatically.
For the same reason the 'flex' versions will be done as a separate
conversion.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This is the result of running the Coccinelle script from
scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to
avoid scalar types (which need careful case-by-case checking), and
instead replace kmalloc-family calls that allocate struct or union
object instances:
Single allocations: kmalloc(sizeof(TYPE), ...)
are replaced with: kmalloc_obj(TYPE, ...)
Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...)
are replaced with: kmalloc_objs(TYPE, COUNT, ...)
Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...)
are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...)
(where TYPE may also be *VAR)
The resulting allocations no longer return "void *", instead returning
"TYPE *".
Signed-off-by: Kees Cook <kees@kernel.org>
The KMSG_COMPONENT macro is a leftover of the s390 specific "kernel
message catalog" which never made it upstream.
Remove the macro in order to get rid of a pointless indirection. Replace
all users with the string it defines. In almost all cases this leads to a
simple replacement like this:
- #define KMSG_COMPONENT "appldata"
- #define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
+ #define pr_fmt(fmt) "appldata: " fmt
Except for some special cases this is just mechanical/scripted work.
Acked-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
The PAI extension 1 control block area is 512 bytes in total.
It currently contains three address pointer which refer to counter
memory blocks followed by a reserved area.
Calculate the reserved area instead of hardcoding its size. This
makes the code more readable and maintainable.
No functional chance.
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Suggested-by: Jan Polensky <japo@linux.ibm.com>
Reviewed-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Rename perf_pai_crypto.c to perf_pai.c. The new perf_pai.c
contains both PAI device drivers:
- pai_crypto for PAI crypto counter set
- pai_ext for PAI NNPA counter set
The rename reflects this common driver supporting both PMUs.
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Reviewed-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
