The gzip step of a kernel build is very lengthy, especially for larger
builds such as allmodconfig.
gzip itself cannot be run in parallel, however an alternative tool exists
that can, providing the same feature set as gzip itself - pigz - which
works as a drop-in replacement.
On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
1.6s, and with pigz it takes 0.09s, so the performance increase is
significant.
Therefore introduce the KPGZIP build variable which specifies the
compressor to use for the kernel image and default it to pigz if available,
or KGZIP otherwise (which itself defaults to gzip).
Modules, packaging and the embedded kernel configuration (CONFIG_IKCONFIG)
continue to use KGZIP which defaults to a single-threaded gzip instance.
Module compression is run in parallel by make, so pigz offers no benefit
there, packaging sits outside the build and the embedded configuration is
too small to matter.
pigz is executed using scripts/jobserver-exec, which reserves the job slots
make has free - necessary, as otherwise pigz would just grab unbounded
CPUs.
The kernel image compression step is always performed at the end of the
build so should typically have all job slots available.
pigz always provides byte-for-byte identical output across runs regardless
of thread count, but pigz and gzip are not guaranteed to generate the same
binary output, so reproducible builds must use the same tools consistently.
Also update Documentation/kbuild/kbuild.rst to describe KPGZIP and
Documentation/process/changes.rst to list the optional pigz dependency.
While there, add entries for the other compressor programs.
Finally, update scripts/ver_linux to include the pigz version there.
Every x86 build ends with the compression of vmlinux.bin, 36MB for
defconfig and over 200MB for allmodconfig, no-op builds are unchanged.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 7.7s 6.0s -1.7s (-22%)
x86 defconfig, touch mm/vma.c, clang 6.9s 5.4s -1.5s (-21%)
x86 defconfig, clean, gcc 28.2s 26.3s -1.8s (-6%)
x86 defconfig, clean, clang 28.7s 27.2s -1.5s (-5%)
x86 allmodconfig, touch mm/vma.c, gcc 23.7s 15.4s -8.3s (-35%)
x86 allmodconfig, touch mm/vma.c, clang 22.5s 15.6s -7.0s (-31%)
x86 allmodconfig, clean, gcc 294.1s 278.6s -15.5s (-5%)
x86 allmodconfig, clean, clang 283.9s 266.8s -17.1s (-6%)
Suggested-by: Kees Cook <kees@kernel.org>
Link: https://zlib.net/pigz/
Assisted-by: LLM
Reviewed-by: Manuel Ebner <manuelebnerli@mailbox.org>
Reviewed-by: Kees Cook <kees@kernel.org>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://patch.msgid.link/20260923-build-speedup-v4-22-73128809a4a4@kernel.org
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
This commit fixes small typos in the docstring of jobserver.py.
Signed-off-by: Ricardo Ungerer <ungerer.ricardo@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20260316220218.568022-1-ungerer.ricardo@gmail.com>
Mauro's work to include documentation from our Python modules. His cover
letter follows:
This is an extended version of:
https://lore.kernel.org/linux-doc/cover.1768488832.git.mchehab+huawei@kernel.org/
It basically adds everything we currently have inside libs/tool/python
to "tools" book inside documentation.
This version should be independent of the other series yet to be merged,
(including the jobserver one).
The vast amount of changes here are docstring cleanups and additions.
They mainly consists on:
- ensuring that every phrase will end with a period, making it uniform
along all files;
- cleaning ups to better uniform docstrings;
- variable descriptions now use "#:" markup, as it allows autodoc to
add them inside the documentation;
- added some missing docstrings;
- some new blank lines at comments to make ReST syntax parser happy;
- add a couple of sphinx markups (mainly, code blocks).
Most of those are minor changes, affecting only comments.
It also has one patch per libarary type, adding them to docs.
For kernel-doc, I did the cleanups first, as there is one code block
inside tools/lib/python/kdoc/latex_fonts.py that would cause a Sphinx
crash without such markups.
The series actually starts with 3 fixes:
- avoid "*" markups on indexes with deep> 3 to override text
- a variable rename to stop abusing doctree name
- don't rely on cwd to get Documentation/ location
patch 4 adds support to document scripts either at:
- tools/
- scripts/
patch 5 contains a CSS to better display autodoc html output.
For those who want to play with documentation, documenting a python
file is very simple. All it takes is to use:
.. automodule:: lib.python.<dir+name>
Usually, we add a couple of control members to it to adjust
the desired documentation scope (add/remove members, showing class
inheritance, showing members that currently don't have
docstrings, etc). That's why we're using:
.. automodule:: lib.python.kdoc.enrich_formatter
:members:
:show-inheritance:
:undoc-members:
(and similar) inside tools/kdoc*.rst.
autodoc allows filtering in/out members, file docstrings, etc.
It also allows documenting just some members or functions with
directives like:
..autofunction:
..automember:
Sphinx also has a helper script to generate .rst files with
documentation:
$ sphinx-apidoc -o foobar tools/lib/python/
which can be helpful to discover what should be documented,
although changes are needed to use what it produces.
The parsing of jobserver options is done in a massive try: block that hides
problems and (perhaps) bugs. Split up that block and make the logic
explicit by moving the initial parsing of MAKEFLAGS out of that block. Add
warnings in the places things can go wrong.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
When using GNU Make's jobserver feature in kernel builds, a bug in MAKEFLAGS
propagation caused "--jobserver-auth=r,w" to reference an unintended file
descriptor. This led to infinite loops in jobserver-exec's os.read() calls
due to empty token.
My shell opened /etc/passwd for some reason without closing it, and as a
result, all child processes inherited this fd 3.
$ ls -l /proc/self/fd
total 0
lrwx------ 1 changbin changbin 64 Dec 25 13:03 0 -> /dev/pts/1
lrwx------ 1 changbin changbin 64 Dec 25 13:03 1 -> /dev/pts/1
lrwx------ 1 changbin changbin 64 Dec 25 13:03 2 -> /dev/pts/1
lr-x------ 1 changbin changbin 64 Dec 25 13:03 3 -> /etc/passwd
lr-x------ 1 changbin changbin 64 Dec 25 13:03 4 -> /proc/1421383/fd
In this case, the `make` should open a new file descriptor for jobserver
control, but clearly, it did not do so and instead still passed fd 3 as
"--jobserver-auth=3,4" in MAKEFLAGS. (The version of my gnu make is 4.3)
This update ensures robustness against invalid jobserver configurations,
even when `make` incorrectly pass non-pipe file descriptors.
* Rejecting empty reads to prevent infinite loops on EOF.
* Clearing `self.jobs` to avoid writing to incorrect files if invalid tokens
are detected.
* Printing detailed error messages to stderr to inform the user.
Cc: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Changbin Du <changbin.du@huawei.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20260108113836.2976527-1-changbin.du@huawei.com>
"scripts/lib" was always a bit of an awkward place for Python modules. We
already have tools/lib; create a tools/lib/python, move the libraries
there, and update the users accordingly.
While at it, move the contents of tools/docs/lib. Rather than make another
directory, just put these documentation-oriented modules under "kdoc".
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251110220430.726665-2-corbet@lwn.net>
