Skip to content
Snippets Groups Projects
Select Git revision
0 results
Search by author
  • Any Author
0 authors
  1. Jun 22, 2023
    • Masahiro Yamada's avatar
      kbuild: implement CONFIG_TRIM_UNUSED_KSYMS without recursion · 5e9e95cc
      Masahiro Yamada authored 
      When CONFIG_TRIM_UNUSED_KSYMS is enabled, Kbuild recursively traverses
the directory tree to determine which EXPORT_SYMBOL to trim. If an
EXPORT_SYMBOL turns out to be unused by anyone, Kbuild begins the
second traverse, where some source files are recompiled with their
EXPORT_SYMBOL() tuned into a no-op.

Linus stated negative opinions about this slowness in commits:

 - 5cf0fd59 ("Kbuild: disable TRIM_UNUSED_KSYMS option")
 - a555bdd0 ("Kbuild: enable TRIM_UNUSED_KSYMS again, with some guarding")

We can do this better now. The final data structures of EXPORT_SYMBOL
are generated by the modpost stage, so modpost can selectively emit
KSYMTAB entries that are really used by modules.

Commit f73edc89 ("kbuild: unify two modpost invocations") is another
ground-work to do this in a one-pass algorithm. With the list of modules,
modpost sets sym->used if it is used by a module. modpost emits KSYMTAB
only for symbols with sym->used==true.

BTW, Nicolas explained why the trimming was implemented with recursion:

  https://lore.kernel.org/all/2o2rpn97-79nq-p7s2-nq5-8p83391473r@syhkavp.arg/



Actually, we never achieved that level of optimization where the chain
reaction of trimming comes into play because:

 - CONFIG_LTO_CLANG cannot remove any unused symbols
 - CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is enabled only for vmlinux,
   but not modules

If deeper trimming is required, we need to revisit this, but I guess
that is unlikely to happen.

Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      5e9e95cc
  3. May 14, 2023
  5. May 07, 2023
  7. Apr 23, 2023
  9. Apr 17, 2023
  11. Apr 16, 2023
  13. Apr 09, 2023
  15. Apr 02, 2023
  17. Mar 26, 2023
  19. Mar 19, 2023
  21. Mar 16, 2023
  23. Mar 15, 2023
  25. Mar 12, 2023
  27. Mar 05, 2023
  29. Feb 19, 2023
  31. Feb 16, 2023
    • Masahiro Yamada's avatar
      kbuild: add a tool to list files ignored by git · 5c3d1d0a
      Masahiro Yamada authored 
      In short, the motivation of this commit is to build a source package
without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before
creating a source tarball. Otherwise build artifacts such as *.o,
*.a, etc. would be included in the tarball. Yet, the tarball ends up
containing several garbage files since 'make clean' does not clean
everything.

Cleaning the tree every time is annoying since it makes the incremental
build impossible. It is desirable to create a source tarball without
cleaning the tree.

In fact, there are some ways to achieve this.

The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses
it, but I do not like it because it works only when the source tree is
managed by git, and all files you want in the tarball must be committed
in advance.

I want to make it work without relying on git. We can do this.

Files that are ignored by git are generated files, so should be excluded
from the source tarball. We can list them out by parsing the .gitignore
files. Of course, .gitignore does not cover all the cases, but it works
well enough.

tar(1) claims to support it:

  --exclude-vcs-ignores

    Exclude files that match patterns read from VCS-specific ignore files.
    Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use 'tar --exclude-vcs-ignores', but this
option does not work. --exclude-vcs-ignore does not understand any of
the negation (!), preceding slash, following slash, etc.. So, this option
is just useless.

Hence, I wrote this gitignore parser. The previous version [1], written
in Python, was so slow. This version is implemented in C, so it works
much faster.

I imported the code from git (commit: 23c56f7bd5f1), so we get the same
result.

This tool traverses the source tree, parsing all .gitignore files, and
prints file paths that are ignored by git.

The output is similar to 'git ls-files --ignored --directory --others
--exclude-per-directory=.gitignore', except

  [1] Not sorted
  [2] No trailing slash for directories

[2] is intentional because tar's --exclude-from option cannot handle
trailing slashes.

[How to test this tool]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ make scripts_package
    HOSTCC  scripts/list-gitignored
  $ scripts/list-gitignored  --prefix=./ -o ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You
see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings

[1]: https://lore.kernel.org/all/20230128173843.765212-1-masahiroy@kernel.org/



Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      5c3d1d0a
  33. Feb 12, 2023
  35. Feb 07, 2023
  37. Feb 05, 2023
    • Linus Torvalds's avatar
      Linux 6.2-rc7 · 4ec5183e
      Linus Torvalds authored
      v6.2-rc7
      4ec5183e
    • Masahiro Yamada's avatar
      kbuild: remove --include-dir MAKEFLAG from top Makefile · 67d7c302
      Masahiro Yamada authored 
      
I added $(srctree)/ to some included Makefiles in the following commits:

 - 3204a7fb ("kbuild: prefix $(srctree)/ to some included Makefiles")
 - d8285639 ("kbuild: do not require sub-make for separate output tree builds")

They were a preparation for removing --include-dir flag.

I have never thought --include-dir useful. Rather, it _is_ harmful.

For example, run the following commands:

  $ make -s ARCH=x86 mrproper defconfig
  $ make ARCH=arm O=foo dtbs
  make[1]: Entering directory '/tmp/linux/foo'
    HOSTCC  scripts/basic/fixdep
  Error: kernelrelease not valid - run 'make prepare' to update it
    UPD     include/config/kernel.release
  make[1]: Leaving directory '/tmp/linux/foo'

The first command configures the source tree for x86. The next command
tries to build ARM device trees in the separate foo/ directory - this
must stop because the directory foo/ has not been configured yet.

However, due to --include-dir=$(abs_srctree), the top Makefile includes
the wrong include/config/auto.conf from the source tree and continues
building. Kbuild traverses the directory tree, but of course it does
not work correctly. The Error message is also pointless - 'make prepare'
does not help at all for fixing the issue.

This commit fixes more arch Makefile, and finally removes --include-dir
from the top Makefile.

There are more breakages under drivers/, but I do not volunteer to fix
them all. I just moved --include-dir to drivers/Makefile.

With this commit, the second command will stop with a sensible message.

  $ make -s ARCH=x86 mrproper defconfig
  $ make ARCH=arm O=foo dtbs
  make[1]: Entering directory '/tmp/linux/foo'
    SYNC    include/config/auto.conf.cmd
  ***
  *** The source tree is not clean, please run 'make ARCH=arm mrproper'
  *** in /tmp/linux
  ***
  make[2]: *** [../Makefile:646: outputmakefile] Error 1
  /tmp/linux/Makefile:770: include/config/auto.conf.cmd: No such file or directory
  make[1]: *** [/tmp/linux/Makefile:793: include/config/auto.conf.cmd] Error 2
  make[1]: Leaving directory '/tmp/linux/foo'
  make: *** [Makefile:226: __sub-make] Error 2

Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      67d7c302
    • Carlos Llamas's avatar
      kbuild: fix trivial typo in comment · feb113ad
      Carlos Llamas authored 
      
Add missing underscore in CONFIG_DEBUG_INFO_BTF_MODULES.

Fixes: f73edc89 ("kbuild: unify two modpost invocations")
Signed-off-by: default avatarCarlos Llamas <cmllamas@google.com>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      feb113ad
    • Nathan Chancellor's avatar
      powerpc/vdso: Filter clang's auto var init zero enabler when linking · 4e3feaad
      Nathan Chancellor authored 
      After commit 8d9acfce ("kbuild: Stop using '-Qunused-arguments' with
clang"), the PowerPC vDSO shows the following error with clang-13 and
older when CONFIG_INIT_STACK_ALL_ZERO is enabled:

  clang: error: argument unused during compilation: '-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang' [-Werror,-Wunused-command-line-argument]

clang-14 added a change to make sure this flag never triggers
-Wunused-command-line-argument, so it is fixed with newer releases. For
older releases that the kernel still supports building with, just filter
out this flag, as has been done for other flags.

Fixes: f0a42fba ("powerpc/vdso: Improve linker flags")
Fixes: 8d9acfce ("kbuild: Stop using '-Qunused-arguments' with clang")
Link: https://github.com/llvm/llvm-project/commit/ca6d5813d17598cd180995fb3bdfca00f364475f


Signed-off-by: default avatarNathan Chancellor <nathan@kernel.org>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      4e3feaad
    • Masahiro Yamada's avatar
      kbuild: save overridden KERNELRELEASE in include/config/kernel.release · 1cb86b6c
      Masahiro Yamada authored 
      
${KERNELRELEASE} is used as a part of the installation path.
(INSTALL_DTBS_PATH, MODLIB, etc.)

When KERNELRELEASE is overridden from the command line, it should be
saved in include/config/kernel.release, so that it will be consistently
used for the installation steps.

Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      1cb86b6c
    • Masahiro Yamada's avatar
      setlocalversion: absorb $(KERNELVERSION) · ec31f868
      Masahiro Yamada authored 
      
Print $(KERNELVERSION) in setlocalversion so that the callers get
simpler.

Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      ec31f868
  39. Jan 29, 2023
  41. Jan 26, 2023
  43. Jan 22, 2023
  45. Jan 15, 2023
  47. Jan 10, 2023
    • Masahiro Yamada's avatar
      kbuild: fix 'make modules' error when CONFIG_DEBUG_INFO_BTF_MODULES=y · 74d3320f
      Masahiro Yamada authored 
      
When CONFIG_DEBUG_INFO_BTF_MODULES=y, running 'make modules'
in the clean kernel tree will get the following error.

  $ grep CONFIG_DEBUG_INFO_BTF_MODULES .config
  CONFIG_DEBUG_INFO_BTF_MODULES=y
  $ make -s clean
  $ make modules
    [snip]
    AR      vmlinux.a
  ar: ./built-in.a: No such file or directory
  make: *** [Makefile:1241: vmlinux.a] Error 1

'modules' depends on 'vmlinux', but builtin objects are not built.

Define KBUILD_BUILTIN.

Fixes: f73edc89 ("kbuild: unify two modpost invocations")
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      74d3320f
    • Masahiro Yamada's avatar
      kbuild: export top-level LDFLAGS_vmlinux only to scripts/Makefile.vmlinux · 8debed3e
      Masahiro Yamada authored 
      Nathan Chancellor reports that $(NM) emits an error message when
GNU Make 4.4 is used to build the ARM zImage.

  $ make-4.4 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- O=build defconfig zImage
    [snip]
    LD      vmlinux
    NM      System.map
    SORTTAB vmlinux
    OBJCOPY arch/arm/boot/Image
    Kernel: arch/arm/boot/Image is ready
  arm-linux-gnueabi-nm: 'arch/arm/boot/compressed/../../../../vmlinux': No such file
  /bin/sh: 1: arithmetic expression: expecting primary: " "
    LDS     arch/arm/boot/compressed/vmlinux.lds
    AS      arch/arm/boot/compressed/head.o
    GZIP    arch/arm/boot/compressed/piggy_data
    AS      arch/arm/boot/compressed/piggy.o
    CC      arch/arm/boot/compressed/misc.o

This occurs since GNU Make commit 98da874c4303 ("[SV 10593] Export
variables to $(shell ...) commands"), and the O= option is needed to
reproduce it. The generated zImage is correct despite the error message.

As the commit description of 98da874c4303 [1] says, exported variables
are passed down to $(shell ) functions, which means exported recursive
variables might be expanded earlier than before, in the parse stage.

The following test code demonstrates the change for GNU Make 4.4.

[Test Makefile]

  $(shell echo hello > foo)
  export foo = $(shell cat bar/../foo)
  $(shell mkdir bar)

  all:
          @echo $(foo)

[GNU Make 4.3]

  $ rm -rf bar; make-4.3
  hello

[GNU Make 4.4]

  $ rm -rf bar; make-4.4
  cat: bar/../foo: No such file or directory
  hello

The 'foo' is a resursively expanded (i.e. lazily expanded) variable.

GNU Make 4.3 expands 'foo' just before running the recipe '@echo $(foo)',
at this point, the directory 'bar' exists.

GNU Make 4.4 expands 'foo' to evaluate $(shell mkdir bar) because it is
exported. At this point, the directory 'bar' does not exit yet. The cat
command cannot resolve the bar/../foo path, hence the error message.

Let's get back to the kernel Makefile.

In arch/arm/boot/compressed/Makefile, KBSS_SZ is referenced by
LDFLAGS_vmlinux, which is recursive and also exported by the top
Makefile.

GNU Make 4.3 expands KBSS_SZ just before running the recipes, so no
error message.

GNU Make 4.4 expands KBSS_SZ in the parse stage, where the directory
arm/arm/boot/compressed does not exit yet. When compiled with O=,
the output directory is created by $(shell mkdir -p $(obj-dirs))
in scripts/Makefile.build.

There are two ways to fix this particular issue:

 - change "$(obj)/../../../../vmlinux" in KBSS_SZ to "vmlinux"
 - unexport LDFLAGS_vmlinux

This commit takes the latter course because it is what I originally
intended.

Commit 3ec8a5b3 ("kbuild: do not export LDFLAGS_vmlinux")
unexported LDFLAGS_vmlinux.

Commit 5d4aeffb ("kbuild: rebuild .vmlinux.export.o when its
prerequisite is updated") accidentally exported it again.

We can clean up arch/arm/boot/compressed/Makefile later.

[1]: https://git.savannah.gnu.org/cgit/make.git/commit/?id=98da874c43035a490cdca81331724f233a3d0c9a

Link: https://lore.kernel.org/all/Y7i8+EjwdnhHtlrr@dev-arch.thelio-3990X/


Fixes: 5d4aeffb ("kbuild: rebuild .vmlinux.export.o when its prerequisite is updated")
Reported-by: default avatarNathan Chancellor <nathan@kernel.org>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNicolas Schier <nicolas@fjasle.eu>
Tested-by: default avatarNathan Chancellor <nathan@kernel.org>
      8debed3e
  49. Jan 08, 2023
  51. Jan 05, 2023