| Message ID | 20240607141126.2983943-1-stefan-koch@siemens.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | [v2] linux-custom: Speedup build of target specific linux-kbuild package | expand |
On 07.06.24 16:11, Stefan Koch wrote: > Avoids time expensive QEMU-emulated merging of kernel config by using > the existing kernel config for the target specific linux-kbuild > > Using DEPENDS instead of RDEPENDS ensures creation of > kernel (including config) before build of target specific linux-kbuild > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > --- > .../linux/files/debian/isar/configure.tmpl | 18 ++++++++++++++---- > meta/recipes-kernel/linux/linux-custom.inc | 10 +++++++--- > 2 files changed, 21 insertions(+), 7 deletions(-) > > diff --git a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl > index 389c9a85..72eae448 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl > @@ -10,10 +10,20 @@ do_configure() { > # Trace what we do here > set -x > > - # Process kernel config target and fragments > - ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} > - ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \ > - ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS} > + kernelconfig="$(find /boot -maxdepth 1 -name "config-${PV}*" -print -quit)" Can we additionally limit this to the case were we expect the config there (kbuild && !kernel)? Just to reduce the risk that some "this should never be there" config is ever silently pulled by the base package build. Only if our dependency is installed, that rather generic config-<version> file is from there, or we failed on installing the dependency. > + if [ -e "${kernelconfig}" ]; then > + # Prefer existing kernel config > + # So, very expensive QEMU-emulated merge_config.sh > + # can be skipped for target specific linux-kbuild package > + mkdir -p ${KERNEL_BUILD_DIR} > + cp "${kernelconfig}" ${KERNEL_BUILD_DIR}/.config > + ${MAKE} O=${KERNEL_BUILD_DIR} olddefconfig > + else > + # Process kernel config target and fragments > + ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} > + ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \ > + ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS} > + fi > > # Stop tracing > set +x > diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc > index 647f09dd..dc354e13 100644 > --- a/meta/recipes-kernel/linux/linux-custom.inc > +++ b/meta/recipes-kernel/linux/linux-custom.inc > @@ -25,6 +25,7 @@ KBUILD_DEPENDS ?= "build-essential:native, \ > flex, \ > git, \ > kmod, \ > + linux-image-${KERNEL_NAME_PROVIDED}:${DISTRO_ARCH} <kbuild !kernel>, \ > rsync," > > KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, \ > @@ -112,7 +113,7 @@ BUILD_PROFILES = "kernel kbuild" > BBCLASSEXTEND:append:cross-profile = " kbuildtarget" > > # When cross-profile is active: > -# build only kernel with the default variant of the recipe > +# build only kernel (including config) with the default variant of the recipe > BUILD_PROFILES:cross-profile = "kernel" > > # -native: kbuild package for host > @@ -122,14 +123,17 @@ RECIPE_PROVIDES:class-native = " \ > linux-kbuild-${KERNEL_NAME_PROVIDED}" > # Use pseudo target to pull in the base variant of the recipe. > # Will be auto-extended with -native by multiarch.bbclass. > -RDEPENDS:class-native += "${BPN}-pseudo" > +# Using DEPENDS, see below > +DEPENDS:class-native += "${BPN}-pseudo" Need to help me with remember my own code: Why do we also need this build dependency? I thought you are only optimizing the kbuildtarget case. > > # -kbuildtarget: kbuild package for target, enforcing non-cross-build > BUILD_PROFILES:class-kbuildtarget = "kbuild" > RECIPE_PROVIDES:class-kbuildtarget = " \ > linux-headers-${KERNEL_NAME_PROVIDED} \ > linux-kbuild-${KERNEL_NAME_PROVIDED}" > -RDEPENDS:class-kbuildtarget = "${BPN}" > +# Using DEPENDS instead of RDEPENDS to ensure creation of kernel including > +# pregenerated kernel config before target specific linux-kbuild package build > +DEPENDS:class-kbuildtarget = "${BPN}" > ISAR_CROSS_COMPILE:class-kbuildtarget = "0" > > # Make bitbake know we will be producing linux-image and linux-headers packages Fine-tuning - this generally looks like a nice optimization now! Jan
On Fri, 2024-06-07 at 17:07 +0200, Jan Kiszka wrote: > On 07.06.24 16:11, Stefan Koch wrote: > > Avoids time expensive QEMU-emulated merging of kernel config by > > using > > the existing kernel config for the target specific linux-kbuild > > > > Using DEPENDS instead of RDEPENDS ensures creation of > > kernel (including config) before build of target specific linux- > > kbuild > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > --- > > .../linux/files/debian/isar/configure.tmpl | 18 > > ++++++++++++++---- > > meta/recipes-kernel/linux/linux-custom.inc | 10 +++++++--- > > 2 files changed, 21 insertions(+), 7 deletions(-) > > > > diff --git a/meta/recipes- > > kernel/linux/files/debian/isar/configure.tmpl b/meta/recipes- > > kernel/linux/files/debian/isar/configure.tmpl > > index 389c9a85..72eae448 100644 > > --- a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl > > +++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl > > @@ -10,10 +10,20 @@ do_configure() { > > # Trace what we do here > > set -x > > > > - # Process kernel config target and fragments > > - ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} > > - ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \ > > - ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS} > > + kernelconfig="$(find /boot -maxdepth 1 -name "config-${PV}*" - > > print -quit)" > > Can we additionally limit this to the case were we expect the config > there (kbuild && !kernel)? Just to reduce the risk that some "this > should never be there" config is ever silently pulled by the base > package build. Only if our dependency is installed, that rather > generic > config-<version> file is from there, or we failed on installing the > dependency. Build profile can be checked here as second condition, too. find with "config-${PV}*" is used because some kernel specify "-rt18" or "-isar" or something else that is not included in PV: /boot/config-5.10.209-audis-rtp-rt18 /boot/config-6.6.11-isar Some kernel do not specify an suffix to ${PV}. Maybe it depends on LOCALVERISON from the kernel config? But as kernel config is not available, we cannot find out this information... find returns the first match. When using the old isar-buildchroot instead of sbuild/schroot: Multiple configs could exist - if multiple kernels are built. But if ${PV} is identical dpkg should have a conflict with overwriting an existing config with the same version? (as long this "-rt18"/"-isar" suffix is not renamed for that build.) See below for dependency problem. > > > + if [ -e "${kernelconfig}" ]; then > > + # Prefer existing kernel config > > + # So, very expensive QEMU-emulated merge_config.sh > > + # can be skipped for target specific linux-kbuild package > > + mkdir -p ${KERNEL_BUILD_DIR} > > + cp "${kernelconfig}" ${KERNEL_BUILD_DIR}/.config > > + ${MAKE} O=${KERNEL_BUILD_DIR} olddefconfig > > + else > > + # Process kernel config target and fragments > > + ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} > > + ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ > > \ > > + ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS} > > + fi > > > > # Stop tracing > > set +x > > diff --git a/meta/recipes-kernel/linux/linux-custom.inc > > b/meta/recipes-kernel/linux/linux-custom.inc > > index 647f09dd..dc354e13 100644 > > --- a/meta/recipes-kernel/linux/linux-custom.inc > > +++ b/meta/recipes-kernel/linux/linux-custom.inc > > @@ -25,6 +25,7 @@ KBUILD_DEPENDS ?= "build-essential:native, \ > > flex, \ > > git, \ > > kmod, \ > > + linux-image- > > ${KERNEL_NAME_PROVIDED}:${DISTRO_ARCH} <kbuild !kernel>, \ That ensures installation of the dependency. > > rsync," > > > > KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, > > \ > > @@ -112,7 +113,7 @@ BUILD_PROFILES = "kernel kbuild" > > BBCLASSEXTEND:append:cross-profile = " kbuildtarget" > > > > # When cross-profile is active: > > -# build only kernel with the default variant of the recipe > > +# build only kernel (including config) with the default variant of > > the recipe > > BUILD_PROFILES:cross-profile = "kernel" > > > > # -native: kbuild package for host > > @@ -122,14 +123,17 @@ RECIPE_PROVIDES:class-native = " \ > > linux-kbuild-${KERNEL_NAME_PROVIDED}" > > # Use pseudo target to pull in the base variant of the recipe. > > # Will be auto-extended with -native by multiarch.bbclass. > > -RDEPENDS:class-native += "${BPN}-pseudo" > > +# Using DEPENDS, see below > > +DEPENDS:class-native += "${BPN}-pseudo" > > Need to help me with remember my own code: Why do we also need this > build dependency? I thought you are only optimizing the kbuildtarget > case. Kernel config is also read from linux-image for linux-kbuild-native. When choosing to call merge_config.sh here again, this change would not be needed. Then a different build profile must be added, because <kernel !kbuild> won't match anymore. > > > > > # -kbuildtarget: kbuild package for target, enforcing non-cross- > > build > > BUILD_PROFILES:class-kbuildtarget = "kbuild" > > RECIPE_PROVIDES:class-kbuildtarget = " \ > > linux-headers-${KERNEL_NAME_PROVIDED} \ > > linux-kbuild-${KERNEL_NAME_PROVIDED}" > > -RDEPENDS:class-kbuildtarget = "${BPN}" > > +# Using DEPENDS instead of RDEPENDS to ensure creation of kernel > > including > > +# pregenerated kernel config before target specific linux-kbuild > > package build > > +DEPENDS:class-kbuildtarget = "${BPN}" > > ISAR_CROSS_COMPILE:class-kbuildtarget = "0" > > > > # Make bitbake know we will be producing linux-image and linux- > > headers packages > > Fine-tuning - this generally looks like a nice optimization now! > > Jan >
diff --git a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl index 389c9a85..72eae448 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl @@ -10,10 +10,20 @@ do_configure() { # Trace what we do here set -x - # Process kernel config target and fragments - ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} - ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \ - ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS} + kernelconfig="$(find /boot -maxdepth 1 -name "config-${PV}*" -print -quit)" + if [ -e "${kernelconfig}" ]; then + # Prefer existing kernel config + # So, very expensive QEMU-emulated merge_config.sh + # can be skipped for target specific linux-kbuild package + mkdir -p ${KERNEL_BUILD_DIR} + cp "${kernelconfig}" ${KERNEL_BUILD_DIR}/.config + ${MAKE} O=${KERNEL_BUILD_DIR} olddefconfig + else + # Process kernel config target and fragments + ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} + ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \ + ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS} + fi # Stop tracing set +x diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc index 647f09dd..dc354e13 100644 --- a/meta/recipes-kernel/linux/linux-custom.inc +++ b/meta/recipes-kernel/linux/linux-custom.inc @@ -25,6 +25,7 @@ KBUILD_DEPENDS ?= "build-essential:native, \ flex, \ git, \ kmod, \ + linux-image-${KERNEL_NAME_PROVIDED}:${DISTRO_ARCH} <kbuild !kernel>, \ rsync," KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, \ @@ -112,7 +113,7 @@ BUILD_PROFILES = "kernel kbuild" BBCLASSEXTEND:append:cross-profile = " kbuildtarget" # When cross-profile is active: -# build only kernel with the default variant of the recipe +# build only kernel (including config) with the default variant of the recipe BUILD_PROFILES:cross-profile = "kernel" # -native: kbuild package for host @@ -122,14 +123,17 @@ RECIPE_PROVIDES:class-native = " \ linux-kbuild-${KERNEL_NAME_PROVIDED}" # Use pseudo target to pull in the base variant of the recipe. # Will be auto-extended with -native by multiarch.bbclass. -RDEPENDS:class-native += "${BPN}-pseudo" +# Using DEPENDS, see below +DEPENDS:class-native += "${BPN}-pseudo" # -kbuildtarget: kbuild package for target, enforcing non-cross-build BUILD_PROFILES:class-kbuildtarget = "kbuild" RECIPE_PROVIDES:class-kbuildtarget = " \ linux-headers-${KERNEL_NAME_PROVIDED} \ linux-kbuild-${KERNEL_NAME_PROVIDED}" -RDEPENDS:class-kbuildtarget = "${BPN}" +# Using DEPENDS instead of RDEPENDS to ensure creation of kernel including +# pregenerated kernel config before target specific linux-kbuild package build +DEPENDS:class-kbuildtarget = "${BPN}" ISAR_CROSS_COMPILE:class-kbuildtarget = "0" # Make bitbake know we will be producing linux-image and linux-headers packages
Avoids time expensive QEMU-emulated merging of kernel config by using the existing kernel config for the target specific linux-kbuild Using DEPENDS instead of RDEPENDS ensures creation of kernel (including config) before build of target specific linux-kbuild Signed-off-by: Stefan Koch <stefan-koch@siemens.com> --- .../linux/files/debian/isar/configure.tmpl | 18 ++++++++++++++---- meta/recipes-kernel/linux/linux-custom.inc | 10 +++++++--- 2 files changed, 21 insertions(+), 7 deletions(-)