| Message ID | 20221109103238.1520091-1-stefan-koch@siemens.com |
|---|---|
| Headers | show |
| Series | linux-custom: Split up binaries from kernel headers to kbuild packages | expand |
On 09.11.22 11:32, Koch, Stefan (DI PA DCP R&D 3) wrote: > Hi > > This patchset is a set of three patches: > - Support overwriting configured schroot dir > - Split up binaries from kernel headers to kbuild packages > - Update custom_kernel docs for split up of kernel scripts and tools > > The main use-case was to swap out the binaries > from the kernel headers package. > They will be swapped out into host and target > specific kernel kbuild packages. > > The main development goals were these: > > 1. Solve already known isar custom kernel > limitations from doc/custom_kernel.inc > - kernel headers package does not support both native > and cross compilation of kernel modules when cross built Right, that requires building packages for multiple architectures. This is the feature we need, and it has nothing to do with the kernel packages per se. Jan > > 2. Honor recommendations for future from doc/custom_kernel.inc > - Generate kernel headers packages for both host and target > when using cross build > > 3. Add extensions known from debian kernel packages structure > - Generate a kernel headers package without binaries > - Create specific kernel kbuild packages that > will ship the "scripts" and "tools" binaries > - Use symlinks to point to the "scripts" and "tools" binaries Keep in mind that we intend to remain compatible with the layout of Debian. Self-built kernels should remain drop-in replacements of the Debian kernel packages. And that is true for development packages as well. So, are your changes working against that, or are they improving existing incompatibility with the debian packages?
On Wed, 2022-11-09 at 12:50 +0100, Jan Kiszka wrote: > On 09.11.22 11:32, Koch, Stefan (DI PA DCP R&D 3) wrote: > > Hi > > > > This patchset is a set of three patches: > > - Support overwriting configured schroot dir > > - Split up binaries from kernel headers to kbuild packages > > - Update custom_kernel docs for split up of kernel scripts and > > tools > > > > The main use-case was to swap out the binaries > > from the kernel headers package. > > They will be swapped out into host and target > > specific kernel kbuild packages. > > > > The main development goals were these: > > > > 1. Solve already known isar custom kernel > > limitations from doc/custom_kernel.inc > > - kernel headers package does not support both native > > and cross compilation of kernel modules when cross built > > Right, that requires building packages for multiple architectures. Important feature... Maybe it should be added to the TODO list in the docs? > This > is the feature we need, and it has nothing to do with the kernel > packages per se. The way that I assume debian will do building the kernel is that they use a generic debian defconfig for each architecture. E. g. linux-config-5.10/config.amd64_none_amd64.xz, linux-config- 5.10/config.arm64_none_arm64.xz ISAR builds are often using specific defconfigs honored by ISAR's linux-custom for the target build. Debian specific architecture builds (e. g. host and target, maybe with sbuild) for one target device, using target specific stripped-down defconfigs, may not working properly. The kernel build for a foreign architecture (e. g. host) may fail because of an incompatible defconfig. Debian provide a generic very similar defconfig for different architectures, and they build the linux-image, linux-headers and linux- kbuild for each architecture: - So I assume the linux-kbuild:amd64 was build with linux-config- 5.10/config.amd64_none_amd64.xz - So I assume the linux-kbuild:arm64 was build with linux-config- 5.10/config.arm64_none_arm64.xz The debian generic defconfig may differ in some arch specific points but you can use the linux-kbuild:amd64 for a cross-build of linux- image:arm64 because the resulting scripts and tools are from the identical set. So mixing linux-headers with different archictecture specific builds from linux-kbuild should work since the defconfigs have a "shared base" (e. g. both define CONFIG_HAVE_C_RECORDMCOUNT thats needed for recordmcount utility as example). > Jan > > > > > 2. Honor recommendations for future from doc/custom_kernel.inc > > - Generate kernel headers packages for both host and target > > when using cross build > > > > 3. Add extensions known from debian kernel packages structure > > - Generate a kernel headers package without binaries > > - Create specific kernel kbuild packages that > > will ship the "scripts" and "tools" binaries > > - Use symlinks to point to the "scripts" and "tools" binaries > > Keep in mind that we intend to remain compatible with the layout of > Debian. Self-built kernels should remain drop-in replacements of the > Debian kernel packages. And that is true for development packages as > well. > > So, are your changes working against that, or are they improving > existing incompatibility with the debian packages? The approach from this patchset does build the kernel in the known ISAR specific way. Debian itself patches the kernel Makefiles to modularize the kernel build what ISAR doesn't do. With this patch the packages linux-kbuild and linux-kbuild-cross are the only new packages that the ISAR build will provide. That means ISAR will provide then linux- headers, linux-image and linux-kbuild. They match on the target with the package name of the respective debian packages. So the existing drop-in should not be affected. In short this patch brings the ISAR linux-custom mechanism a bit nearer to the debian package structure. Since linux-headers behaves then more like the original debian linux-heades to ship the binaries within a separate package -> linux-kbuild. The trick is to build only the scripts and tools with schroot-target (QEMU) in a non-cross way. Since the kernel does not support building the scripts and tools for target architecture when do a cross build without patching the kernel Makefiles. I think that patch was never merged into the kernel: https://patchwork.kernel.org/project/linux-kbuild/patch/1376046432-12588-1-git-send-email-broonie@kernel.org/
On 09.11.22 16:06, Koch, Stefan (DI PA DCP R&D 3) wrote: > On Wed, 2022-11-09 at 12:50 +0100, Jan Kiszka wrote: >> On 09.11.22 11:32, Koch, Stefan (DI PA DCP R&D 3) wrote: >>> Hi >>> >>> This patchset is a set of three patches: >>> - Support overwriting configured schroot dir >>> - Split up binaries from kernel headers to kbuild packages >>> - Update custom_kernel docs for split up of kernel scripts and >>> tools >>> >>> The main use-case was to swap out the binaries >>> from the kernel headers package. >>> They will be swapped out into host and target >>> specific kernel kbuild packages. >>> >>> The main development goals were these: >>> >>> 1. Solve already known isar custom kernel >>> limitations from doc/custom_kernel.inc >>> - kernel headers package does not support both native >>> and cross compilation of kernel modules when cross built >> >> Right, that requires building packages for multiple architectures. > > Important feature... Maybe it should be added to the TODO list in the > docs? > >> This >> is the feature we need, and it has nothing to do with the kernel >> packages per se. > > The way that I assume debian will do building the kernel is that they > use a generic debian defconfig for each architecture. > E. g. linux-config-5.10/config.amd64_none_amd64.xz, linux-config- > 5.10/config.arm64_none_arm64.xz > > ISAR builds are often using specific defconfigs honored by ISAR's > linux-custom for the target build. > > Debian specific architecture builds (e. g. host and target, maybe with > sbuild) for one target device, using target specific stripped-down > defconfigs, may not working properly. The kernel build for a foreign > architecture (e. g. host) may fail because of an incompatible > defconfig. > > Debian provide a generic very similar defconfig for different > architectures, and they build the linux-image, linux-headers and linux- > kbuild for each architecture: > - So I assume the linux-kbuild:amd64 was build with linux-config- > 5.10/config.amd64_none_amd64.xz > - So I assume the linux-kbuild:arm64 was build with linux-config- > 5.10/config.arm64_none_arm64.xz > > The debian generic defconfig may differ in some arch specific points > but you can use the linux-kbuild:amd64 for a cross-build of linux- > image:arm64 because the resulting scripts and tools are from the > identical set. So mixing linux-headers with different archictecture > specific builds from linux-kbuild should work since the defconfigs have > a "shared base" (e. g. both define CONFIG_HAVE_C_RECORDMCOUNT thats > needed for recordmcount utility as example). Sorry, I don't understand what role the config plays here. Isar is used to compile (usually) one specific kernel, Debian compiles a few generic ones. But the fact that Debian compile everything natively (for production) and we do cross-compilation is not related to different configs. The answer to the problem "I need kernel headers on the target" right now is in Isar to build natively the kernel and, thus, the header package. Not optimal, but we still need this case too rarely to have cared about it more so far. > >> Jan >> >>> >>> 2. Honor recommendations for future from doc/custom_kernel.inc >>> - Generate kernel headers packages for both host and target >>> when using cross build >>> >>> 3. Add extensions known from debian kernel packages structure >>> - Generate a kernel headers package without binaries >>> - Create specific kernel kbuild packages that >>> will ship the "scripts" and "tools" binaries >>> - Use symlinks to point to the "scripts" and "tools" binaries >> >> Keep in mind that we intend to remain compatible with the layout of >> Debian. Self-built kernels should remain drop-in replacements of the >> Debian kernel packages. And that is true for development packages as >> well. >> >> So, are your changes working against that, or are they improving >> existing incompatibility with the debian packages? > > The approach from this patchset does build the kernel in the known ISAR > specific way. Debian itself patches the kernel Makefiles to modularize > the kernel build what ISAR doesn't do. With this patch the packages > linux-kbuild and linux-kbuild-cross are the only new packages that the > ISAR build will provide. That means ISAR will provide then linux- > headers, linux-image and linux-kbuild. They match on the target with > the package name of the respective debian packages. So the existing > drop-in should not be affected. > > In short this patch brings the ISAR linux-custom mechanism a bit nearer > to the debian package structure. Since linux-headers behaves then more > like the original debian linux-heades to ship the binaries within a > separate package -> linux-kbuild. > > The trick is to build only the scripts and tools with schroot-target > (QEMU) in a non-cross way. Since the kernel does not support building > the scripts and tools for target architecture when do a cross build > without patching the kernel Makefiles. I think that patch was never > merged into the kernel: > https://patchwork.kernel.org/project/linux-kbuild/patch/1376046432-12588-1-git-send-email-broonie@kernel.org/ > The problem is that patch might have to not provide that use case I wrote about above. It is a rare use case, a non-distro use case, but I wouldn't call it an invalid one. If we can address it by splitting up the Isar kernel build into different packages and then only compile the binary bits once again, now for the target architecture, that would be a nice solution. On top, we should resolve the Isar issue that you can only build one package in one build for one architecture, not multiples. But I don't want special-casing the kernel build beyond that because the problem - I need dev tools both for my Isar build and later also for the target - is generic, even though still rare in your deployments. Jan
On Wed, 2022-11-09 at 17:00 +0100, Jan Kiszka wrote: > On 09.11.22 16:06, Koch, Stefan (DI PA DCP R&D 3) wrote: > > On Wed, 2022-11-09 at 12:50 +0100, Jan Kiszka wrote: > > > On 09.11.22 11:32, Koch, Stefan (DI PA DCP R&D 3) wrote: > > > > Hi > > > > > > > > This patchset is a set of three patches: > > > > - Support overwriting configured schroot dir > > > > - Split up binaries from kernel headers to kbuild packages > > > > - Update custom_kernel docs for split up of kernel scripts and > > > > tools > > > > > > > > The main use-case was to swap out the binaries > > > > from the kernel headers package. > > > > They will be swapped out into host and target > > > > specific kernel kbuild packages. > > > > > > > > The main development goals were these: > > > > > > > > 1. Solve already known isar custom kernel > > > > limitations from doc/custom_kernel.inc > > > > - kernel headers package does not support both native > > > > and cross compilation of kernel modules when cross built > > > > > > Right, that requires building packages for multiple > > > architectures. > > > > Important feature... Maybe it should be added to the TODO list in > > the > > docs? > > > > > This > > > is the feature we need, and it has nothing to do with the kernel > > > packages per se. > > > > The way that I assume debian will do building the kernel is that > > they > > use a generic debian defconfig for each architecture. > > E. g. linux-config-5.10/config.amd64_none_amd64.xz, linux-config- > > 5.10/config.arm64_none_arm64.xz > > > > ISAR builds are often using specific defconfigs honored by ISAR's > > linux-custom for the target build. > > > > Debian specific architecture builds (e. g. host and target, maybe > > with > > sbuild) for one target device, using target specific stripped-down > > defconfigs, may not working properly. The kernel build for a > > foreign > > architecture (e. g. host) may fail because of an incompatible > > defconfig. > > > > Debian provide a generic very similar defconfig for different > > architectures, and they build the linux-image, linux-headers and > > linux- > > kbuild for each architecture: > > - So I assume the linux-kbuild:amd64 was build with linux-config- > > 5.10/config.amd64_none_amd64.xz > > - So I assume the linux-kbuild:arm64 was build with linux-config- > > 5.10/config.arm64_none_arm64.xz > > > > The debian generic defconfig may differ in some arch specific > > points > > but you can use the linux-kbuild:amd64 for a cross-build of linux- > > image:arm64 because the resulting scripts and tools are from the > > identical set. So mixing linux-headers with different archictecture > > specific builds from linux-kbuild should work since the defconfigs > > have > > a "shared base" (e. g. both define CONFIG_HAVE_C_RECORDMCOUNT thats > > needed for recordmcount utility as example). > > Sorry, I don't understand what role the config plays here. Isar is > used > to compile (usually) one specific kernel, Debian compiles a few > generic > ones. But the fact that Debian compile everything natively (for > production) and we do cross-compilation is not related to different > configs. > I just wanted to point out for the future case when ISAR would support a generic way to build packages for multiple architectures: This might not be applicable to all kernel builds. E. g. because of the fact that the used defconfig for a "specific kernel" is not compatible for a multi architecture build, only for the architecture that the defconfig was made for. Yes, it may work if you could specify different defconfigs for each architecture. One goal of this patch was to provide the kbuild scripts and tools just for host and target in a transparent way - without any changes to the build process of existing ISAR users. > The answer to the problem "I need kernel headers on the target" right > now is in Isar to build natively the kernel and, thus, the header > package. Not optimal, but we still need this case too rarely to have > cared about it more so far. > > > > > > Jan > > > > > > > > > > > 2. Honor recommendations for future from doc/custom_kernel.inc > > > > - Generate kernel headers packages for both host and target > > > > when using cross build > > > > > > > > 3. Add extensions known from debian kernel packages structure > > > > - Generate a kernel headers package without binaries > > > > - Create specific kernel kbuild packages that > > > > will ship the "scripts" and "tools" binaries > > > > - Use symlinks to point to the "scripts" and "tools" binaries > > > > > > Keep in mind that we intend to remain compatible with the layout > > > of > > > Debian. Self-built kernels should remain drop-in replacements of > > > the > > > Debian kernel packages. And that is true for development packages > > > as > > > well. > > > > > > So, are your changes working against that, or are they improving > > > existing incompatibility with the debian packages? > > > > The approach from this patchset does build the kernel in the known > > ISAR > > specific way. Debian itself patches the kernel Makefiles to > > modularize > > the kernel build what ISAR doesn't do. With this patch the packages > > linux-kbuild and linux-kbuild-cross are the only new packages that > > the > > ISAR build will provide. That means ISAR will provide then linux- > > headers, linux-image and linux-kbuild. They match on the target > > with > > the package name of the respective debian packages. So the existing > > drop-in should not be affected. > > > > In short this patch brings the ISAR linux-custom mechanism a bit > > nearer > > to the debian package structure. Since linux-headers behaves then > > more > > like the original debian linux-heades to ship the binaries within a > > separate package -> linux-kbuild. > > > > The trick is to build only the scripts and tools with schroot- > > target > > (QEMU) in a non-cross way. Since the kernel does not support > > building > > the scripts and tools for target architecture when do a cross build > > without patching the kernel Makefiles. I think that patch was never > > merged into the kernel: > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.kernel.org%2Fproject%2Flinux-kbuild%2Fpatch%2F1376046432-12588-1-git-send-email-broonie%40kernel.org%2F&data=05%7C01%7Cstefan-koch%40siemens.com%7C5054591149e44251f36d08dac26b7b61%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638036064125400893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iPeO%2FkFjrLNSGQJ83fNsifWNI0wbPzFsWTIi%2B1fZ9PI%3D&reserved=0 > > > > The problem is that patch might have to not provide that use case I > wrote about above. It is a rare use case, a non-distro use case, but > I > wouldn't call it an invalid one. If we can address it by splitting up > the Isar kernel build into different packages and then only compile > the > binary bits once again, now for the target architecture, that would > be a > nice solution. That's what the patch do: - Move the binary scripts and tools from linux-headers to linux-kbuild as debian does it for their kernels - Compile only the scripts and tools for the target using schroot- target (QEMU) and package it as linux-kbuild - Package the binary scripts and tools from the full cross kernel build to linux-kbuild-cross using schroot-host > > On top, we should resolve the Isar issue that you can only build one > package in one build for one architecture, not multiples. But I don't > want special-casing the kernel build beyond that because the problem > - I > need dev tools both for my Isar build and later also for the target - > is > generic, even though still rare in your deployments. This is a nice and important goal. Maybe ISAR is now nearer to this since it has changed to use sbuild now. > > Jan >
On 10.11.22 18:49, Koch, Stefan (DI PA DCP R&D 3) wrote: > On Wed, 2022-11-09 at 17:00 +0100, Jan Kiszka wrote: >> On 09.11.22 16:06, Koch, Stefan (DI PA DCP R&D 3) wrote: >>> On Wed, 2022-11-09 at 12:50 +0100, Jan Kiszka wrote: >>>> On 09.11.22 11:32, Koch, Stefan (DI PA DCP R&D 3) wrote: >>>>> Hi >>>>> >>>>> This patchset is a set of three patches: >>>>> - Support overwriting configured schroot dir >>>>> - Split up binaries from kernel headers to kbuild packages >>>>> - Update custom_kernel docs for split up of kernel scripts and >>>>> tools >>>>> >>>>> The main use-case was to swap out the binaries >>>>> from the kernel headers package. >>>>> They will be swapped out into host and target >>>>> specific kernel kbuild packages. >>>>> >>>>> The main development goals were these: >>>>> >>>>> 1. Solve already known isar custom kernel >>>>> limitations from doc/custom_kernel.inc >>>>> - kernel headers package does not support both native >>>>> and cross compilation of kernel modules when cross built >>>> >>>> Right, that requires building packages for multiple >>>> architectures. >>> >>> Important feature... Maybe it should be added to the TODO list in >>> the >>> docs? >>> >>>> This >>>> is the feature we need, and it has nothing to do with the kernel >>>> packages per se. >>> >>> The way that I assume debian will do building the kernel is that >>> they >>> use a generic debian defconfig for each architecture. >>> E. g. linux-config-5.10/config.amd64_none_amd64.xz, linux-config- >>> 5.10/config.arm64_none_arm64.xz >>> >>> ISAR builds are often using specific defconfigs honored by ISAR's >>> linux-custom for the target build. >>> >>> Debian specific architecture builds (e. g. host and target, maybe >>> with >>> sbuild) for one target device, using target specific stripped-down >>> defconfigs, may not working properly. The kernel build for a >>> foreign >>> architecture (e. g. host) may fail because of an incompatible >>> defconfig. >>> >>> Debian provide a generic very similar defconfig for different >>> architectures, and they build the linux-image, linux-headers and >>> linux- >>> kbuild for each architecture: >>> - So I assume the linux-kbuild:amd64 was build with linux-config- >>> 5.10/config.amd64_none_amd64.xz >>> - So I assume the linux-kbuild:arm64 was build with linux-config- >>> 5.10/config.arm64_none_arm64.xz >>> >>> The debian generic defconfig may differ in some arch specific >>> points >>> but you can use the linux-kbuild:amd64 for a cross-build of linux- >>> image:arm64 because the resulting scripts and tools are from the >>> identical set. So mixing linux-headers with different archictecture >>> specific builds from linux-kbuild should work since the defconfigs >>> have >>> a "shared base" (e. g. both define CONFIG_HAVE_C_RECORDMCOUNT thats >>> needed for recordmcount utility as example). >> >> Sorry, I don't understand what role the config plays here. Isar is >> used >> to compile (usually) one specific kernel, Debian compiles a few >> generic >> ones. But the fact that Debian compile everything natively (for >> production) and we do cross-compilation is not related to different >> configs. >> > > I just wanted to point out for the future case when ISAR would support > a generic way to build packages for multiple architectures: This might > not be applicable to all kernel builds. E. g. because of the fact that > the used defconfig for a "specific kernel" is not compatible for a > multi architecture build, only for the architecture that the defconfig > was made for. > > Yes, it may work if you could specify different defconfigs for each > architecture. One goal of this patch was to provide the kbuild scripts > and tools just for host and target in a transparent way - without any > changes to the build process of existing ISAR users. Misunderstanding: I didn't meant to propose building the kernel-image package for multiple arch, only the then factored-out kbuild packages with the host tool (once for the build env, once for the target). > >> The answer to the problem "I need kernel headers on the target" right >> now is in Isar to build natively the kernel and, thus, the header >> package. Not optimal, but we still need this case too rarely to have >> cared about it more so far. >> >>> >>>> Jan >>>> >>>>> >>>>> 2. Honor recommendations for future from doc/custom_kernel.inc >>>>> - Generate kernel headers packages for both host and target >>>>> when using cross build >>>>> >>>>> 3. Add extensions known from debian kernel packages structure >>>>> - Generate a kernel headers package without binaries >>>>> - Create specific kernel kbuild packages that >>>>> will ship the "scripts" and "tools" binaries >>>>> - Use symlinks to point to the "scripts" and "tools" binaries >>>> >>>> Keep in mind that we intend to remain compatible with the layout >>>> of >>>> Debian. Self-built kernels should remain drop-in replacements of >>>> the >>>> Debian kernel packages. And that is true for development packages >>>> as >>>> well. >>>> >>>> So, are your changes working against that, or are they improving >>>> existing incompatibility with the debian packages? >>> >>> The approach from this patchset does build the kernel in the known >>> ISAR >>> specific way. Debian itself patches the kernel Makefiles to >>> modularize >>> the kernel build what ISAR doesn't do. With this patch the packages >>> linux-kbuild and linux-kbuild-cross are the only new packages that >>> the >>> ISAR build will provide. That means ISAR will provide then linux- >>> headers, linux-image and linux-kbuild. They match on the target >>> with >>> the package name of the respective debian packages. So the existing >>> drop-in should not be affected. >>> >>> In short this patch brings the ISAR linux-custom mechanism a bit >>> nearer >>> to the debian package structure. Since linux-headers behaves then >>> more >>> like the original debian linux-heades to ship the binaries within a >>> separate package -> linux-kbuild. >>> >>> The trick is to build only the scripts and tools with schroot- >>> target >>> (QEMU) in a non-cross way. Since the kernel does not support >>> building >>> the scripts and tools for target architecture when do a cross build >>> without patching the kernel Makefiles. I think that patch was never >>> merged into the kernel: >>> >>> https://patchwork.kernel.org/project/linux-kbuild/patch/1376046432-12588-1-git-send-email-broonie@kernel.org/ >>> >> >> The problem is that patch might have to not provide that use case I >> wrote about above. It is a rare use case, a non-distro use case, but >> I >> wouldn't call it an invalid one. If we can address it by splitting up >> the Isar kernel build into different packages and then only compile >> the >> binary bits once again, now for the target architecture, that would >> be a >> nice solution. > > That's what the patch do: > - Move the binary scripts and tools from linux-headers to linux-kbuild > as debian does it for their kernels > - Compile only the scripts and tools for the target using schroot- > target (QEMU) and package it as linux-kbuild > - Package the binary scripts and tools from the full cross kernel build > to linux-kbuild-cross using schroot-host It's moving in a good direction - but why linux-kbuild-cross? Nothing depends on it in your patches, thus module builds will be broken. I suspect we rather need KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}:<native> and therefore (on amd64) linux-kbuild-${KERNEL_NAME_PROVIDED}:amd64 to be built and then installed for module builds. Jan
On 10.11.22 19:33, Jan Kiszka wrote: > On 10.11.22 18:49, Koch, Stefan (DI PA DCP R&D 3) wrote: >> On Wed, 2022-11-09 at 17:00 +0100, Jan Kiszka wrote: >>> On 09.11.22 16:06, Koch, Stefan (DI PA DCP R&D 3) wrote: >>>> On Wed, 2022-11-09 at 12:50 +0100, Jan Kiszka wrote: >>>>> On 09.11.22 11:32, Koch, Stefan (DI PA DCP R&D 3) wrote: >>>>>> Hi >>>>>> >>>>>> This patchset is a set of three patches: >>>>>> - Support overwriting configured schroot dir >>>>>> - Split up binaries from kernel headers to kbuild packages >>>>>> - Update custom_kernel docs for split up of kernel scripts and >>>>>> tools >>>>>> >>>>>> The main use-case was to swap out the binaries >>>>>> from the kernel headers package. >>>>>> They will be swapped out into host and target >>>>>> specific kernel kbuild packages. >>>>>> >>>>>> The main development goals were these: >>>>>> >>>>>> 1. Solve already known isar custom kernel >>>>>> limitations from doc/custom_kernel.inc >>>>>> - kernel headers package does not support both native >>>>>> and cross compilation of kernel modules when cross built >>>>> >>>>> Right, that requires building packages for multiple >>>>> architectures. >>>> >>>> Important feature... Maybe it should be added to the TODO list in >>>> the >>>> docs? >>>> >>>>> This >>>>> is the feature we need, and it has nothing to do with the kernel >>>>> packages per se. >>>> >>>> The way that I assume debian will do building the kernel is that >>>> they >>>> use a generic debian defconfig for each architecture. >>>> E. g. linux-config-5.10/config.amd64_none_amd64.xz, linux-config- >>>> 5.10/config.arm64_none_arm64.xz >>>> >>>> ISAR builds are often using specific defconfigs honored by ISAR's >>>> linux-custom for the target build. >>>> >>>> Debian specific architecture builds (e. g. host and target, maybe >>>> with >>>> sbuild) for one target device, using target specific stripped-down >>>> defconfigs, may not working properly. The kernel build for a >>>> foreign >>>> architecture (e. g. host) may fail because of an incompatible >>>> defconfig. >>>> >>>> Debian provide a generic very similar defconfig for different >>>> architectures, and they build the linux-image, linux-headers and >>>> linux- >>>> kbuild for each architecture: >>>> - So I assume the linux-kbuild:amd64 was build with linux-config- >>>> 5.10/config.amd64_none_amd64.xz >>>> - So I assume the linux-kbuild:arm64 was build with linux-config- >>>> 5.10/config.arm64_none_arm64.xz >>>> >>>> The debian generic defconfig may differ in some arch specific >>>> points >>>> but you can use the linux-kbuild:amd64 for a cross-build of linux- >>>> image:arm64 because the resulting scripts and tools are from the >>>> identical set. So mixing linux-headers with different archictecture >>>> specific builds from linux-kbuild should work since the defconfigs >>>> have >>>> a "shared base" (e. g. both define CONFIG_HAVE_C_RECORDMCOUNT thats >>>> needed for recordmcount utility as example). >>> >>> Sorry, I don't understand what role the config plays here. Isar is >>> used >>> to compile (usually) one specific kernel, Debian compiles a few >>> generic >>> ones. But the fact that Debian compile everything natively (for >>> production) and we do cross-compilation is not related to different >>> configs. >>> >> >> I just wanted to point out for the future case when ISAR would support >> a generic way to build packages for multiple architectures: This might >> not be applicable to all kernel builds. E. g. because of the fact that >> the used defconfig for a "specific kernel" is not compatible for a >> multi architecture build, only for the architecture that the defconfig >> was made for. >> >> Yes, it may work if you could specify different defconfigs for each >> architecture. One goal of this patch was to provide the kbuild scripts >> and tools just for host and target in a transparent way - without any >> changes to the build process of existing ISAR users. > > Misunderstanding: I didn't meant to propose building the kernel-image > package for multiple arch, only the then factored-out kbuild packages > with the host tool (once for the build env, once for the target). > >> >>> The answer to the problem "I need kernel headers on the target" right >>> now is in Isar to build natively the kernel and, thus, the header >>> package. Not optimal, but we still need this case too rarely to have >>> cared about it more so far. >>> >>>> >>>>> Jan >>>>> >>>>>> >>>>>> 2. Honor recommendations for future from doc/custom_kernel.inc >>>>>> - Generate kernel headers packages for both host and target >>>>>> when using cross build >>>>>> >>>>>> 3. Add extensions known from debian kernel packages structure >>>>>> - Generate a kernel headers package without binaries >>>>>> - Create specific kernel kbuild packages that >>>>>> will ship the "scripts" and "tools" binaries >>>>>> - Use symlinks to point to the "scripts" and "tools" binaries >>>>> >>>>> Keep in mind that we intend to remain compatible with the layout >>>>> of >>>>> Debian. Self-built kernels should remain drop-in replacements of >>>>> the >>>>> Debian kernel packages. And that is true for development packages >>>>> as >>>>> well. >>>>> >>>>> So, are your changes working against that, or are they improving >>>>> existing incompatibility with the debian packages? >>>> >>>> The approach from this patchset does build the kernel in the known >>>> ISAR >>>> specific way. Debian itself patches the kernel Makefiles to >>>> modularize >>>> the kernel build what ISAR doesn't do. With this patch the packages >>>> linux-kbuild and linux-kbuild-cross are the only new packages that >>>> the >>>> ISAR build will provide. That means ISAR will provide then linux- >>>> headers, linux-image and linux-kbuild. They match on the target >>>> with >>>> the package name of the respective debian packages. So the existing >>>> drop-in should not be affected. >>>> >>>> In short this patch brings the ISAR linux-custom mechanism a bit >>>> nearer >>>> to the debian package structure. Since linux-headers behaves then >>>> more >>>> like the original debian linux-heades to ship the binaries within a >>>> separate package -> linux-kbuild. >>>> >>>> The trick is to build only the scripts and tools with schroot- >>>> target >>>> (QEMU) in a non-cross way. Since the kernel does not support >>>> building >>>> the scripts and tools for target architecture when do a cross build >>>> without patching the kernel Makefiles. I think that patch was never >>>> merged into the kernel: >>>> >>>> https://patchwork.kernel.org/project/linux-kbuild/patch/1376046432-12588-1-git-send-email-broonie@kernel.org/ >>>> >>> >>> The problem is that patch might have to not provide that use case I >>> wrote about above. It is a rare use case, a non-distro use case, but >>> I >>> wouldn't call it an invalid one. If we can address it by splitting up >>> the Isar kernel build into different packages and then only compile >>> the >>> binary bits once again, now for the target architecture, that would >>> be a >>> nice solution. >> >> That's what the patch do: >> - Move the binary scripts and tools from linux-headers to linux-kbuild >> as debian does it for their kernels >> - Compile only the scripts and tools for the target using schroot- >> target (QEMU) and package it as linux-kbuild >> - Package the binary scripts and tools from the full cross kernel build >> to linux-kbuild-cross using schroot-host > > It's moving in a good direction - but why linux-kbuild-cross? Nothing > depends on it in your patches, thus module builds will be broken. I > suspect we rather need > > KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}:<native> > > and therefore (on amd64) linux-kbuild-${KERNEL_NAME_PROVIDED}:amd64 to > be built and then installed for module builds. > > Jan > And regarding structure of the series: please split up the creation of kbuild package from also building it for the target arch - if possible. Jan
On Thu, 2022-11-10 at 19:36 +0100, Jan Kiszka wrote: > On 10.11.22 19:33, Jan Kiszka wrote: > > On 10.11.22 18:49, Koch, Stefan (DI PA DCP R&D 3) wrote: > > > On Wed, 2022-11-09 at 17:00 +0100, Jan Kiszka wrote: > > > > On 09.11.22 16:06, Koch, Stefan (DI PA DCP R&D 3) wrote: > > > > > On Wed, 2022-11-09 at 12:50 +0100, Jan Kiszka wrote: > > > > > > On 09.11.22 11:32, Koch, Stefan (DI PA DCP R&D 3) wrote: > > > > > > > Hi > > > > > > > > > > > > > > This patchset is a set of three patches: > > > > > > > - Support overwriting configured schroot dir > > > > > > > - Split up binaries from kernel headers to kbuild > > > > > > > packages > > > > > > > - Update custom_kernel docs for split up of kernel > > > > > > > scripts and > > > > > > > tools > > > > > > > > > > > > > > The main use-case was to swap out the binaries > > > > > > > from the kernel headers package. > > > > > > > They will be swapped out into host and target > > > > > > > specific kernel kbuild packages. > > > > > > > > > > > > > > The main development goals were these: > > > > > > > > > > > > > > 1. Solve already known isar custom kernel > > > > > > > limitations from doc/custom_kernel.inc > > > > > > > - kernel headers package does not support both native > > > > > > > and cross compilation of kernel modules when cross > > > > > > > built > > > > > > > > > > > > Right, that requires building packages for multiple > > > > > > architectures. > > > > > > > > > > Important feature... Maybe it should be added to the TODO > > > > > list in > > > > > the > > > > > docs? > > > > > > > > > > > This > > > > > > is the feature we need, and it has nothing to do with the > > > > > > kernel > > > > > > packages per se. > > > > > > > > > > The way that I assume debian will do building the kernel is > > > > > that > > > > > they > > > > > use a generic debian defconfig for each architecture. > > > > > E. g. linux-config-5.10/config.amd64_none_amd64.xz, linux- > > > > > config- > > > > > 5.10/config.arm64_none_arm64.xz > > > > > > > > > > ISAR builds are often using specific defconfigs honored by > > > > > ISAR's > > > > > linux-custom for the target build. > > > > > > > > > > Debian specific architecture builds (e. g. host and target, > > > > > maybe > > > > > with > > > > > sbuild) for one target device, using target specific > > > > > stripped-down > > > > > defconfigs, may not working properly. The kernel build for a > > > > > foreign > > > > > architecture (e. g. host) may fail because of an incompatible > > > > > defconfig. > > > > > > > > > > Debian provide a generic very similar defconfig for different > > > > > architectures, and they build the linux-image, linux-headers > > > > > and > > > > > linux- > > > > > kbuild for each architecture: > > > > > - So I assume the linux-kbuild:amd64 was build with linux- > > > > > config- > > > > > 5.10/config.amd64_none_amd64.xz > > > > > - So I assume the linux-kbuild:arm64 was build with linux- > > > > > config- > > > > > 5.10/config.arm64_none_arm64.xz > > > > > > > > > > The debian generic defconfig may differ in some arch specific > > > > > points > > > > > but you can use the linux-kbuild:amd64 for a cross-build of > > > > > linux- > > > > > image:arm64 because the resulting scripts and tools are from > > > > > the > > > > > identical set. So mixing linux-headers with different > > > > > archictecture > > > > > specific builds from linux-kbuild should work since the > > > > > defconfigs > > > > > have > > > > > a "shared base" (e. g. both define CONFIG_HAVE_C_RECORDMCOUNT > > > > > thats > > > > > needed for recordmcount utility as example). > > > > > > > > Sorry, I don't understand what role the config plays here. Isar > > > > is > > > > used > > > > to compile (usually) one specific kernel, Debian compiles a few > > > > generic > > > > ones. But the fact that Debian compile everything natively (for > > > > production) and we do cross-compilation is not related to > > > > different > > > > configs. > > > > > > > > > > I just wanted to point out for the future case when ISAR would > > > support > > > a generic way to build packages for multiple architectures: This > > > might > > > not be applicable to all kernel builds. E. g. because of the fact > > > that > > > the used defconfig for a "specific kernel" is not compatible for > > > a > > > multi architecture build, only for the architecture that the > > > defconfig > > > was made for. > > > > > > Yes, it may work if you could specify different defconfigs for > > > each > > > architecture. One goal of this patch was to provide the kbuild > > > scripts > > > and tools just for host and target in a transparent way - without > > > any > > > changes to the build process of existing ISAR users. > > > > Misunderstanding: I didn't meant to propose building the kernel- > > image > > package for multiple arch, only the then factored-out kbuild > > packages > > with the host tool (once for the build env, once for the target). > > Maybe then some logic that detects cross-build could use for the target build with "make scripts tools" only... > > > > > > > The answer to the problem "I need kernel headers on the target" > > > > right > > > > now is in Isar to build natively the kernel and, thus, the > > > > header > > > > package. Not optimal, but we still need this case too rarely to > > > > have > > > > cared about it more so far. > > > > > > > > > > > > > > > Jan > > > > > > > > > > > > > > > > > > > > 2. Honor recommendations for future from > > > > > > > doc/custom_kernel.inc > > > > > > > - Generate kernel headers packages for both host and > > > > > > > target > > > > > > > when using cross build > > > > > > > > > > > > > > 3. Add extensions known from debian kernel packages > > > > > > > structure > > > > > > > - Generate a kernel headers package without binaries > > > > > > > - Create specific kernel kbuild packages that > > > > > > > will ship the "scripts" and "tools" binaries > > > > > > > - Use symlinks to point to the "scripts" and "tools" > > > > > > > binaries > > > > > > > > > > > > Keep in mind that we intend to remain compatible with the > > > > > > layout > > > > > > of > > > > > > Debian. Self-built kernels should remain drop-in > > > > > > replacements of > > > > > > the > > > > > > Debian kernel packages. And that is true for development > > > > > > packages > > > > > > as > > > > > > well. > > > > > > > > > > > > So, are your changes working against that, or are they > > > > > > improving > > > > > > existing incompatibility with the debian packages? > > > > > > > > > > The approach from this patchset does build the kernel in the > > > > > known > > > > > ISAR > > > > > specific way. Debian itself patches the kernel Makefiles to > > > > > modularize > > > > > the kernel build what ISAR doesn't do. With this patch the > > > > > packages > > > > > linux-kbuild and linux-kbuild-cross are the only new packages > > > > > that > > > > > the > > > > > ISAR build will provide. That means ISAR will provide then > > > > > linux- > > > > > headers, linux-image and linux-kbuild. They match on the > > > > > target > > > > > with > > > > > the package name of the respective debian packages. So the > > > > > existing > > > > > drop-in should not be affected. > > > > > > > > > > In short this patch brings the ISAR linux-custom mechanism a > > > > > bit > > > > > nearer > > > > > to the debian package structure. Since linux-headers behaves > > > > > then > > > > > more > > > > > like the original debian linux-heades to ship the binaries > > > > > within a > > > > > separate package -> linux-kbuild. > > > > > > > > > > The trick is to build only the scripts and tools with > > > > > schroot- > > > > > target > > > > > (QEMU) in a non-cross way. Since the kernel does not support > > > > > building > > > > > the scripts and tools for target architecture when do a cross > > > > > build > > > > > without patching the kernel Makefiles. I think that patch was > > > > > never > > > > > merged into the kernel: > > > > > > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.kernel.org%2Fproject%2Flinux-kbuild%2Fpatch%2F1376046432-12588-1-git-send-email-broonie%40kernel.org%2F&data=05%7C01%7Cstefan-koch%40siemens.com%7C8ac82ead88ae49e7854308dac34a8659%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638037022082153110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MgLc%2Bh9ygiMGt2sXXl2CT9rPscIaFFr1f4Y2s5xqGIs%3D&reserved=0 > > > > > > > > > > > > > The problem is that patch might have to not provide that use > > > > case I > > > > wrote about above. It is a rare use case, a non-distro use > > > > case, but > > > > I > > > > wouldn't call it an invalid one. If we can address it by > > > > splitting up > > > > the Isar kernel build into different packages and then only > > > > compile > > > > the > > > > binary bits once again, now for the target architecture, that > > > > would > > > > be a > > > > nice solution. > > > > > > That's what the patch do: > > > - Move the binary scripts and tools from linux-headers to linux- > > > kbuild > > > as debian does it for their kernels > > > - Compile only the scripts and tools for the target using > > > schroot- > > > target (QEMU) and package it as linux-kbuild > > > - Package the binary scripts and tools from the full cross kernel > > > build > > > to linux-kbuild-cross using schroot-host > > > > It's moving in a good direction - but why linux-kbuild-cross? > > Nothing > > depends on it in your patches, thus module builds will be broken. Module build should work since linux-headers package depends on "linux- kbuild-cross | linux-kbuild" I have successfully cross build some module using the linux-module recipe... > > I > > suspect we rather need > > > > KERNEL_PKG_KERN_KBUILD=linux-kbuild- > > ${KERNEL_NAME_PROVIDED}:<native> > > > > and therefore (on amd64) linux-kbuild-${KERNEL_NAME_PROVIDED}:amd64 > > to > > be built and then installed for module builds. > > See my answer to [PATCH 2/3]... > > Jan > > > > And regarding structure of the series: please split up the creation > of > kbuild package from also building it for the target arch - if > possible. - Move the binary scripts and tools from linux-headers to linux-kbuild as debian does it for their kernels - Compile only the scripts and tools for the target using schroot- target (QEMU) and package it as linux-kbuild I would suggest this structure using two instead of one patch... Should the target kbuild creation be toggleable? When yes, enabled or disabled by default? > > Jan >
On 15.11.22 14:44, Uladzimir Bely wrote: > In mail from среда, 9 ноября 2022 г. 13:32:45 +03 user Koch, Stefan wrote: >> Hi >> >> This patchset is a set of three patches: >> - Support overwriting configured schroot dir >> - Split up binaries from kernel headers to kbuild packages >> - Update custom_kernel docs for split up of kernel scripts and tools >> > > Just some thoughts regarding schroot path overwriting... > > What if `linux-custom` recipe could disable specific components? Right, I sill wanted to follow up on that: Debian build profiles can do the trick here. Define one for "host tools only", and only build that in the second step. > > I mean, currently we have this feature only for `linux-libc-dev` package, that > is disabled when "nolibcdev" build profile is selected. If we added similar > build profiles for other components, we could disable also `linux-$ > {KERNEL_NAME_PROVIDED}`, linux-image-{KERNEL_NAME_PROVIDED}, linux-headers- > {KERNEL_NAME_PROVIDED} and `linux-image-${KERNEL_NAME_PROVIDED}-dbg` > > Then, we make `linux-mainline` build all except `linux-kbuild` (if we don't > need cross version). And we could create the similar recipe `linux-kbuild` > that inherits `linux-custom`, but builds only `linux-kbuild` - with > ISAR_CROSS_COMPILE="0". > > Of course, such an approach breaks the feature "build everything from one > recipe", but it doesn't require "schroot" workaround and would make build > process more configurable. On top, we need a loop in the dpkg class the runs the recipe for multiple archs, not just the target one. Then also drop that special and ugly "-cross" package and just built the kbuild thing for the target and the builder arch. Jan
On Fri, 2022-11-18 at 18:11 +0100, Stefan Koch wrote: > On Tue, 2022-11-15 at 18:23 +0100, Jan Kiszka wrote: > > On 15.11.22 14:44, Uladzimir Bely wrote: > > > In mail from среда, 9 ноября 2022 г. 13:32:45 +03 user Koch, > > > Stefan > > > wrote: > > > > Hi > > > > > > > > This patchset is a set of three patches: > > > > - Support overwriting configured schroot dir > > > > - Split up binaries from kernel headers to kbuild packages > > > > - Update custom_kernel docs for split up of kernel scripts and > > > > tools > > > > > > > > > > Just some thoughts regarding schroot path overwriting... > > > > > > What if `linux-custom` recipe could disable specific components? > > > > Right, I sill wanted to follow up on that: Debian build profiles > > can > > do > > the trick here. Define one for "host tools only", and only build > > that > > in > > the second step. > > - I have sucessfully set up a prototype of second patchset that uses > the debian build profiles approach. I'll send a v2 patchset later... That uses build profiles, now. > > > > > > > > > I mean, currently we have this feature only for `linux-libc-dev` > > > package, that > > > is disabled when "nolibcdev" build profile is selected. If we > > > added > > > similar > > > build profiles for other components, we could disable also > > > `linux-$ > > > {KERNEL_NAME_PROVIDED}`, linux-image-{KERNEL_NAME_PROVIDED}, > > > linux- > > > headers- > > > {KERNEL_NAME_PROVIDED} and `linux-image-${KERNEL_NAME_PROVIDED}- > > > dbg` > > > > > > Then, we make `linux-mainline` build all except `linux-kbuild` > > > (if > > > we don't > > > need cross version). And we could create the similar recipe > > > `linux- > > > kbuild` > > > that inherits `linux-custom`, but builds only `linux-kbuild` - > > > with > > > ISAR_CROSS_COMPILE="0". > > > > > @Bely > Your commit fb61feff moves merge_config.sh call from linux-custom.inc > to debian/isar/configure.tmpl > > The merge_config.sh is very slow in target schroot (seems slower than > the build of the scripts/tools itself), so maybe it should be moved > back to linux-custom.inc to run in host schroot. The .config is now reused from first build case now. So merge_config.sh needs not to be rerun anymore. > > > > Of course, such an approach breaks the feature "build everything > > > from one > > > recipe", but it doesn't require "schroot" workaround and would > > > make > > > build > > > process more configurable. > - Prototype keeps build everythong from one recipe approach. > > > > On top, we need a loop in the dpkg class the runs the recipe for > > multiple archs, not just the target one. Then also drop that > > special > > and > > ugly "-cross" package and just built the kbuild thing for the > > target > > and > > the builder arch. > > - Prototype build using dpkg class without the most linux-custom > schroot customizations. Improvment is ongoing... Finished... A loop runs the default build and optionally when appending "target" it creates the kbuild binaries for the target. When appending "host", the ugly -cross package with invalid traget arch will not be created - but instead a regular kbuild package with host architecture will be created. Just append "target" and/or "host" to ISAR_BUILDS in the recipe that includes linux-custom.inc > > What would a beautiful solution to enable a recipe for multiarch in a > generic way? Any ideas? (Specify that target build should run) > > > > > Jan > > > Merry Christmas and a Happy New Year 2023...
Hi This patchset is a set of three patches: - Support overwriting configured schroot dir - Split up binaries from kernel headers to kbuild packages - Update custom_kernel docs for split up of kernel scripts and tools The main use-case was to swap out the binaries from the kernel headers package. They will be swapped out into host and target specific kernel kbuild packages. The main development goals were these: 1. Solve already known isar custom kernel limitations from doc/custom_kernel.inc - kernel headers package does not support both native and cross compilation of kernel modules when cross built 2. Honor recommendations for future from doc/custom_kernel.inc - Generate kernel headers packages for both host and target when using cross build 3. Add extensions known from debian kernel packages structure - Generate a kernel headers package without binaries - Create specific kernel kbuild packages that will ship the "scripts" and "tools" binaries - Use symlinks to point to the "scripts" and "tools" binaries 4. Be user friendly - Avoid usage of separate kbuild bitbake recipe that may enforce redundant configuration of kernel source definitions with user actions to enable kbuild package generation - Use already known way to include linux-custom.inc in just one own bitbake recipe that provides the kernel source definitions - Keep known user behavior for existing build configurations: just update isar, and kbuild packages will be created automatically Best regards Stefan Stefan Koch (3): sbuild: Support overwriting configured schroot dir linux-custom: Split up binaries from kernel headers to kbuild packages docs: Update custom_kernel docs for split up of kernel scripts and tools doc/custom_kernel.md | 13 +-- meta/classes/sbuild.bbclass | 9 +- .../linux/files/debian/control.tmpl | 18 ++- .../linux/files/debian/isar/build.tmpl | 9 +- .../linux/files/debian/isar/common.tmpl | 7 +- .../linux/files/debian/isar/install.tmpl | 64 +++++++++-- .../linux/files/debian/rules.tmpl | 5 +- meta/recipes-kernel/linux/linux-custom.inc | 104 ++++++++++++++++-- 8 files changed, 191 insertions(+), 38 deletions(-)