| Message ID | 20240912151121.212959-2-cedric.hombourger@siemens.com |
|---|---|
| State | Under Review |
| Headers | show |
| Series | dpkg-raw: make sure do_install gets called for -compat/-native builds | expand |
On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote: > With sources being built only once, do_prepare_build (and its dependencies) > are skipped when building -compat or -native packages. This unfortunately > means that do_install is not executed and is causing the package build > to fail as the install step is not finding any files in ${PP}/image. To > avoid changing how do_install is scheduled in the normal case, it is simply > added as an extra dependency to do_dpkg_build to make sure it gets executed > prior to the build of the binary package. What is the use case for raw-compat and -native? Are you actually having a case with different binaries then? > > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> > --- > meta/classes/dpkg-raw.bbclass | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass > index dd7b761f..94aba1aa 100644 > --- a/meta/classes/dpkg-raw.bbclass > +++ b/meta/classes/dpkg-raw.bbclass > @@ -15,6 +15,12 @@ do_install() { > do_install[cleandirs] = "${D}" > addtask install after do_patch do_transform_template before do_prepare_build > > +# make sure do_install gets executed when building -compat / -native packages > +# are built since do_prepare_build would be skipped for those (sources are > +# built only once). To avoid changing how do_install is scheduled in the normal > +# case, it is simply added as an extra dependency to do_dpkg_build > +do_dpkg_build[depends] += "${PN}:do_install" > + So, this class would then violate the rule of shared, identical "source" packages? Does not feel correct. > do_prepare_build[cleandirs] += "${S}/debian" > do_prepare_build() { > cd ${D} Jan
On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote: > > With sources being built only once, do_prepare_build (and its > > dependencies) > > are skipped when building -compat or -native packages. This > > unfortunately > > means that do_install is not executed and is causing the package > > build > > to fail as the install step is not finding any files in > > ${PP}/image. To > > avoid changing how do_install is scheduled in the normal case, it > > is simply > > added as an extra dependency to do_dpkg_build to make sure it gets > > executed > > prior to the build of the binary package. > > What is the use case for raw-compat and -native? Are you actually > having > a case with different binaries then? For my specific case, I had wished -native would not be built because my dpkg-raw package has DPKG_ARCH set to "all". In that case, we should not have a separate build for -native The issue will occur for any "dpkg" package being built for -native when it DEPENDS on a "dpkg-raw" package because our native class automatically changes all dependencies to -native While most "dpkg-raw" packages should have DPKG_ARCH = "all", many recipes did not bother setting this (in most downstream layers that I have seen) and will end up with different binary packages in their - native builds when HOST_ARCH != DISTRO_ARCH To reproduce the issue that I have seen (and which should be fixed IMO): 1. bitbake mc:qemuarm64-bookworm:example-raw 2. bitbake mc:qemuarm64-bookworm:example-raw-native PS: I am planning to investigate how to suppress the duplicate build of "dpkg-raw" package that has DPKG_ARCH set to "all" when a dependent "dpkg" package is built for both variants I believe the issue demonstrated with the above scenario needs fixing (the error message you otherwise get as a user is rather cryptic) > > > > > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> > > --- > > meta/classes/dpkg-raw.bbclass | 6 ++++++ > > 1 file changed, 6 insertions(+) > > > > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg- > > raw.bbclass > > index dd7b761f..94aba1aa 100644 > > --- a/meta/classes/dpkg-raw.bbclass > > +++ b/meta/classes/dpkg-raw.bbclass > > @@ -15,6 +15,12 @@ do_install() { > > do_install[cleandirs] = "${D}" > > addtask install after do_patch do_transform_template before > > do_prepare_build > > > > +# make sure do_install gets executed when building -compat / - > > native packages > > +# are built since do_prepare_build would be skipped for those > > (sources are > > +# built only once). To avoid changing how do_install is scheduled > > in the normal > > +# case, it is simply added as an extra dependency to do_dpkg_build > > +do_dpkg_build[depends] += "${PN}:do_install" > > + > > So, this class would then violate the rule of shared, identical > "source" > packages? Does not feel correct. No same sources are used between non-native and -native. See below Tasks for example-raw: $ cat build-test-raw-native/tmp/work/debian-bookworm-arm64/example- raw/0.3-r0/temp/log.task_order 20240912-143135.942912 do_fetch (171631): log.do_fetch.171631 20240912-143135.967641 do_get_reference_env (171636): log.do_get_reference_env.171636 20240912-143144.778512 do_unpack (171697): log.do_unpack.171697 20240912-143145.280183 do_adjust_git (171767): log.do_adjust_git.171767 20240912-143145.369652 do_transform_template (171780): log.do_transform_template.171780 20240912-143145.477501 do_patch (171793): log.do_patch.171793 20240912-143145.964197 do_install (171849): log.do_install.171849 20240912-143146.108066 do_prepare_build (171883): log.do_prepare_build.171883 20240912-143146.247279 do_dpkg_source (172015): log.do_dpkg_source.172015 20240912-143146.303464 do_local_isarapt (172169): log.do_local_isarapt.172169 20240912-143146.517836 do_deploy_source (172513): log.do_deploy_source.172513 20240912-143457.684464 do_dpkg_build (195129): log.do_dpkg_build.195129 20240912-143527.279878 do_deploy_deb (197150): log.do_deploy_deb.197150 Tasks for example-raw-native (with my changes): $ cat build-test-raw-native/tmp/work/debian-bookworm-arm64/example-raw- native/0.3-r0/temp/log.task_order 20240912-143632.541485 do_fetch (197337): log.do_fetch.197337 20240912-143632.548091 do_get_reference_env (197338): log.do_get_reference_env.197338 20240912-143632.555547 do_local_isarapt (197341): log.do_local_isarapt.197341 20240912-143639.819570 do_unpack (197499): log.do_unpack.197499 20240912-143640.047376 do_adjust_git (197743): log.do_adjust_git.197743 20240912-143640.160769 do_transform_template (198038): log.do_transform_template.198038 20240912-143640.267071 do_patch (198156): log.do_patch.198156 20240912-143640.593967 do_install (198222): log.do_install.198222 20240912-143742.934586 do_fetch_common_source (210951): log.do_fetch_common_source.210951 20240912-143743.720108 do_dpkg_build (211215): log.do_dpkg_build.211215 20240912-143748.362105 do_deploy_deb (212659): log.do_deploy_deb.212659 Without my changes, do_install would not be executed and do_dpkg_build would fail miserably. run.do_dpkg_build will find the source package (more specifically the .dsc file) in the work tree and use it I therefore believe that identical sources are used between -native and non-native Am I missing something? > > > do_prepare_build[cleandirs] += "${S}/debian" > > do_prepare_build() { > > cd ${D} > > Jan >
On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: >> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote: >>> With sources being built only once, do_prepare_build (and its >>> dependencies) >>> are skipped when building -compat or -native packages. This >>> unfortunately >>> means that do_install is not executed and is causing the package >>> build >>> to fail as the install step is not finding any files in >>> ${PP}/image. To >>> avoid changing how do_install is scheduled in the normal case, it >>> is simply >>> added as an extra dependency to do_dpkg_build to make sure it gets >>> executed >>> prior to the build of the binary package. >> >> What is the use case for raw-compat and -native? Are you actually >> having >> a case with different binaries then? > > For my specific case, I had wished -native would not be built because > my dpkg-raw package has DPKG_ARCH set to "all". In that case, we should > not have a separate build for -native > > The issue will occur for any "dpkg" package being built for -native > when it DEPENDS on a "dpkg-raw" package because our native class > automatically changes all dependencies to -native > Then stub-out the other variants, make them depend on the BPN package, but do not further foster their build. Jan
On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote: > > > > With sources being built only once, do_prepare_build (and its > > > > dependencies) > > > > are skipped when building -compat or -native packages. This > > > > unfortunately > > > > means that do_install is not executed and is causing the > > > > package > > > > build > > > > to fail as the install step is not finding any files in > > > > ${PP}/image. To > > > > avoid changing how do_install is scheduled in the normal case, > > > > it > > > > is simply > > > > added as an extra dependency to do_dpkg_build to make sure it > > > > gets > > > > executed > > > > prior to the build of the binary package. > > > > > > What is the use case for raw-compat and -native? Are you actually > > > having > > > a case with different binaries then? > > > > For my specific case, I had wished -native would not be built > > because > > my dpkg-raw package has DPKG_ARCH set to "all". In that case, we > > should > > not have a separate build for -native > > > > The issue will occur for any "dpkg" package being built for -native > > when it DEPENDS on a "dpkg-raw" package because our native class > > automatically changes all dependencies to -native > > > > Then stub-out the other variants, make them depend on the BPN > package, > but do not further foster their build. My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN names But we have multiarch [1] suffixing each of our DEPENDS with -native when built for -native Or maybe I am not understanding your proposal? I am pretty sure we have a bug in Isar. I can build a test case if the illustration with example-raw is not sufficient / convincing [1] https://github.com/ilbers/isar/blob/master/meta/classes/multiarch.bbclass#L73 > > Jan >
On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: >> On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote: >>> On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: >>>> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote: >>>>> With sources being built only once, do_prepare_build (and its >>>>> dependencies) >>>>> are skipped when building -compat or -native packages. This >>>>> unfortunately >>>>> means that do_install is not executed and is causing the >>>>> package >>>>> build >>>>> to fail as the install step is not finding any files in >>>>> ${PP}/image. To >>>>> avoid changing how do_install is scheduled in the normal case, >>>>> it >>>>> is simply >>>>> added as an extra dependency to do_dpkg_build to make sure it >>>>> gets >>>>> executed >>>>> prior to the build of the binary package. >>>> >>>> What is the use case for raw-compat and -native? Are you actually >>>> having >>>> a case with different binaries then? >>> >>> For my specific case, I had wished -native would not be built >>> because >>> my dpkg-raw package has DPKG_ARCH set to "all". In that case, we >>> should >>> not have a separate build for -native >>> >>> The issue will occur for any "dpkg" package being built for -native >>> when it DEPENDS on a "dpkg-raw" package because our native class >>> automatically changes all dependencies to -native >>> >> >> Then stub-out the other variants, make them depend on the BPN >> package, >> but do not further foster their build. > > My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN names > But we have multiarch [1] suffixing each of our DEPENDS with -native > when built for -native > > Or maybe I am not understanding your proposal? I am pretty sure we have > a bug in Isar. I can build a test case if the illustration with > example-raw is not sufficient / convincing Yes, this is an issue in isar. But I would address it there by not doing anything in <raw>-native/compat targets, only making them depend on their base target. IIRC, we already have such a case deep in the kernel build. Rebuilding the same thing is wrong. Jan
On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote: > On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: > > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: > > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote: > > > > > > With sources being built only once, do_prepare_build (and > > > > > > its > > > > > > dependencies) > > > > > > are skipped when building -compat or -native packages. This > > > > > > unfortunately > > > > > > means that do_install is not executed and is causing the > > > > > > package > > > > > > build > > > > > > to fail as the install step is not finding any files in > > > > > > ${PP}/image. To > > > > > > avoid changing how do_install is scheduled in the normal > > > > > > case, > > > > > > it > > > > > > is simply > > > > > > added as an extra dependency to do_dpkg_build to make sure > > > > > > it > > > > > > gets > > > > > > executed > > > > > > prior to the build of the binary package. > > > > > > > > > > What is the use case for raw-compat and -native? Are you > > > > > actually > > > > > having > > > > > a case with different binaries then? > > > > > > > > For my specific case, I had wished -native would not be built > > > > because > > > > my dpkg-raw package has DPKG_ARCH set to "all". In that case, > > > > we > > > > should > > > > not have a separate build for -native > > > > > > > > The issue will occur for any "dpkg" package being built for - > > > > native > > > > when it DEPENDS on a "dpkg-raw" package because our native > > > > class > > > > automatically changes all dependencies to -native > > > > > > > > > > Then stub-out the other variants, make them depend on the BPN > > > package, > > > but do not further foster their build. > > > > My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN > > names > > But we have multiarch [1] suffixing each of our DEPENDS with - > > native > > when built for -native > > > > Or maybe I am not understanding your proposal? I am pretty sure we > > have > > a bug in Isar. I can build a test case if the illustration with > > example-raw is not sufficient / convincing > > Yes, this is an issue in isar. But I would address it there by not > doing > anything in <raw>-native/compat targets, only making them depend on > their base target. IIRC, we already have such a case deep in the > kernel > build. > > Rebuilding the same thing is wrong. Agreed but :) There are two cases to consider 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for - native 2) dpkg-raw produces an architecture-dependent package (Architecture: any) => we would need to build both -native and non-native I hit 1) and I am investigating further (getting myself familiar with the multiarch code) and 2) is IMO addressed with this patch Actually, when I first hit the issue, my dpkg-raw package did not have DPKG_ARCH set to "all" hence my first instinct was to set it to avoid the extra build > > Jan >
On Fri, 2024-09-13 at 13:12 +0200, Cedric Hombourger wrote: > On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote: > > On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > > > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: > > > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) > > > > wrote: > > > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: > > > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users > > > > > > wrote: > > > > > > > With sources being built only once, do_prepare_build (and > > > > > > > its > > > > > > > dependencies) > > > > > > > are skipped when building -compat or -native packages. > > > > > > > This > > > > > > > unfortunately > > > > > > > means that do_install is not executed and is causing the > > > > > > > package > > > > > > > build > > > > > > > to fail as the install step is not finding any files in > > > > > > > ${PP}/image. To > > > > > > > avoid changing how do_install is scheduled in the normal > > > > > > > case, > > > > > > > it > > > > > > > is simply > > > > > > > added as an extra dependency to do_dpkg_build to make > > > > > > > sure > > > > > > > it > > > > > > > gets > > > > > > > executed > > > > > > > prior to the build of the binary package. > > > > > > > > > > > > What is the use case for raw-compat and -native? Are you > > > > > > actually > > > > > > having > > > > > > a case with different binaries then? > > > > > > > > > > For my specific case, I had wished -native would not be built > > > > > because > > > > > my dpkg-raw package has DPKG_ARCH set to "all". In that case, > > > > > we > > > > > should > > > > > not have a separate build for -native > > > > > > > > > > The issue will occur for any "dpkg" package being built for - > > > > > native > > > > > when it DEPENDS on a "dpkg-raw" package because our native > > > > > class > > > > > automatically changes all dependencies to -native > > > > > > > > > > > > > Then stub-out the other variants, make them depend on the BPN > > > > package, > > > > but do not further foster their build. > > > > > > My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN > > > names > > > But we have multiarch [1] suffixing each of our DEPENDS with - > > > native > > > when built for -native > > > > > > Or maybe I am not understanding your proposal? I am pretty sure > > > we > > > have > > > a bug in Isar. I can build a test case if the illustration with > > > example-raw is not sufficient / convincing > > > > Yes, this is an issue in isar. But I would address it there by not > > doing > > anything in <raw>-native/compat targets, only making them depend on > > their base target. IIRC, we already have such a case deep in the > > kernel > > build. > > > > Rebuilding the same thing is wrong. > > Agreed but :) > > There are two cases to consider > > 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for - > native > > 2) dpkg-raw produces an architecture-dependent package (Architecture: > any) => we would need to build both -native and non-native > > I hit 1) and I am investigating further (getting myself familiar with > the multiarch code) and 2) is IMO addressed with this patch > > Actually, when I first hit the issue, my dpkg-raw package did not > have > DPKG_ARCH set to "all" hence my first instinct was to set it to avoid > the extra build > For 1) and if my reading of multiarch is correct, the following should do the trick: diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass index bb0f7983..8aa2de64 100644 --- a/meta/classes/multiarch.bbclass +++ b/meta/classes/multiarch.bbclass @@ -10,7 +10,7 @@ python() { d.appendVar('BBCLASSEXTEND', ' compat') # build native separately only when it differs from the target variant - if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'): + if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH') or d.getVar('DPKG_ARCH') == 'all': pn = d.getVar('PN') if not pn.endswith('-native') and not pn.endswith('-compat'): provides = (d.getVar('PROVIDES') or '').split() This appears to work in the quick test I did but will perform more tests > > > > Jan > > >
On 13.09.24 13:12, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote: >> On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote: >>> On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: >>>> On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote: >>>>> On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: >>>>>> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote: >>>>>>> With sources being built only once, do_prepare_build (and >>>>>>> its >>>>>>> dependencies) >>>>>>> are skipped when building -compat or -native packages. This >>>>>>> unfortunately >>>>>>> means that do_install is not executed and is causing the >>>>>>> package >>>>>>> build >>>>>>> to fail as the install step is not finding any files in >>>>>>> ${PP}/image. To >>>>>>> avoid changing how do_install is scheduled in the normal >>>>>>> case, >>>>>>> it >>>>>>> is simply >>>>>>> added as an extra dependency to do_dpkg_build to make sure >>>>>>> it >>>>>>> gets >>>>>>> executed >>>>>>> prior to the build of the binary package. >>>>>> >>>>>> What is the use case for raw-compat and -native? Are you >>>>>> actually >>>>>> having >>>>>> a case with different binaries then? >>>>> >>>>> For my specific case, I had wished -native would not be built >>>>> because >>>>> my dpkg-raw package has DPKG_ARCH set to "all". In that case, >>>>> we >>>>> should >>>>> not have a separate build for -native >>>>> >>>>> The issue will occur for any "dpkg" package being built for - >>>>> native >>>>> when it DEPENDS on a "dpkg-raw" package because our native >>>>> class >>>>> automatically changes all dependencies to -native >>>>> >>>> >>>> Then stub-out the other variants, make them depend on the BPN >>>> package, >>>> but do not further foster their build. >>> >>> My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN >>> names >>> But we have multiarch [1] suffixing each of our DEPENDS with - >>> native >>> when built for -native >>> >>> Or maybe I am not understanding your proposal? I am pretty sure we >>> have >>> a bug in Isar. I can build a test case if the illustration with >>> example-raw is not sufficient / convincing >> >> Yes, this is an issue in isar. But I would address it there by not >> doing >> anything in <raw>-native/compat targets, only making them depend on >> their base target. IIRC, we already have such a case deep in the >> kernel >> build. >> >> Rebuilding the same thing is wrong. > > Agreed but :) > > There are two cases to consider > > 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for - > native > > 2) dpkg-raw produces an architecture-dependent package (Architecture: > any) => we would need to build both -native and non-native > > I hit 1) and I am investigating further (getting myself familiar with > the multiarch code) and 2) is IMO addressed with this patch No, your patch is even for that case wrong because it enforces source package rebuilds that are no longer expected and supported - same base target, same source package. Jan > > Actually, when I first hit the issue, my dpkg-raw package did not have > DPKG_ARCH set to "all" hence my first instinct was to set it to avoid > the extra build > >> >> Jan >> >
On 13.09.24 13:33, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > On Fri, 2024-09-13 at 13:12 +0200, Cedric Hombourger wrote: >> On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote: >>> On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote: >>>> On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: >>>>> On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) >>>>> wrote: >>>>>> On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: >>>>>>> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users >>>>>>> wrote: >>>>>>>> With sources being built only once, do_prepare_build (and >>>>>>>> its >>>>>>>> dependencies) >>>>>>>> are skipped when building -compat or -native packages. >>>>>>>> This >>>>>>>> unfortunately >>>>>>>> means that do_install is not executed and is causing the >>>>>>>> package >>>>>>>> build >>>>>>>> to fail as the install step is not finding any files in >>>>>>>> ${PP}/image. To >>>>>>>> avoid changing how do_install is scheduled in the normal >>>>>>>> case, >>>>>>>> it >>>>>>>> is simply >>>>>>>> added as an extra dependency to do_dpkg_build to make >>>>>>>> sure >>>>>>>> it >>>>>>>> gets >>>>>>>> executed >>>>>>>> prior to the build of the binary package. >>>>>>> >>>>>>> What is the use case for raw-compat and -native? Are you >>>>>>> actually >>>>>>> having >>>>>>> a case with different binaries then? >>>>>> >>>>>> For my specific case, I had wished -native would not be built >>>>>> because >>>>>> my dpkg-raw package has DPKG_ARCH set to "all". In that case, >>>>>> we >>>>>> should >>>>>> not have a separate build for -native >>>>>> >>>>>> The issue will occur for any "dpkg" package being built for - >>>>>> native >>>>>> when it DEPENDS on a "dpkg-raw" package because our native >>>>>> class >>>>>> automatically changes all dependencies to -native >>>>>> >>>>> >>>>> Then stub-out the other variants, make them depend on the BPN >>>>> package, >>>>> but do not further foster their build. >>>> >>>> My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN >>>> names >>>> But we have multiarch [1] suffixing each of our DEPENDS with - >>>> native >>>> when built for -native >>>> >>>> Or maybe I am not understanding your proposal? I am pretty sure >>>> we >>>> have >>>> a bug in Isar. I can build a test case if the illustration with >>>> example-raw is not sufficient / convincing >>> >>> Yes, this is an issue in isar. But I would address it there by not >>> doing >>> anything in <raw>-native/compat targets, only making them depend on >>> their base target. IIRC, we already have such a case deep in the >>> kernel >>> build. >>> >>> Rebuilding the same thing is wrong. >> >> Agreed but :) >> >> There are two cases to consider >> >> 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for - >> native >> >> 2) dpkg-raw produces an architecture-dependent package (Architecture: >> any) => we would need to build both -native and non-native >> >> I hit 1) and I am investigating further (getting myself familiar with >> the multiarch code) and 2) is IMO addressed with this patch >> >> Actually, when I first hit the issue, my dpkg-raw package did not >> have >> DPKG_ARCH set to "all" hence my first instinct was to set it to avoid >> the extra build >> > > For 1) and if my reading of multiarch is correct, the following should > do the trick: > > diff --git a/meta/classes/multiarch.bbclass > b/meta/classes/multiarch.bbclass > index bb0f7983..8aa2de64 100644 > --- a/meta/classes/multiarch.bbclass > +++ b/meta/classes/multiarch.bbclass > @@ -10,7 +10,7 @@ python() { > d.appendVar('BBCLASSEXTEND', ' compat') > > # build native separately only when it differs from the target > variant > - if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'): > + if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH') or > d.getVar('DPKG_ARCH') == 'all': > pn = d.getVar('PN') > if not pn.endswith('-native') and not pn.endswith('-compat'): > provides = (d.getVar('PROVIDES') or '').split() > > > This appears to work in the quick test I did but will perform more > tests > At least for debianized packages (like dpkg-raw) and for -native. Will probably not address -compat dependencies. Yeah, now it's getting complicated. Jan
On Fri, 2024-09-13 at 14:15 +0200, Jan Kiszka wrote: > On 13.09.24 13:12, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > > On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote: > > > On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > > > > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: > > > > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) > > > > > wrote: > > > > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: > > > > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users > > > > > > > wrote: > > > > > > > > With sources being built only once, do_prepare_build > > > > > > > > (and > > > > > > > > its > > > > > > > > dependencies) > > > > > > > > are skipped when building -compat or -native packages. > > > > > > > > This > > > > > > > > unfortunately > > > > > > > > means that do_install is not executed and is causing > > > > > > > > the > > > > > > > > package > > > > > > > > build > > > > > > > > to fail as the install step is not finding any files in > > > > > > > > ${PP}/image. To > > > > > > > > avoid changing how do_install is scheduled in the > > > > > > > > normal > > > > > > > > case, > > > > > > > > it > > > > > > > > is simply > > > > > > > > added as an extra dependency to do_dpkg_build to make > > > > > > > > sure > > > > > > > > it > > > > > > > > gets > > > > > > > > executed > > > > > > > > prior to the build of the binary package. > > > > > > > > > > > > > > What is the use case for raw-compat and -native? Are you > > > > > > > actually > > > > > > > having > > > > > > > a case with different binaries then? > > > > > > > > > > > > For my specific case, I had wished -native would not be > > > > > > built > > > > > > because > > > > > > my dpkg-raw package has DPKG_ARCH set to "all". In that > > > > > > case, > > > > > > we > > > > > > should > > > > > > not have a separate build for -native > > > > > > > > > > > > The issue will occur for any "dpkg" package being built for > > > > > > - > > > > > > native > > > > > > when it DEPENDS on a "dpkg-raw" package because our native > > > > > > class > > > > > > automatically changes all dependencies to -native > > > > > > > > > > > > > > > > Then stub-out the other variants, make them depend on the BPN > > > > > package, > > > > > but do not further foster their build. > > > > > > > > My "dpkg" recipe is clean, it has a DEPENDS clause with only > > > > BPN > > > > names > > > > But we have multiarch [1] suffixing each of our DEPENDS with - > > > > native > > > > when built for -native > > > > > > > > Or maybe I am not understanding your proposal? I am pretty sure > > > > we > > > > have > > > > a bug in Isar. I can build a test case if the illustration with > > > > example-raw is not sufficient / convincing > > > > > > Yes, this is an issue in isar. But I would address it there by > > > not > > > doing > > > anything in <raw>-native/compat targets, only making them depend > > > on > > > their base target. IIRC, we already have such a case deep in the > > > kernel > > > build. > > > > > > Rebuilding the same thing is wrong. > > > > Agreed but :) > > > > There are two cases to consider > > > > 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for > > - > > native > > > > 2) dpkg-raw produces an architecture-dependent package > > (Architecture: > > any) => we would need to build both -native and non-native > > > > I hit 1) and I am investigating further (getting myself familiar > > with > > the multiarch code) and 2) is IMO addressed with this patch > > No, your patch is even for that case wrong because it enforces source > package rebuilds that are no longer expected and supported - same > base > target, same source package. Are you sure? My understanding of the code and testing (with changes) is showing that do_dpkg_source is only called once as as expected for the non-native variant, for the -native case, do_fetch_common_source is called: the source package to build -native is pulled from isar-apt It is in fact one of the reason this patch is dependent on [2]. Without this, the build will try to get its inputs from /home/builder/example- raw/image because sources used to build -native were created when the non-native package was built but would have been put under /home/builder/example-raw-native/ without [2] [2] https://groups.google.com/g/isar-users/c/md3gmDvhEPQ > > Jan > > > > > Actually, when I first hit the issue, my dpkg-raw package did not > > have > > DPKG_ARCH set to "all" hence my first instinct was to set it to > > avoid > > the extra build > > > > > > > > Jan > > > > > >
On Fri, 2024-09-13 at 14:28 +0200, Jan Kiszka wrote: > On 13.09.24 13:33, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > > On Fri, 2024-09-13 at 13:12 +0200, Cedric Hombourger wrote: > > > On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote: > > > > On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) > > > > wrote: > > > > > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote: > > > > > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) > > > > > > wrote: > > > > > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote: > > > > > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users > > > > > > > > wrote: > > > > > > > > > With sources being built only once, do_prepare_build > > > > > > > > > (and > > > > > > > > > its > > > > > > > > > dependencies) > > > > > > > > > are skipped when building -compat or -native > > > > > > > > > packages. > > > > > > > > > This > > > > > > > > > unfortunately > > > > > > > > > means that do_install is not executed and is causing > > > > > > > > > the > > > > > > > > > package > > > > > > > > > build > > > > > > > > > to fail as the install step is not finding any files > > > > > > > > > in > > > > > > > > > ${PP}/image. To > > > > > > > > > avoid changing how do_install is scheduled in the > > > > > > > > > normal > > > > > > > > > case, > > > > > > > > > it > > > > > > > > > is simply > > > > > > > > > added as an extra dependency to do_dpkg_build to make > > > > > > > > > sure > > > > > > > > > it > > > > > > > > > gets > > > > > > > > > executed > > > > > > > > > prior to the build of the binary package. > > > > > > > > > > > > > > > > What is the use case for raw-compat and -native? Are > > > > > > > > you > > > > > > > > actually > > > > > > > > having > > > > > > > > a case with different binaries then? > > > > > > > > > > > > > > For my specific case, I had wished -native would not be > > > > > > > built > > > > > > > because > > > > > > > my dpkg-raw package has DPKG_ARCH set to "all". In that > > > > > > > case, > > > > > > > we > > > > > > > should > > > > > > > not have a separate build for -native > > > > > > > > > > > > > > The issue will occur for any "dpkg" package being built > > > > > > > for - > > > > > > > native > > > > > > > when it DEPENDS on a "dpkg-raw" package because our > > > > > > > native > > > > > > > class > > > > > > > automatically changes all dependencies to -native > > > > > > > > > > > > > > > > > > > Then stub-out the other variants, make them depend on the > > > > > > BPN > > > > > > package, > > > > > > but do not further foster their build. > > > > > > > > > > My "dpkg" recipe is clean, it has a DEPENDS clause with only > > > > > BPN > > > > > names > > > > > But we have multiarch [1] suffixing each of our DEPENDS with > > > > > - > > > > > native > > > > > when built for -native > > > > > > > > > > Or maybe I am not understanding your proposal? I am pretty > > > > > sure > > > > > we > > > > > have > > > > > a bug in Isar. I can build a test case if the illustration > > > > > with > > > > > example-raw is not sufficient / convincing > > > > > > > > Yes, this is an issue in isar. But I would address it there by > > > > not > > > > doing > > > > anything in <raw>-native/compat targets, only making them > > > > depend on > > > > their base target. IIRC, we already have such a case deep in > > > > the > > > > kernel > > > > build. > > > > > > > > Rebuilding the same thing is wrong. > > > > > > Agreed but :) > > > > > > There are two cases to consider > > > > > > 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild > > > for - > > > native > > > > > > 2) dpkg-raw produces an architecture-dependent package > > > (Architecture: > > > any) => we would need to build both -native and non-native > > > > > > I hit 1) and I am investigating further (getting myself familiar > > > with > > > the multiarch code) and 2) is IMO addressed with this patch > > > > > > Actually, when I first hit the issue, my dpkg-raw package did not > > > have > > > DPKG_ARCH set to "all" hence my first instinct was to set it to > > > avoid > > > the extra build > > > > > > > For 1) and if my reading of multiarch is correct, the following > > should > > do the trick: > > > > diff --git a/meta/classes/multiarch.bbclass > > b/meta/classes/multiarch.bbclass > > index bb0f7983..8aa2de64 100644 > > --- a/meta/classes/multiarch.bbclass > > +++ b/meta/classes/multiarch.bbclass > > @@ -10,7 +10,7 @@ python() { > > d.appendVar('BBCLASSEXTEND', ' compat') > > > > # build native separately only when it differs from the target > > variant > > - if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'): > > + if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH') or > > d.getVar('DPKG_ARCH') == 'all': > > pn = d.getVar('PN') > > if not pn.endswith('-native') and not pn.endswith('- > > compat'): > > provides = (d.getVar('PROVIDES') or '').split() > > > > > > This appears to work in the quick test I did but will perform more > > tests > > > > At least for debianized packages (like dpkg-raw) and for -native. > Will > probably not address -compat dependencies. Yeah, now it's getting > complicated. if DPKG_ARCH == all, wouldn't we expect the recipe to have PROVIDES set to "example-raw-compat example-raw-native" since whatever flavor would result in an _all.deb package being generated? > > Jan >
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass index dd7b761f..94aba1aa 100644 --- a/meta/classes/dpkg-raw.bbclass +++ b/meta/classes/dpkg-raw.bbclass @@ -15,6 +15,12 @@ do_install() { do_install[cleandirs] = "${D}" addtask install after do_patch do_transform_template before do_prepare_build +# make sure do_install gets executed when building -compat / -native packages +# are built since do_prepare_build would be skipped for those (sources are +# built only once). To avoid changing how do_install is scheduled in the normal +# case, it is simply added as an extra dependency to do_dpkg_build +do_dpkg_build[depends] += "${PN}:do_install" + do_prepare_build[cleandirs] += "${S}/debian" do_prepare_build() { cd ${D}
With sources being built only once, do_prepare_build (and its dependencies) are skipped when building -compat or -native packages. This unfortunately means that do_install is not executed and is causing the package build to fail as the install step is not finding any files in ${PP}/image. To avoid changing how do_install is scheduled in the normal case, it is simply added as an extra dependency to do_dpkg_build to make sure it gets executed prior to the build of the binary package. Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com> --- meta/classes/dpkg-raw.bbclass | 6 ++++++ 1 file changed, 6 insertions(+)