| Message ID | 20250307122558.1882902-1-srinuvasan.a@siemens.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2,1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap | expand |
On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > From: srinuvasan <srinuvasan.a@siemens.com> > > Without this option, mmdebstrap deletes the var/log/dpkg.log and its siblings by default, > but this is needed to download deb-src packages for base-apt creation. > Basically, we are referring to the package's status under /var/log/dpkg.log file > and performing the debsrc_download operation. > > Without these changes, bootstrap related deb-src files are not downloaded > under the /downloads/deb-src folder, due to this the repo sanity check > functions failed during base-apt caching stage. > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > --- > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > index 931f6f13..7380a658 100644 > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > @@ -168,6 +168,11 @@ do_bootstrap() { > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > fi > > + cleanup= > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ]; then > + cleanup="--skip=cleanup/reproducible" > + fi > + > # Cleanup mounts if fails > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat "${WORKDIR}/mmtmpdir") \ > @@ -183,6 +188,7 @@ do_bootstrap() { > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \ > $arch_param \ > + $cleanup \ > --mode=unshare \ > ${MMHOOKS} \ > --setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \ What has changed in this version? I'm not seeing my concerns addressed, my questions answered. Jan
On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote: > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > > From: srinuvasan <srinuvasan.a@siemens.com> > > > > Without this option, mmdebstrap deletes the var/log/dpkg.log and > > its siblings by default, > > but this is needed to download deb-src packages for base-apt > > creation. I don't see where deb-src needs dpkg.log. Can you shed some light? Both the old code and the code I recently changed parse the .deb files that were downloaded to determine which source packages should be downloaded Ref: https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y > > Basically, we are referring to the package's status under > > /var/log/dpkg.log file > > and performing the debsrc_download operation. > > > > Without these changes, bootstrap related deb-src files are not > > downloaded > > under the /downloads/deb-src folder, due to this the repo sanity > > check > > functions failed during base-apt caching stage. What are these repo sanity checks? Where do we see the failures? From my base-apt workdir: pwd && grep -ir error <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0-r0/temp run.repo.7287: echo "ERROR: $*" We probably need to be more specific if we are mentioning a failure. Without this, people on this list will not understand what problem(s) you are fixing > > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > > --- > > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++ > > 1 file changed, 6 insertions(+) > > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > index 931f6f13..7380a658 100644 > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > @@ -168,6 +168,11 @@ do_bootstrap() { > > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > > fi > > > > + cleanup= > > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb- > > src', 'yes', 'no', d)}" = "yes" ]; then > > + cleanup="--skip=cleanup/reproducible" > > + fi > > + > > # Cleanup mounts if fails > > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat > > "${WORKDIR}/mmtmpdir") \ > > @@ -183,6 +188,7 @@ do_bootstrap() { > > > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \ > > $arch_param \ > > + $cleanup \ > > --mode=unshare \ > > ${MMHOOKS} \ > > --setup-hook='mkdir -p > > "$1/var/cache/apt/archives/"' \ > > What has changed in this version? I'm not seeing my concerns > addressed, > my questions answered. +1 Please make sure new revisions of your patch series include a changelog (use --cover-letter with git format-patch) > > Jan >
On Saturday, March 8, 2025 at 3:30:04 AM UTC+5:30 Jan Kiszka wrote: On 07.03.25 13:25, srinuv...@siemens.com wrote: > From: srinuvasan <srinuv...@siemens.com> > > Without this option, mmdebstrap deletes the var/log/dpkg.log and its siblings by default, > but this is needed to download deb-src packages for base-apt creation. > Basically, we are referring to the package's status under /var/log/dpkg.log file > and performing the debsrc_download operation. > > Without these changes, bootstrap related deb-src files are not downloaded > under the /downloads/deb-src folder, due to this the repo sanity check > functions failed during base-apt caching stage. > > Signed-off-by: srinuvasan <srinuv...@siemens.com> > --- > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > index 931f6f13..7380a658 100644 > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > @@ -168,6 +168,11 @@ do_bootstrap() { > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > fi > > + cleanup= > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ]; then > + cleanup="--skip=cleanup/reproducible" > + fi > + > # Cleanup mounts if fails > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat "${WORKDIR}/mmtmpdir") \ > @@ -183,6 +188,7 @@ do_bootstrap() { > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \ > $arch_param \ > + $cleanup \ > --mode=unshare \ > ${MMHOOKS} \ > --setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \ What has changed in this version? I'm not seeing my concerns addressed, my questions answered. Hi jan, I will come up with the more details for your query, this version address the failures reported by Uladzimir Bely. Jan
-----Original Message----- From: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com> Sent: 08 March 2025 19:15 To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com> Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com Subject: Re: [PATCH v2 1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote: > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > > From: srinuvasan <srinuvasan.a@siemens.com> > > > > Without this option, mmdebstrap deletes the var/log/dpkg.log and its > > siblings by default, but this is needed to download deb-src packages > > for base-apt creation. I don't see where deb-src needs dpkg.log. Can you shed some light? Both the old code and the code I recently changed parse the .deb files that were downloaded to determine which source packages should be downloaded Ref: https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y > > Basically, we are referring to the package's status under > > /var/log/dpkg.log file and performing the debsrc_download operation. > > > > Without these changes, bootstrap related deb-src files are not > > downloaded under the /downloads/deb-src folder, due to this the repo > > sanity check functions failed during base-apt caching stage. What are these repo sanity checks? Where do we see the failures? From my base-apt workdir: pwd && grep -ir error <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0-r0/temp run.repo.7287: echo "ERROR: $*" We probably need to be more specific if we are mentioning a failure. Without this, people on this list will not understand what problem(s) you are fixing Hi cedric, jan I will explain the issue in the details below: To reproduce the issue ( Ref: https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages ): Trigger the warmup build to download deb and deb-src packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in conf/local.conf) e.x: bitbake mc:qemuamd64-bookworm:isar-image-base Once all the deb and deb-src downloaded for the particular BSP under /download folder Trigger the base-apt target to cache all the deb and deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf) e.x: bitbake base-apt You will see the failures: Logs: NOTE: Resolving any missing task queue dependencies Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% match, 0% complete)############################################# | ETA: 0:00:00 Initialising tasks: 100% |####################################################################################################################| Time: 0:00:00 NOTE: No setscene tasks NOTE: Executing Tasks ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in repo. bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6-2_all.deb bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian-archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9-1_all.deb bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find-rule-perl/libfile-find-rule-perl_0.34-3_all.deb bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber-compare-perl/libnumber-compare-perl_0.03-3_all.deb bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2-6+deb12u1_all.deb bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage-common_3.4-1_all.deb bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob-perl/libtext-glob-perl_0.11-3_all.deb bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules-5.36_5.36.0-7+deb12u1_all.deb bookworm six 1.16.0-4 pool/main/s/six/python3-six_1.16.0-4_all.deb bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b-0+deb12u1_all.deb bookworm usrmerge 37~deb12u1 pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb Just i captured few packages, but many of the deb packages don't have their source files In the base-apt recipes we are calling the repo_sanity_test function finally to make sure all the deb's having the corresponding deb-src packages should be present in the base-apt. Yes, as you said, we are parsing deb files (from /var/cache/apt/archives) that were downloaded to determine which source packages should be downloaded. but in the function "is_not_part_of_current_build "${package}" we are skipping the download-src if parsing deb package (status installed) is not available in the below files ``` "${IMAGE_ROOTFS}"/var/log/dpkg.log \ "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log ``` As I mentioned above, the adduser package was installed in the stage of bootstrap, but that package's status was not captured in the bootstrap's /var/log/dpkg.log file. because mmdebstrap deletes those files by default, hence we didn't capture those bootstrap package status. As we know bootstrap rootfs is used for sbuildchroot creation, but here whatever packages are installed during sbuild-chroot creation, ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the sbuildchroot related package status, but our expectation is it should have bootstrap + sbuildchroot package status available in /var/log/dpkg.log file in sbuildchroot. My first patch solves the above issue with the introduction of "skip=cleanup/reproducible", the bootstrap rootfs generated with /var/log/dpkg.log files, which contain all the bootstrap package status along with that if you use that boostrap for sbuildchroot creation , the sbuildchroot rootfs contains the existing bootstrap plus additionally installed sbuildchroot related packages status is available in sbuldchroot /var/log/dpkg.log file My second patch resolve the below issue: During image creation (IMAGE_FSTYPES = "wic") few packages were installed in the session (e.g: gdisk, bmap-tools) but those packages list available only in the sbuild environment, not outside, carry those packages status to tmp/dpkg_common.log in the rootfs to get the src packages for imager_install related deb packages, presently the cache-deb-src function calls in the ROOTFS_POSTPROCESS_COMMAND but we need to wait till imager generation That's why we pushed that deb-src-caching after image creation. My third path resolve the image rootfs related deb-src missing issue: By default we are removing the /var/log/dpkg.log file in the rootfs_postprocess_clean_log_files function, but this is needed when We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making this function as a configurable one rather than enabled by default. to download IMAGE_ROOTFS related deb-src packages I hope I explain the issue, presently am checking the reproducible part (for jan queries), will update that part very soon. Many thanks, Srinu > > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > > --- > > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++ > > 1 file changed, 6 insertions(+) > > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > index 931f6f13..7380a658 100644 > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > @@ -168,6 +168,11 @@ do_bootstrap() { > > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > > fi > > > > + cleanup= > > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb- > > src', 'yes', 'no', d)}" = "yes" ]; then > > + cleanup="--skip=cleanup/reproducible" > > + fi > > + > > # Cleanup mounts if fails > > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat > > "${WORKDIR}/mmtmpdir") \ > > @@ -183,6 +188,7 @@ do_bootstrap() { > > > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \ > > $arch_param \ > > + $cleanup \ > > --mode=unshare \ > > ${MMHOOKS} \ > > --setup-hook='mkdir -p > > "$1/var/cache/apt/archives/"' \ > > What has changed in this version? I'm not seeing my concerns > addressed, my questions answered. +1 Please make sure new revisions of your patch series include a changelog (use --cover-letter with git format-patch) Sure > > Jan >
On Sat, 2025-03-08 at 18:25 +0000, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote: > > > -----Original Message----- > From: Hombourger, Cedric (FT FDS CES LX) > <cedric.hombourger@siemens.com> > Sent: 08 March 2025 19:15 > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) > <srinuvasan.a@siemens.com> > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com > Subject: Re: [PATCH v2 1/3] mmdebstrap: add > "skip=cleanup/reproducible" option to mmdebstrap > > On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote: > > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > > > From: srinuvasan <srinuvasan.a@siemens.com> > > > > > > Without this option, mmdebstrap deletes the var/log/dpkg.log and > > > its > > > siblings by default, but this is needed to download deb-src > > > packages > > > for base-apt creation. > > I don't see where deb-src needs dpkg.log. Can you shed some light? > Both the old code and the code I recently changed parse the .deb > files that were downloaded to determine which source packages should > be downloaded > > Ref: > https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y > > > > Basically, we are referring to the package's status under > > > /var/log/dpkg.log file and performing the debsrc_download > > > operation. > > > > > > Without these changes, bootstrap related deb-src files are not > > > downloaded under the /downloads/deb-src folder, due to this the > > > repo > > > sanity check functions failed during base-apt caching stage. > > What are these repo sanity checks? > Where do we see the failures? > > From my base-apt workdir: > > pwd && grep -ir error > <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0- > r0/temp > run.repo.7287: echo "ERROR: $*" > > We probably need to be more specific if we are mentioning a failure. > Without this, people on this list will not understand what problem(s) > you are fixing > > > Hi cedric, jan > > I will explain the issue in the details below: > > To reproduce the issue ( Ref: > https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages > ): > > Trigger the warmup build to download deb and deb-src > packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in > conf/local.conf) > > e.x: bitbake mc:qemuamd64-bookworm:isar-image-base > > Once all the deb and deb-src downloaded for the > particular BSP under /download folder > > Trigger the base-apt target to cache all the deb and > deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf) > > e.x: bitbake base-apt > > You will see the failures: > > Logs: > > NOTE: Resolving any missing task queue dependencies > Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% > match, 0% > complete)############################################# | > ETA: 0:00:00 > Initialising tasks: 100% > |#################################################################### > ################################################| Time: 0:00:00 > NOTE: No setscene tasks > NOTE: Executing Tasks > ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in > repo. bookworm adduser 3.134 > pool/main/a/adduser/adduser_3.134_all.deb > bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6- > 2_all.deb > bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb > bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian- > archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb > bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9- > 1_all.deb > bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find- > rule-perl/libfile-find-rule-perl_0.34-3_all.deb > bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber- > compare-perl/libnumber-compare-perl_0.03-3_all.deb > bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2- > 6+deb12u1_all.deb .dsc for pam packages found in our downstream project > bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage- > common_3.4-1_all.deb > bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob- > perl/libtext-glob-perl_0.11-3_all.deb > bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules- > 5.36_5.36.0-7+deb12u1_all.deb > bookworm six 1.16.0-4 pool/main/s/six/python3-six_1.16.0-4_all.deb > bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b- > 0+deb12u1_all.deb > bookworm usrmerge 37~deb12u1 > pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb > bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb > > Just i captured few packages, but many of the deb packages don't have > their source files > > In the base-apt recipes we are calling the repo_sanity_test function > finally to make sure all the deb's having > the corresponding deb-src packages should be present in the base-apt. I checked again log.do_cache in our downstream project and I did not see any of the mentioned errors. $ grep -v 'Export' ./tmp/work/debian-bookworm-any/base-apt/1.0- r0/temp/log.do_cache DEBUG: Executing python function do_cache DEBUG: Using absolute /build/../repo/gnupg/205D601E.pub DEBUG: Executing shell function repo DEBUG: Shell function repo finished DEBUG: Python function do_cache finished That build was however done with the patch I have submitted earlier to this list. It also includes our own layer. I will do a plain Isar build (with my patch) to check if the issue is reproducible. I took a few error cases from your example above and verified that the .dsc files are present in the generated base-apt build. It should be noted that my patch removes "is_not_part_of_current_build()" parsing dpkg.log was IMO not a great idea as I don't think its format is officially documented (I could be wrong though) > > > Yes, as you said, we are parsing deb files (from > /var/cache/apt/archives) that were downloaded to determine which > source packages should be downloaded. > but in the function "is_not_part_of_current_build "${package}" we are > skipping the download-src if parsing deb package (status installed) > is not available in the below files > ``` > "${IMAGE_ROOTFS}"/var/log/dpkg.log \ > "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ > "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ > "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ > "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log > ``` > > As I mentioned above, the adduser package was installed in the stage > of bootstrap, but that package's status was not captured in the > bootstrap's /var/log/dpkg.log file. > because mmdebstrap deletes those files by default, hence we didn't > capture those bootstrap package status. > > As we know bootstrap rootfs is used for sbuildchroot creation, but > here whatever packages are installed during sbuild-chroot creation, > ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the > sbuildchroot related package status, but our expectation is it should > have > bootstrap + sbuildchroot package status available in > /var/log/dpkg.log file in sbuildchroot. > > My first patch solves the above issue > with the introduction of "skip=cleanup/reproducible", the bootstrap > rootfs generated with /var/log/dpkg.log files, which contain all the > bootstrap package status > along with that if you use that boostrap for sbuildchroot creation , > the sbuildchroot rootfs contains the existing bootstrap plus > additionally installed sbuildchroot related packages status is > available in sbuldchroot /var/log/dpkg.log file > > > My second patch resolve the below issue: > > During image creation (IMAGE_FSTYPES = "wic") few packages were > installed in the session (e.g: gdisk, bmap-tools) but those packages > list available > only in the sbuild environment, not outside, carry those packages > status to tmp/dpkg_common.log in the rootfs to get the src packages > for > imager_install related deb packages, presently the cache-deb-src > function calls in the ROOTFS_POSTPROCESS_COMMAND but we need to wait > till imager generation > That's why we pushed that deb-src-caching after image creation. > > My third path resolve the image rootfs related deb-src missing issue: > > By default we are removing the /var/log/dpkg.log file in the > rootfs_postprocess_clean_log_files function, but this is needed when > We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making this > function as a configurable one rather than enabled by default. > to download IMAGE_ROOTFS related deb-src packages > > I hope I explain the issue, presently am checking the reproducible > part (for jan queries), will update that part very soon. > > Many thanks, > Srinu > > > > > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > > > --- > > > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++ > > > 1 file changed, 6 insertions(+) > > > > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar- > > > mmdebstrap.inc > > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > index 931f6f13..7380a658 100644 > > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > @@ -168,6 +168,11 @@ do_bootstrap() { > > > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > > > fi > > > > > > + cleanup= > > > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb- > > > src', 'yes', 'no', d)}" = "yes" ]; then > > > + cleanup="--skip=cleanup/reproducible" > > > + fi > > > + > > > # Cleanup mounts if fails > > > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat > > > "${WORKDIR}/mmtmpdir") \ > > > @@ -183,6 +188,7 @@ do_bootstrap() { > > > > > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args > > > \ > > > $arch_param \ > > > + $cleanup \ > > > --mode=unshare \ > > > ${MMHOOKS} \ > > > --setup-hook='mkdir -p > > > "$1/var/cache/apt/archives/"' \ > > > > What has changed in this version? I'm not seeing my concerns > > addressed, my questions answered. > > +1 > > Please make sure new revisions of your patch series include a > changelog (use --cover-letter with git format-patch) > > Sure > > > > > Jan > > > > -- Cedric Hombourger Siemens AG http://www.siemens.com/
On Sat, 2025-03-08 at 18:25 +0000, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote: > > > -----Original Message----- > From: Hombourger, Cedric (FT FDS CES LX) > <cedric.hombourger@siemens.com> > Sent: 08 March 2025 19:15 > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) > <srinuvasan.a@siemens.com> > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com > Subject: Re: [PATCH v2 1/3] mmdebstrap: add > "skip=cleanup/reproducible" option to mmdebstrap > > On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote: > > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > > > From: srinuvasan <srinuvasan.a@siemens.com> > > > > > > Without this option, mmdebstrap deletes the var/log/dpkg.log and > > > its > > > siblings by default, but this is needed to download deb-src > > > packages > > > for base-apt creation. > > I don't see where deb-src needs dpkg.log. Can you shed some light? > Both the old code and the code I recently changed parse the .deb > files that were downloaded to determine which source packages should > be downloaded > > Ref: > https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y > > > > Basically, we are referring to the package's status under > > > /var/log/dpkg.log file and performing the debsrc_download > > > operation. > > > > > > Without these changes, bootstrap related deb-src files are not > > > downloaded under the /downloads/deb-src folder, due to this the > > > repo > > > sanity check functions failed during base-apt caching stage. > > What are these repo sanity checks? > Where do we see the failures? > > From my base-apt workdir: > > pwd && grep -ir error > <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0- > r0/temp > run.repo.7287: echo "ERROR: $*" > > We probably need to be more specific if we are mentioning a failure. > Without this, people on this list will not understand what problem(s) > you are fixing > > > Hi cedric, jan > > I will explain the issue in the details below: > > To reproduce the issue ( Ref: > https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages > ): > > Trigger the warmup build to download deb and deb-src > packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in > conf/local.conf) > > e.x: bitbake mc:qemuamd64-bookworm:isar-image-base > > Once all the deb and deb-src downloaded for the > particular BSP under /download folder > > Trigger the base-apt target to cache all the deb and > deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf) > > e.x: bitbake base-apt > > You will see the failures: > > Logs: > > NOTE: Resolving any missing task queue dependencies > Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% > match, 0% > complete)############################################# | > ETA: 0:00:00 > Initialising tasks: 100% > |#################################################################### > ################################################| Time: 0:00:00 > NOTE: No setscene tasks > NOTE: Executing Tasks > ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in > repo. bookworm adduser 3.134 > pool/main/a/adduser/adduser_3.134_all.deb .dsc found in our downstream project > bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6- > 2_all.deb > bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb .dsc found in our downstream project > bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian- > archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb > bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9- > 1_all.deb > bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find- > rule-perl/libfile-find-rule-perl_0.34-3_all.deb > bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber- > compare-perl/libnumber-compare-perl_0.03-3_all.deb > bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2- > 6+deb12u1_all.deb > bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage- > common_3.4-1_all.deb > bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob- > perl/libtext-glob-perl_0.11-3_all.deb > bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules- > 5.36_5.36.0-7+deb12u1_all.deb > bookworm six 1.16.0-4 pool/main/s/six/python3-six_1.16.0-4_all.deb > bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b- > 0+deb12u1_all.deb > bookworm usrmerge 37~deb12u1 > pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb > bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb > > Just i captured few packages, but many of the deb packages don't have > their source files > > In the base-apt recipes we are calling the repo_sanity_test function > finally to make sure all the deb's having > the corresponding deb-src packages should be present in the base-apt. > > > Yes, as you said, we are parsing deb files (from > /var/cache/apt/archives) that were downloaded to determine which > source packages should be downloaded. > but in the function "is_not_part_of_current_build "${package}" we are > skipping the download-src if parsing deb package (status installed) > is not available in the below files > ``` > "${IMAGE_ROOTFS}"/var/log/dpkg.log \ > "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ > "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ > "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ > "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log > ``` > > As I mentioned above, the adduser package was installed in the stage > of bootstrap, but that package's status was not captured in the > bootstrap's /var/log/dpkg.log file. > because mmdebstrap deletes those files by default, hence we didn't > capture those bootstrap package status. > > As we know bootstrap rootfs is used for sbuildchroot creation, but > here whatever packages are installed during sbuild-chroot creation, > ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the > sbuildchroot related package status, but our expectation is it should > have > bootstrap + sbuildchroot package status available in > /var/log/dpkg.log file in sbuildchroot. > > My first patch solves the above issue > with the introduction of "skip=cleanup/reproducible", the bootstrap > rootfs generated with /var/log/dpkg.log files, which contain all the > bootstrap package status > along with that if you use that boostrap for sbuildchroot creation , > the sbuildchroot rootfs contains the existing bootstrap plus > additionally installed sbuildchroot related packages status is > available in sbuldchroot /var/log/dpkg.log file > > > My second patch resolve the below issue: > > During image creation (IMAGE_FSTYPES = "wic") few packages were > installed in the session (e.g: gdisk, bmap-tools) but those packages > list available > only in the sbuild environment, not outside, carry those packages > status to tmp/dpkg_common.log in the rootfs to get the src packages > for > imager_install related deb packages, presently the cache-deb-src > function calls in the ROOTFS_POSTPROCESS_COMMAND but we need to wait > till imager generation > That's why we pushed that deb-src-caching after image creation. > > My third path resolve the image rootfs related deb-src missing issue: > > By default we are removing the /var/log/dpkg.log file in the > rootfs_postprocess_clean_log_files function, but this is needed when > We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making this > function as a configurable one rather than enabled by default. > to download IMAGE_ROOTFS related deb-src packages > > I hope I explain the issue, presently am checking the reproducible > part (for jan queries), will update that part very soon. > > Many thanks, > Srinu > > > > > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > > > --- > > > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++ > > > 1 file changed, 6 insertions(+) > > > > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar- > > > mmdebstrap.inc > > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > index 931f6f13..7380a658 100644 > > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > @@ -168,6 +168,11 @@ do_bootstrap() { > > > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > > > fi > > > > > > + cleanup= > > > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb- > > > src', 'yes', 'no', d)}" = "yes" ]; then > > > + cleanup="--skip=cleanup/reproducible" > > > + fi > > > + > > > # Cleanup mounts if fails > > > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat > > > "${WORKDIR}/mmtmpdir") \ > > > @@ -183,6 +188,7 @@ do_bootstrap() { > > > > > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args > > > \ > > > $arch_param \ > > > + $cleanup \ > > > --mode=unshare \ > > > ${MMHOOKS} \ > > > --setup-hook='mkdir -p > > > "$1/var/cache/apt/archives/"' \ > > > > What has changed in this version? I'm not seeing my concerns > > addressed, my questions answered. > > +1 > > Please make sure new revisions of your patch series include a > changelog (use --cover-letter with git format-patch) > > Sure > > > > > Jan > > > > -- Cedric Hombourger Siemens AG http://www.siemens.com/
-----Original Message----- From: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com> Sent: 09 March 2025 00:46 To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com> Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com Subject: Re: [PATCH v2 1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap On Sat, 2025-03-08 at 18:25 +0000, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote: > > > -----Original Message----- > From: Hombourger, Cedric (FT FDS CES LX) > <cedric.hombourger@siemens.com> > Sent: 08 March 2025 19:15 > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) > <srinuvasan.a@siemens.com> > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com > Subject: Re: [PATCH v2 1/3] mmdebstrap: add > "skip=cleanup/reproducible" option to mmdebstrap > > On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote: > > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > > > From: srinuvasan <srinuvasan.a@siemens.com> > > > > > > Without this option, mmdebstrap deletes the var/log/dpkg.log and > > > its siblings by default, but this is needed to download deb-src > > > packages for base-apt creation. > > I don't see where deb-src needs dpkg.log. Can you shed some light? > Both the old code and the code I recently changed parse the .deb files > that were downloaded to determine which source packages should be > downloaded > > Ref: > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgrou > ps.google.com%2Fg%2Fisar-users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Csrinuv > asan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae3bcd9579 > 4fd4addab42e1495d55a%7C1%7C0%7C638770581452433346%7CUnknown%7CTWFpbGZs > b3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj > oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=b49oZTatP9h6CxEys%2BPv%2 > F%2F%2F5v2BmE%2FX1%2FdhDPs%2F1qOY%3D&reserved=0 > > > > Basically, we are referring to the package's status under > > > /var/log/dpkg.log file and performing the debsrc_download > > > operation. > > > > > > Without these changes, bootstrap related deb-src files are not > > > downloaded under the /downloads/deb-src folder, due to this the > > > repo sanity check functions failed during base-apt caching stage. > > What are these repo sanity checks? > Where do we see the failures? > > From my base-apt workdir: > > pwd && grep -ir error > <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0- > r0/temp > run.repo.7287: echo "ERROR: $*" > > We probably need to be more specific if we are mentioning a failure. > Without this, people on this list will not understand what problem(s) > you are fixing > > > Hi cedric, jan > > I will explain the issue in the details below: > > To reproduce the issue ( Ref: > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fdoc%2Fuser_manual.md%23creati > on-of-local-apt-repo-caching-upstream-debian-packages&data=05%7C02%7Cs > rinuvasan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae3bc > d95794fd4addab42e1495d55a%7C1%7C0%7C638770581452457810%7CUnknown%7CTWF > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3XId9a7IuBK9xyzYWnu > dmeG0aKe3awlNSGSmwId7D%2BU%3D&reserved=0 > ): > > Trigger the warmup build to download deb and deb-src > packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in > conf/local.conf) > > e.x: bitbake mc:qemuamd64-bookworm:isar-image-base > > Once all the deb and deb-src downloaded for the > particular BSP under /download folder > > Trigger the base-apt target to cache all the deb and > deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf) > > e.x: bitbake base-apt > > You will see the failures: > > Logs: > > NOTE: Resolving any missing task queue dependencies > Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% > match, 0% complete)############################################# > | > ETA: 0:00:00 > Initialising tasks: 100% > |#################################################################### > ################################################| Time: 0:00:00 > NOTE: No setscene tasks > NOTE: Executing Tasks > ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in > repo. bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb > bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6- > 2_all.deb > bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb > bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian- > archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb > bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9- > 1_all.deb > bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find- > rule-perl/libfile-find-rule-perl_0.34-3_all.deb > bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber- > compare-perl/libnumber-compare-perl_0.03-3_all.deb > bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2- > 6+deb12u1_all.deb .dsc for pam packages found in our downstream project Yes cedric, validated from my end, it is available in downstream project because we have your changes in isar-patches (https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y) But still we can reproduce the issue in vanilla ISAR without your patch, I started working on these changes before you proposed these changes (https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y) Basically, your patches remove the is_not_part_of_current_build() function And refactor the debsrc_download functionality. My patch becomes obsoleted if your (https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y) changes are going to merge in ISAR. Please provide your thoughts? > bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage- > common_3.4-1_all.deb > bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob- > perl/libtext-glob-perl_0.11-3_all.deb > bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules- > 5.36_5.36.0-7+deb12u1_all.deb bookworm six 1.16.0-4 > pool/main/s/six/python3-six_1.16.0-4_all.deb > bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b- > 0+deb12u1_all.deb > bookworm usrmerge 37~deb12u1 > pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb > bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb > > Just i captured few packages, but many of the deb packages don't have > their source files > > In the base-apt recipes we are calling the repo_sanity_test function > finally to make sure all the deb's having the corresponding deb-src > packages should be present in the base-apt. I checked again log.do_cache in our downstream project and I did not see any of the mentioned errors. In downstream project we are not at all calling the repo_sanity_test function due to enabled the cache-deb-src in ROOTFS_POSTPROCESS_COMMAND rather than enabled in conf/local.conf The below condition fails: if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ] that is the reason we are not calling this statement at all " reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing " because we are not enabled the cache-deb-src via BASE_REPO_FEATURES $ grep -v 'Export' ./tmp/work/debian-bookworm-any/base-apt/1.0- r0/temp/log.do_cache DEBUG: Executing python function do_cache DEBUG: Using absolute /build/../repo/gnupg/205D601E.pub DEBUG: Executing shell function repo DEBUG: Shell function repo finished DEBUG: Python function do_cache finished That build was however done with the patch I have submitted earlier to this list. It also includes our own layer. I will do a plain Isar build (with my patch) to check if the issue is reproducible. I took a few error cases from your example above and verified that the .dsc files are present in the generated base-apt build. It should be noted that my patch removes "is_not_part_of_current_build()" parsing dpkg.log was IMO not a great idea as I don't think its format is officially documented (I could be wrong though) > > > Yes, as you said, we are parsing deb files (from > /var/cache/apt/archives) that were downloaded to determine which > source packages should be downloaded. > but in the function "is_not_part_of_current_build "${package}" we are > skipping the download-src if parsing deb package (status installed) is > not available in the below files ``` > "${IMAGE_ROOTFS}"/var/log/dpkg.log \ > "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ > "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ > "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ > "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log > ``` > > As I mentioned above, the adduser package was installed in the stage > of bootstrap, but that package's status was not captured in the > bootstrap's /var/log/dpkg.log file. > because mmdebstrap deletes those files by default, hence we didn't > capture those bootstrap package status. > > As we know bootstrap rootfs is used for sbuildchroot creation, but > here whatever packages are installed during sbuild-chroot creation, > ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the > sbuildchroot related package status, but our expectation is it should > have bootstrap + sbuildchroot package status available in > /var/log/dpkg.log file in sbuildchroot. > > My first patch solves the above issue > with the introduction of "skip=cleanup/reproducible", the bootstrap > rootfs generated with /var/log/dpkg.log files, which contain all the > bootstrap package status along with that if you use that boostrap for > sbuildchroot creation , the sbuildchroot rootfs contains the existing > bootstrap plus additionally installed sbuildchroot related packages > status is available in sbuldchroot /var/log/dpkg.log file > > > My second patch resolve the below issue: > > During image creation (IMAGE_FSTYPES = "wic") few packages were > installed in the session (e.g: gdisk, bmap-tools) but those packages > list available only in the sbuild environment, not outside, carry > those packages status to tmp/dpkg_common.log in the rootfs to get the > src packages for imager_install related deb packages, presently the > cache-deb-src function calls in the ROOTFS_POSTPROCESS_COMMAND but we > need to wait till imager generation That's why we pushed that > deb-src-caching after image creation. > > My third path resolve the image rootfs related deb-src missing issue: > > By default we are removing the /var/log/dpkg.log file in the > rootfs_postprocess_clean_log_files function, but this is needed when > We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making this > function as a configurable one rather than enabled by default. > to download IMAGE_ROOTFS related deb-src packages > > I hope I explain the issue, presently am checking the reproducible > part (for jan queries), will update that part very soon. > > Many thanks, > Srinu > > > > > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > > > --- > > > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++ > > > 1 file changed, 6 insertions(+) > > > > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar- > > > mmdebstrap.inc > > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > index 931f6f13..7380a658 100644 > > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > @@ -168,6 +168,11 @@ do_bootstrap() { > > > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > > > fi > > > > > > + cleanup= > > > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb- > > > src', 'yes', 'no', d)}" = "yes" ]; then > > > + cleanup="--skip=cleanup/reproducible" > > > + fi > > > + > > > # Cleanup mounts if fails > > > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat > > > "${WORKDIR}/mmtmpdir") \ > > > @@ -183,6 +188,7 @@ do_bootstrap() { > > > > > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args > > > \ > > > $arch_param \ > > > + $cleanup \ > > > --mode=unshare \ > > > ${MMHOOKS} \ > > > --setup-hook='mkdir -p > > > "$1/var/cache/apt/archives/"' \ > > > > What has changed in this version? I'm not seeing my concerns > > addressed, my questions answered. > > +1 > > Please make sure new revisions of your patch series include a > changelog (use --cover-letter with git format-patch) > > Sure > > > > > Jan > > > > -- Cedric Hombourger Siemens AG www.siemens.com
On Sun, 2025-03-09 at 08:46 +0000, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote: > > > -----Original Message----- > From: Hombourger, Cedric (FT FDS CES LX) > <cedric.hombourger@siemens.com> > Sent: 09 March 2025 00:46 > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) > <srinuvasan.a@siemens.com> > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com > Subject: Re: [PATCH v2 1/3] mmdebstrap: add > "skip=cleanup/reproducible" option to mmdebstrap > > On Sat, 2025-03-08 at 18:25 +0000, Arjunan, Srinu (FT FDS CES LX PBU > 2) > wrote: > > > > > > -----Original Message----- > > From: Hombourger, Cedric (FT FDS CES LX) > > <cedric.hombourger@siemens.com> > > Sent: 08 March 2025 19:15 > > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) > > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) > > <srinuvasan.a@siemens.com> > > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com > > Subject: Re: [PATCH v2 1/3] mmdebstrap: add > > "skip=cleanup/reproducible" option to mmdebstrap > > > > On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote: > > > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > > > > From: srinuvasan <srinuvasan.a@siemens.com> > > > > > > > > Without this option, mmdebstrap deletes the var/log/dpkg.log > > > > and > > > > its siblings by default, but this is needed to download deb-src > > > > packages for base-apt creation. > > > > I don't see where deb-src needs dpkg.log. Can you shed some light? > > Both the old code and the code I recently changed parse the .deb > > files > > that were downloaded to determine which source packages should be > > downloaded > > > > Ref: > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgrou%2F&data=05%7C02%7Ccedric.hombourger%40siemens.com%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771068307563689%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kPyQeXBw6pYTZng1OieFPozM1xn9YgRdyuC%2FSfGK7o0%3D&reserved=0 > > ps.google.com%2Fg%2Fisar- > > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Csrinuv > > asan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae3bcd9 > > 579 > > 4fd4addab42e1495d55a%7C1%7C0%7C638770581452433346%7CUnknown%7CTWFpb > > GZs > > b3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF > > OIj > > oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=b49oZTatP9h6CxEys%2BP > > v%2 > > F%2F%2F5v2BmE%2FX1%2FdhDPs%2F1qOY%3D&reserved=0 > > > > > > Basically, we are referring to the package's status under > > > > /var/log/dpkg.log file and performing the debsrc_download > > > > operation. > > > > > > > > Without these changes, bootstrap related deb-src files are not > > > > downloaded under the /downloads/deb-src folder, due to this the > > > > repo sanity check functions failed during base-apt caching > > > > stage. > > > > What are these repo sanity checks? > > Where do we see the failures? > > > > From my base-apt workdir: > > > > pwd && grep -ir error > > <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0- > > r0/temp > > run.repo.7287: echo "ERROR: $*" > > > > We probably need to be more specific if we are mentioning a > > failure. > > Without this, people on this list will not understand what > > problem(s) > > you are fixing > > > > > > Hi cedric, jan > > > > I will explain the issue in the details below: > > > > To reproduce the issue ( Ref: > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith%2F&data=05%7C02%7Ccedric.hombourger%40siemens.com%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771068307586848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AUH81pXJwMNzrIIkLA3GkJ2fLWwL9jBFGeltINWR41g%3D&reserved=0 > > ub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fdoc%2Fuser_manual.md%23cre > > ati > > on-of-local-apt-repo-caching-upstream-debian- > > packages&data=05%7C02%7Cs > > rinuvasan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae > > 3bc > > d95794fd4addab42e1495d55a%7C1%7C0%7C638770581452457810%7CUnknown%7C > > TWF > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi > > IsI > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3XId9a7IuBK9xyzY > > Wnu > > dmeG0aKe3awlNSGSmwId7D%2BU%3D&reserved=0 > > ): > > > > Trigger the warmup build to download deb and deb- > > src > > packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in > > conf/local.conf) > > > > e.x: bitbake mc:qemuamd64-bookworm:isar-image-base > > > > Once all the deb and deb-src downloaded for the > > particular BSP under /download folder > > > > Trigger the base-apt target to cache all the deb > > and > > deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf) > > > > e.x: bitbake base-apt > > > > You will see the failures: > > > > Logs: > > > > NOTE: Resolving any missing task queue dependencies > > Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% > > match, 0% > > complete)############################################# > > > > > ETA: 0:00:00 > > Initialising tasks: 100% > > > ################################################################# > > > ### > > ################################################| Time: 0:00:00 > > NOTE: No setscene tasks > > NOTE: Executing Tasks > > ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in > > repo. bookworm adduser 3.134 > > pool/main/a/adduser/adduser_3.134_all.deb > > bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6- > > 2_all.deb > > bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb > > bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian- > > archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb > > bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9- > > 1_all.deb > > bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find- > > rule-perl/libfile-find-rule-perl_0.34-3_all.deb > > bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber- > > compare-perl/libnumber-compare-perl_0.03-3_all.deb > > bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2- > > 6+deb12u1_all.deb > > .dsc for pam packages found in our downstream project > > Yes cedric, validated from my end, it is available in downstream > project because we have your changes in isar-patches > (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr > oups.google.com%2Fg%2Fisar- > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co > m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5 > 5a%7C1%7C0%7C638771068307601863%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc > GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj > oyfQ%3D%3D%7C0%7C%7C%7C&sdata=OoUVTbJLnDaPdXKYmZmOh0Wxn9HpiRis%2FYRaL > YL2Khs%3D&reserved=0) > > But still we can reproduce the issue in vanilla ISAR without your > patch, I started working on these changes before you proposed these > changes > (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr > oups.google.com%2Fg%2Fisar- > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co > m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5 > 5a%7C1%7C0%7C638771068307616236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc > GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj > oyfQ%3D%3D%7C0%7C%7C%7C&sdata=IknRwnuSOZiNUPnx43d4W4dJd2vmylKoUyp%2Bp > T4W6OE%3D&reserved=0) > > Basically, your patches remove the is_not_part_of_current_build() > function > And refactor the debsrc_download functionality. > > My patch becomes obsoleted if your > (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr > oups.google.com%2Fg%2Fisar- > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co > m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5 > 5a%7C1%7C0%7C638771068307630537%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc > GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj > oyfQ%3D%3D%7C0%7C%7C%7C&sdata=vkjpy98ZFZZo4WgKjtwj7UREcd%2B2J4EBcWliJ > eIdHbU%3D&reserved=0) changes are going to merge in ISAR. > > Please provide your thoughts? Well the future of my patch and yours are more for the Isar maintainers to decide. They definitely conflict with each other. > > > bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage- > > common_3.4-1_all.deb > > bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob- > > perl/libtext-glob-perl_0.11-3_all.deb > > bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules- > > 5.36_5.36.0-7+deb12u1_all.deb bookworm six 1.16.0-4 > > pool/main/s/six/python3-six_1.16.0-4_all.deb > > bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b- > > 0+deb12u1_all.deb > > bookworm usrmerge 37~deb12u1 > > pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb > > bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb > > > > Just i captured few packages, but many of the deb packages don't > > have > > their source files > > > > In the base-apt recipes we are calling the repo_sanity_test > > function > > finally to make sure all the deb's having the corresponding deb-src > > packages should be present in the base-apt. > > I checked again log.do_cache in our downstream project and I did not > see any of the mentioned errors. > > In downstream project we are not at all calling the repo_sanity_test > function due to enabled the cache-deb-src in > ROOTFS_POSTPROCESS_COMMAND rather than enabled in conf/local.conf > > The below condition fails: > if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', > 'yes', 'no', d)}" = "yes" ] > > that is the reason we are not calling this statement at all " > reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing " > > because we are not enabled the cache-deb-src via BASE_REPO_FEATURES > > > $ grep -v 'Export' ./tmp/work/debian-bookworm-any/base-apt/1.0- > r0/temp/log.do_cache > DEBUG: Executing python function do_cache > DEBUG: Using absolute /build/../repo/gnupg/205D601E.pub > DEBUG: Executing shell function repo > DEBUG: Shell function repo finished > DEBUG: Python function do_cache finished > > That build was however done with the patch I have submitted earlier > to this list. It also includes our own layer. I will do a plain Isar > build (with my patch) to check if the issue is reproducible. > > I took a few error cases from your example above and verified that > the .dsc files are present in the generated base-apt build. > > It should be noted that my patch removes > "is_not_part_of_current_build()" > > parsing dpkg.log was IMO not a great idea as I don't think its format > is officially documented (I could be wrong though) > > > > > > > Yes, as you said, we are parsing deb files (from > > /var/cache/apt/archives) that were downloaded to determine which > > source packages should be downloaded. > > but in the function "is_not_part_of_current_build "${package}" we > > are > > skipping the download-src if parsing deb package (status installed) > > is > > not available in the below files ``` > > "${IMAGE_ROOTFS}"/var/log/dpkg.log \ > > "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ > > "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ > > "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ > > "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log > > ``` > > > > As I mentioned above, the adduser package was installed in the > > stage > > of bootstrap, but that package's status was not captured in the > > bootstrap's /var/log/dpkg.log file. > > because mmdebstrap deletes those files by default, hence we didn't > > capture those bootstrap package status. > > > > As we know bootstrap rootfs is used for sbuildchroot creation, but > > here whatever packages are installed during sbuild-chroot creation, > > ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the > > sbuildchroot related package status, but our expectation is it > > should > > have bootstrap + sbuildchroot package status available in > > /var/log/dpkg.log file in sbuildchroot. > > > > My first patch solves the above issue > > with the introduction of "skip=cleanup/reproducible", the bootstrap > > rootfs generated with /var/log/dpkg.log files, which contain all > > the > > bootstrap package status along with that if you use that boostrap > > for > > sbuildchroot creation , the sbuildchroot rootfs contains the > > existing > > bootstrap plus additionally installed sbuildchroot related packages > > status is available in sbuldchroot /var/log/dpkg.log file > > > > > > My second patch resolve the below issue: > > > > During image creation (IMAGE_FSTYPES = "wic") few packages were > > installed in the session (e.g: gdisk, bmap-tools) but those > > packages > > list available only in the sbuild environment, not outside, carry > > those packages status to tmp/dpkg_common.log in the rootfs to get > > the > > src packages for imager_install related deb packages, presently the > > cache-deb-src function calls in the ROOTFS_POSTPROCESS_COMMAND but > > we > > need to wait till imager generation That's why we pushed that > > deb-src-caching after image creation. > > > > My third path resolve the image rootfs related deb-src missing > > issue: > > > > By default we are removing the /var/log/dpkg.log file in the > > rootfs_postprocess_clean_log_files function, but this is needed > > when > > We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making > > this > > function as a configurable one rather than enabled by default. > > to download IMAGE_ROOTFS related deb-src packages > > > > I hope I explain the issue, presently am checking the reproducible > > part (for jan queries), will update that part very soon. > > > > Many thanks, > > Srinu > > > > > > > > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > > > > --- > > > > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 > > > > ++++++ > > > > 1 file changed, 6 insertions(+) > > > > > > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar- > > > > mmdebstrap.inc > > > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > > index 931f6f13..7380a658 100644 > > > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > > @@ -168,6 +168,11 @@ do_bootstrap() { > > > > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > > > > fi > > > > > > > > + cleanup= > > > > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache- > > > > deb- > > > > src', 'yes', 'no', d)}" = "yes" ]; then > > > > + cleanup="--skip=cleanup/reproducible" > > > > + fi > > > > + > > > > # Cleanup mounts if fails > > > > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat > > > > "${WORKDIR}/mmtmpdir") \ > > > > @@ -183,6 +188,7 @@ do_bootstrap() { > > > > > > > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap > > > > $bootstrap_args > > > > \ > > > > $arch_param \ > > > > + $cleanup \ > > > > --mode=unshare \ > > > > ${MMHOOKS} \ > > > > --setup-hook='mkdir -p > > > > "$1/var/cache/apt/archives/"' \ > > > > > > What has changed in this version? I'm not seeing my concerns > > > addressed, my questions answered. > > > > +1 > > > > Please make sure new revisions of your patch series include a > > changelog (use --cover-letter with git format-patch) > > > > Sure > > > > > > > > Jan > > > > > > > > > -- > Cedric Hombourger > Siemens AG > www.siemens.com
-----Original Message----- From: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com> Sent: 10 March 2025 14:45 To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com> Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com Subject: Re: [PATCH v2 1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap On Sun, 2025-03-09 at 08:46 +0000, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote: > > > -----Original Message----- > From: Hombourger, Cedric (FT FDS CES LX) > <cedric.hombourger@siemens.com> > Sent: 09 March 2025 00:46 > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) > <srinuvasan.a@siemens.com> > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com > Subject: Re: [PATCH v2 1/3] mmdebstrap: add > "skip=cleanup/reproducible" option to mmdebstrap > > On Sat, 2025-03-08 at 18:25 +0000, Arjunan, Srinu (FT FDS CES LX PBU > 2) > wrote: > > > > > > -----Original Message----- > > From: Hombourger, Cedric (FT FDS CES LX) > > <cedric.hombourger@siemens.com> > > Sent: 08 March 2025 19:15 > > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) > > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) > > <srinuvasan.a@siemens.com> > > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com > > Subject: Re: [PATCH v2 1/3] mmdebstrap: add > > "skip=cleanup/reproducible" option to mmdebstrap > > > > On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote: > > > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote: > > > > From: srinuvasan <srinuvasan.a@siemens.com> > > > > > > > > Without this option, mmdebstrap deletes the var/log/dpkg.log and > > > > its siblings by default, but this is needed to download deb-src > > > > packages for base-apt creation. > > > > I don't see where deb-src needs dpkg.log. Can you shed some light? > > Both the old code and the code I recently changed parse the .deb > > files that were downloaded to determine which source packages should > > be downloaded > > > > Ref: > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr > > ou%2F&data=05%7C02%7Csrinuvasan.a%40siemens.com%7Ccd17f90bd20b4d46a1 > > ff08dd5fb403e6%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C63877194 > > 8975549635%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL > > jAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7 > > C%7C&sdata=oekcbPth33aUQlgM3jGv9ukM1%2BnxgNW1y51931942zE%3D&reserved > > =0 > > ps.google.com%2Fg%2Fisar- > > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Csrinuv > > asan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae3bcd9 > > 579 > > 4fd4addab42e1495d55a%7C1%7C0%7C638770581452433346%7CUnknown%7CTWFpb > > GZs > > b3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF > > OIj > > oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=b49oZTatP9h6CxEys%2BP > > v%2 > > F%2F%2F5v2BmE%2FX1%2FdhDPs%2F1qOY%3D&reserved=0 > > > > > > Basically, we are referring to the package's status under > > > > /var/log/dpkg.log file and performing the debsrc_download > > > > operation. > > > > > > > > Without these changes, bootstrap related deb-src files are not > > > > downloaded under the /downloads/deb-src folder, due to this the > > > > repo sanity check functions failed during base-apt caching > > > > stage. > > > > What are these repo sanity checks? > > Where do we see the failures? > > > > From my base-apt workdir: > > > > pwd && grep -ir error > > <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0- > > r0/temp > > run.repo.7287: echo "ERROR: $*" > > > > We probably need to be more specific if we are mentioning a failure. > > Without this, people on this list will not understand what > > problem(s) > > you are fixing > > > > > > Hi cedric, jan > > > > I will explain the issue in the details below: > > > > To reproduce the issue ( Ref: > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith%2F&data=05%7C02%7Csrinuvasan.a%40siemens.com%7Ccd17f90bd20b4d46a1ff08dd5fb403e6%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771948975572755%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=iTA44lkFK0i58FlzXMRLIDZ5bfgeHTk2cDgv2Q%2BVogU%3D&reserved=0 > > ub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fdoc%2Fuser_manual.md%23cre > > ati > > on-of-local-apt-repo-caching-upstream-debian- > > packages&data=05%7C02%7Cs > > rinuvasan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae > > 3bc > > d95794fd4addab42e1495d55a%7C1%7C0%7C638770581452457810%7CUnknown%7C > > TWF > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi > > IsI > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3XId9a7IuBK9xyzY > > Wnu > > dmeG0aKe3awlNSGSmwId7D%2BU%3D&reserved=0 > > ): > > > > Trigger the warmup build to download deb and deb- > > src > > packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in > > conf/local.conf) > > > > e.x: bitbake mc:qemuamd64-bookworm:isar-image-base > > > > Once all the deb and deb-src downloaded for the > > particular BSP under /download folder > > > > Trigger the base-apt target to cache all the deb > > and > > deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf) > > > > e.x: bitbake base-apt > > > > You will see the failures: > > > > Logs: > > > > NOTE: Resolving any missing task queue dependencies > > Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% > > match, 0% > > complete)############################################# > > > > > ETA: 0:00:00 > > Initialising tasks: 100% > > > ################################################################# > > > ### > > ################################################| Time: 0:00:00 > > NOTE: No setscene tasks > > NOTE: Executing Tasks > > ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in > > repo. bookworm adduser 3.134 > > pool/main/a/adduser/adduser_3.134_all.deb > > bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6- > > 2_all.deb > > bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb > > bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian- > > archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb > > bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9- > > 1_all.deb > > bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find- > > rule-perl/libfile-find-rule-perl_0.34-3_all.deb > > bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber- > > compare-perl/libnumber-compare-perl_0.03-3_all.deb > > bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2- > > 6+deb12u1_all.deb > > .dsc for pam packages found in our downstream project > > Yes cedric, validated from my end, it is available in downstream > project because we have your changes in isar-patches > (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr%2F&data=05%7C02%7Csrinuvasan.a%40siemens.com%7Ccd17f90bd20b4d46a1ff08dd5fb403e6%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771948975584889%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=20glh8vYj2bN2dgIRh6QytScdg1ZjbZzgd9xu6vu18s%3D&reserved=0 > oups.google.com%2Fg%2Fisar- > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co > m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5 > 5a%7C1%7C0%7C638771068307601863%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc > GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj > oyfQ%3D%3D%7C0%7C%7C%7C&sdata=OoUVTbJLnDaPdXKYmZmOh0Wxn9HpiRis%2FYRaL > YL2Khs%3D&reserved=0) > > But still we can reproduce the issue in vanilla ISAR without your > patch, I started working on these changes before you proposed these > changes > (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr%2F&data=05%7C02%7Csrinuvasan.a%40siemens.com%7Ccd17f90bd20b4d46a1ff08dd5fb403e6%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771948975596894%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=RvVRIn9snNRs%2BJJjNsWXAcBrkZcuygNcl28CeWpV4dc%3D&reserved=0 > oups.google.com%2Fg%2Fisar- > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co > m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5 > 5a%7C1%7C0%7C638771068307616236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc > GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj > oyfQ%3D%3D%7C0%7C%7C%7C&sdata=IknRwnuSOZiNUPnx43d4W4dJd2vmylKoUyp%2Bp > T4W6OE%3D&reserved=0) > > Basically, your patches remove the is_not_part_of_current_build() > function > And refactor the debsrc_download functionality. > > My patch becomes obsoleted if your > (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr%2F&data=05%7C02%7Csrinuvasan.a%40siemens.com%7Ccd17f90bd20b4d46a1ff08dd5fb403e6%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771948975609865%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=34iH9h7A%2BEeaokZRn1B8UKOm9iOS2WQlYCuBCyIjgTA%3D&reserved=0 > oups.google.com%2Fg%2Fisar- > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co > m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5 > 5a%7C1%7C0%7C638771068307630537%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc > GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj > oyfQ%3D%3D%7C0%7C%7C%7C&sdata=vkjpy98ZFZZo4WgKjtwj7UREcd%2B2J4EBcWliJ > eIdHbU%3D&reserved=0) changes are going to merge in ISAR. > > Please provide your thoughts? Well the future of my patch and yours are more for the Isar maintainers to decide. They definitely conflict with each other. Now the conflicts are resolved and make the solution from your patch and mine, Please refer this PR: https://groups.google.com/g/isar-users/c/fhinJ7Z6Ro4 Many thanks, Srinu > > > bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage- > > common_3.4-1_all.deb > > bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob- > > perl/libtext-glob-perl_0.11-3_all.deb > > bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules- > > 5.36_5.36.0-7+deb12u1_all.deb bookworm six 1.16.0-4 > > pool/main/s/six/python3-six_1.16.0-4_all.deb > > bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b- > > 0+deb12u1_all.deb > > bookworm usrmerge 37~deb12u1 > > pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb > > bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb > > > > Just i captured few packages, but many of the deb packages don't > > have > > their source files > > > > In the base-apt recipes we are calling the repo_sanity_test > > function > > finally to make sure all the deb's having the corresponding deb-src > > packages should be present in the base-apt. > > I checked again log.do_cache in our downstream project and I did not > see any of the mentioned errors. > > In downstream project we are not at all calling the repo_sanity_test > function due to enabled the cache-deb-src in > ROOTFS_POSTPROCESS_COMMAND rather than enabled in conf/local.conf > > The below condition fails: > if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', > 'yes', 'no', d)}" = "yes" ] > > that is the reason we are not calling this statement at all " > reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing " > > because we are not enabled the cache-deb-src via BASE_REPO_FEATURES > > > $ grep -v 'Export' ./tmp/work/debian-bookworm-any/base-apt/1.0- > r0/temp/log.do_cache > DEBUG: Executing python function do_cache > DEBUG: Using absolute /build/../repo/gnupg/205D601E.pub > DEBUG: Executing shell function repo > DEBUG: Shell function repo finished > DEBUG: Python function do_cache finished > > That build was however done with the patch I have submitted earlier > to this list. It also includes our own layer. I will do a plain Isar > build (with my patch) to check if the issue is reproducible. > > I took a few error cases from your example above and verified that > the .dsc files are present in the generated base-apt build. > > It should be noted that my patch removes > "is_not_part_of_current_build()" > > parsing dpkg.log was IMO not a great idea as I don't think its format > is officially documented (I could be wrong though) > > > > > > > Yes, as you said, we are parsing deb files (from > > /var/cache/apt/archives) that were downloaded to determine which > > source packages should be downloaded. > > but in the function "is_not_part_of_current_build "${package}" we > > are > > skipping the download-src if parsing deb package (status installed) > > is > > not available in the below files ``` > > "${IMAGE_ROOTFS}"/var/log/dpkg.log \ > > "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ > > "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ > > "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ > > "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log > > ``` > > > > As I mentioned above, the adduser package was installed in the > > stage > > of bootstrap, but that package's status was not captured in the > > bootstrap's /var/log/dpkg.log file. > > because mmdebstrap deletes those files by default, hence we didn't > > capture those bootstrap package status. > > > > As we know bootstrap rootfs is used for sbuildchroot creation, but > > here whatever packages are installed during sbuild-chroot creation, > > ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the > > sbuildchroot related package status, but our expectation is it > > should > > have bootstrap + sbuildchroot package status available in > > /var/log/dpkg.log file in sbuildchroot. > > > > My first patch solves the above issue > > with the introduction of "skip=cleanup/reproducible", the bootstrap > > rootfs generated with /var/log/dpkg.log files, which contain all > > the > > bootstrap package status along with that if you use that boostrap > > for > > sbuildchroot creation , the sbuildchroot rootfs contains the > > existing > > bootstrap plus additionally installed sbuildchroot related packages > > status is available in sbuldchroot /var/log/dpkg.log file > > > > > > My second patch resolve the below issue: > > > > During image creation (IMAGE_FSTYPES = "wic") few packages were > > installed in the session (e.g: gdisk, bmap-tools) but those > > packages > > list available only in the sbuild environment, not outside, carry > > those packages status to tmp/dpkg_common.log in the rootfs to get > > the > > src packages for imager_install related deb packages, presently the > > cache-deb-src function calls in the ROOTFS_POSTPROCESS_COMMAND but > > we > > need to wait till imager generation That's why we pushed that > > deb-src-caching after image creation. > > > > My third path resolve the image rootfs related deb-src missing > > issue: > > > > By default we are removing the /var/log/dpkg.log file in the > > rootfs_postprocess_clean_log_files function, but this is needed > > when > > We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making > > this > > function as a configurable one rather than enabled by default. > > to download IMAGE_ROOTFS related deb-src packages > > > > I hope I explain the issue, presently am checking the reproducible > > part (for jan queries), will update that part very soon. > > > > Many thanks, > > Srinu > > > > > > > > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com> > > > > --- > > > > meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 > > > > ++++++ > > > > 1 file changed, 6 insertions(+) > > > > > > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar- > > > > mmdebstrap.inc > > > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > > index 931f6f13..7380a658 100644 > > > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc > > > > @@ -168,6 +168,11 @@ do_bootstrap() { > > > > arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" > > > > fi > > > > > > > > + cleanup= > > > > + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache- > > > > deb- > > > > src', 'yes', 'no', d)}" = "yes" ]; then > > > > + cleanup="--skip=cleanup/reproducible" > > > > + fi > > > > + > > > > # Cleanup mounts if fails > > > > trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > > trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat > > > > "${WORKDIR}/mmtmpdir") \ > > > > @@ -183,6 +188,7 @@ do_bootstrap() { > > > > > > > > sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap > > > > $bootstrap_args > > > > \ > > > > $arch_param \ > > > > + $cleanup \ > > > > --mode=unshare \ > > > > ${MMHOOKS} \ > > > > --setup-hook='mkdir -p > > > > "$1/var/cache/apt/archives/"' \ > > > > > > What has changed in this version? I'm not seeing my concerns > > > addressed, my questions answered. > > > > +1 > > > > Please make sure new revisions of your patch series include a > > changelog (use --cover-letter with git format-patch) > > > > Sure > > > > > > > > Jan > > > > > > > > > -- > Cedric Hombourger > Siemens AG > www.siemens.com
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc index 931f6f13..7380a658 100644 --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc @@ -168,6 +168,11 @@ do_bootstrap() { arch_param="$arch_param,${COMPAT_DISTRO_ARCH}" fi + cleanup= + if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ]; then + cleanup="--skip=cleanup/reproducible" + fi + # Cleanup mounts if fails trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat "${WORKDIR}/mmtmpdir") \ @@ -183,6 +188,7 @@ do_bootstrap() { sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \ $arch_param \ + $cleanup \ --mode=unshare \ ${MMHOOKS} \ --setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \