| Message ID | 20221107082503.14774-1-venkata.pyla@toshiba-tsip.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | image.bbclass: set file timestamps inside the rootfs and initramfs image | expand |
On Mon, 2022-11-07 at 13:55 +0530, venkata.pyla@toshiba-tsip.com wrote: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > As part of reproducible-build work, one of the problem chosen to > solve > is the file time stamps inside rootfs and initramfs are not identical > between two builds. > > With the help of reproducible-builds.org and their suggestions, the > above problem can be fixed using 'SOURCE_DATE_EPOCH' variable [2]. > > In case of rootfs file time-stamps, set all the files and folders > that > are newer than 'SOURCE_DATE_EPOCH' and set it to same. Hi, is there a particular reason why we don't set all dates exactly to SOURCE_DATE_EPOCH? Otherwise we might still have reproducibility issues in case some files are provided by the sstate cache, hence have an older timestamp. Building w/ or w/o cache should not make a difference. Felix > In case of initramfs, regenerate the initramfs image with > 'SOURCE_DATE_EPOCH' variable set as the mkinitramfs script is already > taken care of creating reproducible initramfs image when the > variable is set in the environment[3]. > > The SOURCE_DATE_EPOCH variable should be set to the last modification > of > the git repository as explained in the documentation[2]. > > e.g: > SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > > To know more about the reproducible builds and its goals please refer > [1]. > > [1] https://reproducible-builds.org/ > [2] https://reproducible-builds.org/docs/source-date-epoch/ > [3] > https://manpages.debian.org/bullseye/initramfs-tools-core/mkinitramfs.8.en.html#ENVIRONMENT > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > meta/classes/image.bbclass | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index ccff810..c1bb4fd 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -431,6 +431,21 @@ do_rootfs_finalize() { > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > + > + # Recreate initramfs inorder to set timestamps to > SOURCE_DATE_EPOCH > + # inorder to make reproducible initramfs > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot > "${ROOTFSDIR}" \ > + update-initramfs -u -v > + > + # Set timestamp to files inside the rootfs image inorder to > make > + # reproducible rootfs > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + find ${ROOTFSDIR} -newermt \ > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > %H:%M:%S')" \ > + -printf "%y %p\n" \ > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > + > EOSUDO > } > addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess > -- > 2.20.1 > >
Am Mon, 7 Nov 2022 13:55:03 +0530 schrieb venkata.pyla@toshiba-tsip.com: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > As part of reproducible-build work, one of the problem chosen to solve > is the file time stamps inside rootfs and initramfs are not identical > between two builds. > > With the help of reproducible-builds.org and their suggestions, the > above problem can be fixed using 'SOURCE_DATE_EPOCH' variable [2]. > > In case of rootfs file time-stamps, set all the files and folders that > are newer than 'SOURCE_DATE_EPOCH' and set it to same. > In case of initramfs, regenerate the initramfs image with > 'SOURCE_DATE_EPOCH' variable set as the mkinitramfs script is already > taken care of creating reproducible initramfs image when the > variable is set in the environment[3]. > > The SOURCE_DATE_EPOCH variable should be set to the last modification > of the git repository as explained in the documentation[2]. > > e.g: > SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > > To know more about the reproducible builds and its goals please refer > [1]. > > [1] https://reproducible-builds.org/ > [2] https://reproducible-builds.org/docs/source-date-epoch/ > [3] > https://manpages.debian.org/bullseye/initramfs-tools-core/mkinitramfs.8.en.html#ENVIRONMENT > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > meta/classes/image.bbclass | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index ccff810..c1bb4fd 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -431,6 +431,21 @@ do_rootfs_finalize() { > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > + > + # Recreate initramfs inorder to set timestamps to > SOURCE_DATE_EPOCH > + # inorder to make reproducible initramfs > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot > "${ROOTFSDIR}" \ > + update-initramfs -u -v This should be done where that initial update-initramfs can be found. And not fix things up later. Not every image will have an initrd so that is wrong in multiple ways. > + > + # Set timestamp to files inside the rootfs image inorder to > make > + # reproducible rootfs > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + find ${ROOTFSDIR} -newermt \ > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > %H:%M:%S')" \ > + -printf "%y %p\n" \ > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > + No! Which files do we care about? My guess would content of packages built with isar. We should export the variable in the dpkg class then the packages will be correct and we do not have to mess around with that rootfs and fix problems that should have never been there. I would like to ask for test cases. Ideally first a breaking test and later a commit fixing the issue. Note that deriving the time from git means one needs git. And when using layers just that one Isar git is clearly not good enough. And we will have to check how that works with sstate. I assume any new commit would rebuild all custom packages. Here i see a conflict between regular dev work and repro pedantics. We might need a way to turn that stuff off. regards, Henning > EOSUDO > } > addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
Hi Felix, Thanks for your review, please check my comments below. Thanks, Venkata. >-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Moessbauer, Felix >Sent: 07 November 2022 14:21 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba- >tsip.com>; isar-users@googlegroups.com; Kiszka, Jan <jan.kiszka@siemens.com>; >hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >Schild, Henning <henning.schild@siemens.com> >Subject: Re: [PATCH] image.bbclass: set file timestamps inside the rootfs and >initramfs image > >On Mon, 2022-11-07 at 13:55 +0530, venkata.pyla@toshiba-tsip.com wrote: >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> As part of reproducible-build work, one of the problem chosen to solve >> is the file time stamps inside rootfs and initramfs are not identical >> between two builds. >> >> With the help of reproducible-builds.org and their suggestions, the >> above problem can be fixed using 'SOURCE_DATE_EPOCH' variable [2]. >> >> In case of rootfs file time-stamps, set all the files and folders that >> are newer than 'SOURCE_DATE_EPOCH' and set it to same. > >Hi, > >is there a particular reason why we don't set all dates exactly to >SOURCE_DATE_EPOCH? Otherwise we might still have reproducibility issues in >case some files are provided by the sstate cache, hence have an older >timestamp. Building w/ or w/o cache should not make a difference. It is considered to change only the files that are created during build time and preserve the times in the files that are come with package, anyway the files and their timestamps comes with the package are reproducible. see some discussions about this change in the Debian live-build [1], where it is used for the same purpose to build Debian images reproducibly. Regarding the files comes sstate cache, the reproducible builds should carry out in the fresh build or after cleaned up state, so I think this should not be a problem. [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > >Felix > >> In case of initramfs, regenerate the initramfs image with >> 'SOURCE_DATE_EPOCH' variable set as the mkinitramfs script is already >> taken care of creating reproducible initramfs image when the variable >> is set in the environment[3]. >> >> The SOURCE_DATE_EPOCH variable should be set to the last modification >> of the git repository as explained in the documentation[2]. >> >> e.g: >> SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> >> To know more about the reproducible builds and its goals please refer >> [1]. >> >> [1] https://reproducible-builds.org/ >> [2] https://reproducible-builds.org/docs/source-date-epoch/ >> [3] >> https://manpages.debian.org/bullseye/initramfs-tools-core/mkinitramfs. >> 8.en.html#ENVIRONMENT >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> --- >> meta/classes/image.bbclass | 15 +++++++++++++++ >> 1 file changed, 15 insertions(+) >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index ccff810..c1bb4fd 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -431,6 +431,21 @@ do_rootfs_finalize() { >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> + >> + # Recreate initramfs inorder to set timestamps to >> SOURCE_DATE_EPOCH >> + # inorder to make reproducible initramfs >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot >> "${ROOTFSDIR}" \ >> + update-initramfs -u -v >> + >> + # Set timestamp to files inside the rootfs image inorder to >> make >> + # reproducible rootfs >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + find ${ROOTFSDIR} -newermt \ >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> %H:%M:%S')" \ >> + -printf "%y %p\n" \ >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> + >> EOSUDO >> } >> addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess >> -- >> 2.20.1 >> >> > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar- >users/383fd6d397ccdb132b8e97602784417fcf2cd63d.camel%40siemens.com.
Hi Henning Schild, Thanks for your review, please find my comments below. Thanks, Venkata. >-----Original Message----- >From: Henning Schild <henning.schild@siemens.com> >Sent: 07 November 2022 14:24 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: isar-users@googlegroups.com; jan.kiszka@siemens.com; hayashi kazuhiro(林 >和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳ >ＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> >Subject: Re: [PATCH] image.bbclass: set file timestamps inside the rootfs and >initramfs image > >Am Mon, 7 Nov 2022 13:55:03 +0530 >schrieb venkata.pyla@toshiba-tsip.com: > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> As part of reproducible-build work, one of the problem chosen to solve >> is the file time stamps inside rootfs and initramfs are not identical >> between two builds. >> >> With the help of reproducible-builds.org and their suggestions, the >> above problem can be fixed using 'SOURCE_DATE_EPOCH' variable [2]. >> >> In case of rootfs file time-stamps, set all the files and folders that >> are newer than 'SOURCE_DATE_EPOCH' and set it to same. >> In case of initramfs, regenerate the initramfs image with >> 'SOURCE_DATE_EPOCH' variable set as the mkinitramfs script is already >> taken care of creating reproducible initramfs image when the variable >> is set in the environment[3]. >> >> The SOURCE_DATE_EPOCH variable should be set to the last modification >> of the git repository as explained in the documentation[2]. >> >> e.g: >> SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> >> To know more about the reproducible builds and its goals please refer >> [1]. >> >> [1] https://reproducible-builds.org/ >> [2] https://reproducible-builds.org/docs/source-date-epoch/ >> [3] >> https://manpages.debian.org/bullseye/initramfs-tools-core/mkinitramfs. >> 8.en.html#ENVIRONMENT >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> --- >> meta/classes/image.bbclass | 15 +++++++++++++++ >> 1 file changed, 15 insertions(+) >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index ccff810..c1bb4fd 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -431,6 +431,21 @@ do_rootfs_finalize() { >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> + >> + # Recreate initramfs inorder to set timestamps to >> SOURCE_DATE_EPOCH >> + # inorder to make reproducible initramfs >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot >> "${ROOTFSDIR}" \ >> + update-initramfs -u -v > >This should be done where that initial update-initramfs can be found. >And not fix things up later. Not every image will have an initrd so that is wrong >in multiple ways. Thanks for correcting me, I understood the initrd update should not be in the goal of image class, as some of the images may not be required initrd as you mentioned. I will find the better place to do this and send the another patch. > >> + >> + # Set timestamp to files inside the rootfs image inorder to >> make >> + # reproducible rootfs >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + find ${ROOTFSDIR} -newermt \ >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> %H:%M:%S')" \ >> + -printf "%y %p\n" \ >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> + > >No! Which files do we care about? My guess would content of packages built >with isar. We should export the variable in the dpkg class then the packages will >be correct and we do not have to mess around with that rootfs and fix >problems that should have never been there. This will only change the files are created during build, and not the files come with package. The idea is to set time stamps to the files as same that are modified or added during build time (e.g: /etc/*) and they are newer than SOURCE_DATE_EPOCH date. > >I would like to ask for test cases. Ideally first a breaking test and later a commit >fixing the issue. I executed this in one of the child project (isar-cip-core) and reported the issue here [1], are you expecting to write test cases in isar? [1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/31 > >Note that deriving the time from git means one needs git. And when using >layers just that one Isar git is clearly not good enough. And we will have to >check how that works with sstate. I assume any new commit would rebuild all >custom packages. Here i see a conflict between regular dev work and repro >pedantics. We might need a way to turn that stuff off. > >regards, >Henning > >> EOSUDO >> } >> addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
Am Wed, 9 Nov 2022 09:27:20 +0000 schrieb <Venkata.Pyla@toshiba-tsip.com>: > Hi Henning Schild, > > Thanks for your review, please find my comments below. > > > Thanks, > Venkata. > > >-----Original Message----- > >From: Henning Schild <henning.schild@siemens.com> > >Sent: 07 November 2022 14:24 > >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > >tsip.com> > >Cc: isar-users@googlegroups.com; jan.kiszka@siemens.com; hayashi > >kazuhiro(林和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; > >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) > ><dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] image.bbclass: > >set file timestamps inside the rootfs and initramfs image > > > >Am Mon, 7 Nov 2022 13:55:03 +0530 > >schrieb venkata.pyla@toshiba-tsip.com: > > > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> > >> As part of reproducible-build work, one of the problem chosen to > >> solve is the file time stamps inside rootfs and initramfs are not > >> identical between two builds. > >> > >> With the help of reproducible-builds.org and their suggestions, the > >> above problem can be fixed using 'SOURCE_DATE_EPOCH' variable [2]. > >> > >> In case of rootfs file time-stamps, set all the files and folders > >> that are newer than 'SOURCE_DATE_EPOCH' and set it to same. > >> In case of initramfs, regenerate the initramfs image with > >> 'SOURCE_DATE_EPOCH' variable set as the mkinitramfs script is > >> already taken care of creating reproducible initramfs image when > >> the variable is set in the environment[3]. > >> > >> The SOURCE_DATE_EPOCH variable should be set to the last > >> modification of the git repository as explained in the > >> documentation[2]. > >> > >> e.g: > >> SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > >> > >> To know more about the reproducible builds and its goals please > >> refer [1]. > >> > >> [1] https://reproducible-builds.org/ > >> [2] https://reproducible-builds.org/docs/source-date-epoch/ > >> [3] > >> https://manpages.debian.org/bullseye/initramfs-tools-core/mkinitramfs. > >> 8.en.html#ENVIRONMENT > >> > >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> --- > >> meta/classes/image.bbclass | 15 +++++++++++++++ > >> 1 file changed, 15 insertions(+) > >> > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index ccff810..c1bb4fd 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -431,6 +431,21 @@ do_rootfs_finalize() { > >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >> > >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >> + > >> + # Recreate initramfs inorder to set timestamps to > >> SOURCE_DATE_EPOCH > >> + # inorder to make reproducible initramfs > >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >> + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot > >> "${ROOTFSDIR}" \ > >> + update-initramfs -u -v > > > >This should be done where that initial update-initramfs can be found. > >And not fix things up later. Not every image will have an initrd so > >that is wrong in multiple ways. > > Thanks for correcting me, I understood the initrd update should not > be in the goal of image class, as some of the images may not be > required initrd as you mentioned. I will find the better place to do > this and send the another patch. > > > > >> + > >> + # Set timestamp to files inside the rootfs image inorder > >> to make > >> + # reproducible rootfs > >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >> + find ${ROOTFSDIR} -newermt \ > >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >> %H:%M:%S')" \ > >> + -printf "%y %p\n" \ > >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > >> + > > > >No! Which files do we care about? My guess would content of packages > >built with isar. We should export the variable in the dpkg class > >then the packages will be correct and we do not have to mess around > >with that rootfs and fix problems that should have never been there. > > > > This will only change the files are created during build, and not the > files come with package. All files, or very close to all, come with packages. Because isar builds packages for stuff it does. The wanted timestamp has to be provided at build time of those packages so that later on one does not need to run such a find. You basically first want to make sure the packages (those from isar-apt) are reproducible and only later look at the whole rootfs. That whole rootfs will have remaining differences, just from what maintainer hooks and stuff do. That find violates the rule that everything should come from a package. There are some exceptions to that rule in isar, but those are likely the ones causing repro issues. And anyhow not everyone delivers only the images, some people also deliver the packages. So those need to be reproducible as well ... or even first. > The idea is to set time stamps to the files as same that are modified > or added during build time (e.g: /etc/*) and they are newer than > SOURCE_DATE_EPOCH date. > > > > > >I would like to ask for test cases. Ideally first a breaking test > >and later a commit fixing the issue. > > I executed this in one of the child project (isar-cip-core) and > reported the issue here [1], are you expecting to write test cases in > isar? > > [1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/31 The motivation always has to come with the patch. So that manual diffoscope workflow should be explained in the commit message or documentation. Ideally you would add a test that actually runs that and asserts that there are no problems left. Or ignoring the left ones for starters. That layer and all other layers will later benefit. But whatever you try to fix has to be explained and consistent inside isar only. So issues from some layer clearly do not count. Say we get the debian package feed to become reproducible, i bet we could just compare the repo metadata and the package checksums. Not even dive into the packages and check what might be different. A good way of explaining a change is also to first write the test and later the code to fix the problem a test shows. Everybody would understand easily and if any future change would break it again we would see before a merge. regards, Henning > > > >Note that deriving the time from git means one needs git. And when > >using layers just that one Isar git is clearly not good enough. And > >we will have to check how that works with sstate. I assume any new > >commit would rebuild all custom packages. Here i see a conflict > >between regular dev work and repro pedantics. We might need a way to > >turn that stuff off. > > > >regards, > >Henning > > > >> EOSUDO > >> } > >> addtask rootfs_finalize before do_rootfs after > >> do_rootfs_postprocess >
Hi I am using ISAR build system can you help me how to create a recipe for *Bluez5* * in ISAR build system* *Thanks & Regrads* On Mon, Nov 7, 2022 at 1:55 PM <venkata.pyla@toshiba-tsip.com> wrote: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > As part of reproducible-build work, one of the problem chosen to solve > is the file time stamps inside rootfs and initramfs are not identical > between two builds. > > With the help of reproducible-builds.org and their suggestions, the > above problem can be fixed using 'SOURCE_DATE_EPOCH' variable [2]. > > In case of rootfs file time-stamps, set all the files and folders that > are newer than 'SOURCE_DATE_EPOCH' and set it to same. > In case of initramfs, regenerate the initramfs image with > 'SOURCE_DATE_EPOCH' variable set as the mkinitramfs script is already > taken care of creating reproducible initramfs image when the > variable is set in the environment[3]. > > The SOURCE_DATE_EPOCH variable should be set to the last modification of > the git repository as explained in the documentation[2]. > > e.g: > SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) > > To know more about the reproducible builds and its goals please refer [1]. > > [1] https://reproducible-builds.org/ > [2] https://reproducible-builds.org/docs/source-date-epoch/ > [3] > https://manpages.debian.org/bullseye/initramfs-tools-core/mkinitramfs.8.en.html#ENVIRONMENT > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > meta/classes/image.bbclass | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index ccff810..c1bb4fd 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -431,6 +431,21 @@ do_rootfs_finalize() { > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > + > + # Recreate initramfs inorder to set timestamps to > SOURCE_DATE_EPOCH > + # inorder to make reproducible initramfs > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot "${ROOTFSDIR}" \ > + update-initramfs -u -v > + > + # Set timestamp to files inside the rootfs image inorder to make > + # reproducible rootfs > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + find ${ROOTFSDIR} -newermt \ > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ > + -printf "%y %p\n" \ > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > + > EOSUDO > } > addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess > -- > 2.20.1 > > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar-users/20221107082503.14774-1-venkata.pyla%40toshiba-tsip.com > . >
10.11.2022 12:52, 'Balasubramanian Sundaram' via isar-users wrote: > Hi > > I am using ISAR build system can you help me how to create a recipe > for *Bluez5* > * in ISAR build system* > * > * > *Thanks & Regrads* Hello, Balasubramanian. Current Debian repositories already provide bluez package (v5.55 for bullseye), so you can just add it to IMAGE_INSTALL. If you need to rebuild bluez to make some changes there, SRC_URI = "apt://bluez" can be used (see hello.bb as a reference). P.S. please avoid replying to existing maillist threads with the new questions
Hi Anton Iam getting this error for bluez [image: image.png] Thanks & Regards On Thu, Nov 10, 2022 at 3:33 PM Anton Mikanovich <amikan@ilbers.de> wrote: > 10.11.2022 12:52, 'Balasubramanian Sundaram' via isar-users wrote: > > Hi > > I am using ISAR build system can you help me how to create a recipe for > *Bluez5* > * in ISAR build system* > > *Thanks & Regrads* > > Hello, Balasubramanian. > > Current Debian repositories already provide bluez package (v5.55 for > bullseye), > so you can just add it to IMAGE_INSTALL. > If you need to rebuild bluez to make some changes there, SRC_URI = > "apt://bluez" > can be used (see hello.bb as a reference). > > P.S. please avoid replying to existing maillist threads with the new > questions >
Hi Anton And also could not open hello.bb for reference can you send it again Regards On Thu, Nov 10, 2022 at 3:47 PM Balasubramanian Sundaram < balasubramanian.sundaram@sanmina.com> wrote: > Hi Anton > > Iam getting this error for bluez > [image: image.png] > > Thanks & Regards > > On Thu, Nov 10, 2022 at 3:33 PM Anton Mikanovich <amikan@ilbers.de> wrote: > >> 10.11.2022 12:52, 'Balasubramanian Sundaram' via isar-users wrote: >> >> Hi >> >> I am using ISAR build system can you help me how to create a recipe for >> *Bluez5* >> * in ISAR build system* >> >> *Thanks & Regrads* >> >> Hello, Balasubramanian. >> >> Current Debian repositories already provide bluez package (v5.55 for >> bullseye), >> so you can just add it to IMAGE_INSTALL. >> If you need to rebuild bluez to make some changes there, SRC_URI = >> "apt://bluez" >> can be used (see hello.bb as a reference). >> >> P.S. please avoid replying to existing maillist threads with the new >> questions >> > > > -- > Balasubramanian Sundaram > Firmware Engineer | Sanmina Tech services | India, Chennai | Desk: +91 > 044 61309865 >
10.11.2022 13:17, Balasubramanian Sundaram wrote: > Hi Anton > > Iam getting this error for bluez > image.png > > Thanks & Regards Sorry, I've meant IMAGE_PREINSTALL, not IMAGE_INSTALL. IMAGE_PREINSTALL should be used for any package from external repos. IMAGE_INSTALL should be used in case you will rebuild the package from repo with your custom recipe. hello.bb is located in Isar examples: https://github.com/ilbers/isar/blob/master/meta-isar/recipes-app/hello/hello.bb
Thank you for the clarification. but iam getting following error can you tell what it can be [image: image.png] Regards On Thu, Nov 10, 2022 at 3:58 PM Anton Mikanovich <amikan@ilbers.de> wrote: > 10.11.2022 13:17, Balasubramanian Sundaram wrote: > > Hi Anton > > Iam getting this error for bluez > [image: image.png] > > Thanks & Regards > > Sorry, I've meant IMAGE_PREINSTALL, not IMAGE_INSTALL. > IMAGE_PREINSTALL should be used for any package from external repos. > IMAGE_INSTALL should be used in case you will rebuild the package from repo > with your custom recipe. > > hello.bb is located in Isar examples: > > https://github.com/ilbers/isar/blob/master/meta-isar/recipes-app/hello/hello.bb >
Hey Balasubramanian, you are making some fundamental mistakes here so that is why people might not want to help you or would not know how to do so. When you have a question for help please formulate it clearly and give some context. You just want to install a debian package in your image, use IMAGE_PREINSTALL. You have to modify/rebuild that package ... check out the hello example and use IMAGE_INSTALL. So say you can just use the meta-package "bluetooth" you can have a look at this part of the docs. https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-a-custom-image-recipe And consider giving this whole manual a read. It is not the best and maybe not all clear when you read it first. But over time it should make sense. That said. Please avoid these mistakes: - taking over a random discussion to talk about something totally different, rather open a new one - send debug output as screenshots ... they are hard to read and cite (they all have the same name), and you leak stuff like the model of your cell phone, or that your firmware is "Music" ... send the relevant bits as text and say how you got there. Without your recipe or change it is really hard to put your errors into any context. the last one looks like you modified the image recipe of the cip layer and added bluez as IMAGE_INSTALL where it should have been IMAGE_PREINSTALL - do not send html mails - do not include CONFIDENTIALITY disclaimers, your mails are super public so that does not make any sense regards, Henning Am Thu, 10 Nov 2022 16:21:18 +0530 schrieb Balasubramanian Sundaram <balasubramanian.sundaram@sanmina.com>: > Thank you for the clarification. > > but iam getting following error can you tell what it can be > [image: image.png] > > Regards > > On Thu, Nov 10, 2022 at 3:58 PM Anton Mikanovich <amikan@ilbers.de> > wrote: > > > 10.11.2022 13:17, Balasubramanian Sundaram wrote: > > > > Hi Anton > > > > Iam getting this error for bluez > > [image: image.png] > > > > Thanks & Regards > > > > Sorry, I've meant IMAGE_PREINSTALL, not IMAGE_INSTALL. > > IMAGE_PREINSTALL should be used for any package from external repos. > > IMAGE_INSTALL should be used in case you will rebuild the package > > from repo with your custom recipe. > > > > hello.bb is located in Isar examples: > > > > https://github.com/ilbers/isar/blob/master/meta-isar/recipes-app/hello/hello.bb > > > >
Am Thu, 10 Nov 2022 18:30:58 +0200 schrieb Henning Schild <henning.schild@siemens.com>: > Hey Balasubramanian, > > you are making some fundamental mistakes here so that is why people > might not want to help you or would not know how to do so. > > When you have a question for help please formulate it clearly and > give some context. You just want to install a debian package in your > image, use IMAGE_PREINSTALL. > You have to modify/rebuild that package ... check out the hello > example and use IMAGE_INSTALL. > > So say you can just use the meta-package "bluetooth" you can have a > look at this part of the docs. > https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-a-custom-image-recipe I used plain isar, added "IMAGE_PREINSTALL += "bluetooth"" to the end of meta-isar/recipes-core/images/isar-image-base.bb and did build it with "bitbake mc:qemuamd64-bullseye:isar-image-base". I booted it with qemu-system-x86_64 -smp 2 -enable-kvm -M q35 -m 2G -snapshot build/tmp/deploy/images/qemuamd64/isar-image-base-debian-bullseye-qemuamd64.wic -bios /usr/share/OVMF/OVMF_CODE.fd Works as expected. Well i have the packages like "bluez" installed ... Of cause in a real project one would create a layer with an own image recipe and add the line there. Henning > And consider giving this whole manual a read. It is not the best and > maybe not all clear when you read it first. But over time it should > make sense. > > That said. Please avoid these mistakes: > > - taking over a random discussion to talk about something totally > different, rather open a new one > - send debug output as screenshots ... they are hard to read and cite > (they all have the same name), and you leak stuff like the model of > your cell phone, or that your firmware is "Music" ... send the > relevant bits as text and say how you got there. Without your recipe > or change it is really hard to put your errors into any context. > the last one looks like you modified the image recipe of the cip > layer and added bluez as IMAGE_INSTALL where it should have been > IMAGE_PREINSTALL > - do not send html mails > - do not include CONFIDENTIALITY disclaimers, your mails are super > public so that does not make any sense > > regards, > Henning > > Am Thu, 10 Nov 2022 16:21:18 +0530 > schrieb Balasubramanian Sundaram > <balasubramanian.sundaram@sanmina.com>: > > > Thank you for the clarification. > > > > but iam getting following error can you tell what it can be > > [image: image.png] > > > > Regards > > > > On Thu, Nov 10, 2022 at 3:58 PM Anton Mikanovich <amikan@ilbers.de> > > wrote: > > > > > 10.11.2022 13:17, Balasubramanian Sundaram wrote: > > > > > > Hi Anton > > > > > > Iam getting this error for bluez > > > [image: image.png] > > > > > > Thanks & Regards > > > > > > Sorry, I've meant IMAGE_PREINSTALL, not IMAGE_INSTALL. > > > IMAGE_PREINSTALL should be used for any package from external > > > repos. IMAGE_INSTALL should be used in case you will rebuild the > > > package from repo with your custom recipe. > > > > > > hello.bb is located in Isar examples: > > > > > > https://github.com/ilbers/isar/blob/master/meta-isar/recipes-app/hello/hello.bb > > > > > > > >
>-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Henning Schild >Sent: 10 November 2022 12:42 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: isar-users@googlegroups.com; jan.kiszka@siemens.com; hayashi kazuhiro(林 >和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳ >ＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> >Subject: Re: [PATCH] image.bbclass: set file timestamps inside the rootfs and >initramfs image > >Am Wed, 9 Nov 2022 09:27:20 +0000 >schrieb <Venkata.Pyla@toshiba-tsip.com>: > >> Hi Henning Schild, >> >> Thanks for your review, please find my comments below. >> >> >> Thanks, >> Venkata. >> >> >-----Original Message----- >> >From: Henning Schild <henning.schild@siemens.com> >> >Sent: 07 November 2022 14:24 >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >> >tsip.com> >> >Cc: isar-users@googlegroups.com; jan.kiszka@siemens.com; hayashi >> >kazuhiro(林和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >> >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) >> ><dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] image.bbclass: >> >set file timestamps inside the rootfs and initramfs image >> > >> >Am Mon, 7 Nov 2022 13:55:03 +0530 >> >schrieb venkata.pyla@toshiba-tsip.com: >> > >> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> >> >> As part of reproducible-build work, one of the problem chosen to >> >> solve is the file time stamps inside rootfs and initramfs are not >> >> identical between two builds. >> >> >> >> With the help of reproducible-builds.org and their suggestions, the >> >> above problem can be fixed using 'SOURCE_DATE_EPOCH' variable [2]. >> >> >> >> In case of rootfs file time-stamps, set all the files and folders >> >> that are newer than 'SOURCE_DATE_EPOCH' and set it to same. >> >> In case of initramfs, regenerate the initramfs image with >> >> 'SOURCE_DATE_EPOCH' variable set as the mkinitramfs script is >> >> already taken care of creating reproducible initramfs image when >> >> the variable is set in the environment[3]. >> >> >> >> The SOURCE_DATE_EPOCH variable should be set to the last >> >> modification of the git repository as explained in the >> >> documentation[2]. >> >> >> >> e.g: >> >> SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) >> >> >> >> To know more about the reproducible builds and its goals please >> >> refer [1]. >> >> >> >> [1] https://reproducible-builds.org/ [2] >> >> https://reproducible-builds.org/docs/source-date-epoch/ >> >> [3] >> >> https://manpages.debian.org/bullseye/initramfs-tools-core/mkinitramfs. >> >> 8.en.html#ENVIRONMENT >> >> >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> --- >> >> meta/classes/image.bbclass | 15 +++++++++++++++ >> >> 1 file changed, 15 insertions(+) >> >> >> >> diff --git a/meta/classes/image.bbclass >> >> b/meta/classes/image.bbclass index ccff810..c1bb4fd 100644 >> >> --- a/meta/classes/image.bbclass >> >> +++ b/meta/classes/image.bbclass >> >> @@ -431,6 +431,21 @@ do_rootfs_finalize() { >> >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> >> + >> >> + # Recreate initramfs inorder to set timestamps to >> >> SOURCE_DATE_EPOCH >> >> + # inorder to make reproducible initramfs >> >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> >> + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot >> >> "${ROOTFSDIR}" \ >> >> + update-initramfs -u -v >> > >> >This should be done where that initial update-initramfs can be found. >> >And not fix things up later. Not every image will have an initrd so >> >that is wrong in multiple ways. >> >> Thanks for correcting me, I understood the initrd update should not be >> in the goal of image class, as some of the images may not be required >> initrd as you mentioned. I will find the better place to do this and >> send the another patch. >> >> > >> >> + >> >> + # Set timestamp to files inside the rootfs image inorder >> >> to make >> >> + # reproducible rootfs >> >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> >> + find ${ROOTFSDIR} -newermt \ >> >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> >> %H:%M:%S')" \ >> >> + -printf "%y %p\n" \ >> >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> >> + >> > >> >No! Which files do we care about? My guess would content of packages >> >built with isar. We should export the variable in the dpkg class then >> >the packages will be correct and we do not have to mess around with >> >that rootfs and fix problems that should have never been there. >> > >> >> This will only change the files are created during build, and not the >> files come with package. > >All files, or very close to all, come with packages. Because isar builds packages >for stuff it does. The wanted timestamp has to be provided at build time of >those packages so that later on one does not need to run such a find. > >You basically first want to make sure the packages (those from >isar-apt) are reproducible and only later look at the whole rootfs. That whole >rootfs will have remaining differences, just from what maintainer hooks and >stuff do. > >That find violates the rule that everything should come from a package. >There are some exceptions to that rule in isar, but those are likely the ones >causing repro issues. And anyhow not everyone delivers only the images, some >people also deliver the packages. So those need to be reproducible as well ... or >even first. I am trying to understand Isar and how it installs the debian packages, I think Isar rebuilds some of the packages from the sources using sbuild, If that is so, then sbuild taken care of creating the package reproducibly, if it is not creating then we should definitely check why it is no generating reproducible packages. The find command here mostly fixes the timestamps of files or folders that are created or modified during 'postinst' scripts (or after package installation). > >> The idea is to set time stamps to the files as same that are modified >> or added during build time (e.g: /etc/*) and they are newer than >> SOURCE_DATE_EPOCH date. >> >> >> > >> >I would like to ask for test cases. Ideally first a breaking test and >> >later a commit fixing the issue. >> >> I executed this in one of the child project (isar-cip-core) and >> reported the issue here [1], are you expecting to write test cases in >> isar? >> >> [1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/31 > >The motivation always has to come with the patch. So that manual diffoscope >workflow should be explained in the commit message or documentation. Ideally >you would add a test that actually runs that and asserts that there are no >problems left. Or ignoring the left ones for starters. > >That layer and all other layers will later benefit. But whatever you try to fix has >to be explained and consistent inside isar only. So issues from some layer >clearly do not count. > >Say we get the debian package feed to become reproducible, i bet we could just >compare the repo metadata and the package checksums. Not even dive into the >packages and check what might be different. > >A good way of explaining a change is also to first write the test and later the >code to fix the problem a test shows. Everybody would understand easily and if >any future change would break it again we would see before a merge. I agree with you and I have also sent patch[1] for this that have test script for verifying the reproducibility in Isar images. [1] https://groups.google.com/g/isar-users/c/4ZIuKCOQzVc/m/sPUafWDgAgAJ > >regards, >Henning > >> > >> >Note that deriving the time from git means one needs git. And when >> >using layers just that one Isar git is clearly not good enough. And >> >we will have to check how that works with sstate. I assume any new >> >commit would rebuild all custom packages. Here i see a conflict >> >between regular dev work and repro pedantics. We might need a way to >> >turn that stuff off. >> > >> >regards, >> >Henning >> > >> >> EOSUDO >> >> } >> >> addtask rootfs_finalize before do_rootfs after >> >> do_rootfs_postprocess >> > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar- >users/20221110091226.4a3695f2%40md1za8fc.ad001.siemens.net.
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index ccff810..c1bb4fd 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -431,6 +431,21 @@ do_rootfs_finalize() { "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" rm -f "${ROOTFSDIR}/etc/apt/sources-list" + + # Recreate initramfs inorder to set timestamps to SOURCE_DATE_EPOCH + # inorder to make reproducible initramfs + test ! -z "${SOURCE_DATE_EPOCH}" && \ + SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH} chroot "${ROOTFSDIR}" \ + update-initramfs -u -v + + # Set timestamp to files inside the rootfs image inorder to make + # reproducible rootfs + test ! -z "${SOURCE_DATE_EPOCH}" && \ + find ${ROOTFSDIR} -newermt \ + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ + -printf "%y %p\n" \ + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' + EOSUDO } addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess