| Message ID | 20230102145828.32763-1-venkata.pyla@toshiba-tsip.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | image.bbclass: fix non-reproducible file time-stamps inside rootfs image | expand |
On 02.01.23 15:58, venkata.pyla@toshiba-tsip.com wrote: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > As part of reproducible-build work, the rootfs images generated on same > source should be identical between two builds. > > In this commit it tries to solve one of the non-reproducible problem > i.e. the rootfs file time-stamps generated during build time are not > reproducible, it uses one of the solution provided in the debian > live-build image project (refer [1]), it fixes by finding all the > files/folders that are gernerated newly and set the time-stamp provided > by `SOURCE_DATE_EPOCH` environment variable. > > [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > Just out of curiosity: How does Yocto address this? Jan > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > meta/classes/image.bbclass | 9 +++++++++ > 1 file changed, 9 insertions(+) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 813e1f3..f592a12 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -430,6 +430,15 @@ do_rootfs_finalize() { > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > + > + # Set same time-stamps to the newly generated file/folders in the > + # rootfs image for the purpose of reproducible builds. > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + find ${ROOTFSDIR} -newermt \ > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ > + -printf "%y %p\n" \ > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > + > EOSUDO > } > addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb venkata.pyla@toshiba-tsip.com: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > As part of reproducible-build work, the rootfs images generated on > same source should be identical between two builds. > > In this commit it tries to solve one of the non-reproducible problem > i.e. the rootfs file time-stamps generated during build time are not > reproducible, it uses one of the solution provided in the debian > live-build image project (refer [1]), it fixes by finding all the > files/folders that are gernerated newly and set the time-stamp > provided by `SOURCE_DATE_EPOCH` environment variable. > > [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > meta/classes/image.bbclass | 9 +++++++++ > 1 file changed, 9 insertions(+) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 813e1f3..f592a12 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -430,6 +430,15 @@ do_rootfs_finalize() { > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > + > + # Set same time-stamps to the newly generated file/folders > in the > + # rootfs image for the purpose of reproducible builds. > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + find ${ROOTFSDIR} -newermt \ > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > %H:%M:%S')" \ > + -printf "%y %p\n" \ > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > + This looks like i have seen it before. For me that is _way_ too generic and something that is not a package touches files all over the place. If some package now wants to intentionally bring a file that is from a far away future? Which files are we talking about here? It can basically only be metadata and other little places where we violate our "everything comes from a package" rule. Has this ever been tested against a complex layer, has any of the repro work ever looked at something bigger than the very artificial isar base image? I think a better start would be to bbwarn and only much later move to "-exec touch". We recently added CI tests for reproducible building. Would be nice to the two patches. p1 writes a test-case that goes red, p2 (this one) makes it go green Henning > EOSUDO > } > addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
>-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Jan Kiszka >Sent: 02 January 2023 21:03 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com>; isar-users@googlegroups.com >Cc: amikan@ilbers.de; henning.schild@siemens.com; hayashi kazuhiro(林 和宏 >□ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ >TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> >Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside >rootfs image > >On 02.01.23 15:58, venkata.pyla@toshiba-tsip.com wrote: >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> As part of reproducible-build work, the rootfs images generated on >> same source should be identical between two builds. >> >> In this commit it tries to solve one of the non-reproducible problem >> i.e. the rootfs file time-stamps generated during build time are not >> reproducible, it uses one of the solution provided in the debian >> live-build image project (refer [1]), it fixes by finding all the >> files/folders that are gernerated newly and set the time-stamp >> provided by `SOURCE_DATE_EPOCH` environment variable. >> >> [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 >> > >Just out of curiosity: How does Yocto address this? Yocto also does similar approach as debian live-build but in yocto it set the timestamp `REPRODUCIBLE_TIMESTAMP_ROOTFS` to all the rootfs files See the reference code https://github.com/yoctoproject/poky/blob/master/meta/classes-recipe/image.bbclass#L673 whereas in debian live-build it assumes the files from the debian packages are reproducible so only the newly changed or created files are set to the timestamp `SOURCE_DATE_EPOCH` > >Jan > >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> --- >> meta/classes/image.bbclass | 9 +++++++++ >> 1 file changed, 9 insertions(+) >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index 813e1f3..f592a12 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> + >> + # Set same time-stamps to the newly generated file/folders in the >> + # rootfs image for the purpose of reproducible builds. >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + find ${ROOTFSDIR} -newermt \ >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ >> + -printf "%y %p\n" \ >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> + >> EOSUDO >> } >> addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess > >-- >Siemens AG, Technology >Competence Center Embedded Linux > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar-users/f5a8e3e8-30ad-9039-836f- >5ce42b25567f%40siemens.com.
On 02.01.23 17:44, Henning Schild wrote: > Am Mon, 2 Jan 2023 20:28:28 +0530 > schrieb venkata.pyla@toshiba-tsip.com: > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> As part of reproducible-build work, the rootfs images generated on >> same source should be identical between two builds. >> >> In this commit it tries to solve one of the non-reproducible problem >> i.e. the rootfs file time-stamps generated during build time are not >> reproducible, it uses one of the solution provided in the debian >> live-build image project (refer [1]), it fixes by finding all the >> files/folders that are gernerated newly and set the time-stamp >> provided by `SOURCE_DATE_EPOCH` environment variable. >> >> [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> --- >> meta/classes/image.bbclass | 9 +++++++++ >> 1 file changed, 9 insertions(+) >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index 813e1f3..f592a12 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> + >> + # Set same time-stamps to the newly generated file/folders >> in the >> + # rootfs image for the purpose of reproducible builds. >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + find ${ROOTFSDIR} -newermt \ >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> %H:%M:%S')" \ >> + -printf "%y %p\n" \ >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> + > > This looks like i have seen it before. For me that is _way_ too generic > and something that is not a package touches files all over the place. > If some package now wants to intentionally bring a file that is from a > far away future? Then debian-live would have the same problem - I don't think following that pattern is a bad idea as we are not alone. Jan
>-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Henning Schild >Sent: 02 January 2023 22:14 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; jan.kiszka@siemens.com; >hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> >Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside >rootfs image > >Am Mon, 2 Jan 2023 20:28:28 +0530 >schrieb venkata.pyla@toshiba-tsip.com: > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> As part of reproducible-build work, the rootfs images generated on >> same source should be identical between two builds. >> >> In this commit it tries to solve one of the non-reproducible problem >> i.e. the rootfs file time-stamps generated during build time are not >> reproducible, it uses one of the solution provided in the debian >> live-build image project (refer [1]), it fixes by finding all the >> files/folders that are gernerated newly and set the time-stamp >> provided by `SOURCE_DATE_EPOCH` environment variable. >> >> [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> --- >> meta/classes/image.bbclass | 9 +++++++++ >> 1 file changed, 9 insertions(+) >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index 813e1f3..f592a12 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> + >> + # Set same time-stamps to the newly generated file/folders >> in the >> + # rootfs image for the purpose of reproducible builds. >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + find ${ROOTFSDIR} -newermt \ >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> %H:%M:%S')" \ >> + -printf "%y %p\n" \ >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> + > >This looks like i have seen it before. For me that is _way_ too generic and >something that is not a package touches files all over the place. >If some package now wants to intentionally bring a file that is from a far away >future? > >Which files are we talking about here? It can basically only be metadata and >other little places where we violate our "everything comes from a package" rule. files/folder/symbolic-link that are modified or generated during build time like /etc/os-release /etc/hostname . . . /var/lib/dpkg/info/* /var/cache/* --- I have printed all the files that modified during build by executing below command find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y %p\n" > modified_files_times.txt attached modified_files_times.txt for your reference, all these files/folders/symbolic-link are generated or modified during build time. > >Has this ever been tested against a complex layer, has any of the repro work >ever looked at something bigger than the very artificial isar base image? Similar implementations were used in the projects like Debian/live-build[1] and poky[2] [1] https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/build/efi-image#L57 [2] https://github.com/yoctoproject/poky/blob/master/meta/classes-recipe/image.bbclass#L673 > >I think a better start would be to bbwarn and only much later move to "-exec >touch". > >We recently added CI tests for reproducible building. Would be nice to the two >patches. p1 writes a test-case that goes red, p2 (this one) makes it go green The patch (p1) already been sent [1], for writing test-case to verify reproducible build problems, the test-case is getting failed (shows non-reproducible time-stamp problem in build/diffoscope-output.txt) when executed without this patch, and after applying this patch it will still show fail (because there are other reproducible problems to fix) but the non-reproducible timestamp problem in rootfs shows fixed. [1] https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb58b24e953cb9 > >Henning > >> EOSUDO >> } >> addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar- >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net.
Am Tue, 3 Jan 2023 09:05:53 +0100 schrieb Jan Kiszka <jan.kiszka@siemens.com>: > On 02.01.23 17:44, Henning Schild wrote: > > Am Mon, 2 Jan 2023 20:28:28 +0530 > > schrieb venkata.pyla@toshiba-tsip.com: > > > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> > >> As part of reproducible-build work, the rootfs images generated on > >> same source should be identical between two builds. > >> > >> In this commit it tries to solve one of the non-reproducible > >> problem i.e. the rootfs file time-stamps generated during build > >> time are not reproducible, it uses one of the solution provided in > >> the debian live-build image project (refer [1]), it fixes by > >> finding all the files/folders that are gernerated newly and set > >> the time-stamp provided by `SOURCE_DATE_EPOCH` environment > >> variable. > >> > >> [1] > >> https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > >> > >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> --- > >> meta/classes/image.bbclass | 9 +++++++++ > >> 1 file changed, 9 insertions(+) > >> > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { > >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >> > >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >> + > >> + # Set same time-stamps to the newly generated file/folders > >> in the > >> + # rootfs image for the purpose of reproducible builds. > >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >> + find ${ROOTFSDIR} -newermt \ > >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >> %H:%M:%S')" \ > >> + -printf "%y %p\n" \ > >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > >> + > > > > This looks like i have seen it before. For me that is _way_ too > > generic and something that is not a package touches files all over > > the place. If some package now wants to intentionally bring a file > > that is from a far away future? > > Then debian-live would have the same problem - I don't think following > that pattern is a bad idea as we are not alone. debian-live is a closed quality thing with likely very little hacks that do not come from packages. So ... kind of ... is isar core. But in the layers we do not really know what people do. Note that this code should somehow be colocated with do_rootfs_quality_check where we already check how people messed around in postproc based on time of files. And i still vote that we bring it as bbwarn only, telling people that a file is causing issues with repro build. And maybe a list of files that people have to append to to have them touched ... not touch everything we find by default. Henning > Jan >
Am Tue, 3 Jan 2023 14:10:14 +0000 schrieb <Venkata.Pyla@toshiba-tsip.com>: > >-----Original Message----- > >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >Behalf Of Henning Schild > >Sent: 02 January 2023 22:14 > >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > >tsip.com> > >Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG > >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >image.bbclass: fix non-reproducible file time-stamps inside rootfs > >image > > > >Am Mon, 2 Jan 2023 20:28:28 +0530 > >schrieb venkata.pyla@toshiba-tsip.com: > > > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> > >> As part of reproducible-build work, the rootfs images generated on > >> same source should be identical between two builds. > >> > >> In this commit it tries to solve one of the non-reproducible > >> problem i.e. the rootfs file time-stamps generated during build > >> time are not reproducible, it uses one of the solution provided in > >> the debian live-build image project (refer [1]), it fixes by > >> finding all the files/folders that are gernerated newly and set > >> the time-stamp provided by `SOURCE_DATE_EPOCH` environment > >> variable. > >> > >> [1] > >> https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > >> > >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> --- > >> meta/classes/image.bbclass | 9 +++++++++ > >> 1 file changed, 9 insertions(+) > >> > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { > >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >> > >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >> + > >> + # Set same time-stamps to the newly generated file/folders > >> in the > >> + # rootfs image for the purpose of reproducible builds. > >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >> + find ${ROOTFSDIR} -newermt \ > >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >> %H:%M:%S')" \ > >> + -printf "%y %p\n" \ > >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > >> + > > > >This looks like i have seen it before. For me that is _way_ too > >generic and something that is not a package touches files all over > >the place. If some package now wants to intentionally bring a file > >that is from a far away future? > > > >Which files are we talking about here? It can basically only be > >metadata and other little places where we violate our "everything > >comes from a package" rule. > > files/folder/symbolic-link that are modified or generated during > build time like /etc/os-release > /etc/hostname > . > . > . > /var/lib/dpkg/info/* > /var/cache/* > --- > > I have printed all the files that modified during build by executing > below command find ${ROOTFSDIR} -newermt "$(date > -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y %p\n" > > modified_files_times.txt > > attached modified_files_times.txt for your reference, all these > files/folders/symbolic-link are generated or modified during build > time. So i am guessing it is about anything that comes out of postprocess functions and maintainer scripts like postinst ? > > > >Has this ever been tested against a complex layer, has any of the > >repro work ever looked at something bigger than the very artificial > >isar base image? > Similar implementations were used in the projects like > Debian/live-build[1] and poky[2] > > [1] > https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/build/efi-image#L57 > [2] > https://github.com/yoctoproject/poky/blob/master/meta/classes-recipe/image.bbclass#L673 I am talking about isar layers not other projects. > > > >I think a better start would be to bbwarn and only much later move > >to "-exec touch". > > > >We recently added CI tests for reproducible building. Would be nice > >to the two patches. p1 writes a test-case that goes red, p2 (this > >one) makes it go green > The patch (p1) already been sent [1], for writing test-case to verify > reproducible build problems, the test-case is getting failed (shows > non-reproducible time-stamp problem in build/diffoscope-output.txt) > when executed without this patch, and after applying this patch it > will still show fail (because there are other reproducible problems > to fix) but the non-reproducible timestamp problem in rootfs shows > fixed. So the code is there but not used because it goes failed anyhow? In that case i would suggest to rewrite it in a way that it does not fail and can run in CI. It could for example ignore "known issues" and be enabled, where then this very series would remove a few "known issues" in p2 (p1 would be to introduce all currently known issues and enable that thing in CI) and p3 would be what we look at here Not sure i understood but it sounds like we have a test that we only run manually, and i want it to run in CI. In fact a script to check for repro issues should be so generic that any layer can use it. So your vision would be that isar core brings that script and i.e. jailhouse-images can run it on all its images. And a few calls to the script will also be in testsuite/ and be called in pipelines regularly. Henning > [1] > https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb58b24e953cb9 > > > > >Henning > > > >> EOSUDO > >> } > >> addtask rootfs_finalize before do_rootfs after > >> do_rootfs_postprocess > > > >-- > >You received this message because you are subscribed to the Google > >Groups "isar-users" group. > >To unsubscribe from this group and stop receiving emails from it, > >send an email to isar-users+unsubscribe@googlegroups.com. > >To view this discussion on the web visit > >https://groups.google.com/d/msgid/isar- > >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net.
>-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Henning Schild >Sent: 04 January 2023 00:36 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; jan.kiszka@siemens.com; >hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> >Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside >rootfs image > >Am Tue, 3 Jan 2023 14:10:14 +0000 >schrieb <Venkata.Pyla@toshiba-tsip.com>: > >> >-----Original Message----- >> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >> >Behalf Of Henning Schild >> >Sent: 02 January 2023 22:14 >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >> >tsip.com> >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; >> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >> >image.bbclass: fix non-reproducible file time-stamps inside rootfs >> >image >> > >> >Am Mon, 2 Jan 2023 20:28:28 +0530 >> >schrieb venkata.pyla@toshiba-tsip.com: >> > >> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> >> >> As part of reproducible-build work, the rootfs images generated on >> >> same source should be identical between two builds. >> >> >> >> In this commit it tries to solve one of the non-reproducible >> >> problem i.e. the rootfs file time-stamps generated during build >> >> time are not reproducible, it uses one of the solution provided in >> >> the debian live-build image project (refer [1]), it fixes by >> >> finding all the files/folders that are gernerated newly and set the >> >> time-stamp provided by `SOURCE_DATE_EPOCH` environment variable. >> >> >> >> [1] >> >> https://salsa.debian.org/live-team/live-build/-/merge_requests/218 >> >> >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> --- >> >> meta/classes/image.bbclass | 9 +++++++++ >> >> 1 file changed, 9 insertions(+) >> >> >> >> diff --git a/meta/classes/image.bbclass >> >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 >> >> --- a/meta/classes/image.bbclass >> >> +++ b/meta/classes/image.bbclass >> >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >> >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> >> + >> >> + # Set same time-stamps to the newly generated file/folders >> >> in the >> >> + # rootfs image for the purpose of reproducible builds. >> >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> >> + find ${ROOTFSDIR} -newermt \ >> >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> >> %H:%M:%S')" \ >> >> + -printf "%y %p\n" \ >> >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> >> + >> > >> >This looks like i have seen it before. For me that is _way_ too >> >generic and something that is not a package touches files all over >> >the place. If some package now wants to intentionally bring a file >> >that is from a far away future? >> > >> >Which files are we talking about here? It can basically only be >> >metadata and other little places where we violate our "everything >> >comes from a package" rule. >> >> files/folder/symbolic-link that are modified or generated during build >> time like /etc/os-release /etc/hostname . >> . >> . >> /var/lib/dpkg/info/* >> /var/cache/* >> --- >> >> I have printed all the files that modified during build by executing >> below command find ${ROOTFSDIR} -newermt "$(date >> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y %p\n" >> >> modified_files_times.txt >> >> attached modified_files_times.txt for your reference, all these >> files/folders/symbolic-link are generated or modified during build >> time. > >So i am guessing it is about anything that comes out of postprocess functions >and maintainer scripts like postinst ? Yes, those are the files that are modified by the packages and have different timestamp on each build. I think it is okay to set reproducible times-tamp for such files, as they are not coming from the packages but are modified at build time. > >> > >> >Has this ever been tested against a complex layer, has any of the >> >repro work ever looked at something bigger than the very artificial >> >isar base image? >> Similar implementations were used in the projects like >> Debian/live-build[1] and poky[2] >> >> [1] >> https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/bu >> ild/efi-image#L57 >> [2] >> https://github.com/yoctoproject/poky/blob/master/meta/classes-recipe/i >> mage.bbclass#L673 > >I am talking about isar layers not other projects. > >> > >> >I think a better start would be to bbwarn and only much later move to >> >"-exec touch". >> > >> >We recently added CI tests for reproducible building. Would be nice >> >to the two patches. p1 writes a test-case that goes red, p2 (this >> >one) makes it go green >> The patch (p1) already been sent [1], for writing test-case to verify >> reproducible build problems, the test-case is getting failed (shows >> non-reproducible time-stamp problem in build/diffoscope-output.txt) >> when executed without this patch, and after applying this patch it >> will still show fail (because there are other reproducible problems to >> fix) but the non-reproducible timestamp problem in rootfs shows fixed. > >So the code is there but not used because it goes failed anyhow? In that case i >would suggest to rewrite it in a way that it does not fail and can run in CI. It >could for example ignore "known issues" and be enabled, where then this very >series would remove a few "known issues" >in p2 (p1 would be to introduce all currently known issues and enable that thing >in CI) and p3 would be what we look at here You mean run the test and just ignore the results? > >Not sure i understood but it sounds like we have a test that we only run >manually, and i want it to run in CI. In fact a script to check for repro issues >should be so generic that any layer can use it. Eventually the repro-build test-case should go to CI, I will plan to do it. Can I know where the isar CI pipe lines are running? I couldn't see in isar-github may be running some other place? > >So your vision would be that isar core brings that script and i.e. >jailhouse-images can run it on all its images. And a few calls to the script will >also be in testsuite/ and be called in pipelines regularly. > >Henning > >> [1] >> https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb58b2 >> 4e953cb9 >> >> > >> >Henning >> > >> >> EOSUDO >> >> } >> >> addtask rootfs_finalize before do_rootfs after >> >> do_rootfs_postprocess >> > >> >-- >> >You received this message because you are subscribed to the Google >> >Groups "isar-users" group. >> >To unsubscribe from this group and stop receiving emails from it, >> >send an email to isar-users+unsubscribe@googlegroups.com. >> >To view this discussion on the web visit >> >https://groups.google.com/d/msgid/isar- >> >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net. > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar- >users/20230103200543.07e987ba%40md1za8fc.ad001.siemens.net.
Am Wed, 4 Jan 2023 07:54:44 +0000 schrieb <Venkata.Pyla@toshiba-tsip.com>: > >-----Original Message----- > >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >Behalf Of Henning Schild > >Sent: 04 January 2023 00:36 > >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > >tsip.com> > >Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG > >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >image.bbclass: fix non-reproducible file time-stamps inside rootfs > >image > > > >Am Tue, 3 Jan 2023 14:10:14 +0000 > >schrieb <Venkata.Pyla@toshiba-tsip.com>: > > > >> >-----Original Message----- > >> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >> >Behalf Of Henning Schild > >> >Sent: 02 January 2023 22:14 > >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >> ><Venkata.Pyla@toshiba- tsip.com> > >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > >> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG > >> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >> >image.bbclass: fix non-reproducible file time-stamps inside rootfs > >> >image > >> > > >> >Am Mon, 2 Jan 2023 20:28:28 +0530 > >> >schrieb venkata.pyla@toshiba-tsip.com: > >> > > >> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> >> > >> >> As part of reproducible-build work, the rootfs images generated > >> >> on same source should be identical between two builds. > >> >> > >> >> In this commit it tries to solve one of the non-reproducible > >> >> problem i.e. the rootfs file time-stamps generated during build > >> >> time are not reproducible, it uses one of the solution provided > >> >> in the debian live-build image project (refer [1]), it fixes by > >> >> finding all the files/folders that are gernerated newly and set > >> >> the time-stamp provided by `SOURCE_DATE_EPOCH` environment > >> >> variable. > >> >> > >> >> [1] > >> >> https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > >> >> > >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> >> --- > >> >> meta/classes/image.bbclass | 9 +++++++++ > >> >> 1 file changed, 9 insertions(+) > >> >> > >> >> diff --git a/meta/classes/image.bbclass > >> >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 > >> >> --- a/meta/classes/image.bbclass > >> >> +++ b/meta/classes/image.bbclass > >> >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { > >> >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >> >> > >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >> >> + > >> >> + # Set same time-stamps to the newly generated > >> >> file/folders in the > >> >> + # rootfs image for the purpose of reproducible builds. > >> >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >> >> + find ${ROOTFSDIR} -newermt \ > >> >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >> >> %H:%M:%S')" \ > >> >> + -printf "%y %p\n" \ > >> >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > >> >> + > >> > > >> >This looks like i have seen it before. For me that is _way_ too > >> >generic and something that is not a package touches files all over > >> >the place. If some package now wants to intentionally bring a file > >> >that is from a far away future? > >> > > >> >Which files are we talking about here? It can basically only be > >> >metadata and other little places where we violate our "everything > >> >comes from a package" rule. > >> > >> files/folder/symbolic-link that are modified or generated during > >> build time like /etc/os-release /etc/hostname . > >> . > >> . > >> /var/lib/dpkg/info/* > >> /var/cache/* > >> --- > >> > >> I have printed all the files that modified during build by > >> executing below command find ${ROOTFSDIR} -newermt "$(date > >> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y %p\n" > >> > >> modified_files_times.txt > >> > >> attached modified_files_times.txt for your reference, all these > >> files/folders/symbolic-link are generated or modified during build > >> time. > > > >So i am guessing it is about anything that comes out of postprocess > >functions and maintainer scripts like postinst ? > > Yes, those are the files that are modified by the packages and have > different timestamp on each build. I think it is okay to set > reproducible times-tamp for such files, as they are not coming from > the packages but are modified at build time. Sure they are OK. But if i read your patch correctly you will potentially adjust times on ALL files, not just some for which it might be OK. And it uses a variable which does not have a default, should we not set that to something? Or help users to choose a good value for it. Without this variable and a clue on a value to pick, this is all dead code. > > > >> > > >> >Has this ever been tested against a complex layer, has any of the > >> >repro work ever looked at something bigger than the very > >> >artificial isar base image? > >> Similar implementations were used in the projects like > >> Debian/live-build[1] and poky[2] > >> > >> [1] > >> https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/bu > >> ild/efi-image#L57 > >> [2] > >> https://github.com/yoctoproject/poky/blob/master/meta/classes-recipe/i > >> mage.bbclass#L673 > > > >I am talking about isar layers not other projects. > > > >> > > >> >I think a better start would be to bbwarn and only much later > >> >move to "-exec touch". > >> > > >> >We recently added CI tests for reproducible building. Would be > >> >nice to the two patches. p1 writes a test-case that goes red, p2 > >> >(this one) makes it go green > >> The patch (p1) already been sent [1], for writing test-case to > >> verify reproducible build problems, the test-case is getting > >> failed (shows non-reproducible time-stamp problem in > >> build/diffoscope-output.txt) when executed without this patch, and > >> after applying this patch it will still show fail (because there > >> are other reproducible problems to fix) but the non-reproducible > >> timestamp problem in rootfs shows fixed. > > > >So the code is there but not used because it goes failed anyhow? In > >that case i would suggest to rewrite it in a way that it does not > >fail and can run in CI. It could for example ignore "known issues" > >and be enabled, where then this very series would remove a few > >"known issues" in p2 (p1 would be to introduce all currently known > >issues and enable that thing in CI) and p3 would be what we look at > >here > > You mean run the test and just ignore the results? Ignore some, the ones we know as violators that we still need to work on. But do not ignore anything coming up as new problems, maybe/likely in layers. Again, has this ever been tested on anything but the very boring isar core? isar-cip-core could be interesting as well, especially the image cip-core-image-security > > > >Not sure i understood but it sounds like we have a test that we only > >run manually, and i want it to run in CI. In fact a script to check > >for repro issues should be so generic that any layer can use it. > > Eventually the repro-build test-case should go to CI, I will plan to > do it. > > Can I know where the isar CI pipe lines are running? I couldn't see > in isar-github may be running some other place? Unfortunately the two places that run it are not public and would require you to receive an account. You could ask Baurzhan if you can get one, or you just run it yourself manually while working on the patches. If it is enabled and will ever fail, you will likely get reports from CI sent to you. Henning > > > >So your vision would be that isar core brings that script and i.e. > >jailhouse-images can run it on all its images. And a few calls to > >the script will also be in testsuite/ and be called in pipelines > >regularly. > > > >Henning > > > >> [1] > >> https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb58b2 > >> 4e953cb9 > >> > >> > > >> >Henning > >> > > >> >> EOSUDO > >> >> } > >> >> addtask rootfs_finalize before do_rootfs after > >> >> do_rootfs_postprocess > >> > > >> >-- > >> >You received this message because you are subscribed to the Google > >> >Groups "isar-users" group. > >> >To unsubscribe from this group and stop receiving emails from it, > >> >send an email to isar-users+unsubscribe@googlegroups.com. > >> >To view this discussion on the web visit > >> >https://groups.google.com/d/msgid/isar- > >> >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net. > > > >-- > >You received this message because you are subscribed to the Google > >Groups "isar-users" group. > >To unsubscribe from this group and stop receiving emails from it, > >send an email to isar-users+unsubscribe@googlegroups.com. > >To view this discussion on the web visit > >https://groups.google.com/d/msgid/isar- > >users/20230103200543.07e987ba%40md1za8fc.ad001.siemens.net.
>-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Henning Schild >Sent: 04 January 2023 15:00 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; jan.kiszka@siemens.com; >hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; >Baurzhan Ismagulov <ibr@radix50.net> >Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside >rootfs image > >Am Wed, 4 Jan 2023 07:54:44 +0000 >schrieb <Venkata.Pyla@toshiba-tsip.com>: > >> >-----Original Message----- >> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >> >Behalf Of Henning Schild >> >Sent: 04 January 2023 00:36 >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >> >tsip.com> >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; >> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >> >image.bbclass: fix non-reproducible file time-stamps inside rootfs >> >image >> > >> >Am Tue, 3 Jan 2023 14:10:14 +0000 >> >schrieb <Venkata.Pyla@toshiba-tsip.com>: >> > >> >> >-----Original Message----- >> >> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >> >> >Behalf Of Henning Schild >> >> >Sent: 02 January 2023 22:14 >> >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >> >> ><Venkata.Pyla@toshiba- tsip.com> >> >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; >> >> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >> >> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >> >> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >> >> >image.bbclass: fix non-reproducible file time-stamps inside rootfs >> >> >image >> >> > >> >> >Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb >> >> >venkata.pyla@toshiba-tsip.com: >> >> > >> >> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> >> >> >> >> As part of reproducible-build work, the rootfs images generated >> >> >> on same source should be identical between two builds. >> >> >> >> >> >> In this commit it tries to solve one of the non-reproducible >> >> >> problem i.e. the rootfs file time-stamps generated during build >> >> >> time are not reproducible, it uses one of the solution provided >> >> >> in the debian live-build image project (refer [1]), it fixes by >> >> >> finding all the files/folders that are gernerated newly and set >> >> >> the time-stamp provided by `SOURCE_DATE_EPOCH` environment >> >> >> variable. >> >> >> >> >> >> [1] >> >> >> https://salsa.debian.org/live-team/live-build/-/merge_requests/2 >> >> >> 18 >> >> >> >> >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> >> --- >> >> >> meta/classes/image.bbclass | 9 +++++++++ >> >> >> 1 file changed, 9 insertions(+) >> >> >> >> >> >> diff --git a/meta/classes/image.bbclass >> >> >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 >> >> >> --- a/meta/classes/image.bbclass >> >> >> +++ b/meta/classes/image.bbclass >> >> >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >> >> >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> >> >> >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> >> >> + >> >> >> + # Set same time-stamps to the newly generated >> >> >> file/folders in the >> >> >> + # rootfs image for the purpose of reproducible builds. >> >> >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> >> >> + find ${ROOTFSDIR} -newermt \ >> >> >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> >> >> %H:%M:%S')" \ >> >> >> + -printf "%y %p\n" \ >> >> >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >> >> >> + >> >> > >> >> >This looks like i have seen it before. For me that is _way_ too >> >> >generic and something that is not a package touches files all over >> >> >the place. If some package now wants to intentionally bring a file >> >> >that is from a far away future? >> >> > >> >> >Which files are we talking about here? It can basically only be >> >> >metadata and other little places where we violate our "everything >> >> >comes from a package" rule. >> >> >> >> files/folder/symbolic-link that are modified or generated during >> >> build time like /etc/os-release /etc/hostname . >> >> . >> >> . >> >> /var/lib/dpkg/info/* >> >> /var/cache/* >> >> --- >> >> >> >> I have printed all the files that modified during build by >> >> executing below command find ${ROOTFSDIR} -newermt "$(date >> >> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y >%p\n" >> >> >> >> modified_files_times.txt >> >> >> >> attached modified_files_times.txt for your reference, all these >> >> files/folders/symbolic-link are generated or modified during build >> >> time. >> > >> >So i am guessing it is about anything that comes out of postprocess >> >functions and maintainer scripts like postinst ? >> >> Yes, those are the files that are modified by the packages and have >> different timestamp on each build. I think it is okay to set >> reproducible times-tamp for such files, as they are not coming from >> the packages but are modified at build time. > >Sure they are OK. But if i read your patch correctly you will potentially adjust >times on ALL files, not just some for which it might be OK. > >And it uses a variable which does not have a default, should we not set that to >something? Or help users to choose a good value for it. Setting default value, I didn't think of it because reproducible builds may not be required for regular builds, however when someone wants it can be enable by setting the value to it. So as you mentioned it is good to mention somewhere to help users to choose good value for it, I am thinking to add it in doc/user_manual.md > >Without this variable and a clue on a value to pick, this is all dead code. > >> > >> >> > >> >> >Has this ever been tested against a complex layer, has any of the >> >> >repro work ever looked at something bigger than the very >> >> >artificial isar base image? >> >> Similar implementations were used in the projects like >> >> Debian/live-build[1] and poky[2] >> >> >> >> [1] >> >> https://salsa.debian.org/live-team/live-build/-/blob/master/scripts >> >> /bu >> >> ild/efi-image#L57 >> >> [2] >> >> https://github.com/yoctoproject/poky/blob/master/meta/classes-recip >> >> e/i >> >> mage.bbclass#L673 >> > >> >I am talking about isar layers not other projects. >> > >> >> > >> >> >I think a better start would be to bbwarn and only much later move >> >> >to "-exec touch". >> >> > >> >> >We recently added CI tests for reproducible building. Would be >> >> >nice to the two patches. p1 writes a test-case that goes red, p2 >> >> >(this one) makes it go green >> >> The patch (p1) already been sent [1], for writing test-case to >> >> verify reproducible build problems, the test-case is getting failed >> >> (shows non-reproducible time-stamp problem in >> >> build/diffoscope-output.txt) when executed without this patch, and >> >> after applying this patch it will still show fail (because there >> >> are other reproducible problems to fix) but the non-reproducible >> >> timestamp problem in rootfs shows fixed. >> > >> >So the code is there but not used because it goes failed anyhow? In >> >that case i would suggest to rewrite it in a way that it does not >> >fail and can run in CI. It could for example ignore "known issues" >> >and be enabled, where then this very series would remove a few "known >> >issues" in p2 (p1 would be to introduce all currently known issues >> >and enable that thing in CI) and p3 would be what we look at here >> >> You mean run the test and just ignore the results? > >Ignore some, the ones we know as violators that we still need to work on. But >do not ignore anything coming up as new problems, maybe/likely in layers. > >Again, has this ever been tested on anything but the very boring isar core? isar- >cip-core could be interesting as well, especially the image cip-core-image- >security > I have executed reproducible build tests in isar-cip-core image manually (not with the test-suite) and reported the problem as well here [1] [1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/31 >> > >> >Not sure i understood but it sounds like we have a test that we only >> >run manually, and i want it to run in CI. In fact a script to check >> >for repro issues should be so generic that any layer can use it. >> >> Eventually the repro-build test-case should go to CI, I will plan to >> do it. >> >> Can I know where the isar CI pipe lines are running? I couldn't see in >> isar-github may be running some other place? > >Unfortunately the two places that run it are not public and would require you >to receive an account. You could ask Baurzhan if you can get one, or you just >run it yourself manually while working on the patches. If it is enabled and will >ever fail, you will likely get reports from CI sent to you. Got it, then I will test them manually as usual. > >Henning > >> > >> >So your vision would be that isar core brings that script and i.e. >> >jailhouse-images can run it on all its images. And a few calls to the >> >script will also be in testsuite/ and be called in pipelines >> >regularly. >> > >> >Henning >> > >> >> [1] >> >> https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb5 >> >> 8b2 >> >> 4e953cb9 >> >> >> >> > >> >> >Henning >> >> > >> >> >> EOSUDO >> >> >> } >> >> >> addtask rootfs_finalize before do_rootfs after >> >> >> do_rootfs_postprocess >> >> > >> >> >-- >> >> >You received this message because you are subscribed to the Google >> >> >Groups "isar-users" group. >> >> >To unsubscribe from this group and stop receiving emails from it, >> >> >send an email to isar-users+unsubscribe@googlegroups.com. >> >> >To view this discussion on the web visit >> >> >https://groups.google.com/d/msgid/isar- >> >> >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net. >> > >> >-- >> >You received this message because you are subscribed to the Google >> >Groups "isar-users" group. >> >To unsubscribe from this group and stop receiving emails from it, >> >send an email to isar-users+unsubscribe@googlegroups.com. >> >To view this discussion on the web visit >> >https://groups.google.com/d/msgid/isar- >> >users/20230103200543.07e987ba%40md1za8fc.ad001.siemens.net. > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar- >users/20230104102934.042477a8%40md1za8fc.ad001.siemens.net.
Am Wed, 4 Jan 2023 13:48:10 +0000 schrieb <Venkata.Pyla@toshiba-tsip.com>: > >-----Original Message----- > >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >Behalf Of Henning Schild > >Sent: 04 January 2023 15:00 > >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > >tsip.com> > >Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG > >Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov > ><ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix > >non-reproducible file time-stamps inside rootfs image > > > >Am Wed, 4 Jan 2023 07:54:44 +0000 > >schrieb <Venkata.Pyla@toshiba-tsip.com>: > > > >> >-----Original Message----- > >> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >> >Behalf Of Henning Schild > >> >Sent: 04 January 2023 00:36 > >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >> ><Venkata.Pyla@toshiba- tsip.com> > >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > >> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG > >> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >> >image.bbclass: fix non-reproducible file time-stamps inside rootfs > >> >image > >> > > >> >Am Tue, 3 Jan 2023 14:10:14 +0000 > >> >schrieb <Venkata.Pyla@toshiba-tsip.com>: > >> > > >> >> >-----Original Message----- > >> >> >From: isar-users@googlegroups.com > >> >> ><isar-users@googlegroups.com> On Behalf Of Henning Schild > >> >> >Sent: 02 January 2023 22:14 > >> >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >> >> ><Venkata.Pyla@toshiba- tsip.com> > >> >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >> >> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 > >> >> >□ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > >> >> >kumar(ＴＳＩＰ TMIEC ODG Porting) > >> >> ><dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >> >> >image.bbclass: fix non-reproducible file time-stamps inside > >> >> >rootfs image > >> >> > > >> >> >Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb > >> >> >venkata.pyla@toshiba-tsip.com: > >> >> > > >> >> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> >> >> > >> >> >> As part of reproducible-build work, the rootfs images > >> >> >> generated on same source should be identical between two > >> >> >> builds. > >> >> >> > >> >> >> In this commit it tries to solve one of the non-reproducible > >> >> >> problem i.e. the rootfs file time-stamps generated during > >> >> >> build time are not reproducible, it uses one of the solution > >> >> >> provided in the debian live-build image project (refer [1]), > >> >> >> it fixes by finding all the files/folders that are > >> >> >> gernerated newly and set the time-stamp provided by > >> >> >> `SOURCE_DATE_EPOCH` environment variable. > >> >> >> > >> >> >> [1] > >> >> >> https://salsa.debian.org/live-team/live-build/-/merge_requests/2 > >> >> >> 18 > >> >> >> > >> >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> >> >> --- > >> >> >> meta/classes/image.bbclass | 9 +++++++++ > >> >> >> 1 file changed, 9 insertions(+) > >> >> >> > >> >> >> diff --git a/meta/classes/image.bbclass > >> >> >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 > >> >> >> --- a/meta/classes/image.bbclass > >> >> >> +++ b/meta/classes/image.bbclass > >> >> >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { > >> >> >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >> >> >> > >> >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >> >> >> + > >> >> >> + # Set same time-stamps to the newly generated > >> >> >> file/folders in the > >> >> >> + # rootfs image for the purpose of reproducible > >> >> >> builds. > >> >> >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >> >> >> + find ${ROOTFSDIR} -newermt \ > >> >> >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >> >> >> %H:%M:%S')" \ > >> >> >> + -printf "%y %p\n" \ > >> >> >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} > >> >> >> ';' > >> >> >> + > >> >> > > >> >> >This looks like i have seen it before. For me that is _way_ too > >> >> >generic and something that is not a package touches files all > >> >> >over the place. If some package now wants to intentionally > >> >> >bring a file that is from a far away future? > >> >> > > >> >> >Which files are we talking about here? It can basically only be > >> >> >metadata and other little places where we violate our > >> >> >"everything comes from a package" rule. > >> >> > >> >> files/folder/symbolic-link that are modified or generated during > >> >> build time like /etc/os-release /etc/hostname . > >> >> . > >> >> . > >> >> /var/lib/dpkg/info/* > >> >> /var/cache/* > >> >> --- > >> >> > >> >> I have printed all the files that modified during build by > >> >> executing below command find ${ROOTFSDIR} -newermt "$(date > >> >> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y > >%p\n" > >> >> > >> >> modified_files_times.txt > >> >> > >> >> attached modified_files_times.txt for your reference, all these > >> >> files/folders/symbolic-link are generated or modified during > >> >> build time. > >> > > >> >So i am guessing it is about anything that comes out of > >> >postprocess functions and maintainer scripts like postinst ? > >> > >> Yes, those are the files that are modified by the packages and have > >> different timestamp on each build. I think it is okay to set > >> reproducible times-tamp for such files, as they are not coming from > >> the packages but are modified at build time. > > > >Sure they are OK. But if i read your patch correctly you will > >potentially adjust times on ALL files, not just some for which it > >might be OK. > > > >And it uses a variable which does not have a default, should we not > >set that to something? Or help users to choose a good value for it. > > Setting default value, I didn't think of it because reproducible > builds may not be required for regular builds, however when someone > wants it can be enable by setting the value to it. So as you > mentioned it is good to mention somewhere to help users to choose > good value for it, I am thinking to add it in doc/user_manual.md Yes, user manual or maybe local.conf.sample but commented out. But some value that works and makes sense needs to be given to people. Otherwise i would set SOURCE_DATE_EPOCH="just a guess 42" which might not work Henning > > > >Without this variable and a clue on a value to pick, this is all > >dead code. > > > >> > > >> >> > > >> >> >Has this ever been tested against a complex layer, has any of > >> >> >the repro work ever looked at something bigger than the very > >> >> >artificial isar base image? > >> >> Similar implementations were used in the projects like > >> >> Debian/live-build[1] and poky[2] > >> >> > >> >> [1] > >> >> https://salsa.debian.org/live-team/live-build/-/blob/master/scripts > >> >> /bu > >> >> ild/efi-image#L57 > >> >> [2] > >> >> https://github.com/yoctoproject/poky/blob/master/meta/classes-recip > >> >> e/i > >> >> mage.bbclass#L673 > >> > > >> >I am talking about isar layers not other projects. > >> > > >> >> > > >> >> >I think a better start would be to bbwarn and only much later > >> >> >move to "-exec touch". > >> >> > > >> >> >We recently added CI tests for reproducible building. Would be > >> >> >nice to the two patches. p1 writes a test-case that goes red, > >> >> >p2 (this one) makes it go green > >> >> The patch (p1) already been sent [1], for writing test-case to > >> >> verify reproducible build problems, the test-case is getting > >> >> failed (shows non-reproducible time-stamp problem in > >> >> build/diffoscope-output.txt) when executed without this patch, > >> >> and after applying this patch it will still show fail (because > >> >> there are other reproducible problems to fix) but the > >> >> non-reproducible timestamp problem in rootfs shows fixed. > >> > > >> >So the code is there but not used because it goes failed anyhow? > >> >In that case i would suggest to rewrite it in a way that it does > >> >not fail and can run in CI. It could for example ignore "known > >> >issues" and be enabled, where then this very series would remove > >> >a few "known issues" in p2 (p1 would be to introduce all > >> >currently known issues and enable that thing in CI) and p3 would > >> >be what we look at here > >> > >> You mean run the test and just ignore the results? > > > >Ignore some, the ones we know as violators that we still need to > >work on. But do not ignore anything coming up as new problems, > >maybe/likely in layers. > > > >Again, has this ever been tested on anything but the very boring > >isar core? isar- cip-core could be interesting as well, especially > >the image cip-core-image- security > > > I have executed reproducible build tests in isar-cip-core image > manually (not with the test-suite) and reported the problem as well > here [1] [1] > https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/31 > > >> > > >> >Not sure i understood but it sounds like we have a test that we > >> >only run manually, and i want it to run in CI. In fact a script > >> >to check for repro issues should be so generic that any layer can > >> >use it. > >> > >> Eventually the repro-build test-case should go to CI, I will plan > >> to do it. > >> > >> Can I know where the isar CI pipe lines are running? I couldn't > >> see in isar-github may be running some other place? > > > >Unfortunately the two places that run it are not public and would > >require you to receive an account. You could ask Baurzhan if you can > >get one, or you just run it yourself manually while working on the > >patches. If it is enabled and will ever fail, you will likely get > >reports from CI sent to you. > > Got it, then I will test them manually as usual. > > > > >Henning > > > >> > > >> >So your vision would be that isar core brings that script and i.e. > >> >jailhouse-images can run it on all its images. And a few calls to > >> >the script will also be in testsuite/ and be called in pipelines > >> >regularly. > >> > > >> >Henning > >> > > >> >> [1] > >> >> https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb5 > >> >> 8b2 > >> >> 4e953cb9 > >> >> > >> >> > > >> >> >Henning > >> >> > > >> >> >> EOSUDO > >> >> >> } > >> >> >> addtask rootfs_finalize before do_rootfs after > >> >> >> do_rootfs_postprocess > >> >> > > >> >> >-- > >> >> >You received this message because you are subscribed to the > >> >> >Google Groups "isar-users" group. > >> >> >To unsubscribe from this group and stop receiving emails from > >> >> >it, send an email to isar-users+unsubscribe@googlegroups.com. > >> >> >To view this discussion on the web visit > >> >> >https://groups.google.com/d/msgid/isar- > >> >> >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net. > >> > > >> >-- > >> >You received this message because you are subscribed to the Google > >> >Groups "isar-users" group. > >> >To unsubscribe from this group and stop receiving emails from it, > >> >send an email to isar-users+unsubscribe@googlegroups.com. > >> >To view this discussion on the web visit > >> >https://groups.google.com/d/msgid/isar- > >> >users/20230103200543.07e987ba%40md1za8fc.ad001.siemens.net. > > > >-- > >You received this message because you are subscribed to the Google > >Groups "isar-users" group. > >To unsubscribe from this group and stop receiving emails from it, > >send an email to isar-users+unsubscribe@googlegroups.com. > >To view this discussion on the web visit > >https://groups.google.com/d/msgid/isar- > >users/20230104102934.042477a8%40md1za8fc.ad001.siemens.net. >
On 04.01.23 14:53, Henning Schild wrote: > Am Wed, 4 Jan 2023 13:48:10 +0000 > schrieb <Venkata.Pyla@toshiba-tsip.com>: > >>> -----Original Message----- >>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >>> Behalf Of Henning Schild >>> Sent: 04 January 2023 15:00 >>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >>> tsip.com> >>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov >>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix >>> non-reproducible file time-stamps inside rootfs image >>> >>> Am Wed, 4 Jan 2023 07:54:44 +0000 >>> schrieb <Venkata.Pyla@toshiba-tsip.com>: >>> >>>>> -----Original Message----- >>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >>>>> Behalf Of Henning Schild >>>>> Sent: 04 January 2023 00:36 >>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >>>>> <Venkata.Pyla@toshiba- tsip.com> >>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >>>>> Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >>>>> image.bbclass: fix non-reproducible file time-stamps inside rootfs >>>>> image >>>>> >>>>> Am Tue, 3 Jan 2023 14:10:14 +0000 >>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: >>>>> >>>>>>> -----Original Message----- >>>>>>> From: isar-users@googlegroups.com >>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild >>>>>>> Sent: 02 January 2023 22:14 >>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >>>>>>> <Venkata.Pyla@toshiba- tsip.com> >>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 >>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh >>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) >>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >>>>>>> image.bbclass: fix non-reproducible file time-stamps inside >>>>>>> rootfs image >>>>>>> >>>>>>> Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb >>>>>>> venkata.pyla@toshiba-tsip.com: >>>>>>> >>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >>>>>>>> >>>>>>>> As part of reproducible-build work, the rootfs images >>>>>>>> generated on same source should be identical between two >>>>>>>> builds. >>>>>>>> >>>>>>>> In this commit it tries to solve one of the non-reproducible >>>>>>>> problem i.e. the rootfs file time-stamps generated during >>>>>>>> build time are not reproducible, it uses one of the solution >>>>>>>> provided in the debian live-build image project (refer [1]), >>>>>>>> it fixes by finding all the files/folders that are >>>>>>>> gernerated newly and set the time-stamp provided by >>>>>>>> `SOURCE_DATE_EPOCH` environment variable. >>>>>>>> >>>>>>>> [1] >>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/2 >>>>>>>> 18 >>>>>>>> >>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >>>>>>>> --- >>>>>>>> meta/classes/image.bbclass | 9 +++++++++ >>>>>>>> 1 file changed, 9 insertions(+) >>>>>>>> >>>>>>>> diff --git a/meta/classes/image.bbclass >>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 >>>>>>>> --- a/meta/classes/image.bbclass >>>>>>>> +++ b/meta/classes/image.bbclass >>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >>>>>>>> >>>>>>>> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >>>>>>>> + >>>>>>>> + # Set same time-stamps to the newly generated >>>>>>>> file/folders in the >>>>>>>> + # rootfs image for the purpose of reproducible >>>>>>>> builds. >>>>>>>> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >>>>>>>> + find ${ROOTFSDIR} -newermt \ >>>>>>>> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >>>>>>>> %H:%M:%S')" \ >>>>>>>> + -printf "%y %p\n" \ >>>>>>>> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} >>>>>>>> ';' >>>>>>>> + >>>>>>> >>>>>>> This looks like i have seen it before. For me that is _way_ too >>>>>>> generic and something that is not a package touches files all >>>>>>> over the place. If some package now wants to intentionally >>>>>>> bring a file that is from a far away future? >>>>>>> >>>>>>> Which files are we talking about here? It can basically only be >>>>>>> metadata and other little places where we violate our >>>>>>> "everything comes from a package" rule. >>>>>> >>>>>> files/folder/symbolic-link that are modified or generated during >>>>>> build time like /etc/os-release /etc/hostname . >>>>>> . >>>>>> . >>>>>> /var/lib/dpkg/info/* >>>>>> /var/cache/* >>>>>> --- >>>>>> >>>>>> I have printed all the files that modified during build by >>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date >>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y >>> %p\n" >>>>>> >>>>>> modified_files_times.txt >>>>>> >>>>>> attached modified_files_times.txt for your reference, all these >>>>>> files/folders/symbolic-link are generated or modified during >>>>>> build time. >>>>> >>>>> So i am guessing it is about anything that comes out of >>>>> postprocess functions and maintainer scripts like postinst ? >>>> >>>> Yes, those are the files that are modified by the packages and have >>>> different timestamp on each build. I think it is okay to set >>>> reproducible times-tamp for such files, as they are not coming from >>>> the packages but are modified at build time. >>> >>> Sure they are OK. But if i read your patch correctly you will >>> potentially adjust times on ALL files, not just some for which it >>> might be OK. >>> >>> And it uses a variable which does not have a default, should we not >>> set that to something? Or help users to choose a good value for it. >> >> Setting default value, I didn't think of it because reproducible >> builds may not be required for regular builds, however when someone >> wants it can be enable by setting the value to it. So as you >> mentioned it is good to mention somewhere to help users to choose >> good value for it, I am thinking to add it in doc/user_manual.md > > Yes, user manual or maybe local.conf.sample but commented out. But some > value that works and makes sense needs to be given to people. Otherwise > i would set > > SOURCE_DATE_EPOCH="just a guess 42" > > which might not work If I understand this patch correctly, any file older than SOURCE_DATE_EPOCH will not be touched, any newer one will get that date. So proposing some concrete date here could eventually overshoot. What would be better is to document the format along the commented-out variable. Jan
>-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Jan Kiszka >Sent: 04 January 2023 20:06 >To: Henning Schild <henning.schild@siemens.com>; pyla venkata(ＴＳＩＰ >TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; hayashi kazuhiro(林 和宏 >□ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ >TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; ibr@radix50.net >Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside >rootfs image > >On 04.01.23 14:53, Henning Schild wrote: >> Am Wed, 4 Jan 2023 13:48:10 +0000 >> schrieb <Venkata.Pyla@toshiba-tsip.com>: >> >>>> -----Original Message----- >>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >>>> Behalf Of Henning Schild >>>> Sent: 04 January 2023 15:00 >>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >>>> tsip.com> >>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >>>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov >>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix >>>> non-reproducible file time-stamps inside rootfs image >>>> >>>> Am Wed, 4 Jan 2023 07:54:44 +0000 >>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: >>>> >>>>>> -----Original Message----- >>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >>>>>> Behalf Of Henning Schild >>>>>> Sent: 04 January 2023 00:36 >>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >>>>>> <Venkata.Pyla@toshiba- tsip.com> >>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >>>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >>>>>> Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >>>>>> image.bbclass: fix non-reproducible file time-stamps inside rootfs >>>>>> image >>>>>> >>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000 >>>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: >>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> >>>>>>>> On Behalf Of Henning Schild >>>>>>>> Sent: 02 January 2023 22:14 >>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >>>>>>>> <Venkata.Pyla@toshiba- tsip.com> >>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 >>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh >>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) >>>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside >>>>>>>> rootfs image >>>>>>>> >>>>>>>> Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb >>>>>>>> venkata.pyla@toshiba-tsip.com: >>>>>>>> >>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >>>>>>>>> >>>>>>>>> As part of reproducible-build work, the rootfs images generated >>>>>>>>> on same source should be identical between two builds. >>>>>>>>> >>>>>>>>> In this commit it tries to solve one of the non-reproducible >>>>>>>>> problem i.e. the rootfs file time-stamps generated during build >>>>>>>>> time are not reproducible, it uses one of the solution provided >>>>>>>>> in the debian live-build image project (refer [1]), it fixes by >>>>>>>>> finding all the files/folders that are gernerated newly and set >>>>>>>>> the time-stamp provided by `SOURCE_DATE_EPOCH` environment >>>>>>>>> variable. >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/ >>>>>>>>> 2 >>>>>>>>> 18 >>>>>>>>> >>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >>>>>>>>> --- >>>>>>>>> meta/classes/image.bbclass | 9 +++++++++ >>>>>>>>> 1 file changed, 9 insertions(+) >>>>>>>>> >>>>>>>>> diff --git a/meta/classes/image.bbclass >>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 >>>>>>>>> --- a/meta/classes/image.bbclass >>>>>>>>> +++ b/meta/classes/image.bbclass >>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >>>>>>>>> >>>>>>>>> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >>>>>>>>> + >>>>>>>>> + # Set same time-stamps to the newly generated >>>>>>>>> file/folders in the >>>>>>>>> + # rootfs image for the purpose of reproducible >>>>>>>>> builds. >>>>>>>>> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >>>>>>>>> + find ${ROOTFSDIR} -newermt \ >>>>>>>>> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >>>>>>>>> %H:%M:%S')" \ >>>>>>>>> + -printf "%y %p\n" \ >>>>>>>>> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} >>>>>>>>> ';' >>>>>>>>> + >>>>>>>> >>>>>>>> This looks like i have seen it before. For me that is _way_ too >>>>>>>> generic and something that is not a package touches files all >>>>>>>> over the place. If some package now wants to intentionally bring >>>>>>>> a file that is from a far away future? >>>>>>>> >>>>>>>> Which files are we talking about here? It can basically only be >>>>>>>> metadata and other little places where we violate our >>>>>>>> "everything comes from a package" rule. >>>>>>> >>>>>>> files/folder/symbolic-link that are modified or generated during >>>>>>> build time like /etc/os-release /etc/hostname . >>>>>>> . >>>>>>> . >>>>>>> /var/lib/dpkg/info/* >>>>>>> /var/cache/* >>>>>>> --- >>>>>>> >>>>>>> I have printed all the files that modified during build by >>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date >>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y >>>> %p\n" >>>>>>> >>>>>>> modified_files_times.txt >>>>>>> >>>>>>> attached modified_files_times.txt for your reference, all these >>>>>>> files/folders/symbolic-link are generated or modified during >>>>>>> build time. >>>>>> >>>>>> So i am guessing it is about anything that comes out of >>>>>> postprocess functions and maintainer scripts like postinst ? >>>>> >>>>> Yes, those are the files that are modified by the packages and have >>>>> different timestamp on each build. I think it is okay to set >>>>> reproducible times-tamp for such files, as they are not coming from >>>>> the packages but are modified at build time. >>>> >>>> Sure they are OK. But if i read your patch correctly you will >>>> potentially adjust times on ALL files, not just some for which it >>>> might be OK. >>>> >>>> And it uses a variable which does not have a default, should we not >>>> set that to something? Or help users to choose a good value for it. >>> >>> Setting default value, I didn't think of it because reproducible >>> builds may not be required for regular builds, however when someone >>> wants it can be enable by setting the value to it. So as you >>> mentioned it is good to mention somewhere to help users to choose >>> good value for it, I am thinking to add it in doc/user_manual.md >> >> Yes, user manual or maybe local.conf.sample but commented out. But >> some value that works and makes sense needs to be given to people. >> Otherwise i would set >> >> SOURCE_DATE_EPOCH="just a guess 42" >> >> which might not work > >If I understand this patch correctly, any file older than SOURCE_DATE_EPOCH >will not be touched, any newer one will get that date. >So proposing some concrete date here could eventually overshoot. What would >be better is to document the format along the commented-out variable. The good value for it is the date of the last change to the source (git log -1 --pretty=%ct) > >Jan > >-- >Siemens AG, Technology >Competence Center Embedded Linux > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar-users/832199ad-53c9-dcd2-a7e7- >30c61fe92c92%40siemens.com.
Am Wed, 4 Jan 2023 15:35:56 +0100 schrieb Jan Kiszka <jan.kiszka@siemens.com>: > On 04.01.23 14:53, Henning Schild wrote: > > Am Wed, 4 Jan 2023 13:48:10 +0000 > > schrieb <Venkata.Pyla@toshiba-tsip.com>: > > > >>> -----Original Message----- > >>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >>> Behalf Of Henning Schild > >>> Sent: 04 January 2023 15:00 > >>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >>> <Venkata.Pyla@toshiba- tsip.com> > >>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > >>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG > >>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov > >>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix > >>> non-reproducible file time-stamps inside rootfs image > >>> > >>> Am Wed, 4 Jan 2023 07:54:44 +0000 > >>> schrieb <Venkata.Pyla@toshiba-tsip.com>: > >>> > >>>>> -----Original Message----- > >>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> > >>>>> On Behalf Of Henning Schild > >>>>> Sent: 04 January 2023 00:36 > >>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >>>>> <Venkata.Pyla@toshiba- tsip.com> > >>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > >>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC > >>>>> ODG Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: > >>>>> [PATCH] image.bbclass: fix non-reproducible file time-stamps > >>>>> inside rootfs image > >>>>> > >>>>> Am Tue, 3 Jan 2023 14:10:14 +0000 > >>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: isar-users@googlegroups.com > >>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild > >>>>>>> Sent: 02 January 2023 22:14 > >>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >>>>>>> <Venkata.Pyla@toshiba- tsip.com> > >>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 > >>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > >>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) > >>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >>>>>>> image.bbclass: fix non-reproducible file time-stamps inside > >>>>>>> rootfs image > >>>>>>> > >>>>>>> Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb > >>>>>>> venkata.pyla@toshiba-tsip.com: > >>>>>>> > >>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >>>>>>>> > >>>>>>>> As part of reproducible-build work, the rootfs images > >>>>>>>> generated on same source should be identical between two > >>>>>>>> builds. > >>>>>>>> > >>>>>>>> In this commit it tries to solve one of the non-reproducible > >>>>>>>> problem i.e. the rootfs file time-stamps generated during > >>>>>>>> build time are not reproducible, it uses one of the solution > >>>>>>>> provided in the debian live-build image project (refer [1]), > >>>>>>>> it fixes by finding all the files/folders that are > >>>>>>>> gernerated newly and set the time-stamp provided by > >>>>>>>> `SOURCE_DATE_EPOCH` environment variable. > >>>>>>>> > >>>>>>>> [1] > >>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/2 > >>>>>>>> 18 > >>>>>>>> > >>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >>>>>>>> --- > >>>>>>>> meta/classes/image.bbclass | 9 +++++++++ > >>>>>>>> 1 file changed, 9 insertions(+) > >>>>>>>> > >>>>>>>> diff --git a/meta/classes/image.bbclass > >>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 > >>>>>>>> --- a/meta/classes/image.bbclass > >>>>>>>> +++ b/meta/classes/image.bbclass > >>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() { > >>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >>>>>>>> > >>>>>>>> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >>>>>>>> + > >>>>>>>> + # Set same time-stamps to the newly generated > >>>>>>>> file/folders in the > >>>>>>>> + # rootfs image for the purpose of reproducible > >>>>>>>> builds. > >>>>>>>> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >>>>>>>> + find ${ROOTFSDIR} -newermt \ > >>>>>>>> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >>>>>>>> %H:%M:%S')" \ > >>>>>>>> + -printf "%y %p\n" \ > >>>>>>>> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} > >>>>>>>> ';' > >>>>>>>> + > >>>>>>> > >>>>>>> This looks like i have seen it before. For me that is _way_ > >>>>>>> too generic and something that is not a package touches files > >>>>>>> all over the place. If some package now wants to intentionally > >>>>>>> bring a file that is from a far away future? > >>>>>>> > >>>>>>> Which files are we talking about here? It can basically only > >>>>>>> be metadata and other little places where we violate our > >>>>>>> "everything comes from a package" rule. > >>>>>> > >>>>>> files/folder/symbolic-link that are modified or generated > >>>>>> during build time like /etc/os-release /etc/hostname . > >>>>>> . > >>>>>> . > >>>>>> /var/lib/dpkg/info/* > >>>>>> /var/cache/* > >>>>>> --- > >>>>>> > >>>>>> I have printed all the files that modified during build by > >>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date > >>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y > >>>>>> > >>> %p\n" > >>>>>> > >>>>>> modified_files_times.txt > >>>>>> > >>>>>> attached modified_files_times.txt for your reference, all these > >>>>>> files/folders/symbolic-link are generated or modified during > >>>>>> build time. > >>>>> > >>>>> So i am guessing it is about anything that comes out of > >>>>> postprocess functions and maintainer scripts like postinst ? > >>>> > >>>> Yes, those are the files that are modified by the packages and > >>>> have different timestamp on each build. I think it is okay to set > >>>> reproducible times-tamp for such files, as they are not coming > >>>> from the packages but are modified at build time. > >>> > >>> Sure they are OK. But if i read your patch correctly you will > >>> potentially adjust times on ALL files, not just some for which it > >>> might be OK. > >>> > >>> And it uses a variable which does not have a default, should we > >>> not set that to something? Or help users to choose a good value > >>> for it. > >> > >> Setting default value, I didn't think of it because reproducible > >> builds may not be required for regular builds, however when someone > >> wants it can be enable by setting the value to it. So as you > >> mentioned it is good to mention somewhere to help users to choose > >> good value for it, I am thinking to add it in doc/user_manual.md > > > > Yes, user manual or maybe local.conf.sample but commented out. But > > some value that works and makes sense needs to be given to people. > > Otherwise i would set > > > > SOURCE_DATE_EPOCH="just a guess 42" > > > > which might not work > > If I understand this patch correctly, any file older than > SOURCE_DATE_EPOCH will not be touched, any newer one will get that > date. So proposing some concrete date here could eventually > overshoot. What would be better is to document the format along the > commented-out variable. That wild touching is exactly why i would like to have all that more verbose. Log out which files would be touched, or write it to a file. If anything goes wrong here it will mess up all times in the rootfs, basically unnoticed. And since that value is hard to get right, and can not be static ... things will go wrong. Touching files that according to dpkg are owned by a package indicates a potential problem, maybe except they live in /etc or /var Files not owned by any package should be reviewed if they should not better be removed instead of touched. But maybe it does not hurt to touch too much, in that case i propose we skip the find and set all files to 1.1.1970 ... just kidding ;) Henning > Jan >
Am Wed, 4 Jan 2023 14:50:44 +0000 schrieb <Venkata.Pyla@toshiba-tsip.com>: > >-----Original Message----- > >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >Behalf Of Jan Kiszka > >Sent: 04 January 2023 20:06 > >To: Henning Schild <henning.schild@siemens.com>; pyla > >venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com> > >Cc: isar-users@googlegroups.com; amikan@ilbers.de; hayashi > >kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; > >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) > ><dinesh.kumar@toshiba-tsip.com>; ibr@radix50.net Subject: Re: > >[PATCH] image.bbclass: fix non-reproducible file time-stamps inside > >rootfs image > > > >On 04.01.23 14:53, Henning Schild wrote: > >> Am Wed, 4 Jan 2023 13:48:10 +0000 > >> schrieb <Venkata.Pyla@toshiba-tsip.com>: > >> > >>>> -----Original Message----- > >>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> > >>>> On Behalf Of Henning Schild > >>>> Sent: 04 January 2023 15:00 > >>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >>>> <Venkata.Pyla@toshiba- tsip.com> > >>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > >>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC > >>>> ODG Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov > >>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix > >>>> non-reproducible file time-stamps inside rootfs image > >>>> > >>>> Am Wed, 4 Jan 2023 07:54:44 +0000 > >>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: > >>>> > >>>>>> -----Original Message----- > >>>>>> From: isar-users@googlegroups.com > >>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild > >>>>>> Sent: 04 January 2023 00:36 > >>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >>>>>> <Venkata.Pyla@toshiba- tsip.com> > >>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 > >>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > >>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) > >>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >>>>>> image.bbclass: fix non-reproducible file time-stamps inside > >>>>>> rootfs image > >>>>>> > >>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000 > >>>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: > >>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: isar-users@googlegroups.com > >>>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild > >>>>>>>> Sent: 02 January 2023 22:14 > >>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > >>>>>>>> <Venkata.Pyla@toshiba- tsip.com> > >>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 > >>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > >>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) > >>>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] > >>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside > >>>>>>>> rootfs image > >>>>>>>> > >>>>>>>> Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb > >>>>>>>> venkata.pyla@toshiba-tsip.com: > >>>>>>>> > >>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >>>>>>>>> > >>>>>>>>> As part of reproducible-build work, the rootfs images > >>>>>>>>> generated on same source should be identical between two > >>>>>>>>> builds. > >>>>>>>>> > >>>>>>>>> In this commit it tries to solve one of the non-reproducible > >>>>>>>>> problem i.e. the rootfs file time-stamps generated during > >>>>>>>>> build time are not reproducible, it uses one of the > >>>>>>>>> solution provided in the debian live-build image project > >>>>>>>>> (refer [1]), it fixes by finding all the files/folders that > >>>>>>>>> are gernerated newly and set the time-stamp provided by > >>>>>>>>> `SOURCE_DATE_EPOCH` environment variable. > >>>>>>>>> > >>>>>>>>> [1] > >>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/ > >>>>>>>>> 2 > >>>>>>>>> 18 > >>>>>>>>> > >>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >>>>>>>>> --- > >>>>>>>>> meta/classes/image.bbclass | 9 +++++++++ > >>>>>>>>> 1 file changed, 9 insertions(+) > >>>>>>>>> > >>>>>>>>> diff --git a/meta/classes/image.bbclass > >>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 > >>>>>>>>> --- a/meta/classes/image.bbclass > >>>>>>>>> +++ b/meta/classes/image.bbclass > >>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() { > >>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >>>>>>>>> > >>>>>>>>> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >>>>>>>>> + > >>>>>>>>> + # Set same time-stamps to the newly generated > >>>>>>>>> file/folders in the > >>>>>>>>> + # rootfs image for the purpose of reproducible > >>>>>>>>> builds. > >>>>>>>>> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >>>>>>>>> + find ${ROOTFSDIR} -newermt \ > >>>>>>>>> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >>>>>>>>> %H:%M:%S')" \ > >>>>>>>>> + -printf "%y %p\n" \ > >>>>>>>>> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} > >>>>>>>>> ';' > >>>>>>>>> + > >>>>>>>> > >>>>>>>> This looks like i have seen it before. For me that is _way_ > >>>>>>>> too generic and something that is not a package touches > >>>>>>>> files all over the place. If some package now wants to > >>>>>>>> intentionally bring a file that is from a far away future? > >>>>>>>> > >>>>>>>> Which files are we talking about here? It can basically only > >>>>>>>> be metadata and other little places where we violate our > >>>>>>>> "everything comes from a package" rule. > >>>>>>> > >>>>>>> files/folder/symbolic-link that are modified or generated > >>>>>>> during build time like /etc/os-release /etc/hostname . > >>>>>>> . > >>>>>>> . > >>>>>>> /var/lib/dpkg/info/* > >>>>>>> /var/cache/* > >>>>>>> --- > >>>>>>> > >>>>>>> I have printed all the files that modified during build by > >>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date > >>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y > >>>>>>> > >>>> %p\n" > >>>>>>> > >>>>>>> modified_files_times.txt > >>>>>>> > >>>>>>> attached modified_files_times.txt for your reference, all > >>>>>>> these files/folders/symbolic-link are generated or modified > >>>>>>> during build time. > >>>>>> > >>>>>> So i am guessing it is about anything that comes out of > >>>>>> postprocess functions and maintainer scripts like postinst ? > >>>>> > >>>>> Yes, those are the files that are modified by the packages and > >>>>> have different timestamp on each build. I think it is okay to > >>>>> set reproducible times-tamp for such files, as they are not > >>>>> coming from the packages but are modified at build time. > >>>> > >>>> Sure they are OK. But if i read your patch correctly you will > >>>> potentially adjust times on ALL files, not just some for which it > >>>> might be OK. > >>>> > >>>> And it uses a variable which does not have a default, should we > >>>> not set that to something? Or help users to choose a good value > >>>> for it. > >>> > >>> Setting default value, I didn't think of it because reproducible > >>> builds may not be required for regular builds, however when > >>> someone wants it can be enable by setting the value to it. So as > >>> you mentioned it is good to mention somewhere to help users to > >>> choose good value for it, I am thinking to add it in > >>> doc/user_manual.md > >> > >> Yes, user manual or maybe local.conf.sample but commented out. But > >> some value that works and makes sense needs to be given to people. > >> Otherwise i would set > >> > >> SOURCE_DATE_EPOCH="just a guess 42" > >> > >> which might not work > > > >If I understand this patch correctly, any file older than > >SOURCE_DATE_EPOCH will not be touched, any newer one will get that > >date. So proposing some concrete date here could eventually > >overshoot. What would be better is to document the format along the > >commented-out variable. > > The good value for it is the date of the last change to the source > (git log -1 --pretty=%ct) And i assume if one wanted to release tarballs one would have to include a value in such a release, derived from version control. I am not sure anyone really puts isar layers into tarballs but wanted to remind that git is not all, the documentation can and should mention git first though. Maybe with a line that actually calls that very git command once comment marks have been removed. Henning > > > >Jan > > > >-- > >Siemens AG, Technology > >Competence Center Embedded Linux > > > >-- > >You received this message because you are subscribed to the Google > >Groups "isar-users" group. > >To unsubscribe from this group and stop receiving emails from it, > >send an email to isar-users+unsubscribe@googlegroups.com. > >To view this discussion on the web visit > >https://groups.google.com/d/msgid/isar-users/832199ad-53c9-dcd2-a7e7- > >30c61fe92c92%40siemens.com. >
On 04.01.23 16:01, Henning Schild wrote: > Am Wed, 4 Jan 2023 15:35:56 +0100 > schrieb Jan Kiszka <jan.kiszka@siemens.com>: > >> On 04.01.23 14:53, Henning Schild wrote: >>> Am Wed, 4 Jan 2023 13:48:10 +0000 >>> schrieb <Venkata.Pyla@toshiba-tsip.com>: >>> >>>>> -----Original Message----- >>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >>>>> Behalf Of Henning Schild >>>>> Sent: 04 January 2023 15:00 >>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >>>>> <Venkata.Pyla@toshiba- tsip.com> >>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >>>>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov >>>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix >>>>> non-reproducible file time-stamps inside rootfs image >>>>> >>>>> Am Wed, 4 Jan 2023 07:54:44 +0000 >>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: >>>>> >>>>>>> -----Original Message----- >>>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> >>>>>>> On Behalf Of Henning Schild >>>>>>> Sent: 04 January 2023 00:36 >>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >>>>>>> <Venkata.Pyla@toshiba- tsip.com> >>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >>>>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC >>>>>>> ODG Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: >>>>>>> [PATCH] image.bbclass: fix non-reproducible file time-stamps >>>>>>> inside rootfs image >>>>>>> >>>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000 >>>>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>: >>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: isar-users@googlegroups.com >>>>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild >>>>>>>>> Sent: 02 January 2023 22:14 >>>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >>>>>>>>> <Venkata.Pyla@toshiba- tsip.com> >>>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >>>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 >>>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh >>>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) >>>>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >>>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside >>>>>>>>> rootfs image >>>>>>>>> >>>>>>>>> Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb >>>>>>>>> venkata.pyla@toshiba-tsip.com: >>>>>>>>> >>>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >>>>>>>>>> >>>>>>>>>> As part of reproducible-build work, the rootfs images >>>>>>>>>> generated on same source should be identical between two >>>>>>>>>> builds. >>>>>>>>>> >>>>>>>>>> In this commit it tries to solve one of the non-reproducible >>>>>>>>>> problem i.e. the rootfs file time-stamps generated during >>>>>>>>>> build time are not reproducible, it uses one of the solution >>>>>>>>>> provided in the debian live-build image project (refer [1]), >>>>>>>>>> it fixes by finding all the files/folders that are >>>>>>>>>> gernerated newly and set the time-stamp provided by >>>>>>>>>> `SOURCE_DATE_EPOCH` environment variable. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/2 >>>>>>>>>> 18 >>>>>>>>>> >>>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >>>>>>>>>> --- >>>>>>>>>> meta/classes/image.bbclass | 9 +++++++++ >>>>>>>>>> 1 file changed, 9 insertions(+) >>>>>>>>>> >>>>>>>>>> diff --git a/meta/classes/image.bbclass >>>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 >>>>>>>>>> --- a/meta/classes/image.bbclass >>>>>>>>>> +++ b/meta/classes/image.bbclass >>>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >>>>>>>>>> >>>>>>>>>> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >>>>>>>>>> + >>>>>>>>>> + # Set same time-stamps to the newly generated >>>>>>>>>> file/folders in the >>>>>>>>>> + # rootfs image for the purpose of reproducible >>>>>>>>>> builds. >>>>>>>>>> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >>>>>>>>>> + find ${ROOTFSDIR} -newermt \ >>>>>>>>>> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >>>>>>>>>> %H:%M:%S')" \ >>>>>>>>>> + -printf "%y %p\n" \ >>>>>>>>>> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} >>>>>>>>>> ';' >>>>>>>>>> + >>>>>>>>> >>>>>>>>> This looks like i have seen it before. For me that is _way_ >>>>>>>>> too generic and something that is not a package touches files >>>>>>>>> all over the place. If some package now wants to intentionally >>>>>>>>> bring a file that is from a far away future? >>>>>>>>> >>>>>>>>> Which files are we talking about here? It can basically only >>>>>>>>> be metadata and other little places where we violate our >>>>>>>>> "everything comes from a package" rule. >>>>>>>> >>>>>>>> files/folder/symbolic-link that are modified or generated >>>>>>>> during build time like /etc/os-release /etc/hostname . >>>>>>>> . >>>>>>>> . >>>>>>>> /var/lib/dpkg/info/* >>>>>>>> /var/cache/* >>>>>>>> --- >>>>>>>> >>>>>>>> I have printed all the files that modified during build by >>>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date >>>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y >>>>>>>> >>>>> %p\n" >>>>>>>> >>>>>>>> modified_files_times.txt >>>>>>>> >>>>>>>> attached modified_files_times.txt for your reference, all these >>>>>>>> files/folders/symbolic-link are generated or modified during >>>>>>>> build time. >>>>>>> >>>>>>> So i am guessing it is about anything that comes out of >>>>>>> postprocess functions and maintainer scripts like postinst ? >>>>>> >>>>>> Yes, those are the files that are modified by the packages and >>>>>> have different timestamp on each build. I think it is okay to set >>>>>> reproducible times-tamp for such files, as they are not coming >>>>>> from the packages but are modified at build time. >>>>> >>>>> Sure they are OK. But if i read your patch correctly you will >>>>> potentially adjust times on ALL files, not just some for which it >>>>> might be OK. >>>>> >>>>> And it uses a variable which does not have a default, should we >>>>> not set that to something? Or help users to choose a good value >>>>> for it. >>>> >>>> Setting default value, I didn't think of it because reproducible >>>> builds may not be required for regular builds, however when someone >>>> wants it can be enable by setting the value to it. So as you >>>> mentioned it is good to mention somewhere to help users to choose >>>> good value for it, I am thinking to add it in doc/user_manual.md >>> >>> Yes, user manual or maybe local.conf.sample but commented out. But >>> some value that works and makes sense needs to be given to people. >>> Otherwise i would set >>> >>> SOURCE_DATE_EPOCH="just a guess 42" >>> >>> which might not work >> >> If I understand this patch correctly, any file older than >> SOURCE_DATE_EPOCH will not be touched, any newer one will get that >> date. So proposing some concrete date here could eventually >> overshoot. What would be better is to document the format along the >> commented-out variable. > > That wild touching is exactly why i would like to have all that more > verbose. Log out which files would be touched, or write it to a file. > If anything goes wrong here it will mess up all times in the rootfs, > basically unnoticed. > > And since that value is hard to get right, and can not be static ... > things will go wrong. > > Touching files that according to dpkg are owned by a package indicates > a potential problem, maybe except they live in /etc or /var > Files not owned by any package should be reviewed if they should not > better be removed instead of touched. It would be nice to also catch packaging issues this way, but that requires careful thoughts to avoid making this step needlessly expensive. The value of having a reproducible image is clearly higher than that. Jan
>-----Original Message----- >From: Henning Schild <henning.schild@siemens.com> >Sent: 04 January 2023 20:37 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: jan.kiszka@siemens.com; isar-users@googlegroups.com; amikan@ilbers.de; >hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; >ibr@radix50.net >Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside >rootfs image > >Am Wed, 4 Jan 2023 14:50:44 +0000 >schrieb <Venkata.Pyla@toshiba-tsip.com>: > >> >-----Original Message----- >> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On >> >Behalf Of Jan Kiszka >> >Sent: 04 January 2023 20:06 >> >To: Henning Schild <henning.schild@siemens.com>; pyla >> >venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com> >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; hayashi >> >kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >> >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) >> ><dinesh.kumar@toshiba-tsip.com>; ibr@radix50.net Subject: Re: >> >[PATCH] image.bbclass: fix non-reproducible file time-stamps inside >> >rootfs image >> > >> >On 04.01.23 14:53, Henning Schild wrote: >> >> Am Wed, 4 Jan 2023 13:48:10 +0000 >> >> schrieb <Venkata.Pyla@toshiba-tsip.com>: >> >> >> >>>> -----Original Message----- >> >>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> >> >>>> On Behalf Of Henning Schild >> >>>> Sent: 04 January 2023 15:00 >> >>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >> >>>> <Venkata.Pyla@toshiba- tsip.com> >> >>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >> >>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) >> >>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG >> >>>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov >> >>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix >> >>>> non-reproducible file time-stamps inside rootfs image >> >>>> >> >>>> Am Wed, 4 Jan 2023 07:54:44 +0000 schrieb >> >>>> <Venkata.Pyla@toshiba-tsip.com>: >> >>>> >> >>>>>> -----Original Message----- >> >>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> >> >>>>>> On Behalf Of Henning Schild >> >>>>>> Sent: 04 January 2023 00:36 >> >>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >> >>>>>> <Venkata.Pyla@toshiba- tsip.com> >> >>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >> >>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 >> >>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh >> >>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) >> >>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH] >> >>>>>> image.bbclass: fix non-reproducible file time-stamps inside >> >>>>>> rootfs image >> >>>>>> >> >>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000 schrieb >> >>>>>> <Venkata.Pyla@toshiba-tsip.com>: >> >>>>>> >> >>>>>>>> -----Original Message----- >> >>>>>>>> From: isar-users@googlegroups.com >> >>>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild >> >>>>>>>> Sent: 02 January 2023 22:14 >> >>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) >> >>>>>>>> <Venkata.Pyla@toshiba- tsip.com> >> >>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de; >> >>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 >> >>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh >> >>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba- >tsip.com> >> >>>>>>>> Subject: Re: [PATCH] >> >>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside >> >>>>>>>> rootfs image >> >>>>>>>> >> >>>>>>>> Am Mon, 2 Jan 2023 20:28:28 +0530 schrieb >> >>>>>>>> venkata.pyla@toshiba-tsip.com: >> >>>>>>>> >> >>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >>>>>>>>> >> >>>>>>>>> As part of reproducible-build work, the rootfs images >> >>>>>>>>> generated on same source should be identical between two >> >>>>>>>>> builds. >> >>>>>>>>> >> >>>>>>>>> In this commit it tries to solve one of the non-reproducible >> >>>>>>>>> problem i.e. the rootfs file time-stamps generated during >> >>>>>>>>> build time are not reproducible, it uses one of the solution >> >>>>>>>>> provided in the debian live-build image project (refer [1]), >> >>>>>>>>> it fixes by finding all the files/folders that are >> >>>>>>>>> gernerated newly and set the time-stamp provided by >> >>>>>>>>> `SOURCE_DATE_EPOCH` environment variable. >> >>>>>>>>> >> >>>>>>>>> [1] >> >>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_reques >> >>>>>>>>> ts/ >> >>>>>>>>> 2 >> >>>>>>>>> 18 >> >>>>>>>>> >> >>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >>>>>>>>> --- >> >>>>>>>>> meta/classes/image.bbclass | 9 +++++++++ >> >>>>>>>>> 1 file changed, 9 insertions(+) >> >>>>>>>>> >> >>>>>>>>> diff --git a/meta/classes/image.bbclass >> >>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 >> >>>>>>>>> --- a/meta/classes/image.bbclass >> >>>>>>>>> +++ b/meta/classes/image.bbclass >> >>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >> >>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >>>>>>>>> >> >>>>>>>>> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> >>>>>>>>> + >> >>>>>>>>> + # Set same time-stamps to the newly generated >> >>>>>>>>> file/folders in the >> >>>>>>>>> + # rootfs image for the purpose of reproducible >> >>>>>>>>> builds. >> >>>>>>>>> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> >>>>>>>>> + find ${ROOTFSDIR} -newermt \ >> >>>>>>>>> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> >>>>>>>>> %H:%M:%S')" \ >> >>>>>>>>> + -printf "%y %p\n" \ >> >>>>>>>>> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} >> >>>>>>>>> ';' >> >>>>>>>>> + >> >>>>>>>> >> >>>>>>>> This looks like i have seen it before. For me that is _way_ >> >>>>>>>> too generic and something that is not a package touches files >> >>>>>>>> all over the place. If some package now wants to >> >>>>>>>> intentionally bring a file that is from a far away future? >> >>>>>>>> >> >>>>>>>> Which files are we talking about here? It can basically only >> >>>>>>>> be metadata and other little places where we violate our >> >>>>>>>> "everything comes from a package" rule. >> >>>>>>> >> >>>>>>> files/folder/symbolic-link that are modified or generated >> >>>>>>> during build time like /etc/os-release /etc/hostname . >> >>>>>>> . >> >>>>>>> . >> >>>>>>> /var/lib/dpkg/info/* >> >>>>>>> /var/cache/* >> >>>>>>> --- >> >>>>>>> >> >>>>>>> I have printed all the files that modified during build by >> >>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date >> >>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t >%y >> >>>>>>> >> >>>> %p\n" >> >>>>>>> >> >>>>>>> modified_files_times.txt >> >>>>>>> >> >>>>>>> attached modified_files_times.txt for your reference, all >> >>>>>>> these files/folders/symbolic-link are generated or modified >> >>>>>>> during build time. >> >>>>>> >> >>>>>> So i am guessing it is about anything that comes out of >> >>>>>> postprocess functions and maintainer scripts like postinst ? >> >>>>> >> >>>>> Yes, those are the files that are modified by the packages and >> >>>>> have different timestamp on each build. I think it is okay to >> >>>>> set reproducible times-tamp for such files, as they are not >> >>>>> coming from the packages but are modified at build time. >> >>>> >> >>>> Sure they are OK. But if i read your patch correctly you will >> >>>> potentially adjust times on ALL files, not just some for which it >> >>>> might be OK. >> >>>> >> >>>> And it uses a variable which does not have a default, should we >> >>>> not set that to something? Or help users to choose a good value >> >>>> for it. >> >>> >> >>> Setting default value, I didn't think of it because reproducible >> >>> builds may not be required for regular builds, however when >> >>> someone wants it can be enable by setting the value to it. So as >> >>> you mentioned it is good to mention somewhere to help users to >> >>> choose good value for it, I am thinking to add it in >> >>> doc/user_manual.md >> >> >> >> Yes, user manual or maybe local.conf.sample but commented out. But >> >> some value that works and makes sense needs to be given to people. >> >> Otherwise i would set >> >> >> >> SOURCE_DATE_EPOCH="just a guess 42" >> >> >> >> which might not work >> > >> >If I understand this patch correctly, any file older than >> >SOURCE_DATE_EPOCH will not be touched, any newer one will get that >> >date. So proposing some concrete date here could eventually >> >overshoot. What would be better is to document the format along the >> >commented-out variable. >> >> The good value for it is the date of the last change to the source >> (git log -1 --pretty=%ct) > >And i assume if one wanted to release tarballs one would have to include a >value in such a release, derived from version control. > >I am not sure anyone really puts isar layers into tarballs but wanted to remind >that git is not all, the documentation can and should mention git first though. >Maybe with a line that actually calls that very git command once comment >marks have been removed. In that case in the comment, we can suggest to the users for the repository users Can use git command or the # # Uncomment for reproducible builds # Git repository users can use value from 'git log -1 --pretty=%ct' # Not repository users can use value from 'stat -c%Y ChangeLog' #SOURCE_DATE_EPOCH = > >Henning > >> > >> >Jan >> > >> >-- >> >Siemens AG, Technology >> >Competence Center Embedded Linux >> > >> >-- >> >You received this message because you are subscribed to the Google >> >Groups "isar-users" group. >> >To unsubscribe from this group and stop receiving emails from it, >> >send an email to isar-users+unsubscribe@googlegroups.com. >> >To view this discussion on the web visit >> >https://groups.google.com/d/msgid/isar-users/832199ad-53c9-dcd2-a7e7- >> >30c61fe92c92%40siemens.com. >>
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 813e1f3..f592a12 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -430,6 +430,15 @@ do_rootfs_finalize() { "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" rm -f "${ROOTFSDIR}/etc/apt/sources-list" + + # Set same time-stamps to the newly generated file/folders in the + # rootfs image for the purpose of reproducible builds. + test ! -z "${SOURCE_DATE_EPOCH}" && \ + find ${ROOTFSDIR} -newermt \ + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ + -printf "%y %p\n" \ + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' + EOSUDO } addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess