| Message ID | 20230105061857.14993-2-venkata.pyla@toshiba-tsip.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | [v2,1/1] image.bbclass: fix non-reproducible file time-stamps inside rootfs | expand |
Am Thu, 5 Jan 2023 11:48:57 +0530 schrieb venkata.pyla@toshiba-tsip.com: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > As part of reproducible-build work, the rootfs images generated on > same source should be identical between two builds. > > In this commit it tries to solve one of the non-reproducible problem > i.e. the rootfs file time-stamps generated during build time are not > reproducible, it uses one of the solution provided in the debian > live-build image project (refer [1]), it fixes by finding all the > files/folders that are gernerated newly and set the time-stamp > provided by `SOURCE_DATE_EPOCH` environment variable. > > [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > meta-isar/conf/local.conf.sample | 10 ++++++++++ > meta/classes/image.bbclass | 9 +++++++++ > 2 files changed, 19 insertions(+) > > diff --git a/meta-isar/conf/local.conf.sample > b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644 > --- a/meta-isar/conf/local.conf.sample > +++ b/meta-isar/conf/local.conf.sample > @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password" > #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache" > # Enable ccache debug mode > #CCACHE_DEBUG = "1" > + > +# Uncommnet and add value to it to build images reproducibly > +# > +# The value for `SOURCE_DATE_EPOCH` should be latest source change > time in +# seconds since the Epoch. > +# Git repository users can use value from 'git log -1 --pretty=%ct' > +# Non git repository users can use value from 'stat -c%Y ChangeLog' > +# To know more details about this variable and how to set the value > refer below +# https://reproducible-builds.org/docs/source-date-epoch/ > +#SOURCE_DATE_EPOCH = ${@bb.process.run(git log ...)} would be nice here. So once uncommented it will keep moving as people commit. > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 813e1f3..38a9adf 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -430,6 +430,15 @@ do_rootfs_finalize() { > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > + > + # Set same time-stamps to the newly generated file/folders > in the > + # rootfs image for the purpose of reproducible builds. > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > + find ${ROOTFSDIR} -newermt \ > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > %H:%M:%S')" \ > + -printf "%y %p\n" \ > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps + > EOSUDO I would suggest to at least display a bbwarn if "wc -l" of that file exceeds some number ... say 50. I guess if SOURCE_DATE_EPOCH was too old, say 01.01.1990 the whole filesystem would be touched which might indicate a problem. Not sure what a good number would be. We could also check for certain files to _not_ be in there for sure. I might give that patch a try and see for myself what a too old value would do. But right now i will keep going with the expectation that it would "touch all files without big warning" and the thing might still boot but the broken metadata could cause any kind of problems in applications that can get confused by that big change. Henning > } > addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
On Thu, 2023-01-05 at 09:19 +0100, Henning Schild wrote: > Am Thu, 5 Jan 2023 11:48:57 +0530 > schrieb venkata.pyla@toshiba-tsip.com: > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > > As part of reproducible-build work, the rootfs images generated on > > same source should be identical between two builds. > > > > In this commit it tries to solve one of the non-reproducible > > problem > > i.e. the rootfs file time-stamps generated during build time are > > not > > reproducible, it uses one of the solution provided in the debian > > live-build image project (refer [1]), it fixes by finding all the > > files/folders that are gernerated newly and set the time-stamp > > provided by `SOURCE_DATE_EPOCH` environment variable. > > > > [1] > > https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > > > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > > --- > > meta-isar/conf/local.conf.sample | 10 ++++++++++ > > meta/classes/image.bbclass | 9 +++++++++ > > 2 files changed, 19 insertions(+) > > > > diff --git a/meta-isar/conf/local.conf.sample > > b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644 > > --- a/meta-isar/conf/local.conf.sample > > +++ b/meta-isar/conf/local.conf.sample > > @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password" > > #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache" > > # Enable ccache debug mode > > #CCACHE_DEBUG = "1" > > + > > +# Uncommnet and add value to it to build images reproducibly > > +# > > +# The value for `SOURCE_DATE_EPOCH` should be latest source change > > time in +# seconds since the Epoch. > > +# Git repository users can use value from 'git log -1 -- > > pretty=%ct' > > +# Non git repository users can use value from 'stat -c%Y > > ChangeLog' > > +# To know more details about this variable and how to set the > > value > > refer below +# > > https://reproducible-builds.org/docs/source-date-epoch/ > > +#SOURCE_DATE_EPOCH = > > ${@bb.process.run(git log ...)} > > would be nice here. So once uncommented it will keep moving as people > commit. Be careful with this pattern as it bypasses the Bitbake signatures, leading to non-deterministic signatures errors. We had the same problem with the ISAR release command. Maybe the immediate expansion operator works in the local conf, but I did not check that. It would be better to inject that from the outside, e.g. via KAS or an environment variable. And in addition, we need to make sure that we only forward this variable to non-sbuild tasks. The sbuild tasks should get the SOURCE_DATE_EPOCH from the latest entry in the changelog. However, for auto-generated changelogs, the date should be the value of SOURCE_DATE_EPOCH. Felix > > > diff --git a/meta/classes/image.bbclass > > b/meta/classes/image.bbclass > > index 813e1f3..38a9adf 100644 > > --- a/meta/classes/image.bbclass > > +++ b/meta/classes/image.bbclass > > @@ -430,6 +430,15 @@ do_rootfs_finalize() { > > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > > + > > + # Set same time-stamps to the newly generated file/folders > > in the > > + # rootfs image for the purpose of reproducible builds. > > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > > + find ${ROOTFSDIR} -newermt \ > > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > > %H:%M:%S')" \ > > + -printf "%y %p\n" \ > > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > > > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps + > > EOSUDO > > I would suggest to at least display a bbwarn if "wc -l" of that file > exceeds some number ... say 50. I guess if SOURCE_DATE_EPOCH was too > old, say 01.01.1990 the whole filesystem would be touched which might > indicate a problem. > > Not sure what a good number would be. We could also check for certain > files to _not_ be in there for sure. > > I might give that patch a try and see for myself what a too old value > would do. But right now i will keep going with the expectation that > it > would "touch all files without big warning" and the thing might still > boot but the broken metadata could cause any kind of problems in > applications that can get confused by that big change. > > Henning > > > } > > addtask rootfs_finalize before do_rootfs after > > do_rootfs_postprocess >
>-----Original Message----- >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf >Of Henning Schild >Sent: 05 January 2023 13:49 >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- >tsip.com> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; jan.kiszka@siemens.com; >hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> >Subject: Re: [PATCH v2 1/1] image.bbclass: fix non-reproducible file time-stamps >inside rootfs > >Am Thu, 5 Jan 2023 11:48:57 +0530 >schrieb venkata.pyla@toshiba-tsip.com: > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> >> >> As part of reproducible-build work, the rootfs images generated on >> same source should be identical between two builds. >> >> In this commit it tries to solve one of the non-reproducible problem >> i.e. the rootfs file time-stamps generated during build time are not >> reproducible, it uses one of the solution provided in the debian >> live-build image project (refer [1]), it fixes by finding all the >> files/folders that are gernerated newly and set the time-stamp >> provided by `SOURCE_DATE_EPOCH` environment variable. >> >> [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218 >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> >> --- >> meta-isar/conf/local.conf.sample | 10 ++++++++++ >> meta/classes/image.bbclass | 9 +++++++++ >> 2 files changed, 19 insertions(+) >> >> diff --git a/meta-isar/conf/local.conf.sample >> b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644 >> --- a/meta-isar/conf/local.conf.sample >> +++ b/meta-isar/conf/local.conf.sample >> @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password" >> #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache" >> # Enable ccache debug mode >> #CCACHE_DEBUG = "1" >> + >> +# Uncommnet and add value to it to build images reproducibly # # The >> +value for `SOURCE_DATE_EPOCH` should be latest source change >> time in +# seconds since the Epoch. >> +# Git repository users can use value from 'git log -1 --pretty=%ct' >> +# Non git repository users can use value from 'stat -c%Y ChangeLog' >> +# To know more details about this variable and how to set the value >> refer below +# https://reproducible-builds.org/docs/source-date-epoch/ >> +#SOURCE_DATE_EPOCH = > >${@bb.process.run(git log ...)} > >would be nice here. So once uncommented it will keep moving as people >commit. > >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index 813e1f3..38a9adf 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" >> >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" >> + >> + # Set same time-stamps to the newly generated file/folders >> in the >> + # rootfs image for the purpose of reproducible builds. >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ >> + find ${ROOTFSDIR} -newermt \ >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d >> %H:%M:%S')" \ >> + -printf "%y %p\n" \ >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > >> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps + EOSUDO > >I would suggest to at least display a bbwarn if "wc -l" of that file exceeds some >number ... say 50. I guess if SOURCE_DATE_EPOCH was too old, say 01.01.1990 >the whole filesystem would be touched which might indicate a problem. 'SOURCE_DATE_EPOCH' is not set by default and regular builds will not have any problem for sure, but when someone uses it then I think they know what they are doing and can see the results. I will also add bbwarn about this files time modification and print all files that are modified. > >Not sure what a good number would be. We could also check for certain files to >_not_ be in there for sure. > >I might give that patch a try and see for myself what a too old value would do. >But right now i will keep going with the expectation that it would "touch all files >without big warning" and the thing might still boot but the broken metadata >could cause any kind of problems in applications that can get confused by that >big change. > >Henning > >> } >> addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess > >-- >You received this message because you are subscribed to the Google Groups >"isar-users" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to isar-users+unsubscribe@googlegroups.com. >To view this discussion on the web visit >https://groups.google.com/d/msgid/isar- >users/20230105091904.530199bb%40md1za8fc.ad001.siemens.net.
Am Thu, 5 Jan 2023 13:52:51 +0000 schrieb <Venkata.Pyla@toshiba-tsip.com>: > >-----Original Message----- > >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > >Behalf Of Henning Schild > >Sent: 05 January 2023 13:49 > >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > >tsip.com> > >Cc: isar-users@googlegroups.com; amikan@ilbers.de; > >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) > ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG > >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH v2 1/1] > >image.bbclass: fix non-reproducible file time-stamps inside rootfs > > > >Am Thu, 5 Jan 2023 11:48:57 +0530 > >schrieb venkata.pyla@toshiba-tsip.com: > > > >> From: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> > >> As part of reproducible-build work, the rootfs images generated on > >> same source should be identical between two builds. > >> > >> In this commit it tries to solve one of the non-reproducible > >> problem i.e. the rootfs file time-stamps generated during build > >> time are not reproducible, it uses one of the solution provided in > >> the debian live-build image project (refer [1]), it fixes by > >> finding all the files/folders that are gernerated newly and set > >> the time-stamp provided by `SOURCE_DATE_EPOCH` environment > >> variable. > >> > >> [1] > >> https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > >> > >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > >> --- > >> meta-isar/conf/local.conf.sample | 10 ++++++++++ > >> meta/classes/image.bbclass | 9 +++++++++ > >> 2 files changed, 19 insertions(+) > >> > >> diff --git a/meta-isar/conf/local.conf.sample > >> b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644 > >> --- a/meta-isar/conf/local.conf.sample > >> +++ b/meta-isar/conf/local.conf.sample > >> @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password" > >> #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache" > >> # Enable ccache debug mode > >> #CCACHE_DEBUG = "1" > >> + > >> +# Uncommnet and add value to it to build images reproducibly # # > >> The +value for `SOURCE_DATE_EPOCH` should be latest source change > >> time in +# seconds since the Epoch. > >> +# Git repository users can use value from 'git log -1 > >> --pretty=%ct' +# Non git repository users can use value from 'stat > >> -c%Y ChangeLog' +# To know more details about this variable and > >> how to set the value refer below +# > >> https://reproducible-builds.org/docs/source-date-epoch/ > >> +#SOURCE_DATE_EPOCH = > > > >${@bb.process.run(git log ...)} > > > >would be nice here. So once uncommented it will keep moving as people > >commit. > > > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index 813e1f3..38a9adf 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -430,6 +430,15 @@ do_rootfs_finalize() { > >> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > >> > >> rm -f "${ROOTFSDIR}/etc/apt/sources-list" > >> + > >> + # Set same time-stamps to the newly generated file/folders > >> in the > >> + # rootfs image for the purpose of reproducible builds. > >> + test ! -z "${SOURCE_DATE_EPOCH}" && \ > >> + find ${ROOTFSDIR} -newermt \ > >> + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > >> %H:%M:%S')" \ > >> + -printf "%y %p\n" \ > >> + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > > >> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps + EOSUDO > > > >I would suggest to at least display a bbwarn if "wc -l" of that file > >exceeds some number ... say 50. I guess if SOURCE_DATE_EPOCH was too > >old, say 01.01.1990 the whole filesystem would be touched which > >might indicate a problem. > 'SOURCE_DATE_EPOCH' is not set by default and regular builds will not > have any problem for sure, but when someone uses it then I think they > know what they are doing and can see the results. We eventually want to enable it by default i guess and so we can already think about what not so experienced users could stumble over and how to help them. > I will also add bbwarn about this files time modification and print > all files that are modified. Thanks. Henning > > > >Not sure what a good number would be. We could also check for > >certain files to _not_ be in there for sure. > > > >I might give that patch a try and see for myself what a too old > >value would do. But right now i will keep going with the expectation > >that it would "touch all files without big warning" and the thing > >might still boot but the broken metadata could cause any kind of > >problems in applications that can get confused by that big change. > > > >Henning > > > >> } > >> addtask rootfs_finalize before do_rootfs after > >> do_rootfs_postprocess > > > >-- > >You received this message because you are subscribed to the Google > >Groups "isar-users" group. > >To unsubscribe from this group and stop receiving emails from it, > >send an email to isar-users+unsubscribe@googlegroups.com. > >To view this discussion on the web visit > >https://groups.google.com/d/msgid/isar- > >users/20230105091904.530199bb%40md1za8fc.ad001.siemens.net. >
Am Thu, 5 Jan 2023 10:50:36 +0100 schrieb "Moessbauer, Felix (T CED INW-CN)" <felix.moessbauer@siemens.com>: > On Thu, 2023-01-05 at 09:19 +0100, Henning Schild wrote: > > Am Thu, 5 Jan 2023 11:48:57 +0530 > > schrieb venkata.pyla@toshiba-tsip.com: > > > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > > > > As part of reproducible-build work, the rootfs images generated on > > > same source should be identical between two builds. > > > > > > In this commit it tries to solve one of the non-reproducible > > > problem > > > i.e. the rootfs file time-stamps generated during build time are > > > not > > > reproducible, it uses one of the solution provided in the debian > > > live-build image project (refer [1]), it fixes by finding all the > > > files/folders that are gernerated newly and set the time-stamp > > > provided by `SOURCE_DATE_EPOCH` environment variable. > > > > > > [1] > > > https://salsa.debian.org/live-team/live-build/-/merge_requests/218 > > > > > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > --- > > > meta-isar/conf/local.conf.sample | 10 ++++++++++ > > > meta/classes/image.bbclass | 9 +++++++++ > > > 2 files changed, 19 insertions(+) > > > > > > diff --git a/meta-isar/conf/local.conf.sample > > > b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644 > > > --- a/meta-isar/conf/local.conf.sample > > > +++ b/meta-isar/conf/local.conf.sample > > > @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password" > > > #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache" > > > # Enable ccache debug mode > > > #CCACHE_DEBUG = "1" > > > + > > > +# Uncommnet and add value to it to build images reproducibly > > > +# > > > +# The value for `SOURCE_DATE_EPOCH` should be latest source > > > change time in +# seconds since the Epoch. > > > +# Git repository users can use value from 'git log -1 -- > > > pretty=%ct' > > > +# Non git repository users can use value from 'stat -c%Y > > > ChangeLog' > > > +# To know more details about this variable and how to set the > > > value > > > refer below +# > > > https://reproducible-builds.org/docs/source-date-epoch/ > > > +#SOURCE_DATE_EPOCH = > > > > ${@bb.process.run(git log ...)} > > > > would be nice here. So once uncommented it will keep moving as > > people commit. > > Be careful with this pattern as it bypasses the Bitbake signatures, > leading to non-deterministic signatures errors. We had the same > problem with the ISAR release command. Maybe the immediate expansion > operator works in the local conf, but I did not check that. The value is only ever used in image.bbclass in the finalize function. So maybe not a problem to expect with sstate. > It would be better to inject that from the outside, e.g. via KAS or an > environment variable. Yes but we also need to document sane values for it, maybe in the user doc after all and not the local.conf.sample. > And in addition, we need to make sure that we > only forward this variable to non-sbuild tasks. The sbuild tasks > should get the SOURCE_DATE_EPOCH from the latest entry in the > changelog. However, for auto-generated changelogs, the date should be > the value of SOURCE_DATE_EPOCH. I sure hope this one does not go into the env of the package building, but it should be double checked. For auto-generated changelogs we have another solution based on last debian changelog entry or 1.1.1970 if there is none. https://github.com/ilbers/isar/commit/53d315fdd3f3dcf70ac3f257d9431ba522e86eb4 Henning > Felix > > > > > > diff --git a/meta/classes/image.bbclass > > > b/meta/classes/image.bbclass > > > index 813e1f3..38a9adf 100644 > > > --- a/meta/classes/image.bbclass > > > +++ b/meta/classes/image.bbclass > > > @@ -430,6 +430,15 @@ do_rootfs_finalize() { > > > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > > > > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > > > + > > > + # Set same time-stamps to the newly generated > > > file/folders in the > > > + # rootfs image for the purpose of reproducible builds. > > > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > > > + find ${ROOTFSDIR} -newermt \ > > > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > > > %H:%M:%S')" \ > > > + -printf "%y %p\n" \ > > > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > > > > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps + > > > EOSUDO > > > > I would suggest to at least display a bbwarn if "wc -l" of that file > > exceeds some number ... say 50. I guess if SOURCE_DATE_EPOCH was too > > old, say 01.01.1990 the whole filesystem would be touched which > > might indicate a problem. > > > > Not sure what a good number would be. We could also check for > > certain files to _not_ be in there for sure. > > > > I might give that patch a try and see for myself what a too old > > value would do. But right now i will keep going with the > > expectation that it > > would "touch all files without big warning" and the thing might > > still boot but the broken metadata could cause any kind of problems > > in applications that can get confused by that big change. > > > > Henning > > > > > } > > > addtask rootfs_finalize before do_rootfs after > > > do_rootfs_postprocess > > >
On Thu, 2023-01-05 at 18:05 +0100, Henning Schild wrote: > > > > https://reproducible-builds.org/docs/source-date-epoch/ > > > > +#SOURCE_DATE_EPOCH = > > > > > > ${@bb.process.run(git log ...)} > > > > > > would be nice here. So once uncommented it will keep moving as > > > people commit. > > > > Be careful with this pattern as it bypasses the Bitbake signatures, > > leading to non-deterministic signatures errors. We had the same > > problem with the ISAR release command. Maybe the immediate > > expansion > > operator works in the local conf, but I did not check that. > > The value is only ever used in image.bbclass in the finalize > function. > So maybe not a problem to expect with sstate. This is totally independent of sstate. Bitbake internally requires that the values of all variables are deterministic. This is required because the set of signatures of a task is used to decide if a task is re- executed or not. And changing signatures without changing the "input" is a bitbake error. > > > It would be better to inject that from the outside, e.g. via KAS or > > an > > environment variable. > > Yes but we also need to document sane values for it, maybe in the > user > doc after all and not the local.conf.sample. > > > And in addition, we need to make sure that we > > only forward this variable to non-sbuild tasks. The sbuild tasks > > should get the SOURCE_DATE_EPOCH from the latest entry in the > > changelog. However, for auto-generated changelogs, the date should > > be > > the value of SOURCE_DATE_EPOCH. > > I sure hope this one does not go into the env of the package > building, > but it should be double checked. > > For auto-generated changelogs we have another solution based on last > debian changelog entry or 1.1.1970 if there is none. > > https://github.com/ilbers/isar/commit/53d315fdd3f3dcf70ac3f257d9431ba522e86eb4 IMHO this pattern is a hacky workaround. As we now have the SOURCE_DATE_EPOCH, we should also use that for the changelog. But better not as part of this series. It is already getting too long. @venkata: When sending a new version of the series, please do not send it as "in-reply-to ...", as this makes it harder and harder to read it between all the replies. Better simply send it directly. Felix > > Henning > > > Felix > > > > > > > > > diff --git a/meta/classes/image.bbclass > > > > b/meta/classes/image.bbclass > > > > index 813e1f3..38a9adf 100644 > > > > --- a/meta/classes/image.bbclass > > > > +++ b/meta/classes/image.bbclass > > > > @@ -430,6 +430,15 @@ do_rootfs_finalize() { > > > > > > > > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" > > > > > > > > rm -f "${ROOTFSDIR}/etc/apt/sources-list" > > > > + > > > > + # Set same time-stamps to the newly generated > > > > file/folders in the > > > > + # rootfs image for the purpose of reproducible builds. > > > > + test ! -z "${SOURCE_DATE_EPOCH}" && \ > > > > + find ${ROOTFSDIR} -newermt \ > > > > + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d > > > > %H:%M:%S')" \ > > > > + -printf "%y %p\n" \ > > > > + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} > > > > ';' > > > > > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps + > > > > EOSUDO > > > > > > I would suggest to at least display a bbwarn if "wc -l" of that > > > file > > > exceeds some number ... say 50. I guess if SOURCE_DATE_EPOCH was > > > too > > > old, say 01.01.1990 the whole filesystem would be touched which > > > might indicate a problem. > > > > > > Not sure what a good number would be. We could also check for > > > certain files to _not_ be in there for sure. > > > > > > I might give that patch a try and see for myself what a too old > > > value would do. But right now i will keep going with the > > > expectation that it > > > would "touch all files without big warning" and the thing might > > > still boot but the broken metadata could cause any kind of > > > problems > > > in applications that can get confused by that big change. > > > > > > Henning > > > > > > > } > > > > addtask rootfs_finalize before do_rootfs after > > > > do_rootfs_postprocess > > > > > >
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644 --- a/meta-isar/conf/local.conf.sample +++ b/meta-isar/conf/local.conf.sample @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password" #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache" # Enable ccache debug mode #CCACHE_DEBUG = "1" + +# Uncommnet and add value to it to build images reproducibly +# +# The value for `SOURCE_DATE_EPOCH` should be latest source change time in +# seconds since the Epoch. +# Git repository users can use value from 'git log -1 --pretty=%ct' +# Non git repository users can use value from 'stat -c%Y ChangeLog' +# To know more details about this variable and how to set the value refer below +# https://reproducible-builds.org/docs/source-date-epoch/ +#SOURCE_DATE_EPOCH = diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 813e1f3..38a9adf 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -430,6 +430,15 @@ do_rootfs_finalize() { "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" rm -f "${ROOTFSDIR}/etc/apt/sources-list" + + # Set same time-stamps to the newly generated file/folders in the + # rootfs image for the purpose of reproducible builds. + test ! -z "${SOURCE_DATE_EPOCH}" && \ + find ${ROOTFSDIR} -newermt \ + "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ + -printf "%y %p\n" \ + -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps + EOSUDO } addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess