| Message ID | 20231207154114.20400-4-venkata.pyla@toshiba-tsip.com |
|---|---|
| State | New |
| Headers | show |
| Series | Make ext4 images reproducible | expand |
On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > system. > hash_seed: creates reproducible directory indexes in the file system. > > Reference commit in e2fsprogs: > e1f7100643a46456be107b33098f6034b0835e6d Hi! Did you also send this patch to OE? We need to integrate it there as well, as this is the basis for the ISAR plugins. Best regards, Felix > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > scripts/lib/wic/partition.py | 11 +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/scripts/lib/wic/partition.py > b/scripts/lib/wic/partition.py > index e50871b8..90b2c037 100644 > --- a/scripts/lib/wic/partition.py > +++ b/scripts/lib/wic/partition.py > @@ -280,6 +280,17 @@ class Partition(): > > extraopts = self.mkfs_extraopts or "-F -i 8192" > > + if os.getenv('SOURCE_DATE_EPOCH'): > + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) > + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, > pseudo) > + > + # Set hash_seed to generate deterministic directory > indexes > + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9- > 2f2fdf33d460") > + if self.fsuuid: > + namespace = uuid.UUID(self.fsuuid) > + hash_seed = str(uuid.uuid5(namespace, str(sde_time))) > + extraopts += " -E hash_seed=%s" % hash_seed > + > label_str = "" > if self.label: > label_str = "-L %s" % self.label
> -----Original Message----- > From: 'MOESSBAUER, Felix' via isar-users <isar-users@googlegroups.com> > Sent: Friday, December 8, 2023 4:04 PM > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > <Venkata.Pyla@toshiba-tsip.com> > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to > generate reproducible ext4 images > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote: > > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > > system. > > hash_seed: creates reproducible directory indexes in the file system. > > > > Reference commit in e2fsprogs: > > e1f7100643a46456be107b33098f6034b0835e6d > > Hi! > > Did you also send this patch to OE? > We need to integrate it there as well, as this is the basis for the ISAR plugins. Not shared yet, I will send to OE as well. > > Best regards, > Felix > > > > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > > --- > > scripts/lib/wic/partition.py | 11 +++++++++++ > > 1 file changed, 11 insertions(+) > > > > diff --git a/scripts/lib/wic/partition.py > > b/scripts/lib/wic/partition.py index e50871b8..90b2c037 100644 > > --- a/scripts/lib/wic/partition.py > > +++ b/scripts/lib/wic/partition.py > > @@ -280,6 +280,17 @@ class Partition(): > > > > extraopts = self.mkfs_extraopts or "-F -i 8192" > > > > + if os.getenv('SOURCE_DATE_EPOCH'): > > + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) > > + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, > > pseudo) > > + > > + # Set hash_seed to generate deterministic directory > > indexes > > + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9- > > 2f2fdf33d460") > > + if self.fsuuid: > > + namespace = uuid.UUID(self.fsuuid) > > + hash_seed = str(uuid.uuid5(namespace, str(sde_time))) > > + extraopts += " -E hash_seed=%s" % hash_seed > > + > > label_str = "" > > if self.label: > > label_str = "-L %s" % self.label > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/isar- > users/813c16c90a28fd387d5dbdb51aa02f98b1ebfa57.camel%40siemens.com.
On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote: > > -----Original Message----- > > From: 'MOESSBAUER, Felix' via isar-users <isar-users@googlegroups.com> > > Sent: Friday, December 8, 2023 4:04 PM > > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG Porting) > > <Venkata.Pyla@toshiba-tsip.com> > > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to > > generate reproducible ext4 images > > > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote: > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > > > system. > > > hash_seed: creates reproducible directory indexes in the file system. > > > > > > Reference commit in e2fsprogs: > > > e1f7100643a46456be107b33098f6034b0835e6d > > > > Hi! > > > > Did you also send this patch to OE? > > We need to integrate it there as well, as this is the basis for the ISAR plugins. > > Not shared yet, I will send to OE as well. We should follow "upstream first" here. Once OE has accepted the patch we will inherit it during next OE -> Isar update cycle. Maintainers: Do not apply it to Isar directly. > > > > > Best regards, > > Felix > > > > >
> -----Original Message----- > From: 'Florian Bezdeka' via isar-users <isar-users@googlegroups.com> > Sent: Saturday, December 9, 2023 4:38 PM > To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to > generate reproducible ext4 images > > On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote: > > > -----Original Message----- > > > From: 'MOESSBAUER, Felix' via isar-users > > > <isar-users@googlegroups.com> > > > Sent: Friday, December 8, 2023 4:04 PM > > > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG > > > Porting) <Venkata.Pyla@toshiba-tsip.com> > > > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 > ＤＭＥ > > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed > > > to generate reproducible ext4 images > > > > > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com > wrote: > > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > > > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > > > > system. > > > > hash_seed: creates reproducible directory indexes in the file system. > > > > > > > > Reference commit in e2fsprogs: > > > > e1f7100643a46456be107b33098f6034b0835e6d > > > > > > Hi! > > > > > > Did you also send this patch to OE? > > > We need to integrate it there as well, as this is the basis for the ISAR > plugins. > > > > Not shared yet, I will send to OE as well. > > We should follow "upstream first" here. Once OE has accepted the patch we > will inherit it during next OE -> Isar update cycle. I have shared this patch to OE [1], hope I will get feedback from the community and will support to fix this issue in the OE itself. [1] https://lists.openembedded.org/g/openembedded-core/message/192192 > > Maintainers: Do not apply it to Isar directly. > > > > > > > > > Best regards, > > > Felix > > > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar- > users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com > .
> -----Original Message----- > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf > Of Venkata.Pyla@toshiba-tsip.com > Sent: Tuesday, December 12, 2023 7:37 PM > To: florian.bezdeka@siemens.com; felix.moessbauer@siemens.com; isar- > users@googlegroups.com > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > Subject: RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to > generate reproducible ext4 images > > > > > -----Original Message----- > > From: 'Florian Bezdeka' via isar-users <isar-users@googlegroups.com> > > Sent: Saturday, December 9, 2023 4:38 PM > > To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > > tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com > > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to > > generate reproducible ext4 images > > > > On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote: > > > > -----Original Message----- > > > > From: 'MOESSBAUER, Felix' via isar-users > > > > <isar-users@googlegroups.com> > > > > Sent: Friday, December 8, 2023 4:04 PM > > > > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG > > > > Porting) <Venkata.Pyla@toshiba-tsip.com> > > > > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 > > ＤＭＥ > > > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > > > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > > > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and > > > > hash_seed to generate reproducible ext4 images > > > > > > > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com > > wrote: > > > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > > > > > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > > > > > system. > > > > > hash_seed: creates reproducible directory indexes in the file system. > > > > > > > > > > Reference commit in e2fsprogs: > > > > > e1f7100643a46456be107b33098f6034b0835e6d > > > > > > > > Hi! > > > > > > > > Did you also send this patch to OE? > > > > We need to integrate it there as well, as this is the basis for > > > > the ISAR > > plugins. > > > > > > Not shared yet, I will send to OE as well. > > > > We should follow "upstream first" here. Once OE has accepted the patch > > we will inherit it during next OE -> Isar update cycle. > > I have shared this patch to OE [1], hope I will get feedback from the community > and will support to fix this issue in the OE itself. > > [1] https://lists.openembedded.org/g/openembedded-core/message/192192 This patch [1] has been merged now in upstream OE-core, now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually? [1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0 > > > > Maintainers: Do not apply it to Isar directly. > > > > > > > > > > > > > Best regards, > > > > Felix > > > > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to isar-users+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/isar- > > users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com > > . > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/isar- > users/OS3PR01MB68566626DF8F78E4F73D2A45A48EA%40OS3PR01MB6856.jpn > prd01.prod.outlook.com.
On Thu, 2023-12-28 at 06:02 +0000, Venkata.Pyla@toshiba-tsip.com wrote: > > -----Original Message----- > > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf > > Of Venkata.Pyla@toshiba-tsip.com > > Sent: Tuesday, December 12, 2023 7:37 PM > > To: florian.bezdeka@siemens.com; felix.moessbauer@siemens.com; isar- > > users@googlegroups.com > > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > > Subject: RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to > > generate reproducible ext4 images > > > > > > > > > -----Original Message----- > > > From: 'Florian Bezdeka' via isar-users <isar-users@googlegroups.com> > > > Sent: Saturday, December 9, 2023 4:38 PM > > > To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- > > > tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com > > > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ > > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to > > > generate reproducible ext4 images > > > > > > On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote: > > > > > -----Original Message----- > > > > > From: 'MOESSBAUER, Felix' via isar-users > > > > > <isar-users@googlegroups.com> > > > > > Sent: Friday, December 8, 2023 4:04 PM > > > > > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG > > > > > Porting) <Venkata.Pyla@toshiba-tsip.com> > > > > > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 > > > ＤＭＥ > > > > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh > > > > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com> > > > > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and > > > > > hash_seed to generate reproducible ext4 images > > > > > > > > > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com > > > wrote: > > > > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > > > > > > > > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > > > > > > system. > > > > > > hash_seed: creates reproducible directory indexes in the file system. > > > > > > > > > > > > Reference commit in e2fsprogs: > > > > > > e1f7100643a46456be107b33098f6034b0835e6d > > > > > > > > > > Hi! > > > > > > > > > > Did you also send this patch to OE? > > > > > We need to integrate it there as well, as this is the basis for > > > > > the ISAR > > > plugins. > > > > > > > > Not shared yet, I will send to OE as well. > > > > > > We should follow "upstream first" here. Once OE has accepted the patch > > > we will inherit it during next OE -> Isar update cycle. > > > > I have shared this patch to OE [1], hope I will get feedback from the community > > and will support to fix this issue in the OE itself. > > > > [1] https://lists.openembedded.org/g/openembedded-core/message/192192 > > This patch [1] has been merged now in upstream OE-core, > now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually? > > [1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0 That's great! Thanks for taking care. I added two guys from Ilbers, maybe they have an update pending already or can comment on the timeline. Anton, Uladzimir any comments? In general it's a good idea to come up with a patch to speed things up. The last update was done in 64af29bcc4b0 ("wic: Update to the latest revision"). Might be used as inspiration. Florian > > > > > > > Maintainers: Do not apply it to Isar directly. > > > > > > > > > > > > > > > > > Best regards, > > > > > Felix > > > > > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "isar-users" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to isar-users+unsubscribe@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/isar- > > > users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to > > isar-users+unsubscribe@googlegroups.com. > > To view this discussion on the web visit https://groups.google.com/d/msgid/isar- > > users/OS3PR01MB68566626DF8F78E4F73D2A45A48EA%40OS3PR01MB6856.jpn > > prd01.prod.outlook.com.
28/12/2023 10:47, Florian Bezdeka wrote: > On Thu, 2023-12-28 at 06:02 +0000, Venkata.Pyla@toshiba-tsip.com wrote: >> This patch [1] has been merged now in upstream OE-core, >> now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually? >> >> [1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0 > That's great! Thanks for taking care. > > I added two guys from Ilbers, maybe they have an update pending already > or can comment on the timeline. Anton, Uladzimir any comments? > > In general it's a good idea to come up with a patch to speed things up. > > The last update was done in 64af29bcc4b0 ("wic: Update to the latest > revision"). Might be used as inspiration. > > Florian Hello, We are going to make v0.10 release soon and probably can take care of externals (like WIC, OE libs and Bitbake) update right after that.
Hello Anton, Hope this message finds you well. Just wanted to follow up on a previous mail regarding wic and other related updates from openembedded-core. You had mentioned about the updates from openembedded-core to be handled right after the release of ISAR v0.10 Would like to understand if there are any plans for wic and other related updates from openembedded-core. Thanks and Regards, Adithya Balakumar -----Original Message----- From: Anton Mikanovich <amikan@ilbers.de> Sent: Thursday, December 28, 2023 3:40 PM To: Florian Bezdeka <florian.bezdeka@siemens.com>; pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com; Uladzimir Bely <ubely@ilbers.de> Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; venkatapyla4@gmail.com; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com> Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images 28/12/2023 10:47, Florian Bezdeka wrote: > On Thu, 2023-12-28 at 06:02 +0000, Venkata.Pyla@toshiba-tsip.com wrote: >> This patch [1] has been merged now in upstream OE-core, >> now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually? >> >> [1] >> https://github.com/openembedded/openembedded-core/commit/bb822ab75de0 >> 020572058090439b93cc56bbf7e0 > That's great! Thanks for taking care. > > I added two guys from Ilbers, maybe they have an update pending > already or can comment on the timeline. Anton, Uladzimir any comments? > > In general it's a good idea to come up with a patch to speed things up. > > The last update was done in 64af29bcc4b0 ("wic: Update to the latest > revision"). Might be used as inspiration. > > Florian Hello, We are going to make v0.10 release soon and probably can take care of externals (like WIC, OE libs and Bitbake) update right after that.
13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote: > Hello Anton, > > Hope this message finds you well. > > Just wanted to follow up on a previous mail regarding wic and other related updates from openembedded-core. You had mentioned about the updates from openembedded-core to be handled right after the release of ISAR v0.10 > > Would like to understand if there are any plans for wic and other related updates from openembedded-core. > > Thanks and Regards, > Adithya Balakumar Hello Adithya, We've checked and the next OE LTS using bitbake 2.8 should be released in April. We could start preparations for the bitbake upgrade and update OE after that. Would that suit your needs?
Hello Anton, Thanks for the quick update. Updating OE after the update to bitbake 2.8 should be fine. Thanks and Regards, Adithya Balakumar -----Original Message----- From: Anton Mikanovich <amikan@ilbers.de> Sent: Wednesday, March 13, 2024 9:30 PM To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; florian.bezdeka@siemens.com; venkata1.pyla <venkata1.pyla@toshibaap.onmicrosoft.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com; ubely@ilbers.de Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; venkatapyla4@gmail.com Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images 13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote: > Hello Anton, > > Hope this message finds you well. > > Just wanted to follow up on a previous mail regarding wic and other > related updates from openembedded-core. You had mentioned about the > updates from openembedded-core to be handled right after the release > of ISAR v0.10 > > Would like to understand if there are any plans for wic and other related updates from openembedded-core. > > Thanks and Regards, > Adithya Balakumar Hello Adithya, We've checked and the next OE LTS using bitbake 2.8 should be released in April. We could start preparations for the bitbake upgrade and update OE after that. Would that suit your needs?
On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote: > From: venkata pyla <venkata.pyla@toshiba-tsip.com> > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > system. > hash_seed: creates reproducible directory indexes in the file system. > > Reference commit in e2fsprogs: > e1f7100643a46456be107b33098f6034b0835e6d > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com> > --- > scripts/lib/wic/partition.py | 11 +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/scripts/lib/wic/partition.py > b/scripts/lib/wic/partition.py > index e50871b8..90b2c037 100644 > --- a/scripts/lib/wic/partition.py > +++ b/scripts/lib/wic/partition.py > @@ -280,6 +280,17 @@ class Partition(): > > extraopts = self.mkfs_extraopts or "-F -i 8192" > > + if os.getenv('SOURCE_DATE_EPOCH'): > + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) > + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, > pseudo) > + > + # Set hash_seed to generate deterministic directory > indexes > + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9- > 2f2fdf33d460") > + if self.fsuuid: > + namespace = uuid.UUID(self.fsuuid) > + hash_seed = str(uuid.uuid5(namespace, str(sde_time))) > + extraopts += " -E hash_seed=%s" % hash_seed > + Hi, while reworking the SDE in ISAR, I stumbled upon this as well. This patch only covers the .wic part, but we need a similar patch for the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but I was not able to make the .ext4 build reproducible (the diff indicated that the inodes are still shuffled around). This makes me wonder if mke2fs.ext4 even supports producing a reproducible rootfs. Have you been able to create a bit-by-bit identical .wic of an ext4 partition? Best regards, Felix > label_str = "" > if self.label: > label_str = "-L %s" % self.label
On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote: > 13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote: > > Hello Anton, > > > > Hope this message finds you well. > > > > Just wanted to follow up on a previous mail regarding wic and other > > related updates from openembedded-core. You had mentioned about the > > updates from openembedded-core to be handled right after the > > release of ISAR v0.10 > > > > Would like to understand if there are any plans for wic and other > > related updates from openembedded-core. > > > > Thanks and Regards, > > Adithya Balakumar > > Hello Adithya, > > We've checked and the next OE LTS using bitbake 2.8 should be > released > in April. Hi, bitbake 2.8 has been released last week. > We could start preparations for the bitbake upgrade and update OE > after > that. It would be great if we could have bitbake 2.8 in a timely manner. This release contains many of our contributed fixes (e.g. for reproducible builds, better estimation of available CPUs on large servers). Felix > Would that suit your needs? >
On 25.04.24 08:24, Moessbauer, Felix (T CED OES-DE) wrote: > On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote: >> 13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote: >>> Hello Anton, >>> >>> Hope this message finds you well. >>> >>> Just wanted to follow up on a previous mail regarding wic and other >>> related updates from openembedded-core. You had mentioned about the >>> updates from openembedded-core to be handled right after the >>> release of ISAR v0.10 >>> >>> Would like to understand if there are any plans for wic and other >>> related updates from openembedded-core. >>> >>> Thanks and Regards, >>> Adithya Balakumar >> >> Hello Adithya, >> >> We've checked and the next OE LTS using bitbake 2.8 should be >> released >> in April. > > Hi, bitbake 2.8 has been released last week. > >> We could start preparations for the bitbake upgrade and update OE >> after >> that. > > It would be great if we could have bitbake 2.8 in a timely manner. This > release contains many of our contributed fixes (e.g. for reproducible > builds, better estimation of available CPUs on large servers). > Any impact on recipe syntax etc. again? Jan
On Thu, 2024-04-25 at 08:30 +0200, Jan Kiszka wrote: > On 25.04.24 08:24, Moessbauer, Felix (T CED OES-DE) wrote: > > On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote: > > > 13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote: > > > > Hello Anton, > > > > > > > > Hope this message finds you well. > > > > > > > > Just wanted to follow up on a previous mail regarding wic and > > > > other > > > > related updates from openembedded-core. You had mentioned about > > > > the > > > > updates from openembedded-core to be handled right after the > > > > release of ISAR v0.10 > > > > > > > > Would like to understand if there are any plans for wic and > > > > other > > > > related updates from openembedded-core. > > > > > > > > Thanks and Regards, > > > > Adithya Balakumar > > > > > > Hello Adithya, > > > > > > We've checked and the next OE LTS using bitbake 2.8 should be > > > released > > > in April. > > > > Hi, bitbake 2.8 has been released last week. > > > > > We could start preparations for the bitbake upgrade and update OE > > > after > > > that. > > > > It would be great if we could have bitbake 2.8 in a timely manner. > > This > > release contains many of our contributed fixes (e.g. for > > reproducible > > builds, better estimation of available CPUs on large servers). > > > > Any impact on recipe syntax etc. again? I just did a quick test of bitbake 2.8 in ISAR and that (almost) worked out of the box. Only required change was to move the kbuildtarget.bbclass into the meta/classes dir. Probably the only relevant changes are around the handling of PR: - The PR value can no longer be set from the recipe file name - this was rarely used, but in any case is no longer supported. - PE and PR are no longer included in the work directory path (WORKDIR). This may break some tool assumptions about directory paths, but those should really be querying paths from the build system (or not poking into WORKDIR externally). https://docs.yoctoproject.org/4.3.4/migration-guides/migration-4.3.html#versioning-changes Felix > > Jan >
25/04/2024 09:24, MOESSBAUER, Felix wrote: > On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote: >> Hello Adithya, >> >> We've checked and the next OE LTS using bitbake 2.8 should be >> released >> in April. > Hi, bitbake 2.8 has been released last week. > >> We could start preparations for the bitbake upgrade and update OE >> after >> that. > It would be great if we could have bitbake 2.8 in a timely manner. This > release contains many of our contributed fixes (e.g. for reproducible > builds, better estimation of available CPUs on large servers). > > Felix > >> Would that suit your needs? >> Hello, I've already started working on Bitbake and OE libs update. As you've already mentioned there are no much changes, so hope will send it soon.
Hi Felix, I saw your mail regarding your attempt to make ext4 filesystem images from IMAGE_CMD:ext4 reproducible. If you don't mind, could you briefly explain what was the problem you faced in achieving this? I am also trying to understand on how to achieve the same. Thanks and Regards, Adithya Balakumar On Tuesday, April 23, 2024 at 2:47:11 PM UTC+5:30 MOESSBAUER, Felix wrote: On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote: > From: venkata pyla <venkat...@toshiba-tsip.com> > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > system. > hash_seed: creates reproducible directory indexes in the file system. > > Reference commit in e2fsprogs: > e1f7100643a46456be107b33098f6034b0835e6d > > Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com> > --- > scripts/lib/wic/partition.py | 11 +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/scripts/lib/wic/partition.py > b/scripts/lib/wic/partition.py > index e50871b8..90b2c037 100644 > --- a/scripts/lib/wic/partition.py > +++ b/scripts/lib/wic/partition.py > @@ -280,6 +280,17 @@ class Partition(): > > extraopts = self.mkfs_extraopts or "-F -i 8192" > > + if os.getenv('SOURCE_DATE_EPOCH'): > + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) > + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, > pseudo) > + > + # Set hash_seed to generate deterministic directory > indexes > + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9- > 2f2fdf33d460") > + if self.fsuuid: > + namespace = uuid.UUID(self.fsuuid) > + hash_seed = str(uuid.uuid5(namespace, str(sde_time))) > + extraopts += " -E hash_seed=%s" % hash_seed > + Hi, while reworking the SDE in ISAR, I stumbled upon this as well. This patch only covers the .wic part, but we need a similar patch for the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but I was not able to make the .ext4 build reproducible (the diff indicated that the inodes are still shuffled around). This makes me wonder if mke2fs.ext4 even supports producing a reproducible rootfs. Have you been able to create a bit-by-bit identical .wic of an ext4 partition? Best regards, Felix > label_str = "" > if self.label: > label_str = "-L %s" % self.label
On Fri, 2024-07-05 at 05:13 -0700, Adithya Balakumar wrote: > Hi Felix, > > I saw your mail regarding your attempt to make ext4 filesystem images > from IMAGE_CMD:ext4 reproducible. > If you don't mind, could you briefly explain what was the problem you > faced in achieving this? Hi, the problem is stated below: > the diff indicated that the inodes are still shuffled around). This makes me wonder if mke2fs.ext4 even supports producing a reproducible rootfs. I just copied the pattern from wic, but for whatever reason the inodes still were not deterministic. Are you sure, that this patch is sufficient to make the ext4 reproducible? Felix > I am also trying to understand on how to achieve the same. > > Thanks and Regards, > Adithya Balakumar > > > On Tuesday, April 23, 2024 at 2:47:11 PM UTC+5:30 MOESSBAUER, Felix > wrote: > > On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com > > wrote: > > > From: venkata pyla <venkat...@toshiba-tsip.com> > > > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > > > system. > > > hash_seed: creates reproducible directory indexes in the file > > > system. > > > > > > Reference commit in e2fsprogs: > > > e1f7100643a46456be107b33098f6034b0835e6d > > > > > > Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com> > > > --- > > > scripts/lib/wic/partition.py | 11 +++++++++++ > > > 1 file changed, 11 insertions(+) > > > > > > diff --git a/scripts/lib/wic/partition.py > > > b/scripts/lib/wic/partition.py > > > index e50871b8..90b2c037 100644 > > > --- a/scripts/lib/wic/partition.py > > > +++ b/scripts/lib/wic/partition.py > > > @@ -280,6 +280,17 @@ class Partition(): > > > > > > extraopts = self.mkfs_extraopts or "-F -i 8192" > > > > > > + if os.getenv('SOURCE_DATE_EPOCH'): > > > + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) > > > + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % > > > (sde_time, > > > pseudo) > > > + > > > + # Set hash_seed to generate deterministic directory > > > indexes > > > + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9- > > > 2f2fdf33d460") > > > + if self.fsuuid: > > > + namespace = uuid.UUID(self.fsuuid) > > > + hash_seed = str(uuid.uuid5(namespace, > > > str(sde_time))) > > > + extraopts += " -E hash_seed=%s" % hash_seed > > > + > > > > Hi, while reworking the SDE in ISAR, I stumbled upon this as well. > > This patch only covers the .wic part, but we need a similar patch > > for > > the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but > > I > > was not able to make the .ext4 build reproducible (the diff > > indicated > > that the inodes are still shuffled around). This makes me wonder if > > mke2fs.ext4 even supports producing a reproducible rootfs. > > > > Have you been able to create a bit-by-bit identical .wic of an ext4 > > partition? > > > > Best regards, > > Felix > > > > > label_str = "" > > > if self.label: > > > label_str = "-L %s" % self.label > > > > -- > > Siemens AG, Technology > > Linux Expert Center > > > >
Hi Felix, To answer your question, No, these patches alone are not enough to make ext4 filesystem images reproducible. These patches were mainly tested with isar-cip-core security images which has /home ad /var mounted as ext4 filesystem images. With these patches I noticed the /home ext4 partition is reproducible but /var required an extra fix (https://groups.google.com/g/isar-users/c/RsjRjzigLOE) But, I see reproducibility problems when entire rootfs is deployed in an ext4 filesystem (in the case of wic and with IMAGE_CMD:ext4). I even raised this point in the ISAR ML (https://groups.google.com/g/isar-users/c/Ll7t4G41Lfo) That’s when I saw your previous mail. Thanks and Regards, Adithya Balakumar From: Adithya Balakumar <adithya190298@gmail.com> Sent: Monday, July 8, 2024 4:28 PM To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com> Subject: Fwd: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images ---------- Forwarded message --------- From: MOESSBAUER, Felix <felix.moessbauer@siemens.com<mailto:felix.moessbauer@siemens.com>> Date: Mon, 8 Jul, 2024, 16:13 Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images To: isar-users@googlegroups.com<mailto:isar-users@googlegroups.com> <isar-users@googlegroups.com<mailto:isar-users@googlegroups.com>>, adithya190298@gmail.com<mailto:adithya190298@gmail.com> <adithya190298@gmail.com<mailto:adithya190298@gmail.com>> On Fri, 2024-07-05 at 05:13 -0700, Adithya Balakumar wrote: > Hi Felix, > > I saw your mail regarding your attempt to make ext4 filesystem images > from IMAGE_CMD:ext4 reproducible. > If you don't mind, could you briefly explain what was the problem you > faced in achieving this? Hi, the problem is stated below: > the diff indicated that the inodes are still shuffled around). This makes me wonder if mke2fs.ext4 even supports producing a reproducible rootfs. I just copied the pattern from wic, but for whatever reason the inodes still were not deterministic. Are you sure, that this patch is sufficient to make the ext4 reproducible? Felix > I am also trying to understand on how to achieve the same. > > Thanks and Regards, > Adithya Balakumar > > > On Tuesday, April 23, 2024 at 2:47:11 PM UTC+5:30 MOESSBAUER, Felix > wrote: > > On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com<mailto:venkat...@toshiba-tsip.com> > > wrote: > > > From: venkata pyla <venkat...@toshiba-tsip.com<mailto:venkat...@toshiba-tsip.com>> > > > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file > > > system. > > > hash_seed: creates reproducible directory indexes in the file > > > system. > > > > > > Reference commit in e2fsprogs: > > > e1f7100643a46456be107b33098f6034b0835e6d > > > > > > Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com<mailto:venkat...@toshiba-tsip.com>> > > > --- > > > scripts/lib/wic/partition.py | 11 +++++++++++ > > > 1 file changed, 11 insertions(+) > > > > > > diff --git a/scripts/lib/wic/partition.py > > > b/scripts/lib/wic/partition.py > > > index e50871b8..90b2c037 100644 > > > --- a/scripts/lib/wic/partition.py > > > +++ b/scripts/lib/wic/partition.py > > > @@ -280,6 +280,17 @@ class Partition(): > > > > > > extraopts = self.mkfs_extraopts or "-F -i 8192" > > > > > > + if os.getenv('SOURCE_DATE_EPOCH'): > > > + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) > > > + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % > > > (sde_time, > > > pseudo) > > > + > > > + # Set hash_seed to generate deterministic directory > > > indexes > > > + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9- > > > 2f2fdf33d460") > > > + if self.fsuuid: > > > + namespace = uuid.UUID(self.fsuuid) > > > + hash_seed = str(uuid.uuid5(namespace, > > > str(sde_time))) > > > + extraopts += " -E hash_seed=%s" % hash_seed > > > + > > > > Hi, while reworking the SDE in ISAR, I stumbled upon this as well. > > This patch only covers the .wic part, but we need a similar patch > > for > > the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but > > I > > was not able to make the .ext4 build reproducible (the diff > > indicated > > that the inodes are still shuffled around). This makes me wonder if > > mke2fs.ext4 even supports producing a reproducible rootfs. > > > > Have you been able to create a bit-by-bit identical .wic of an ext4 > > partition? > > > > Best regards, > > Felix > > > > > label_str = "" > > > if self.label: > > > label_str = "-L %s" % self.label > > > > -- > > Siemens AG, Technology > > Linux Expert Center > > > > -- Siemens AG, Technology Linux Expert Center
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index e50871b8..90b2c037 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -280,6 +280,17 @@ class Partition(): extraopts = self.mkfs_extraopts or "-F -i 8192" + if os.getenv('SOURCE_DATE_EPOCH'): + sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) + pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, pseudo) + + # Set hash_seed to generate deterministic directory indexes + namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460") + if self.fsuuid: + namespace = uuid.UUID(self.fsuuid) + hash_seed = str(uuid.uuid5(namespace, str(sde_time))) + extraopts += " -E hash_seed=%s" % hash_seed + label_str = "" if self.label: label_str = "-L %s" % self.label