[v3,1/1] image.bbclass: fix non-reproducible file time-stamps inside rootfs

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

As part of reproducible-build work, the rootfs images generated on same
source should be identical between two builds.

In this commit it tries to solve one of the non-reproducible problem
i.e. the rootfs file time-stamps generated during build time are not
reproducible, it uses one of the solution provided in the debian
live-build image project (refer [1]), it fixes by finding all the
files/folders that are gernerated newly and set the time-stamp provided
by `SOURCE_DATE_EPOCH` environment variable.

[1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 meta-isar/conf/local.conf.sample | 10 ++++++++++
 meta/classes/image.bbclass       | 10 ++++++++++
 2 files changed, 20 insertions(+)

On Thu, 2023-01-05 at 20:42 +0530, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> As part of reproducible-build work, the rootfs images generated on
> same
> source should be identical between two builds.
> 
> In this commit it tries to solve one of the non-reproducible problem
> i.e. the rootfs file time-stamps generated during build time are not
> reproducible, it uses one of the solution provided in the debian
> live-build image project (refer [1]), it fixes by finding all the
> files/folders that are gernerated newly and set the time-stamp
> provided
> by `SOURCE_DATE_EPOCH` environment variable.
> 
> [1] 
> https://salsa.debian.org/live-team/live-build/-/merge_requests/218
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  meta-isar/conf/local.conf.sample | 10 ++++++++++
>  meta/classes/image.bbclass       | 10 ++++++++++
>  2 files changed, 20 insertions(+)
> 
> diff --git a/meta-isar/conf/local.conf.sample b/meta-
> isar/conf/local.conf.sample
> index 57d0620..3c4a473 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password"
>  #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
>  # Enable ccache debug mode
>  #CCACHE_DEBUG = "1"
> +
> +# Uncommnet and add value to it to build images reproducibly
> +#
> +# The value for `SOURCE_DATE_EPOCH` should be latest source change
> time in
> +# seconds since the Epoch.
> +# Git repository users can use value from 'git log -1 --pretty=%ct'
> +# Non git repository users can use value from 'stat -c%Y ChangeLog'
> +# To know more details about this variable and how to set the value
> refer below
> +# https://reproducible-builds.org/docs/source-date-epoch/
> +#SOURCE_DATE_EPOCH =
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 813e1f3..8371ecd 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -431,6 +431,16 @@ do_rootfs_finalize() {
>  
>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>  EOSUDO
> +
> +    # Set same time-stamps to the newly generated file/folders in
> the
> +    # rootfs image for the purpose of reproducible builds.
> +    test ! -z "${SOURCE_DATE_EPOCH}" && \
> +        sudo find ${ROOTFSDIR} -newermt \
> +            "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
> +            -printf "%y %p\n" \
> +            -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >
> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \
> +            bbwarn "$(cat
> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file
> timestamps to build image reproducibly"
> +

Hi, I just tested this code and found the following issues:

This does not rebuild cleanly, as in general the do_rootfs_finalize
cannot be re-executed. IMHO this is an ISAR bug and not a problem of
your patch:

do_rootfs_finalize
mv: cannot stat '/build/tmp/work/debian-bookworm-amd64/img-<...>/1.0-
r0/rootfs/etc/apt/sources-list': No such file or directory

Second, it is a bit hard to use with automatic data from git.
I solved it the following way, which works so far:

SOURCE_DATE_EPOCH := "${@ bb.process.run('git -C ${LAYERDIR_project}
log -1 --pretty=\%ct')[0].strip() }"

One problem here is, that this can hardly be generalized as we have to
feed in the path of the main layer - which is not known by ISAR.

The list of touched files is quite long, but basically most of it falls
in the following classes:

- PKG info: /var/lib/dpkg/info/
- Pycache: /usr/lib/python3.10/curses/__pycache__ This anyways needs to
be further investigated as my understanding is that we do not want to
distribute the pycache
- directories and symlinks:  These always have to be fixed as the
creation date depends on the install order of packages. I would not
warn on these

Best regards,
Felix

>  }
>  addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
>  
> -- 
> 2.20.1
> 
>

>-----Original Message-----
>From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf
>Of Moessbauer, Felix
>Sent: 06 January 2023 15:15
>To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
><Venkata.Pyla@toshiba-tsip.com>
>Cc: amikan@ilbers.de; Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(
>林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(Ｔ
>ＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; Schild, Henning
><henning.schild@siemens.com>
>Subject: Re: [PATCH v3 1/1] image.bbclass: fix non-reproducible file time-stamps
>inside rootfs
>
>On Thu, 2023-01-05 at 20:42 +0530, venkata.pyla@toshiba-tsip.com wrote:
>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>
>> As part of reproducible-build work, the rootfs images generated on
>> same source should be identical between two builds.
>>
>> In this commit it tries to solve one of the non-reproducible problem
>> i.e. the rootfs file time-stamps generated during build time are not
>> reproducible, it uses one of the solution provided in the debian
>> live-build image project (refer [1]), it fixes by finding all the
>> files/folders that are gernerated newly and set the time-stamp
>> provided by `SOURCE_DATE_EPOCH` environment variable.
>>
>> [1]
>> https://salsa.debian.org/live-team/live-build/-/merge_requests/218
>>
>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> ---
>>  meta-isar/conf/local.conf.sample | 10 ++++++++++
>>  meta/classes/image.bbclass       | 10 ++++++++++
>>  2 files changed, 20 insertions(+)
>>
>> diff --git a/meta-isar/conf/local.conf.sample b/meta-
>> isar/conf/local.conf.sample index 57d0620..3c4a473 100644
>> --- a/meta-isar/conf/local.conf.sample
>> +++ b/meta-isar/conf/local.conf.sample
>> @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password"
>>  #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
>>  # Enable ccache debug mode
>>  #CCACHE_DEBUG = "1"
>> +
>> +# Uncommnet and add value to it to build images reproducibly # # The
>> +value for `SOURCE_DATE_EPOCH` should be latest source change
>> time in
>> +# seconds since the Epoch.
>> +# Git repository users can use value from 'git log -1 --pretty=%ct'
>> +# Non git repository users can use value from 'stat -c%Y ChangeLog'
>> +# To know more details about this variable and how to set the value
>> refer below
>> +# https://reproducible-builds.org/docs/source-date-epoch/
>> +#SOURCE_DATE_EPOCH =
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 813e1f3..8371ecd 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -431,6 +431,16 @@ do_rootfs_finalize() {
>>
>>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>>  EOSUDO
>> +
>> +    # Set same time-stamps to the newly generated file/folders in
>> the
>> +    # rootfs image for the purpose of reproducible builds.
>> +    test ! -z "${SOURCE_DATE_EPOCH}" && \
>> +        sudo find ${ROOTFSDIR} -newermt \
>> +            "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
>> +            -printf "%y %p\n" \
>> +            -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >
>> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \
>> +            bbwarn "$(cat
>> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file
>> timestamps to build image reproducibly"
>> +
>
>Hi, I just tested this code and found the following issues:
>
>This does not rebuild cleanly, as in general the do_rootfs_finalize cannot be re-
>executed. IMHO this is an ISAR bug and not a problem of your patch:
>
>do_rootfs_finalize
>mv: cannot stat '/build/tmp/work/debian-bookworm-amd64/img-<...>/1.0-
>r0/rootfs/etc/apt/sources-list': No such file or directory

Yes, I also faced this problem, but in my case for reproducible build test I have to delete the build/tmp and build/sstate-cache, so this issue is not seen.

>
>Second, it is a bit hard to use with automatic data from git.
>I solved it the following way, which works so far:
>
>SOURCE_DATE_EPOCH := "${@ bb.process.run('git -C ${LAYERDIR_project} log -1 -
>-pretty=\%ct')[0].strip() }"
>
>One problem here is, that this can hardly be generalized as we have to feed in
>the path of the main layer - which is not known by ISAR.

Your solution is good, but as you mentioned this will be difficult get the layer path, I think it is better document in the user_manual.md and let the main layer defines this.

>
>The list of touched files is quite long, but basically most of it falls in the
>following classes:
>
>- PKG info: /var/lib/dpkg/info/
>- Pycache: /usr/lib/python3.10/curses/__pycache__ This anyways needs to be
>further investigated as my understanding is that we do not want to distribute
>the pycache
I think this should be deleted instead modifying the time, this may be generated while executing python scripts part of package installation (postinst script)

>- directories and symlinks:  These always have to be fixed as the creation date
>depends on the install order of packages. I would not warn on these
Filtering out will be little complex, but let me try.

>
>Best regards,
>Felix
>
>>  }
>>  addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
>>
>> --
>> 2.20.1
>>
>>
>
>--
>You received this message because you are subscribed to the Google Groups
>"isar-users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email
>to isar-users+unsubscribe@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/isar-
>users/074dd25bf429e532368b0c93ed8ec908bafb7941.camel%40siemens.com.