| Message ID | 20240403141231.179832-1-felix.moessbauer@siemens.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | [v2,1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set | expand |
On 03.04.24 16:12, Felix Moessbauer wrote: > In case the SOURCE_DATE_EPOCH variable is set, we switch the debian > mirror to a snapshot mirror. The used date is derived from the value of > SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is > only injected temporarily during the build. > > To further control the behavior, we introduce the following variables: > > - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used > - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to > snapshot-cloudflare.debian.org) > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > Changes since v1: > > - disable valid-until checking for both bootstrapping and schroot > - conditionally make the SOURCE_DATE_EPOCH a vardep of bootstrap > - move ISAR_USE_DEBIAN_SNAPSHOTS to the bitbake.conf as it is used > both in bootstrap, as well as in rootfs. > > Best regards, > Felix Moessbauer > Siemens AG > > RECIPE-API-CHANGELOG.md | 6 ++++++ > doc/user_manual.md | 3 +++ > meta/classes/rootfs.bbclass | 3 +++ > meta/conf/bitbake.conf | 1 + > .../isar-bootstrap/isar-bootstrap.inc | 17 +++++++++++++++-- > 5 files changed, 28 insertions(+), 2 deletions(-) > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index 6653ab43..c146d60c 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. > To simplify downstream kernel module builds, we automatically turn of cross > compilation for a user-provided module when building it for a distro kernel. > > + > +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set > + > +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot > +mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`. > +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 419d5339..227ce5f9 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -425,12 +425,15 @@ Some other variables include: > > - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information. > - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number. > + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional. > - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional. > - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional. > - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional. > - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. > - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. > - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. > + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional. > + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`. > - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. > - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. > - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > index fb14f3ca..0d7744f7 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -112,6 +112,9 @@ rootfs_configure_apt() { > mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' > { > echo 'Acquire::Retries "3";' > + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then > + echo 'Acquire::Check-Valid-Until "false";' > + fi > echo 'APT::Install-Recommends "0";' > echo 'APT::Install-Suggests "0";' > } > '${ROOTFSDIR}/etc/apt/apt.conf.d/50isar' > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 91c5c815..84c9a9bb 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -68,6 +68,7 @@ KERNEL_FILE ?= "vmlinuz" > KERNEL_FILE:mipsel ?= "vmlinux" > KERNEL_FILE:riscv64 ?= "vmlinux" > KERNEL_FILE:arm64 ?= "vmlinux" > +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}" > > MACHINEOVERRIDES ?= "${MACHINE}" > DISTROOVERRIDES ?= "${DISTRO}" > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index f548e202..d44876a4 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -32,6 +32,8 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR > BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" > BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" > FILESEXTRAPATHS:append = ":${BBPATH}" > +# reproducible builds > +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" > > inherit deb-dl-dir > > @@ -111,9 +113,15 @@ def parse_aptsources_list_line(source_list_line): > > def get_apt_source_mirror(d, aptsources_entry_list): > import re > + import time > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" > + elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): > + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') > + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time()) > + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) > + premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) > else: > premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" > mirror_list = [entry.split() > @@ -265,6 +273,7 @@ def get_host_release(): > return rel > > do_bootstrap[vardeps] += " \ > + ${@'SOURCE_DATE_EPOCH' if bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')) else ''} \ > DISTRO_APT_PREMIRRORS \ > ISAR_ENABLE_COMPAT_ARCH \ > ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ > @@ -382,8 +391,12 @@ do_bootstrap() { > chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} > fi > > - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ > - -o APT::Update::Error-Mode=any > + > + APT_UPDATE_OPTS="-o APT::Update::Error-Mode=any" > + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then > + APT_UPDATE_OPTS="${APT_UPDATE_OPTS} -o Acquire::Check-Valid-Until=false" > + fi > + chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y ${APT_UPDATE_OPTS} > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > -o Debug::pkgProblemResolver=yes This can be a valuable addition but it needs more thoughts first: How do you plan to handle snapshot.ubuntu.com? And what will you when the repo list contains a third-party repo for which there is no snapshot? That should at least raise a warning, rather than silently suggesting to produce a reproducible image. BTW, we should likely also double-check how we define the SOURCE_DATE_EPOCH value, in comparison to OE. See my reply on your kas patch. Jan
On Mon, 2024-04-08 at 07:02 +0200, Jan Kiszka wrote: > On 03.04.24 16:12, Felix Moessbauer wrote: > > In case the SOURCE_DATE_EPOCH variable is set, we switch the debian > > mirror to a snapshot mirror. The used date is derived from the > > value of > > SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this > > mirror is > > only injected temporarily during the build. > > > > To further control the behavior, we introduce the following > > variables: > > > > - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used > > - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults > > to > > snapshot-cloudflare.debian.org) > > > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > > --- > > Changes since v1: > > > > - disable valid-until checking for both bootstrapping and schroot > > - conditionally make the SOURCE_DATE_EPOCH a vardep of bootstrap > > - move ISAR_USE_DEBIAN_SNAPSHOTS to the bitbake.conf as it is used > > both in bootstrap, as well as in rootfs. > > > > Best regards, > > Felix Moessbauer > > Siemens AG > > > > RECIPE-API-CHANGELOG.md | 6 ++++++ > > doc/user_manual.md | 3 +++ > > meta/classes/rootfs.bbclass | 3 +++ > > meta/conf/bitbake.conf | 1 + > > .../isar-bootstrap/isar-bootstrap.inc | 17 > > +++++++++++++++-- > > 5 files changed, 28 insertions(+), 2 deletions(-) > > > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > > index 6653ab43..c146d60c 100644 > > --- a/RECIPE-API-CHANGELOG.md > > +++ b/RECIPE-API-CHANGELOG.md > > @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro > > kernels is not supported in debian. > > To simplify downstream kernel module builds, we automatically turn > > of cross > > compilation for a user-provided module when building it for a > > distro kernel. > > > > + > > +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is > > set > > + > > +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian > > snapshot > > +mirror is used. This can be overwritten with > > `ISAR_USE_DEBIAN_SNAPSHOTS`. > > +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. > > diff --git a/doc/user_manual.md b/doc/user_manual.md > > index 419d5339..227ce5f9 100644 > > --- a/doc/user_manual.md > > +++ b/doc/user_manual.md > > @@ -425,12 +425,15 @@ Some other variables include: > > > > - `IMAGE_INSTALL` - The list of custom packages to build and > > install to target image, please refer to relative chapter for more > > information. > > - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be > > run in parallel. Please set this option according to your host CPU > > cores number. > > + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling > > to make the results reproducible. This variable is optional. > > - `HOST_DISTRO` - The distro to use for SDK root filesystem. This > > variable is optional. > > - `HOST_ARCH` - The Debian architecture of SDK root filesystem > > (e.g., `amd64`). By default set to current Debian host > > architecture. This variable is optional. > > - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK > > root filesystem. This variable is optional. > > - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files > > for SDK root filesystem. This variable is optional. > > - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to > > DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt > > bootstrap repo for the host. > > - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to > > the default URI in the format `ftp.debian.org my.preferred.mirror`. > > This variable is optional. PREMIRRORS will be used only for the > > build. The final images will have the sources list as mentioned in > > DISTRO_APT_SOURCES. > > + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot > > instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is > > set. Optional. > > + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. > > Defaults to `snapshot-cloudflare.debian.org`. > > - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify > > apt repos for apt installation after bootstrapping. > > - `FILESEXTRAPATHS` - The default directories BitBake uses when > > it processes recipes are initially defined by the FILESPATH > > variable. You can extend FILESPATH variable by using > > FILESEXTRAPATHS. > > - `FILESOVERRIDES` - A subset of OVERRIDES used by the build > > system for creating FILESPATH. The FILESOVERRIDES variable uses > > overrides to automatically extend the FILESPATH variable. > > diff --git a/meta/classes/rootfs.bbclass > > b/meta/classes/rootfs.bbclass > > index fb14f3ca..0d7744f7 100644 > > --- a/meta/classes/rootfs.bbclass > > +++ b/meta/classes/rootfs.bbclass > > @@ -112,6 +112,9 @@ rootfs_configure_apt() { > > mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' > > { > > echo 'Acquire::Retries "3";' > > + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then > > + echo 'Acquire::Check-Valid-Until "false";' > > + fi > > echo 'APT::Install-Recommends "0";' > > echo 'APT::Install-Suggests "0";' > > } > '${ROOTFSDIR}/etc/apt/apt.conf.d/50isar' > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > > index 91c5c815..84c9a9bb 100644 > > --- a/meta/conf/bitbake.conf > > +++ b/meta/conf/bitbake.conf > > @@ -68,6 +68,7 @@ KERNEL_FILE ?= "vmlinuz" > > KERNEL_FILE:mipsel ?= "vmlinux" > > KERNEL_FILE:riscv64 ?= "vmlinux" > > KERNEL_FILE:arm64 ?= "vmlinux" > > +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if > > d.getVar('SOURCE_DATE_EPOCH') else '0'}" > > > > MACHINEOVERRIDES ?= "${MACHINE}" > > DISTROOVERRIDES ?= "${DISTRO}" > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > index f548e202..d44876a4 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -32,6 +32,8 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if > > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR > > BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if > > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else > > 'DISTRO')}" > > BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if > > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else > > 'BASE_DISTRO')}" > > FILESEXTRAPATHS:append = ":${BBPATH}" > > +# reproducible builds > > +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" > > > > inherit deb-dl-dir > > > > @@ -111,9 +113,15 @@ def > > parse_aptsources_list_line(source_list_line): > > > > def get_apt_source_mirror(d, aptsources_entry_list): > > import re > > + import time > > > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > > premirrors = "\S* > > file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" > > + elif > > bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): > > + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') > > + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or > > int(time.time()) > > + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', > > time.gmtime(int(source_date_epoch))) > > + premirrors = 'deb.debian.org/(.*) > > {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) > > else: > > premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" > > mirror_list = [entry.split() > > @@ -265,6 +273,7 @@ def get_host_release(): > > return rel > > > > do_bootstrap[vardeps] += " \ > > + ${@'SOURCE_DATE_EPOCH' if > > bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')) else ''} > > \ > > DISTRO_APT_PREMIRRORS \ > > ISAR_ENABLE_COMPAT_ARCH \ > > ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ > > @@ -382,8 +391,12 @@ do_bootstrap() { > > chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture > > ${COMPAT_DISTRO_ARCH} > > fi > > > > - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ > > - -o APT::Update::Error-Mode=any > > + > > + APT_UPDATE_OPTS="-o APT::Update::Error-Mode=any" > > + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then > > + APT_UPDATE_OPTS="${APT_UPDATE_OPTS} -o Acquire::Check- > > Valid-Until=false" > > + fi > > + chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > > ${APT_UPDATE_OPTS} > > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > > -o Debug::pkgProblemResolver=yes > > This can be a valuable addition but it needs more thoughts first: > > How do you plan to handle snapshot.ubuntu.com? And what will you when > the repo list contains a third-party repo for which there is no Good point. How about moving the ISAR_DEBIAN_SNAPSHOT_MIRROR to the distro.conf? > snapshot? That should at least raise a warning, rather than silently > suggesting to produce a reproducible image. Hmm... I don't want to raise a warning in this case, as this can very well be a stable source. We simply can't tell by just looking at the sources. Even having a stable mirror does not mean we can reproduce the image: It is just another piece in the big puzzle to make it finally work. It is currently accepted (across various projects, including docker buildx), that reproducibility is something that needs to be checked manually and by that is stated by a user. > > BTW, we should likely also double-check how we define the > SOURCE_DATE_EPOCH value, in comparison to OE. See my reply on your > kas > patch. Right. According to [1]->Current Development->Timestamps, [2], the SDE is injected into the build as an environment variable. However, this can and should be overwritten on a per-recipe basis by using the bitbake variable SOURCE_DATE_EPOCH. Given that, the best way to inject is via env + addition to BB_ENV_PASSTHROUGH_ADDITIONS. CC'ing Quirin, as we now have a similar discussion on the CIP ML as well. Felix [1] https://wiki.yoctoproject.org/wiki/Reproducible_Builds [2] https://github.com/openembedded/openembedded-core/blob/486ed69939f1b7572f121960c2dde246032686e7/meta/conf/bitbake.conf#L686 > > Jan >
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 6653ab43..c146d60c 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. To simplify downstream kernel module builds, we automatically turn of cross compilation for a user-provided module when building it for a distro kernel. + +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set + +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot +mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`. +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. diff --git a/doc/user_manual.md b/doc/user_manual.md index 419d5339..227ce5f9 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -425,12 +425,15 @@ Some other variables include: - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information. - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number. + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional. - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional. - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional. - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional. + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`. - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index fb14f3ca..0d7744f7 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -112,6 +112,9 @@ rootfs_configure_apt() { mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' { echo 'Acquire::Retries "3";' + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then + echo 'Acquire::Check-Valid-Until "false";' + fi echo 'APT::Install-Recommends "0";' echo 'APT::Install-Suggests "0";' } > '${ROOTFSDIR}/etc/apt/apt.conf.d/50isar' diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 91c5c815..84c9a9bb 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -68,6 +68,7 @@ KERNEL_FILE ?= "vmlinuz" KERNEL_FILE:mipsel ?= "vmlinux" KERNEL_FILE:riscv64 ?= "vmlinux" KERNEL_FILE:arm64 ?= "vmlinux" +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}" MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index f548e202..d44876a4 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -32,6 +32,8 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" FILESEXTRAPATHS:append = ":${BBPATH}" +# reproducible builds +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" inherit deb-dl-dir @@ -111,9 +113,15 @@ def parse_aptsources_list_line(source_list_line): def get_apt_source_mirror(d, aptsources_entry_list): import re + import time if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" + elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time()) + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) + premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" mirror_list = [entry.split() @@ -265,6 +273,7 @@ def get_host_release(): return rel do_bootstrap[vardeps] += " \ + ${@'SOURCE_DATE_EPOCH' if bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')) else ''} \ DISTRO_APT_PREMIRRORS \ ISAR_ENABLE_COMPAT_ARCH \ ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ @@ -382,8 +391,12 @@ do_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any + + APT_UPDATE_OPTS="-o APT::Update::Error-Mode=any" + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then + APT_UPDATE_OPTS="${APT_UPDATE_OPTS} -o Acquire::Check-Valid-Until=false" + fi + chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y ${APT_UPDATE_OPTS} chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ -o Debug::pkgProblemResolver=yes
In case the SOURCE_DATE_EPOCH variable is set, we switch the debian mirror to a snapshot mirror. The used date is derived from the value of SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is only injected temporarily during the build. To further control the behavior, we introduce the following variables: - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to snapshot-cloudflare.debian.org) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- Changes since v1: - disable valid-until checking for both bootstrapping and schroot - conditionally make the SOURCE_DATE_EPOCH a vardep of bootstrap - move ISAR_USE_DEBIAN_SNAPSHOTS to the bitbake.conf as it is used both in bootstrap, as well as in rootfs. Best regards, Felix Moessbauer Siemens AG RECIPE-API-CHANGELOG.md | 6 ++++++ doc/user_manual.md | 3 +++ meta/classes/rootfs.bbclass | 3 +++ meta/conf/bitbake.conf | 1 + .../isar-bootstrap/isar-bootstrap.inc | 17 +++++++++++++++-- 5 files changed, 28 insertions(+), 2 deletions(-)