| Message ID | 20240403113310.135008-1-felix.moessbauer@siemens.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | [1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set | expand |
On Wed, 2024-04-03 at 13:33 +0200, Felix Moessbauer wrote: > In case the SOURCE_DATE_EPOCH variable is set, we switch the debian > mirror to a snapshot mirror. The used date is derived from the value > of > SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror > is > only injected temporarily during the build. > > To further control the behavior, we introduce the following > variables: > > - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used > - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults > to > snapshot-cloudflare.debian.org) I just noticed, that we also need to disable the valid-until check. Will fix that in a v2. Best regards, Felix > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > Dear maintainers, > > I'm not quite sure if the introduced variables also need to be added > to the vardeps of e.g. the bootstrap task. Please double check this. > > Best regards, > Felix Moessbauer > Siemens AG > > RECIPE-API-CHANGELOG.md | 6 ++++++ > doc/user_manual.md | 3 +++ > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 +++++++++ > 3 files changed, 18 insertions(+) > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index 6653ab43..c146d60c 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels > is not supported in debian. > To simplify downstream kernel module builds, we automatically turn > of cross > compilation for a user-provided module when building it for a distro > kernel. > > + > +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set > + > +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian > snapshot > +mirror is used. This can be overwritten with > `ISAR_USE_DEBIAN_SNAPSHOTS`. > +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 419d5339..227ce5f9 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -425,12 +425,15 @@ Some other variables include: > > - `IMAGE_INSTALL` - The list of custom packages to build and > install to target image, please refer to relative chapter for more > information. > - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be > run in parallel. Please set this option according to your host CPU > cores number. > + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to > make the results reproducible. This variable is optional. > - `HOST_DISTRO` - The distro to use for SDK root filesystem. This > variable is optional. > - `HOST_ARCH` - The Debian architecture of SDK root filesystem > (e.g., `amd64`). By default set to current Debian host architecture. > This variable is optional. > - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root > filesystem. This variable is optional. > - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for > SDK root filesystem. This variable is optional. > - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to > DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt > bootstrap repo for the host. > - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the > default URI in the format `ftp.debian.org my.preferred.mirror`. This > variable is optional. PREMIRRORS will be used only for the build. The > final images will have the sources list as mentioned in > DISTRO_APT_SOURCES. > + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot > instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is > set. Optional. > + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. > Defaults to `snapshot-cloudflare.debian.org`. > - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt > repos for apt installation after bootstrapping. > - `FILESEXTRAPATHS` - The default directories BitBake uses when it > processes recipes are initially defined by the FILESPATH variable. > You can extend FILESPATH variable by using FILESEXTRAPATHS. > - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system > for creating FILESPATH. The FILESOVERRIDES variable uses overrides to > automatically extend the FILESPATH variable. > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index f548e202..1e5a2911 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -32,6 +32,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR > BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" > BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else > 'BASE_DISTRO')}" > FILESEXTRAPATHS:append = ":${BBPATH}" > +# reproducible builds > +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if > d.getVar('SOURCE_DATE_EPOCH') else '0'}" > +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" > > inherit deb-dl-dir > > @@ -111,9 +114,15 @@ def > parse_aptsources_list_line(source_list_line): > > def get_apt_source_mirror(d, aptsources_entry_list): > import re > + import time > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > premirrors = "\S* > file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" > + elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): > + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') > + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or > int(time.time()) > + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', > time.gmtime(int(source_date_epoch))) > + premirrors = 'deb.debian.org/(.*) > {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) > else: > premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" > mirror_list = [entry.split()
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 6653ab43..c146d60c 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. To simplify downstream kernel module builds, we automatically turn of cross compilation for a user-provided module when building it for a distro kernel. + +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set + +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot +mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`. +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. diff --git a/doc/user_manual.md b/doc/user_manual.md index 419d5339..227ce5f9 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -425,12 +425,15 @@ Some other variables include: - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information. - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number. + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional. - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional. - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional. - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional. + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`. - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index f548e202..1e5a2911 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -32,6 +32,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" FILESEXTRAPATHS:append = ":${BBPATH}" +# reproducible builds +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}" +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" inherit deb-dl-dir @@ -111,9 +114,15 @@ def parse_aptsources_list_line(source_list_line): def get_apt_source_mirror(d, aptsources_entry_list): import re + import time if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" + elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time()) + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) + premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" mirror_list = [entry.split()
In case the SOURCE_DATE_EPOCH variable is set, we switch the debian mirror to a snapshot mirror. The used date is derived from the value of SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is only injected temporarily during the build. To further control the behavior, we introduce the following variables: - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to snapshot-cloudflare.debian.org) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- Dear maintainers, I'm not quite sure if the introduced variables also need to be added to the vardeps of e.g. the bootstrap task. Please double check this. Best regards, Felix Moessbauer Siemens AG RECIPE-API-CHANGELOG.md | 6 ++++++ doc/user_manual.md | 3 +++ meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 +++++++++ 3 files changed, 18 insertions(+)