From patchwork Tue Apr 9 15:55:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 3488 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 09 Apr 2024 17:56:13 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-pj1-f60.google.com (mail-pj1-f60.google.com [209.85.216.60]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 439FuBeH022954 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 9 Apr 2024 17:56:12 +0200 Received: by mail-pj1-f60.google.com with SMTP id 98e67ed59e1d1-2a2c2b0d82asf5199846a91.3 for ; Tue, 09 Apr 2024 08:56:12 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712678165; cv=pass; d=google.com; s=arc-20160816; b=B+mbH3SXXcwoBJODbamMK66c8NbePOhe26QpGPAoT+oLaaLsQHassLxq7JNR3dkAT+ gQh0BTNvhixdV0PJ3KDCr2OpmleYmQPQmx9mD9wZ0cskmWzj5ECuW5iFdKOGQkjB7o9l OBlwkGOjmHe7euG2OJIpLS3yb4g3J033xeFNxtdZrlS4EUP4zHTy9ATFIaIMlr6DdKAT Va3Jr3GhxCmYHlYPEvvE8tN7gxTNPtb+90v6Ds1dkDXvxCaH0U4kyX5Wx024n7DxXjPy UdExkW/9KBlpw9D05VTTfIkgzG2jsOLb2pZURMUgawNDgCXLqO+5p952/jvdpCKucL5j zrHw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=0IWWmF4q/vlKzzC6OiXmIj1nrKq9AqoRdWAEa32xC3o=; fh=z70l4yF9oGbrYh4Q4jHu+xSbANNdMEW34xb4IWwS1qk=; b=A9cILA62sjYThIS6d0VprIRMoMS8gZXEbcx0+e26xlvrpe7faNlf64WvKj6t7LlpVG WGcqj0URaEkBKIRwkuq4syCzTbpDq0Mp2M/m/ikp4l+oE4j05C06Z6ckY4hf+cX/Ehzy PfnZUBhFkUINO1ivYy74H9nspe/gSLWITRmv+om59eqqshY23iT7tZ34Zgbh6cdDROtx S4LC0CFDloZW496rCyN3/EghsdlSiHAm/ztthqaZR5mSoiXFa/KXOkmu9vp9NRXnwsZE rU4CahwigPhh3RFvU4IGsdck5a86Hm0+tSQ8AxaPFlrXHT2X4+GRMUce34VNkEesKkYT HFkQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=J+cL1W+A; spf=pass (google.com: domain of fm-1321639-20240409155601abb2507ec2bc3d7b23-x_jmhs@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1321639-20240409155601abb2507ec2bc3d7b23-X_jMHs@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1712678165; x=1713282965; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0IWWmF4q/vlKzzC6OiXmIj1nrKq9AqoRdWAEa32xC3o=; b=nUHSccij2ZDVEPG5KYtczRyQ4dGDCCMGvZweCiM7uqf3GhniHKI6Ez79RZDzzTHzks 9P8QNGAnP1n5ccgV0W82FlDYfmtFYJ3tB5RE46nL/X7dT3l90aISXXGICKMjKDN8jDpt 1jLQXl0TkS7e1GVJG2gWncfhwjjrSlWJLv+DIiy3sapTAoxu+zUzcoUOAZpeFhmlR074 0LlsCcrYiRw9pfm1BD+Dpk/sEoAbZo3JWmg/j0rC6CZRG5GcAQIXAlp2g/8Mlpu9Tgd1 4tkeRvxYOTtpD/OFUhBpRgV8tB+wP+JpLYiRJSUzHzq97rjzxBihXBQJbHiy+FNqZ7Xf v45A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712678165; x=1713282965; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-beenthere:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=0IWWmF4q/vlKzzC6OiXmIj1nrKq9AqoRdWAEa32xC3o=; b=woaXghlw4dZM9XcDNLGWishGXgNP8YDPWKMILToD3qz1aG7VCHlaGP5UErTHF5Ih0c AHIG9CvmSp72Owt3JA2JPPVr1WCbtW0MBBKUuvZCTDp1kaHJn0ZImnYA+9x7UVA8zjFV uT5Dx9tnTfapCtSKUzLQh5RGwSfDmeanSQqUcI3OwhY1Bh5gfGB008tKv2g6UOW6Q6SB qV3LxnMksubCS/QWQ5WE1mJpVz8Ige0DnAkO1bJgoS4JQgQsf2wG3eAExDORmtARv4Gd sumk9sCbTypiH5Q5osZ4i7FeLdf2TflUpQlMEYHZJKKsXkhqXzxpnEFRSOezCgSq+mQh dhoQ== X-Forwarded-Encrypted: i=2; AJvYcCXvou8UChpbVxjSWqa8Hoe3Xq/wVRtF7gjxIuOM0Tp0O6sIXjBlkE9AXlvZgaeVvpmKaK9EXfaiYi9cz2UsBJxIjI6QDMQ= X-Gm-Message-State: AOJu0Yz0eKjQfVHekV1vIYTXkI0l552zAn2MGoUFX6tOpeYQH7oWGsVr C82J8iYwUU6cHta1K18jXXslWNMTPBHALS9rd/LWDu09VXuT58bR X-Google-Smtp-Source: AGHT+IHnDvN3CXo6Wn4r+TpuxGwyLq41sAR0SbmmLggPGM1T+DnXVdH0EGattOYIzAj9MCIZLKq4MA== X-Received: by 2002:a17:90a:c16:b0:2a3:be59:e969 with SMTP id 22-20020a17090a0c1600b002a3be59e969mr15152pjs.47.1712678165132; Tue, 09 Apr 2024 08:56:05 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90a:9cc:b0:2a0:9806:220f with SMTP id 70-20020a17090a09cc00b002a09806220fls2938197pjo.0.-pod-prod-09-us; Tue, 09 Apr 2024 08:56:04 -0700 (PDT) X-Received: by 2002:a17:90a:d493:b0:2a3:4544:ea80 with SMTP id s19-20020a17090ad49300b002a34544ea80mr47123pju.5.1712678163735; Tue, 09 Apr 2024 08:56:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712678163; cv=none; d=google.com; s=arc-20160816; b=BbFyWQgW9mAfoZl5WHgeuh30bsbvh1+O/a/HrUzgwstf3pV9nXI2Y2qIqgWpe5pVo9 yO2rNfDLD6QThQVC106oAXv2L84AuS7CO1qNGGxqbNUPondngbqWQgMccrTcsouWaD0/ 3Ib61PYJJyqFe88GrDueTQj6OWk8R2v21RwywIOGxeMjt6bRY7B79W0Ibukc1zg+sFaw UNoBnV/1QGAMoGzkhbbt9dH5t4DR6MZU9p1+sKQW9mEGo4dwLFjBQC+aAxgtDgmJPkC+ ffNzHY/1NHHDDk6ucvcst59XMgbGhQUzdbzLPlpyA4cxljul3PyrvuRjysrbt7JwYOI8 K+Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=QRZZOm4OI7CdXbKlI70a6zKsDBS65rwqhT4WNgY1gv0=; fh=WkhL8kaJc+l2wQon1t06Ej3uvBGj9sVhNcE8PaS/XbI=; b=p+CANZqzf8g9aqvbY5cjzLFXrZ3QZaOQqbMV/PS5YHkLUTdB2R6PqsICDIY3QFSVm6 Zho4/2XAfZhFPJuXBjXkWu36k3Axvp+qkmRWDZNi9Li8ZKzzRp4bVB2zCntd+4PwWk+A 5OnWtRMOdtciApuTI4FLZr8G2NpP9ENNotIYsg19+Cj4pnKd42878CEadzjGtrYvEi+m r00Xbtu/2e6qkmBGj1wWAX6b34LWKs3f5Y47f30uH4CQH90zDUNUGClsRuNPyCrl5WJ/ IgFVX0Xl/FidzXRQHa6THAYmACZWhx44mL9ppWi94sDGgNz7GCZxGBUEJSw2hwfmvMBS HNKA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=J+cL1W+A; spf=pass (google.com: domain of fm-1321639-20240409155601abb2507ec2bc3d7b23-x_jmhs@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1321639-20240409155601abb2507ec2bc3d7b23-X_jMHs@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id w12-20020a17090a8a0c00b002a290bec184si518118pjn.2.2024.04.09.08.56.03 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2024 08:56:03 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20240409155601abb2507ec2bc3d7b23-x_jmhs@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 20240409155601abb2507ec2bc3d7b23 for ; Tue, 09 Apr 2024 17:56:01 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, quirin.gylstorff@siemens.com, Felix Moessbauer Subject: [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set Date: Tue, 9 Apr 2024 17:55:46 +0200 Message-Id: <20240409155549.826454-3-felix.moessbauer@siemens.com> In-Reply-To: <20240409155549.826454-1-felix.moessbauer@siemens.com> References: <20240409155549.826454-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=J+cL1W+A; spf=pass (google.com: domain of fm-1321639-20240409155601abb2507ec2bc3d7b23-x_jmhs@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1321639-20240409155601abb2507ec2bc3d7b23-X_jMHs@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch adds infrastructure to switch the apt sources to a frozen snapshot mirror. To build against a mirror, set ISAR_USE_APT_SNAPSHOT=1. As the mirror is distro specific, it is configured in the distro config for all supported distros (currently only debian and ubuntu). For unsupported distros, a meaningful error message is emitted on enabling the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this mirror is only injected temporarily and does not end up in the final apt sources list. To further control the behavior, we introduce the following variables: - DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax identical to DISTRO_APT_PREMIRRORS. - ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This is automatically derived from the SOURCE_DATE_EPOCH if not set. Signed-off-by: Felix Moessbauer --- RECIPE-API-CHANGELOG.md | 6 ++++++ doc/user_manual.md | 3 +++ meta-isar/conf/distro/ubuntu-common.inc | 3 +++ meta/conf/bitbake.conf | 3 +++ meta/conf/distro/debian-common.conf | 3 +++ .../isar-bootstrap/isar-bootstrap.inc | 16 ++++++++++++++++ 6 files changed, 34 insertions(+) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 6653ab43..e6861523 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. To simplify downstream kernel module builds, we automatically turn of cross compilation for a user-provided module when building it for a distro kernel. + +### Build against debian snapshot mirror + +To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`. +The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually +pre-defined in the distro config. diff --git a/doc/user_manual.md b/doc/user_manual.md index 419d5339..70741968 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -431,6 +431,9 @@ Some other variables include: - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. + - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional. + - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros. + - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc index 9d8a843b..54bb747a 100644 --- a/meta-isar/conf/distro/ubuntu-common.inc +++ b/meta-isar/conf/distro/ubuntu-common.inc @@ -32,3 +32,6 @@ IMAGE_PREINSTALL += "init" IMAGE_PREINSTALL += "initramfs-tools" IMAGER_INSTALL:wic += "python3-distutils" + +# snapshot mirror for reproducible builds +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 1da3ecac..4cfa8b10 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -145,6 +145,9 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}" # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE # ISAR: set value to date of latest release SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251" +# Debian snapshots +ISAR_USE_APT_SNAPSHOT ??= "0" +ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}" # Default parallelism and resource usage for xz XZ_MEMLIMIT ?= "50%" diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf index 1e1dfc83..db538510 100644 --- a/meta/conf/distro/debian-common.conf +++ b/meta/conf/distro/debian-common.conf @@ -39,3 +39,6 @@ SYSTEMD_BOOTLOADER_INSTALL:sid = "systemd-boot-efi:${DISTRO_ARCH}" COMPAT_DISTRO_ARCH:amd64 = "i386" COMPAT_DISTRO_ARCH:arm64 = "armhf" + +# snapshot mirror for reproducible builds +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot-cloudflare.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n" \ No newline at end of file diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 17f19fd8..733a23df 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -30,6 +30,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" FILESEXTRAPATHS:append = ":${BBPATH}" +# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT +ISAR_APT_SNAPSHOT_MIRROR ??= "" +APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}" inherit deb-dl-dir @@ -107,11 +110,21 @@ def parse_aptsources_list_line(source_list_line): return [type, options, source, suite, components] +def get_apt_snapshot_date(d): + import time + source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP') + return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) + def get_apt_source_mirror(d, aptsources_entry_list): import re + # this is executed during parsing. No error checking possible + use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) + snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR') if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" + elif use_snapshot and snapshot_mirror: + premirrors = snapshot_mirror else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" mirror_list = [entry.split() @@ -126,6 +139,8 @@ def get_apt_source_mirror(d, aptsources_entry_list): new_aptsources_entry_list[2] = re.sub(regex, replace, aptsources_entry_list[2], count = 1) + if use_snapshot: + new_aptsources_entry_list[1] = "[check-valid-until=no]" return new_aptsources_entry_list return aptsources_entry_list @@ -240,6 +255,7 @@ do_apt_config_prepare[vardeps] += " \ APTSRCS \ ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ DEPLOY_ISAR_BOOTSTRAP \ + ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \ " python do_apt_config_prepare() { apt_preferences_out = d.getVar("APTPREFS")