From patchwork Wed Apr 3 14:12:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 3473 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 03 Apr 2024 16:12:54 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-pg1-f188.google.com (mail-pg1-f188.google.com [209.85.215.188]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 433ECq1i024207 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 3 Apr 2024 16:12:53 +0200 Received: by mail-pg1-f188.google.com with SMTP id 41be03b00d2f7-5cfc2041cdfsf4344776a12.2 for ; Wed, 03 Apr 2024 07:12:53 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712153567; cv=pass; d=google.com; s=arc-20160816; b=Xds0g56myck8ELBluGb7F/CfkgF0UKbWT/OSHsayBfdfe8vGy4HeA6cgrpgQmRgpKP 2qavnAZpOMp0Ur+YBWNb/uRA1NOn7x/iPlDutpzwV5BIOAwx3ykjQtceIsqCL74NqZ/Y WdEgaB2/3e/hIrwBBJTac60cYece23iXVq3y6H4aDqrQ1E0CZk00VGXlRzgDrI+w7U39 lzLcIzl3WFssvsAoeLFR08++uY4ceb/ffArem+8cvDWMFuI1LqEnvLQehUpurMw/C4EW acvEMdJOhddJfh9nWrqRVwFqsBRPFZNNCaW3XEUlANj5ljFyeqQS43XFjf/o3ObGU9OA QjuQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=4oT8YbxDz/HK31fzra+B/Pl5poJFLcODoax/urrUORE=; fh=Czg7lpfF8Us39pgcemC9L+wlDMt69iJ7VC9K4U07Axg=; b=S6rr4KXcVIN5N4OpHSaTCGFuTzjAzxdw0fXSM+JxepGRGJYzuEL0YFb+q9yMWVuvKF y0qYhGkTFI50tdCUpL4S2pxpJnTSNdQONAd/GqwRmxuvdblhWdDXdQh6BGAUagHspVN6 78twhhUu5QqE9jD2BbxgzU3qWLveWRobydXNwbgbczRlWFXwqK16ObpC574xR7lMGsm2 kUGjJsZuP9Gq2H0nDuAvidagRbQGfIyRB4BrOjgSCPdaHu2ldeqHDfJJ4obQfivEVS/M e+ePUkN1ta6KHAPycePUFG76IqoTtoz3ZssPgPUvSucY46ln5d/q3dfFnVklMIzkIB8r txaw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=DSfPr5d6; spf=pass (google.com: domain of fm-1321639-20240403141242d117168b20d1b8cbf5-tgd4ku@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-20240403141242d117168b20d1b8cbf5-Tgd4Ku@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1712153567; x=1712758367; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=4oT8YbxDz/HK31fzra+B/Pl5poJFLcODoax/urrUORE=; b=RidrQOziI7jm4VeHZotNGafum6tCQ4d35RV2jDMfagt4T4GbojdzUTKNKEl46Jo+2r YK9RQvdU9RzH40thVY+Hz4JhcLe7WsDdjhg2okcPdX9q7UpHSHGqMLj5eUGL7voLaabn 8e9W+72WJ1TpqGblZkwaqPU6UfaOqS7SB21N34Q0v22DhxMt9wm5LfGe9rUWzFDPo/Mm qwooaPVhXr//shaU2rzT9G7yMvTABHnGt929d1b6WZ0PUU2JiOL+mLSXTGhnffZdYqS7 1NHwF0Lg6IDEhDONJtKWFMOAq7xICCNqLxyWDHmn82mgjrEKVMCyQZ/viIePTbujt4YP yoUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712153567; x=1712758367; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=4oT8YbxDz/HK31fzra+B/Pl5poJFLcODoax/urrUORE=; b=r/AdD+l23LVQwo/le3vmVXSgO1CEW8zCOAfMs/bM7rywEWULq/zgwHYzOtvxBwRq2j 6dIoIfTwIm+g7+rOPegsBqOC7xzlkkPuoLh8kYPWU+KiKhG0+WoydxKXaZfkQ32RWa5I OCUzsvdPRuLDyTDIuO30QOZatSqwyqDKeOQX531T+JeH2RY0KaSzoPYucQ5EQR24NM+R 8VWirONL7C4CpIfH4mwzBxA1uBZwNSmaL8tpgAaTCzmV+rAfF4kV3k+f4clOq7IfRT5W /fYw2P1H91dWpbwoYv6DqqQ1620NfD5GgwpcpusnHMFAqIqfb07VWjk9o1tWh2H48rUt zJIg== X-Forwarded-Encrypted: i=2; AJvYcCWUDTX+fxuAi2EBTJxnhbbOqCQ/EEzK+35UAq6mKGtmM1dbhGsVM7pP26oHIHQeyv7Jl66IifyFQvOdUbu+MTSauvvAViM= X-Gm-Message-State: AOJu0YwIPSBNx4QviZXDVX0vZqChCjAIe07N9HMQXtLAipegtxjvdZva A2ZKSHg65j9Lg3u3ZqhMCihjJs3NyL3FWaz2Jj0cPaHmPKOa+eSe X-Google-Smtp-Source: AGHT+IGn5b119CEJvU9OW8yHtAT5BJ6xEhVlPIawJPVe+FV+CtvT8LUdpeLzNilFi720/IR8sZn49Q== X-Received: by 2002:a17:90a:43c4:b0:2a2:672f:ef66 with SMTP id r62-20020a17090a43c400b002a2672fef66mr3637481pjg.7.1712153566713; Wed, 03 Apr 2024 07:12:46 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90a:d410:b0:2a0:4f1a:6a0c with SMTP id r16-20020a17090ad41000b002a04f1a6a0cls3681239pju.2.-pod-prod-04-us; Wed, 03 Apr 2024 07:12:45 -0700 (PDT) X-Received: by 2002:a05:6a21:1f17:b0:1a3:3db3:48c4 with SMTP id ry23-20020a056a211f1700b001a33db348c4mr11024248pzb.18.1712153565162; Wed, 03 Apr 2024 07:12:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712153565; cv=none; d=google.com; s=arc-20160816; b=H27quJx3OCBEFxcyNoxuXINgncSCLzOQTlzbbd4lmT+qMCh3bY1nXwfxGPQ8q+PlvC /kld5VQt7QqI9Z133VADpvx/B+ykCINzhwViKCdL1P6Zfi/1J9u8kM4PNExfaCZBh2Ds zzrKcsVGZfBYzIBZrNtt+8esRwqkpeaYMwLHDaSNm8GWfH1qEPPHFwgY4f+KZkwIHjHu Lae/6flmrKsIUCa90oz9yrIYViRXtEo+Io6HPzVa3NFsW7aOZI2Hnwi13A6+H8n0FGen vz8yMLns2c9WNHCvTDqu+LZoUg4u7xjxyAlG27FDrNdDB6PNb+133bNWcxZKURMZ3QZp C6+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=PEwwPHw4NS4yjC0k4/2ZlqVxGxEH5YrV+QPBoEmgimU=; fh=bVX03jNaRhKZGJkxT8AB1TU04tIU5gqw6O8qRk+YAyc=; b=VXYCJnDoGrJtZ5iotSaOjbQf6XsNmvWkCFJMqi91/7z78BdcwDnT2zreFTXxB8hCjt QDUuHf/4lH4KS3eH/AsYof1blRLRyoAdhGokaLd2lOWYPY0iY47BW1vRtuSk4DQC8k/1 oGwr2jAp6+SdqwHZX2n1Vk/cq5uiffoMpRTqRZaFBdScMGwTDLshqF15InFvXeEvfECA aA/c6c0Y8sFiBvAsCwnoTF6ckoW9lkf3yLlQ746oDVJHgQB+RnmuWdMq/NOqc63reyzW ze1bLPLufsSaO+k2G8FkknfjNpoIR+369WoEpslVeWaQrCokiOd6w1QuAZNsGClg+sK2 2cTg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=DSfPr5d6; spf=pass (google.com: domain of fm-1321639-20240403141242d117168b20d1b8cbf5-tgd4ku@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-20240403141242d117168b20d1b8cbf5-Tgd4Ku@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net. [185.136.64.227]) by gmr-mx.google.com with ESMTPS id q21-20020a056a00085500b006eac41e9673si762745pfk.2.2024.04.03.07.12.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2024 07:12:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20240403141242d117168b20d1b8cbf5-tgd4ku@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) client-ip=185.136.64.227; Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 20240403141242d117168b20d1b8cbf5 for ; Wed, 03 Apr 2024 16:12:42 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, venkata.pyla@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp, dinesh.kumar@toshiba-tsip.com, Felix Moessbauer Subject: [PATCH v2 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Date: Wed, 3 Apr 2024 16:12:31 +0200 Message-Id: <20240403141231.179832-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=DSfPr5d6; spf=pass (google.com: domain of fm-1321639-20240403141242d117168b20d1b8cbf5-tgd4ku@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-20240403141242d117168b20d1b8cbf5-Tgd4Ku@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= In case the SOURCE_DATE_EPOCH variable is set, we switch the debian mirror to a snapshot mirror. The used date is derived from the value of SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is only injected temporarily during the build. To further control the behavior, we introduce the following variables: - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to snapshot-cloudflare.debian.org) Signed-off-by: Felix Moessbauer --- Changes since v1: - disable valid-until checking for both bootstrapping and schroot - conditionally make the SOURCE_DATE_EPOCH a vardep of bootstrap - move ISAR_USE_DEBIAN_SNAPSHOTS to the bitbake.conf as it is used both in bootstrap, as well as in rootfs. Best regards, Felix Moessbauer Siemens AG RECIPE-API-CHANGELOG.md | 6 ++++++ doc/user_manual.md | 3 +++ meta/classes/rootfs.bbclass | 3 +++ meta/conf/bitbake.conf | 1 + .../isar-bootstrap/isar-bootstrap.inc | 17 +++++++++++++++-- 5 files changed, 28 insertions(+), 2 deletions(-) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 6653ab43..c146d60c 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. To simplify downstream kernel module builds, we automatically turn of cross compilation for a user-provided module when building it for a distro kernel. + +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set + +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot +mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`. +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. diff --git a/doc/user_manual.md b/doc/user_manual.md index 419d5339..227ce5f9 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -425,12 +425,15 @@ Some other variables include: - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information. - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number. + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional. - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional. - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional. - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional. + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`. - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index fb14f3ca..0d7744f7 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -112,6 +112,9 @@ rootfs_configure_apt() { mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' { echo 'Acquire::Retries "3";' + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then + echo 'Acquire::Check-Valid-Until "false";' + fi echo 'APT::Install-Recommends "0";' echo 'APT::Install-Suggests "0";' } > '${ROOTFSDIR}/etc/apt/apt.conf.d/50isar' diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 91c5c815..84c9a9bb 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -68,6 +68,7 @@ KERNEL_FILE ?= "vmlinuz" KERNEL_FILE:mipsel ?= "vmlinux" KERNEL_FILE:riscv64 ?= "vmlinux" KERNEL_FILE:arm64 ?= "vmlinux" +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}" MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index f548e202..d44876a4 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -32,6 +32,8 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" FILESEXTRAPATHS:append = ":${BBPATH}" +# reproducible builds +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" inherit deb-dl-dir @@ -111,9 +113,15 @@ def parse_aptsources_list_line(source_list_line): def get_apt_source_mirror(d, aptsources_entry_list): import re + import time if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" + elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time()) + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) + premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" mirror_list = [entry.split() @@ -265,6 +273,7 @@ def get_host_release(): return rel do_bootstrap[vardeps] += " \ + ${@'SOURCE_DATE_EPOCH' if bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')) else ''} \ DISTRO_APT_PREMIRRORS \ ISAR_ENABLE_COMPAT_ARCH \ ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ @@ -382,8 +391,12 @@ do_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any + + APT_UPDATE_OPTS="-o APT::Update::Error-Mode=any" + if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then + APT_UPDATE_OPTS="${APT_UPDATE_OPTS} -o Acquire::Check-Valid-Until=false" + fi + chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y ${APT_UPDATE_OPTS} chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ -o Debug::pkgProblemResolver=yes