From patchwork Wed Apr 3 11:33:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 3472 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 03 Apr 2024 13:33:36 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-oi1-f190.google.com (mail-oi1-f190.google.com [209.85.167.190]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 433BXYAO023499 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 3 Apr 2024 13:33:34 +0200 Received: by mail-oi1-f190.google.com with SMTP id 5614622812f47-3c3bc0de8f8sf4574260b6e.3 for ; Wed, 03 Apr 2024 04:33:34 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712144008; cv=pass; d=google.com; s=arc-20160816; b=u6PQLmII6PVhLGUfxXLHz/u4hedxfBUq7kASW8hx6Dajfi7ACUJ0I8oVLesJ37gFHn F3CAMbcVj/p8/h/7eH6eyJLz0a/ArWrTnBH+/+LXIPh8jhHFlAQ5tu2bZGA3kcFYGpF/ L+80/6xlKyJanEzGqk0BDtIPCnhhYkD9yHQx4SW4Ff3D2yPUwDYeKCKhsXkd05UaSAY1 QUcSc7owp++KDyfGGueXh6KfgVR0gI4BCEkF0wK0A8xpeAGCVkmQA8iNMXH3wgUwAJJL y20p6l4OEyKEpdGcia9tMFA61uqikjAuUWPongaoUMo0XdWMnN6XD5aKk42Pboat7O42 fP2A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=Pea41gKZDZlzOcq2ZYcfVStNorAsk931L/ANRC6RzPs=; fh=8x6rVF4Zqnmk9Wo2e8q4PdmvJbDhZlEkbcRQgNSAZtY=; b=pvS4/kAd1VLvtGpxIZaufYxZFapm+ecXsM90obfSLQ12DV5KuuleJcyAwObWfabMYf cOTAaMCU0fodPoqruchDB9baZyKsLIWL5acGJdP+slIL6DlsIr3Kez78bTg/CGn20fFa rAcCGjEfNAP/OBMsiXwiBStXeNaRldDD7WWxpgDg3KGAqUW7zg/eSf6LlcdLAc0Td5DH zATI8alEr4QrcyYn8gmmE1B89gGL1kevLUGMUg/X4WigQ58V1JkndYtd8lSQVv+cnBC5 FT9GRLxmU7bfp77L8yHXhnCNPJHbKYafeDadIzcSNFrkchfy74tXkwHAFn/gPK6GxGA+ bezA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=kJnzafQq; spf=pass (google.com: domain of fm-1321639-20240403113324f382b7e4d6db6c350e-6tehyk@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20240403113324f382b7e4d6db6c350e-6tEHYK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1712144008; x=1712748808; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Pea41gKZDZlzOcq2ZYcfVStNorAsk931L/ANRC6RzPs=; b=ExjJnurwtcT+l9h/VTOXpEwOyMWUqyWLY2uFxdXetFmEgzL1UX3ODUOfxJPYzFcLoN nT3og5aThKcCPfT7MWjV3rZ/vmbq4b1dWbKu0FhYnVZf7Q5nqW+jPPxUHVqS4cJyrBUR yHPbMnJcenwJJgOrGy+lk+v1VQrDOGPxeyx72AlAS9yqi+d+20RskinnSVmmJMsCFg3x C1Pfx0QLoJMAw1e2BkQEdLx2wrwUleEpq/U88Y8lNHovMDBS8iEbst6z8xWLEgoKv9uM OZ4Sn1aOJrThgh2tvY+xTRA2hr9Gpg6EmrceTENtJjD8+GUDODTeYLtKpC6hf50CxQXp ZXkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712144008; x=1712748808; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Pea41gKZDZlzOcq2ZYcfVStNorAsk931L/ANRC6RzPs=; b=n4YH8sIsFEHU2WkmxdqgqKvPtBFZwK4czXai3UI7rI0XPbkdbCZCR7LSLaf/qwH6xV hTz2iBO25qURVlb22xxu5uXN0gkY9i/jNPOOAi7N5pZRR0V4ZrTWW7rMjJws8/4JQgzq o306rDgTODqGCjqo4qj5cx9N5JXURCHnX1auyHaXcLBiVE/VYdz4vUVv+22Soqd/UK8Y RLLAsQ/CoFEEwRQmTqv4clQ2CJxc1rfMRyn5oz2tD/bqHw59nV/0Fnzmcp+tz6W/GRhl 0BxXDD2Cey06zokvW0G+ugkONSAD8iBlTvsLLplGtswllgzzQPCJ2+hYG0XI9QaA7K07 t7Ow== X-Forwarded-Encrypted: i=2; AJvYcCUF3ysK0dD04MGMDVtdFMFdL/2rP1AfrZW4lWqcyDk/5GiMtfnQLcrkRsXYIk/lAekujD+x0KJnoyr25FBRsFqvfIVIUYs= X-Gm-Message-State: AOJu0YxEMFu/BynSqzUhReB6XXYmbMl0DqHm3lHb9lnKUQ1YF4A0mC4N K4wSf72MEcrQTgBepL1b46PCRSCwrOBgRJQpMW+emQZBYebwvI8v X-Google-Smtp-Source: AGHT+IGKWBf37z/dE8IuTh7qmHhcAvcDXHILE7l2ulJWNQUhaVzXrYDHVxV8RwKL7XMwVEgafHhyuQ== X-Received: by 2002:a05:6808:11c6:b0:3c5:d3fc:d8ae with SMTP id p6-20020a05680811c600b003c5d3fcd8aemr785665oiv.52.1712144007230; Wed, 03 Apr 2024 04:33:27 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac8:5d92:0:b0:431:3419:79d3 with SMTP id d18-20020ac85d92000000b00431341979d3ls8405810qtx.0.-pod-prod-01-us; Wed, 03 Apr 2024 04:33:26 -0700 (PDT) X-Received: by 2002:a05:622a:60c:b0:432:bad8:ba70 with SMTP id z12-20020a05622a060c00b00432bad8ba70mr16335370qta.32.1712144005878; Wed, 03 Apr 2024 04:33:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712144005; cv=none; d=google.com; s=arc-20160816; b=uP/LXqPz/J9rEL1ZpEHplhrxoNB6zMUn15fUap1KPe2ARBgCmXJD9jd4d2VVj6svEo Dma62oZMNzRBqyEZODAVCkKk/6FinXfHE4kjLO3jn6AkhG9t+BkVVwGjtbeEPPBvTxgH 3GxGZLcEY0LmBP34Jj4aL5pcjVhUGcs6e/3PHmMEWUh1E8aaqTYLGJoz8qaI4y/Y25iK 86iwZGEHJJ04jAIzCFoRwpeYxLOE9G5xI19DkyuuNdtRw8VHw50LHJrU1r7f99reYHWe YTH4wu52MyTf0CMM3VIyRUmDbgTTwmfqve+7AZc07OqxAN40Bxq77/1Jt7KgVa1aiyib Iv9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=qOhpFwKRAwtV47eKubkbS54J351QNBrku6i2nS0TnDc=; fh=bVX03jNaRhKZGJkxT8AB1TU04tIU5gqw6O8qRk+YAyc=; b=tgNtTAam5X8VMJhDRV+7AyZJo7T3eK+i8x0e5NQKGZthT7/HtlOMbDy7BBbLm8sz/b fbQks4YVxPcfm4swXQ+lXjtNzVShR5KBW27hKKvmVYG6AIF7Xm+ITzqYICmE+T3acYbe IEMLiFMfvHAYbHQq+qaJl7cgVqcnWlTMwiW7PWBcpvtv21Un7sEDP1eKsHDg+IOri101 u00COLD2EQyvcrYCsWbXry3GjDLZFcdAGDul3Ob30uINBeTDtHb/r9Nogsrw1J2tn48j NmMclAn9UzAzl95JDS8VbaY/EU+6Iy/QTpndBnWGo6rCW4ylUO5yAghOp3lfFZnMZeOP DtkQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=kJnzafQq; spf=pass (google.com: domain of fm-1321639-20240403113324f382b7e4d6db6c350e-6tehyk@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20240403113324f382b7e4d6db6c350e-6tEHYK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id eq25-20020a05622a5e1900b0043140b1ef8fsi1180513qtb.3.2024.04.03.04.33.25 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2024 04:33:25 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20240403113324f382b7e4d6db6c350e-6tehyk@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20240403113324f382b7e4d6db6c350e for ; Wed, 03 Apr 2024 13:33:24 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, venkata.pyla@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp, dinesh.kumar@toshiba-tsip.com, Felix Moessbauer Subject: [PATCH 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Date: Wed, 3 Apr 2024 13:33:10 +0200 Message-Id: <20240403113310.135008-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=kJnzafQq; spf=pass (google.com: domain of fm-1321639-20240403113324f382b7e4d6db6c350e-6tehyk@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20240403113324f382b7e4d6db6c350e-6tEHYK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= In case the SOURCE_DATE_EPOCH variable is set, we switch the debian mirror to a snapshot mirror. The used date is derived from the value of SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is only injected temporarily during the build. To further control the behavior, we introduce the following variables: - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to snapshot-cloudflare.debian.org) Signed-off-by: Felix Moessbauer --- Dear maintainers, I'm not quite sure if the introduced variables also need to be added to the vardeps of e.g. the bootstrap task. Please double check this. Best regards, Felix Moessbauer Siemens AG RECIPE-API-CHANGELOG.md | 6 ++++++ doc/user_manual.md | 3 +++ meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 +++++++++ 3 files changed, 18 insertions(+) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 6653ab43..c146d60c 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. To simplify downstream kernel module builds, we automatically turn of cross compilation for a user-provided module when building it for a distro kernel. + +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set + +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot +mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`. +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. diff --git a/doc/user_manual.md b/doc/user_manual.md index 419d5339..227ce5f9 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -425,12 +425,15 @@ Some other variables include: - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information. - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number. + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional. - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional. - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional. - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional. + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`. - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index f548e202..1e5a2911 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -32,6 +32,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" FILESEXTRAPATHS:append = ":${BBPATH}" +# reproducible builds +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}" +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" inherit deb-dl-dir @@ -111,9 +114,15 @@ def parse_aptsources_list_line(source_list_line): def get_apt_source_mirror(d, aptsources_entry_list): import re + import time if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" + elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time()) + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) + premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" mirror_list = [entry.split()