From patchwork Fri Jan 13 17:47:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2490 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 13 Jan 2023 18:47:53 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-pl1-f185.google.com (mail-pl1-f185.google.com [209.85.214.185]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 30DHlpcr030324 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 13 Jan 2023 18:47:52 +0100 Received: by mail-pl1-f185.google.com with SMTP id u6-20020a170903124600b00188cd4769bcsf15131081plh.0; Fri, 13 Jan 2023 09:47:52 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673632065; cv=pass; d=google.com; s=arc-20160816; b=K7E/eIwdeJIOXPK9nvQcvbhMEZbjVD+5nSKa0MicnS8CaFgLIyqmqmRvVw50Cy5+Z0 JtMjv8Pr+e1BRupocrHbTDtqTTBNAjOUCgr9K41L0LCf+OByjlt5USbCk2igLC+I2h2J zhLd6LF/XyHlL///EH28HlwiF7qa5ojCxZo4glksx+ROBKaA6btOt+IYP9KPJFMN77FB azE3+nINqqmb+bafhYOF9Rt36VTL+2JhQDVOM6Tc1a2v6fCRkN4ZfrfnXqPNfmmUs4PV A1s8dFH/K/ApQYda58yeERlakWGBr5qAB8LtjypEA0tV8h7/7tT1QenPTr0fvNqc8fTr D6WQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=Ov11d0QQzhWge2Kzx7SydNU1A4zZhgDHH7dfkExrb9Y=; b=Uoaxdcmlmxt5eKSo4iE3Et3OuHDrtsdWulrivs01wtMWuiRIZhnFsVb8EFAtH5j7Nr TVhA1aHnbhCVmvhS7dZ8Ylf+jR/TnQv76QtL9dZMuZ8qBl7OcMtlnKZrGSbWBw0rdcxA mow5pToN4HkDAG8evyYdkM4XsKkl78KBYtEps6xMyWqNPiul3xlCQD+oNDJ/F0pdCSv4 dvgC36m1O0eB1Yr6Utjyyv0v7HGpLDTEBdEi+EfjgkJlraNP4Ns3yAtWHSwr7gVETNjP akyvuLD0UHSXYzLBjlTPq6Y64krMBTNvPy8YlTcsYp2S1pKcX8qy7NV8lnuprb3gf09U ngfw== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::164 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=Ov11d0QQzhWge2Kzx7SydNU1A4zZhgDHH7dfkExrb9Y=; b=EW4nKdnQJHCucT/24zWEyydMw3q1NFoznBdFaGRqSp9cq8LaOAtrhTvkFVOFzG+iJu x2TLF16QfHQbs2b3KeC31FOB1uTMxGgdMiKa+K+whUQGvgdueCTGC0Ri7cjQRMV9PUec KXxWbVxVnvTCfJuniwMhfai3VGV9xfvnnC9C3TsBVs7k3ISp5IoLtQO2Yb3+Sf+kYWZ1 +eUNwfnH8S7zKauRosdnNip4vvoNgVOhBaA1zpHad0Lda5QtJ8R8OtdE5TFPPKGzOGjl qOA2xKEtzBPAS0nsL35KXaqcrxKqX0jCox7u8tC38NycgjX+GLLylzmYdZ7fTXMK+Ryt 9Xog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=Ov11d0QQzhWge2Kzx7SydNU1A4zZhgDHH7dfkExrb9Y=; b=7a4+8/DgBc19oxAC8KiZ2i/XK2rU4Y6xAY+uf0UwMjUJ9mmiRb6F4hv4tfgZv+qmON br3IdOFLiWX/Ibw/Sz2vLgGahMXaOL0q3m+91IUpxFS2jPCmSpyLw4mYdyKZEfY5zT2M vA8PatSJybbpMYnUHVFt1K5zTmGak5g/i55+HX4Wa74/eorQgarH4rjF6y7ElBXUnLCm JBLUlzu7gRD+uX7b4NuIX/ZjWSuBHGzLlNbyz5G0TN7uw8O13nqS3SqG6thjZWdzFijx eIvgsGbmxQp2GrLvgtprk0DE/JvDap7eea0r2/cvNbs8W+E07Wr3jyG77lHQdrTzjPYM v1EA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AFqh2kruNqeGfMZ2pNbxKvAzRI13Ol61LfyCwA4CELlKQVxg2vksA6q6 e40MhonlANDbrOsOQTYOElg= X-Google-Smtp-Source: AMrXdXvyVqjOewWINUrhtv+8xxerU03kKUXxPPkkGF1KSNHgCLk8AAMqrwPcyGHwpiA5Z4ALnc/1rg== X-Received: by 2002:a17:903:2612:b0:186:9c2b:3a39 with SMTP id jd18-20020a170903261200b001869c2b3a39mr34067plb.115.1673632065631; Fri, 13 Jan 2023 09:47:45 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90a:a50d:b0:226:42ea:28cc with SMTP id a13-20020a17090aa50d00b0022642ea28ccls5018522pjq.0.-pod-preprod-gmail; Fri, 13 Jan 2023 09:47:44 -0800 (PST) X-Received: by 2002:a17:902:d50e:b0:191:4378:ec06 with SMTP id b14-20020a170902d50e00b001914378ec06mr14963020plg.61.1673632064689; Fri, 13 Jan 2023 09:47:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673632064; cv=none; d=google.com; s=arc-20160816; b=jjELuJp/xVXhj7zbay/3lcK5hHrSS5epZmBcKDK95bF8kqwoYmlR7t305UXoiJMylG eieDb8tyxvri5P2WmR4YINUqBPUbtUSlZCpPqFkiok5/3+ips2nw0LGJymEN6KxO+K15 Pb6y/RVEG+BMJ87JTOblRVJZOnu3WyzA5mrP3L7FO2U6eCseK0INVHCEIQNMcHJHbR/F SGysm6BqDNX3tw0iEpiouprrdRAUL80WvN9NmdNsvG3DOMgHh5lE8wtdxz4KOschkLED JU8dbouLoE7dcX3x8t8D9KXa/xC5ZUKuph+6Gj+vTbBjRFTP7HGLeAgtLgUKt/OrsFNN 5Gew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=fvDd9ccOF6PdU+Gu9JGMDKWUUMLpFBh2nCm0zj00lLc=; b=ORlfktIr8iGCWVTEdBcAK6Ww/cvByx9sMk6Nd5wOmJ8zR/8ST9KaRgA3RkeLEC9AWm ntALnMtquylr4k4ffVHe/1UW5UTJ/rTNQYsPxT8UOfI+SLXfRMhGVHtRT5gEnJq6XadG aw4MPZsuuqt/Gdq/BJ7H6huZ22xYRaVDCDYtpiRo2Tqb/e4WoS2JSnhCdGyD5lBZOdSA z9a5K6xf9vUwK6S4lKTY79fRRFnqEG8z5HuLyNbDUxLQGf9/2yN4K1jAlw6KoXY1PoXg T2yJs8vV+2p7uwLNA10fQ8HxC20LVGq6Xg0bV3mHvMDaVhH6Lobw0j49kfJzfv+bkeo1 xTpA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::164 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from relay03.th.seeweb.it (relay03.th.seeweb.it. [2001:4b7a:2000:18::164]) by gmr-mx.google.com with ESMTPS id d17-20020a170902f15100b00188c5696675si2338076plb.6.2023.01.13.09.47.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 09:47:44 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::164 as permitted sender) client-ip=2001:4b7a:2000:18::164; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:cd05:a8a7:67a9:5f16]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id DDB3E1F8A1; Fri, 13 Jan 2023 18:47:41 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v5] suggested changes for reproducibility patchset v5 Date: Fri, 13 Jan 2023 18:47:37 +0100 Message-Id: <20230113174737.281104-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::164 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" suggested changes for reproducibility patchset WARNING: eval-image-1.0-r0 do_rootfs_finalize: modified timestamp (1673628837) of 3 files for image reproducibly List of files modified could be found here: ./build/tmp/deploy/images/debx86/files.modified_timestamps v.2: rebased on current ilbers:next v.3: new script added: wic-extract-rootfs-partition.sh [image.wic] v.4: example with for epoch generation from git v.5: reverted the example and rework some few code Signed-off-by: Roberto A. Foglietta --- meta-isar/conf/local.conf.sample | 2 +- meta/classes/image-account-extension.bbclass | 6 +-- meta/classes/image.bbclass | 21 ++++---- meta/classes/initramfs.bbclass | 4 +- wic-extract-rootfs-partition.sh | 52 ++++++++++++++++++++ 5 files changed, 70 insertions(+), 15 deletions(-) create mode 100755 wic-extract-rootfs-partition.sh diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample index 6208623..1d7e178 100644 --- a/meta-isar/conf/local.conf.sample +++ b/meta-isar/conf/local.conf.sample @@ -257,4 +257,4 @@ USER_isar[flags] += "clear-text-password" # Non git repository users can use value from 'stat -c%Y ChangeLog' # To know more details about this variable and how to set the value refer below # https://reproducible-builds.org/docs/source-date-epoch/ -#SOURCE_DATE_EPOCH = +#SOURCE_DATE_EPOCH = "" diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass index bb173b1..1d49054 100644 --- a/meta/classes/image-account-extension.bbclass +++ b/meta/classes/image-account-extension.bbclass @@ -256,11 +256,11 @@ image_postprocess_accounts() { # chpasswd adds a random salt when running against a clear-text password. # For reproducible images, we manually generate the password and use the # SOURCE_DATE_EPOCH to generate the salt in a deterministic way. - if [ -z "${SOURCE_DATE_EPOCH}"]; then + if [ -z "${SOURCE_DATE_EPOCH}" ]; then chpasswd_args="" else - salt="$(echo "${SOURCE_DATE_EPOCH}" | sha256sum -z | cut -c 1-15)" - password="$(openssl passwd -6 -salt $salt "$password")" + salt="$(echo ${SOURCE_DATE_EPOCH} | sha256sum -z | cut -c 1-15)" + password="$(openssl passwd -6 -salt $salt $password)" fi fi printf '%s:%s' "$name" "$password" | sudo chroot '${ROOTFSDIR}' \ diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 063b9a3..944733b 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -310,8 +310,8 @@ python() { # invalidate the SSTATE entries for most packages, even if they don't use the # global SOURCE_DATE_EPOCH variable. rootfs_install_pkgs_install_prepend() { - if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then - export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}" + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + export SOURCE_DATE_EPOCH fi } @@ -443,13 +443,16 @@ EOSUDO # Set same time-stamps to the newly generated file/folders in the # rootfs image for the purpose of reproducible builds. - test ! -z "${SOURCE_DATE_EPOCH}" && \ - sudo find ${ROOTFSDIR} -newermt \ - "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ - -printf "%y %p\n" \ - -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \ - bbwarn "$(grep ^f ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file timestamps to build image reproducibly" - + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps" + if sudo find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ + -printf "%y %p\n" -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >"$fn"; then + if [ -e "$fn" ]; then + bbwarn "modified timestamp (${SOURCE_DATE_EPOCH}) of $(egrep ^f "$fn" | wc -l) files for image reproducibly\n " \ + "List of files modified could be found here: .${DEPLOY_DIR_IMAGE}/files.modified_timestamps" + fi + fi + fi } addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass index db28334..1b98bc0 100644 --- a/meta/classes/initramfs.bbclass +++ b/meta/classes/initramfs.bbclass @@ -33,8 +33,8 @@ do_generate_initramfs() { rootfs_do_qemu # generate reproducible initrd if requested - if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then - export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}" + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + export SOURCE_DATE_EPOCH fi sudo -E chroot "${INITRAMFS_ROOTFS}" \ diff --git a/wic-extract-rootfs-partition.sh b/wic-extract-rootfs-partition.sh new file mode 100755 index 0000000..48de0d3 --- /dev/null +++ b/wic-extract-rootfs-partition.sh @@ -0,0 +1,52 @@ +#!/bin/bash +# +# Copyright (c) Roberto A. Foglietta, 2023 +# +# Authors: +# Roberto A. Foglietta +# +# SPDX-License-Identifier: MIT +# +#set -ex + +if [ "$(whoami)" != "root" ]; then + echo + echo "WARNING: this script should run as root, sudo!" + sudo -E $0 "$@" + exit $? +fi + +if [ -e "$1" ]; then + fimg=$(readlink -e $1) +fi + +cd $(dirname $0) + +if [ ! -n "$1" -a ! -e "$fimg" ]; then + fimg=$(ls -1 build/tmp/deploy/images/*/*.wic) + n=( $fimg ) + if [ ${#n[@]} -gt 1 ]; then + echo + echo "WARNING: more than one image found, choose one:" + echo + echo "$fimg" + echo + exit 1 + fi +fi + +if [ ! -e "$fimg" ]; then + echo + echo "ERROR: no any image or block device ${1:+'$1' }found, abort!" + echo + exit 1 +fi + +wicf=$fimg +losetup -Pf $wicf +ldev=$(losetup -j $wicf | cut -d: -f1 | tail -n1) +echo loopdev:$ldev +dd if=${ldev}p2 bs=1M of=${wicf/.wic/.rootfs} status=progress +chown $(id -u).$(id -g) ${wicf/.wic/.rootfs} +du -ms ${wicf/.wic/.rootfs} +losetup -d $ldev