From patchwork Sun Jan 15 22:17:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2494 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Sun, 15 Jan 2023 23:17:47 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ot1-f63.google.com (mail-ot1-f63.google.com [209.85.210.63]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 30FMHkEP008092 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 15 Jan 2023 23:17:46 +0100 Received: by mail-ot1-f63.google.com with SMTP id cz20-20020a0568306a1400b006849b669d65sf9482223otb.10; Sun, 15 Jan 2023 14:17:46 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673821060; cv=pass; d=google.com; s=arc-20160816; b=wXtSrJgcQMDjx+0NJ96Ins0pGa+K6m1kqzgLSNy54f5956MPtCo89m/abuvx2EFdU/ x82nNSlT1KM11P1zbY85hAO2+STaWBHz88xC0jcI3XEE0fFAeQXNlpgo7J7rVBYITEJw par5Vwx2Z5LIVA+MmmY347ruUBrjEwbGQaogiH/pb0SdGYaFxhHTjgUOt6HwgUXPGFoE GGAKz/PNMCAfAFBcLrUYQWTLV0Ev8iBxYf3J6hNWjwAm3S1bKqNwxioK/vMw4dlj0Lqf q/4SI4KJ2TdToD901ZZFp5CwUzvGyUKPmwdahqJReSvwrttDMzLUF4SZ4M1cy6YBTpmM 8eIQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=M1VmKAfdZ/EUGZliI6WwQoC8i4DVic4AIght9EvdF84=; b=guyr0GMR/qfGxHcwhBYSVpr9Yx/lM9dy5tlTYjaojpSyoe4poNWBrJFfJutKe/55U9 9h1VvAaXWuBnRkp4RX1NAhU5neeFnY+zgzebrr/Jv6xXPx0E+3F6907OkQT43V6vcla3 tT/lJJIukngbuVbos1Y6IuiyQm+cYO1h4x2PzPAwYVjSVacWj5j6i8X8vSsCB+Toe9yV karFMDz3E++rOyRt/N5M17PIZMG3KCwgyirRqiznyp3nHAF2WhVuXPwhckZb5wMyQPMz tg0vBwajtZp/aIwKhs2HrxX0/JWi+U9FwMTr1+DOf2nPTkC8lid13Uq3mJBgx0N8kP+z 795g== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=M1VmKAfdZ/EUGZliI6WwQoC8i4DVic4AIght9EvdF84=; b=h0PSyEwy8rrXac7ZeRh4nd76enOp+jGoQpoIWkmY33zMQrgEZwBZktiL+6alzU0u08 evfYYUj4nLijiKTuzWP3WiN9awua5l/aVnbSFNcvly8GUeY3mEr+UWapkF85A883lOgL uAM9r1Que4SpqRghIx4XJl1lZQ5XfgAShRfywjzDVJ7iy8za2yrnd2sjdMpmTVy4eZsj p7PAmHBHv68naT1BtdRYYfBMUtCqmcUvd5Uec0ka4GWq5xFt1f8O96iwg3t+fBdPfNPD VklzhZszTd9O+tj3hB4pwzXcPyxB7TcBvWcXuhoYQbSJw0imZgVPtzqkUGGha3CEgxCT Cqvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=M1VmKAfdZ/EUGZliI6WwQoC8i4DVic4AIght9EvdF84=; b=mF8d0QwxtZWO/cXnqJvxGmlcgKEhoFaYY888UMzPUqbPIMoM1vnxwCOMKyubzSlVUe bSiU1qJBkGsnFbIZFPSCsOjlkbYPJisM+L8HX9A3edW5a5RGjLuphgHTy26S2qazY6Hd fFkR79mnsjJSU2d5Je06HcmrV3Aa3buWYADxaAiez2+pCTlt0sYcTm6uumnA4wBgmgZ2 cazzPCqRP8unAOy+ejUvNg4CdumgTV1+m32Xr3GnPiAj8cqNIJsl9lTWpiNNRCkWRTsz LailL0ACZNYzxdQcwImrJLfUiJ7yu5TUq75BpDLdCZcAKCM9M0raIvb9zNmk6fX5z44a ShLw== Sender: isar-users@googlegroups.com X-Gm-Message-State: AFqh2krHFbRXQI5UlxKHBcNZ4QJ01CbLG0/JbyS88nBBiOJVV4T3yI0M 9hI1Jj23tyZraX7ZcJlYY1M= X-Google-Smtp-Source: AMrXdXtfavcWNeL79p0C5miAU3jx5ITDjLFdUjzI0lyFOoIYME59L/AkDQH39Bw7qA549v9vPGoG2A== X-Received: by 2002:a05:6830:2653:b0:677:916f:9767 with SMTP id f19-20020a056830265300b00677916f9767mr5680600otu.58.1673821060460; Sun, 15 Jan 2023 14:17:40 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6830:621b:b0:661:b84b:eb5e with SMTP id cd27-20020a056830621b00b00661b84beb5els2159045otb.3.-pod-prod-gmail; Sun, 15 Jan 2023 14:17:40 -0800 (PST) X-Received: by 2002:a05:6830:1107:b0:684:c286:11f0 with SMTP id w7-20020a056830110700b00684c28611f0mr7971197otq.7.1673821060071; Sun, 15 Jan 2023 14:17:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673821060; cv=none; d=google.com; s=arc-20160816; b=e9AcuyOnSjsxmCRcyEtqXRgA43EJ2F9RnK4DeHnf1gtqtEOmcpvDKBNRNabZo3vQ// +W+OZbhNBBAkAspeq+j8+zGvFFLkV4s+hBKzHf0VbyZL1EBWeDLEVCHiLaN/dWW/wcK4 2WbJYsCj4L3SiUkMxYIYPCw9dHY2d3Y6bMA21QX/wQ3Ea31LVGIR1h8NUTvodwS0fUVG VD7QfJg5YPueum2zCEAgRLJFtlLh28qQ6MfKTcN6Knbdb91MpXjDzCr9WBNrZ2kNdzZG dAoi+zwpNuB+MKCb3fKGbLksvwaULQ1/gdR3jBOfAwO7+kOB3Bo2+vLXXlxOfkepIOoy rOZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=F+h3td5glBULsVmz6382NtZhDdO9d/2r+BdAZgyRHmY=; b=i6PwaP55/VKSBWoK+q52Namhs9zJtI62BMUMhHZ1jQNz0kn3QzkSs9yutynH9Y8hvp aZYmzTfxllwKnezHJeOUWd2divqhsbzVTqO9TIMAgyXv1URhezaz08n4tw50Miq20Isb z6RWNzLOyOpVwwJhvohEi2YwGiKg0GjWRIx6esh5fQpvcHB/MjePkXZ5Vs+pCFVP7Ggb qq4E/OKbYxr93Enjj+zDJls04o+sRL1em5mIQuXZkAekFXo2TzmqWoxSVVdHwOTNGZHI UteRuqLbD6XDbwHfrB5+1KwI7kbqXJn6souVJsaDceu5L9maBWqAex7SARhBm80ruA7d OFSg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from relay02.th.seeweb.it (relay02.th.seeweb.it. [2001:4b7a:2000:18::163]) by gmr-mx.google.com with ESMTPS id e21-20020a9d0195000000b00684db315574si524113ote.1.2023.01.15.14.17.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Jan 2023 14:17:39 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) client-ip=2001:4b7a:2000:18::163; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:8d79:8896:56be:2c19]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id 480DE1F58D; Sun, 15 Jan 2023 23:17:38 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v7] suggested changes for reproducibility patchset v7 Date: Sun, 15 Jan 2023 23:17:34 +0100 Message-Id: <20230115221734.741365-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" suggested changes for reproducibility patchset WARNING: eval-image-1.0-r0 do_rootfs_finalize: modified timestamp (1673628837) of 3 files for image reproducibly List of files modified could be found here: ./build/tmp/deploy/images/debx86/files.modified_timestamps v.2: rebased on current ilbers:next v.3: new script added: wic-extract-rootfs-partition.sh [image.wic] v.4: example with for epoch generation from git v.5: reverted the example and rework some few code v.6: the 1st part of the warning shows up each time the epoch is used while the 2nd line appears only when some files has been touched This allows the user to know the current situation aboat epoch. v.7: forgot to commit before producing the patch v6 but sent! Signed-off-by: Roberto A. Foglietta produc Signed-off-by: Roberto A. Foglietta --- meta-isar/conf/local.conf.sample | 2 +- meta/classes/image-account-extension.bbclass | 6 +-- meta/classes/image.bbclass | 22 +++++---- meta/classes/initramfs.bbclass | 4 +- wic-extract-rootfs-partition.sh | 52 ++++++++++++++++++++ 5 files changed, 71 insertions(+), 15 deletions(-) create mode 100755 wic-extract-rootfs-partition.sh diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample index 6208623e..1d7e178a 100644 --- a/meta-isar/conf/local.conf.sample +++ b/meta-isar/conf/local.conf.sample @@ -257,4 +257,4 @@ USER_isar[flags] += "clear-text-password" # Non git repository users can use value from 'stat -c%Y ChangeLog' # To know more details about this variable and how to set the value refer below # https://reproducible-builds.org/docs/source-date-epoch/ -#SOURCE_DATE_EPOCH = +#SOURCE_DATE_EPOCH = "" diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass index bb173b14..1d49054c 100644 --- a/meta/classes/image-account-extension.bbclass +++ b/meta/classes/image-account-extension.bbclass @@ -256,11 +256,11 @@ image_postprocess_accounts() { # chpasswd adds a random salt when running against a clear-text password. # For reproducible images, we manually generate the password and use the # SOURCE_DATE_EPOCH to generate the salt in a deterministic way. - if [ -z "${SOURCE_DATE_EPOCH}"]; then + if [ -z "${SOURCE_DATE_EPOCH}" ]; then chpasswd_args="" else - salt="$(echo "${SOURCE_DATE_EPOCH}" | sha256sum -z | cut -c 1-15)" - password="$(openssl passwd -6 -salt $salt "$password")" + salt="$(echo ${SOURCE_DATE_EPOCH} | sha256sum -z | cut -c 1-15)" + password="$(openssl passwd -6 -salt $salt $password)" fi fi printf '%s:%s' "$name" "$password" | sudo chroot '${ROOTFSDIR}' \ diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 063b9a3b..191c3940 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -310,8 +310,8 @@ python() { # invalidate the SSTATE entries for most packages, even if they don't use the # global SOURCE_DATE_EPOCH variable. rootfs_install_pkgs_install_prepend() { - if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then - export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}" + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + export SOURCE_DATE_EPOCH fi } @@ -443,13 +443,17 @@ EOSUDO # Set same time-stamps to the newly generated file/folders in the # rootfs image for the purpose of reproducible builds. - test ! -z "${SOURCE_DATE_EPOCH}" && \ - sudo find ${ROOTFSDIR} -newermt \ - "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ - -printf "%y %p\n" \ - -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \ - bbwarn "$(grep ^f ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file timestamps to build image reproducibly" - + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps" + sudo find ${ROOTFSDIR} -newermt "$(date -d@"${SOURCE_DATE_EPOCH}" '+%Y-%m-%d %H:%M:%S')" \ + -printf "%y %p\n" -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >"$fn" + msg="" + ncfs=$(egrep ^f "$fn" | wc -l) + if [ $ncfs -gt 0 ]; then + msg="\n List of files modified could be found here: ."${DEPLOY_DIR_IMAGE}"/files.modified_timestamps" + fi + bbwarn "Modified timestamp ("${SOURCE_DATE_EPOCH}") of "$ncfs" files for image reproducibly.$msg" + fi } addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass index db283347..1b98bc06 100644 --- a/meta/classes/initramfs.bbclass +++ b/meta/classes/initramfs.bbclass @@ -33,8 +33,8 @@ do_generate_initramfs() { rootfs_do_qemu # generate reproducible initrd if requested - if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then - export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}" + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + export SOURCE_DATE_EPOCH fi sudo -E chroot "${INITRAMFS_ROOTFS}" \ diff --git a/wic-extract-rootfs-partition.sh b/wic-extract-rootfs-partition.sh new file mode 100755 index 00000000..48de0d3a --- /dev/null +++ b/wic-extract-rootfs-partition.sh @@ -0,0 +1,52 @@ +#!/bin/bash +# +# Copyright (c) Roberto A. Foglietta, 2023 +# +# Authors: +# Roberto A. Foglietta +# +# SPDX-License-Identifier: MIT +# +#set -ex + +if [ "$(whoami)" != "root" ]; then + echo + echo "WARNING: this script should run as root, sudo!" + sudo -E $0 "$@" + exit $? +fi + +if [ -e "$1" ]; then + fimg=$(readlink -e $1) +fi + +cd $(dirname $0) + +if [ ! -n "$1" -a ! -e "$fimg" ]; then + fimg=$(ls -1 build/tmp/deploy/images/*/*.wic) + n=( $fimg ) + if [ ${#n[@]} -gt 1 ]; then + echo + echo "WARNING: more than one image found, choose one:" + echo + echo "$fimg" + echo + exit 1 + fi +fi + +if [ ! -e "$fimg" ]; then + echo + echo "ERROR: no any image or block device ${1:+'$1' }found, abort!" + echo + exit 1 +fi + +wicf=$fimg +losetup -Pf $wicf +ldev=$(losetup -j $wicf | cut -d: -f1 | tail -n1) +echo loopdev:$ldev +dd if=${ldev}p2 bs=1M of=${wicf/.wic/.rootfs} status=progress +chown $(id -u).$(id -g) ${wicf/.wic/.rootfs} +du -ms ${wicf/.wic/.rootfs} +losetup -d $ldev