From patchwork Sun Jan 15 21:53:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2493 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Sun, 15 Jan 2023 22:53:22 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-qk1-f186.google.com (mail-qk1-f186.google.com [209.85.222.186]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 30FLrLdw007965 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 15 Jan 2023 22:53:21 +0100 Received: by mail-qk1-f186.google.com with SMTP id m3-20020a05620a24c300b006fee2294e97sf19510959qkn.11; Sun, 15 Jan 2023 13:53:21 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673819595; cv=pass; d=google.com; s=arc-20160816; b=ShrnYFgJy+q86BZMa7Bq/NzoGKBvhOLx7vGJk0TTOjhzG9Na0d/6CsMNYLYkFf8xYx OXMotvEjfnUA+9wgVVavPmk85Xx2sw3Cd/RLqQzUkqUi/fZWrPRCZo5F7tcOknzaudIV ytYVqXMe+4NLuVUdQvlS1n9iy3ffTqjZyPSmmJ22ywhO5uok/DiMA3SnDBh8S4QiXUs1 jc6n//FMzW5B4hbewEGK1HSt/GKocB2HR7XxXfDJz6cefksI9LKfv03lzAxdB97zPZzG ZwnZKYaqXM5GKbFyzlsOUEvSACDLdFre49t5PH5v6UZgQTzCtuWEZRQbB5CxE0/2oyox xoBA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=Y6TzXdh+6jFWiYlS04TkAxlRE3Z3sbUJEMljLj25xHI=; b=is/aniMUhG5NY8/64WOjA1jFk2mZg+F1tyDEqAeogo+73CJk7asoivo4pwKawkloyx cuXm7hfs7JI4jJGtUS8+H8SW3iX0XJp1ea0GEdirqGZxRVgrlTbJf6fb9s3c3Wr6FOg/ JZbTkMtK6QJvmnc4SYiSd1o4LAnD1VSglsDXXP18uaqFfSOTPg/aC5NeGbB5j+5lxaM1 Utc4OZ1227R4YBgar+eTt+BmFIAISd+3M4RmwXPSnpmNvkDxIkHMmur1GCc/O0EI12Cb IaQwhdYg+wk3acjqqbU7pUZUi4rVJMTVzuZxWYX5pI8G+Uuz3lbNRlFjIlQ+2rVYvjWO lW9w== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=Y6TzXdh+6jFWiYlS04TkAxlRE3Z3sbUJEMljLj25xHI=; b=ff55JtRhdCheLmezh0UDu0CdfQQ5uysH7EZWK8hjGG45/dgnRpoetDtUVGmw2wZq2v dzQgu2RhJ37o2qrviUPXylclOoDiQcllfipJfb9gJtcMypnzp5Dm1ak2vVSR+ecAK1to LenfMnKItZujIDiar+mX4RXlSRgV3g0P+pHziuUuJ05tB8eHd8exbcNpIUQEsfC55Hjd hMKNJZCVtDw0+vZ5hXPa1j8NKRpKht2Peom98A8s+FwH45M2duk21K+P3+d22U0ryf3L 2QBPlRhbiG9gkzlD/ldL0aZGLX6n6L7JB1F2l2VgkYfNEKxG54AFo0O4z6wqTYSymwsv UAwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=Y6TzXdh+6jFWiYlS04TkAxlRE3Z3sbUJEMljLj25xHI=; b=LNTWJTH67NwR58QcmVytRXBT/mk//SJ28lufaiSkIn6VEd5zOhDnm9G+aRHPAb+UOK 0AS4vsjK1mCyxGeQ+73U4ePVRvjwjKcmjYO742OdBmKRG9Q5YzXAPJtK5aJKe4pOcI9b 7tzXZmexrNKM03PMW5VqXHREHGVzMK3I7V4YT4DMCFH8Wabbfw3dljyR7gf7Rbu9Axsa GQrqIvPICc2LI/n0JvJ1M3z/gjNrawOV5SpCPZKUQHKIlT4aIiBgD3/iRM2k8MU1Yxum yRCi9SNU0TjK58c3umuKGx0hfnYkf9RmvAfafCbM4Kzc+RPuUXnVuVvHXaXrmdNuU2rZ R3cA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AFqh2ko1S0XR0N1fmacWMLJNpzEqp8KbeFRdIqp52OVT+SEe9jPP+Y6E UvUvqAXIwbnzvXslDzFNvcA= X-Google-Smtp-Source: AMrXdXt0Dv1mFjnnWA3/ndN/IbWyuc0OfALvBjSdcZAHlDfaShpKHP+B+gCiYYiF8GdJ8/0FmHfusg== X-Received: by 2002:ac8:7208:0:b0:3b6:fb4:81c0 with SMTP id a8-20020ac87208000000b003b60fb481c0mr149818qtp.285.1673819595611; Sun, 15 Jan 2023 13:53:15 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:622a:418a:b0:3b0:5130:7690 with SMTP id cd10-20020a05622a418a00b003b051307690ls8935857qtb.1.-pod-prod-gmail; Sun, 15 Jan 2023 13:53:15 -0800 (PST) X-Received: by 2002:ac8:1344:0:b0:3ab:5d1e:a775 with SMTP id f4-20020ac81344000000b003ab5d1ea775mr114977538qtj.12.1673819594966; Sun, 15 Jan 2023 13:53:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673819594; cv=none; d=google.com; s=arc-20160816; b=XAwuuaGp0zWyHh/78FBsIzpa0wpGAsoeYolxA4bdm++61rBfraZeSXt0jRiwQZXbeW A/hPfPtBw5ZUtGXnCujegm02C5Z4GY+vSXN3EXUPW6C80LprG6sgGL1I8fL8nlfDIZHR JJCsg+cCmetABDmYcoGODJMb0MfyPnA8WMGvcUzqJhVueOHevO0Yd6FWuQJb3UKbU3nm 8C287jRKwowTZxKGEsgs+rxmAPJgW4rn5B9vk1jO71S0Lu1CqV/pjPZDDC9wlLTyjwqR L3ga5lmtslH6FH0S/wyfl2qHLTr/hdI7y8z/5oUdwrCdJ/MEU9RBUpwCK9KoxD8DqKc1 /3Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=GktxAIWFPS/zaGID773MkyVILTkdvWxw1CPRXlNzL+s=; b=SW5uPhGbhMWaD8sdpwLdH/Ul/XTJi3PzYtx5z/SQ9eImQUXyyb7oO44DCf0YFDKip2 bgkVYEXLaX4jCl0hhKRuNahdiABJgoSff6/RCDJL/cMCr9BOUiirOoL0mJ6QZCrUAw2Q OoiBwSI+bYqUZ0g7+HA/S7Y5nmN5WYxR32XPvc285pDQ4BlducFdML4CgMcavnpjNwnq kI0mZ0oMsop28SGAwzIY1vRCil1PNADjIVGrDxFmNH/YZ13IOI4lshR66zqaZGETsHGH PZdxNU00QVa/DbqHi6EVZYA0uegypsOGCAQmRXUvDZ0U1oszMehPpgeoCkrxAPpUVq6d wllA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from relay04.th.seeweb.it (relay04.th.seeweb.it. [5.144.164.165]) by gmr-mx.google.com with ESMTPS id a20-20020ac84d94000000b003a694b5199csi118640qtw.3.2023.01.15.13.53.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Jan 2023 13:53:14 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.165 as permitted sender) client-ip=5.144.164.165; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:8d79:8896:56be:2c19]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id 7C2191F4C7; Sun, 15 Jan 2023 22:53:12 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v6] suggested changes for reproducibility patchset v6 Date: Sun, 15 Jan 2023 22:53:10 +0100 Message-Id: <20230115215310.732295-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" suggested changes for reproducibility patchset WARNING: eval-image-1.0-r0 do_rootfs_finalize: modified timestamp (1673628837) of 3 files for image reproducibly List of files modified could be found here: ./build/tmp/deploy/images/debx86/files.modified_timestamps v.2: rebased on current ilbers:next v.3: new script added: wic-extract-rootfs-partition.sh [image.wic] v.4: example with for epoch generation from git v.5: reverted the example and rework some few code v.6: the 1st part of the warning shows up each time the epoch is used while the 2nd line appears only when some files has been touched This allows the user to know the current situation aboat epoch. Signed-off-by: Roberto A. Foglietta --- meta-isar/conf/local.conf.sample | 2 +- meta/classes/image-account-extension.bbclass | 6 +-- meta/classes/image.bbclass | 20 ++++---- meta/classes/initramfs.bbclass | 4 +- wic-extract-rootfs-partition.sh | 52 ++++++++++++++++++++ 5 files changed, 69 insertions(+), 15 deletions(-) create mode 100755 wic-extract-rootfs-partition.sh diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample index 6208623e..1d7e178a 100644 --- a/meta-isar/conf/local.conf.sample +++ b/meta-isar/conf/local.conf.sample @@ -257,4 +257,4 @@ USER_isar[flags] += "clear-text-password" # Non git repository users can use value from 'stat -c%Y ChangeLog' # To know more details about this variable and how to set the value refer below # https://reproducible-builds.org/docs/source-date-epoch/ -#SOURCE_DATE_EPOCH = +#SOURCE_DATE_EPOCH = "" diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass index bb173b14..1d49054c 100644 --- a/meta/classes/image-account-extension.bbclass +++ b/meta/classes/image-account-extension.bbclass @@ -256,11 +256,11 @@ image_postprocess_accounts() { # chpasswd adds a random salt when running against a clear-text password. # For reproducible images, we manually generate the password and use the # SOURCE_DATE_EPOCH to generate the salt in a deterministic way. - if [ -z "${SOURCE_DATE_EPOCH}"]; then + if [ -z "${SOURCE_DATE_EPOCH}" ]; then chpasswd_args="" else - salt="$(echo "${SOURCE_DATE_EPOCH}" | sha256sum -z | cut -c 1-15)" - password="$(openssl passwd -6 -salt $salt "$password")" + salt="$(echo ${SOURCE_DATE_EPOCH} | sha256sum -z | cut -c 1-15)" + password="$(openssl passwd -6 -salt $salt $password)" fi fi printf '%s:%s' "$name" "$password" | sudo chroot '${ROOTFSDIR}' \ diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 063b9a3b..bf3dfea8 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -310,8 +310,8 @@ python() { # invalidate the SSTATE entries for most packages, even if they don't use the # global SOURCE_DATE_EPOCH variable. rootfs_install_pkgs_install_prepend() { - if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then - export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}" + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + export SOURCE_DATE_EPOCH fi } @@ -443,13 +443,15 @@ EOSUDO # Set same time-stamps to the newly generated file/folders in the # rootfs image for the purpose of reproducible builds. - test ! -z "${SOURCE_DATE_EPOCH}" && \ - sudo find ${ROOTFSDIR} -newermt \ - "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ - -printf "%y %p\n" \ - -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \ - bbwarn "$(grep ^f ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file timestamps to build image reproducibly" - + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + msg="" + fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps" + if sudo find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \ + -printf "%y %p\n" -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >"$fn"; then + msg="\n List of files modified could be found here: .${DEPLOY_DIR_IMAGE}/files.modified_timestamps" + fi + bbwarn "Modified timestamp (${SOURCE_DATE_EPOCH}) of $(egrep ^f '$fn' | wc -l) files for image reproducibly.$msg" + fi } addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass index db283347..1b98bc06 100644 --- a/meta/classes/initramfs.bbclass +++ b/meta/classes/initramfs.bbclass @@ -33,8 +33,8 @@ do_generate_initramfs() { rootfs_do_qemu # generate reproducible initrd if requested - if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then - export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}" + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + export SOURCE_DATE_EPOCH fi sudo -E chroot "${INITRAMFS_ROOTFS}" \ diff --git a/wic-extract-rootfs-partition.sh b/wic-extract-rootfs-partition.sh new file mode 100755 index 00000000..48de0d3a --- /dev/null +++ b/wic-extract-rootfs-partition.sh @@ -0,0 +1,52 @@ +#!/bin/bash +# +# Copyright (c) Roberto A. Foglietta, 2023 +# +# Authors: +# Roberto A. Foglietta +# +# SPDX-License-Identifier: MIT +# +#set -ex + +if [ "$(whoami)" != "root" ]; then + echo + echo "WARNING: this script should run as root, sudo!" + sudo -E $0 "$@" + exit $? +fi + +if [ -e "$1" ]; then + fimg=$(readlink -e $1) +fi + +cd $(dirname $0) + +if [ ! -n "$1" -a ! -e "$fimg" ]; then + fimg=$(ls -1 build/tmp/deploy/images/*/*.wic) + n=( $fimg ) + if [ ${#n[@]} -gt 1 ]; then + echo + echo "WARNING: more than one image found, choose one:" + echo + echo "$fimg" + echo + exit 1 + fi +fi + +if [ ! -e "$fimg" ]; then + echo + echo "ERROR: no any image or block device ${1:+'$1' }found, abort!" + echo + exit 1 +fi + +wicf=$fimg +losetup -Pf $wicf +ldev=$(losetup -j $wicf | cut -d: -f1 | tail -n1) +echo loopdev:$ldev +dd if=${ldev}p2 bs=1M of=${wicf/.wic/.rootfs} status=progress +chown $(id -u).$(id -g) ${wicf/.wic/.rootfs} +du -ms ${wicf/.wic/.rootfs} +losetup -d $ldev