From patchwork Mon Jul 8 06:17:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 3671 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 08 Jul 2024 08:18:06 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f188.google.com (mail-lj1-f188.google.com [209.85.208.188]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 4686I5Du030928 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 8 Jul 2024 08:18:05 +0200 Received: by mail-lj1-f188.google.com with SMTP id 38308e7fff4ca-2ee87546e5esf38428821fa.0 for ; Sun, 07 Jul 2024 23:18:05 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1720419479; cv=pass; d=google.com; s=arc-20160816; b=LH1vso9R/94bHf18dWNkWkNrGg8cB1xQWtKx/3c1nGIbUmmOph02Iw30furHXv2d7q vebngFBVJQHN1x9d7QPz6U88jhcACDCjaCF2jnr6nvTV8zilRrgMpElUbvj206PHJ01F NRolu0u8gwdS6xrL+7/LhrL70oVBEWJFDwDrCrSxnwiJiUYh59f8EelIscDs/2yjJFEq P/6/kctQTaQdD0lZ2IHM2SWa5frDov1QmDcDJt+6hxo4c4j+g58Pizv1V6GtV3ZG+qru fk9+SjbJ48JHb/1StRL9IREqDUBltth0Oq5Xu55N9XU7r8THCWmNEfHgi8vImNfTevre nGHQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=6KeDG8EnGr4dmaEUO0kinkyPEGDCXbH9Nk1rp98cbZA=; fh=raoxeF1K3M0mLn5ZagIrqNNYn5ahm7Ggn4Hb1O4Z9tk=; b=waSTaDZvwJUVDizlYEZfGXNwxsFFSZ9F7X5J+a6CKWRHKq/WnFaj7ZYATzd1Lf6Lz1 nxTjI3dwa8Hvj8JNVuPXLnRSZoSrD3WuzWSfAEPF5XC87QUALxGtUrjqYkP1IuzI7R86 oa04Duvio05VJE1Nbe1gK02WVqcNKRps95CjkwHBkstG7GMO3CzzRn5kXj7aqQi6J/Ro dkvnYwgRyZ1elVj0BHIG7dnxbFVQY/zoGi62sa7BDyoDTklhcBwm3pEW7NLSHDMODgtk PUk4Z827zbWc85NNg4cEtS9vhrojkyT4o8SyUGF2D2z6J/mXrvt/+pHDraV6EA32YHfs dF9g==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1720419479; x=1721024279; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=6KeDG8EnGr4dmaEUO0kinkyPEGDCXbH9Nk1rp98cbZA=; b=smFWOVy65+u5bhrwqASOUvFrzbwqaTQUWz0az+TaBzLMzMmevnhjDmW+xdtsBmTX7q 859fN/FSxSqxM2hLwwRHaGE/ONmOwVgMqXZza/jZmRDYPCzZUJxBspng3Pm3VbF5GkST 8z59oWHrA1dWSCUZ6d1NooYj1h8IMWoelLVLIcq/YAKwvwpfrnhZvweEWKM6y6jpW+2G zqJF01i3P7qARQyh9guB+G+3ixCyy5j7ytVmRLdPB4O4a5dEsgZKpGcDj0cSv6225Bn5 6JEKyfQGHEHHJD61ODrz8NQ/XuuCNhotl/FSK2W39D4ESCV3Prg++hIvnxg/QRDGiM8W y1rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720419479; x=1721024279; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to; bh=6KeDG8EnGr4dmaEUO0kinkyPEGDCXbH9Nk1rp98cbZA=; b=pqRCSWRztnZarOvbB+ZPsKWB5C1QjB7nPGgQKD/Dah96MyuDtFxukYYjrR++l+eNI/ kfQsBSZ7ELbRNiP0Y6VtjpUfTeRqYCmSdn4Tzbfy1IXdemEGFxo1mXMzHtFWMdFMU4uC r0Ndhi2wP4mOIP4XcmuwKp/IQuMwAu9S2iHV+ZQzzxC/19atzEtoHCmZCttDEojL9ord /rQ7FtvskT8glxo8FdrMkcCt3ZLBh6YVhGOHobkFIzWZoJe/y6amP4aSy5/Vvok7fS7k jDnXaKljjFgwOkDrjBB/wWQWS9xb1QggFf1CaKyIK1TtQEWxchIdUKrcFtSoO0eyEfyJ 7Aew== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVNGg8JqkQdjnKboz0rytiO88aAta6DXeOR67q+teN78kbiLBTC70kR7ZmLMoQUXA8a+dl5mK7cgs9jn60/GejYFnoohD8= X-Gm-Message-State: AOJu0Yz4YwfsTwerMYoC69DO/0cD6DMcTuzXOpKx63Ge4ScjHlT2l6p1 pUpSKiSeijER3gevA/cnb3t7gtxIZL4epf37EoU3ewAVC0T63CCj X-Google-Smtp-Source: AGHT+IH1cSi6pnyB0DQca0odZnarkpLruuw2hkFU6xt7XxZCB4Snxr1NyH7XzhG1e3rMOQhmVf/3xg== X-Received: by 2002:a2e:a58a:0:b0:2ee:7d37:498b with SMTP id 38308e7fff4ca-2ee8ed8cf11mr92339071fa.14.1720419478929; Sun, 07 Jul 2024 23:17:58 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:aa26:0:b0:2ee:89a7:12ac with SMTP id 38308e7fff4ca-2ee8cd96adfls18073181fa.1.-pod-prod-06-eu; Sun, 07 Jul 2024 23:17:57 -0700 (PDT) X-Received: by 2002:a05:651c:601:b0:2ee:4f22:33f9 with SMTP id 38308e7fff4ca-2ee8eda42f0mr63774631fa.24.1720419475918; Sun, 07 Jul 2024 23:17:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1720419475; cv=none; d=google.com; s=arc-20160816; b=ix4HviUdwA1EEZBEKxNsNpDOVRLzCbKrvcOTu6v0/xTlSIigFpi40t9EbatepSw6tA H6FvyLdaP3zpPRoruJyw/0hfqs8Dr6gDVuwv0icahguesEUF646caU4+DQrLuuaPp8ID 8JjeVHEXxlzAfvoshj+I+IXw3IRVGPW54OVZV9k0RD9PDSMUaAxKHM1O/fYZ8hj1eUhH GDjdnqMrfmzfXiYNsRtfUc7S+PBJp9RIQ/X7fGpyNjddD0PGmkRZheLtUgZpMUx1QOuv X3UL7Vc+z9qSgHhhKvykd4zKFiBSTKwrR7AyIKJKCMBh66JDfjhIlJHBvvH3L7Jv3dWT F8bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=5nPiJF+WzumJAVVCCb60FHItiJ2GgvRB52oI2WIygYs=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=lOyxHloPrbAZEsLO9mcuwW/9SRoTRWS2SOoo7xdMHG2POubl8oCclgeZAbZLsv2dy2 HuNly0y2tRRSi1DwEHnLpHrur2m+fY1pB0oaZee1bo0TMgYJ3ErcrLV03IsarsZ7rgV1 LP5D2SEnDldsZ4TfT4monbYOizoYDFVqxizMZeSgvpCz2deLtQMUB62+/T5f+dIp6P7h V/yKNXn/W65DHCSgRrbfx8kemcGB2slqJRc3GhPqGoKRL6QaV/OKs3kEpPwHETl69/Ia hjMXVRTHavwL4ubUewuauFA0hnckG14foXTBIQjt2A5DclJ93XSeRG7E/Y3vWLZdr7yg I0NQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2ee941090e9si1239231fa.6.2024.07.07.23.17.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 07 Jul 2024 23:17:55 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from user-B660.promwad.corp ([159.148.83.114]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 4686HmUx030688 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 8 Jul 2024 08:17:54 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH v10 7/8] isar-bootstrap: Use tar output instead of directory Date: Mon, 8 Jul 2024 09:17:37 +0300 Message-Id: <20240708061738.311938-8-amikan@ilbers.de> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240708061738.311938-1-amikan@ilbers.de> References: <20240708061738.311938-1-amikan@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Do not unpack tar with rootfs content inside bootstrap recipe. This change can allow to remove sudo usage during the bootstrap later. Using tar output requires moving chroot prepare step to the later unpack step. Signed-off-by: Anton Mikanovich --- meta/classes/rootfs.bbclass | 11 +++++---- .../isar-bootstrap/isar-bootstrap.inc | 23 ++++++++++--------- .../isar-mmdebstrap/isar-mmdebstrap.inc | 23 ++++--------------- 3 files changed, 24 insertions(+), 33 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index aa3f6baa..0a1e64e5 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -73,12 +73,15 @@ rootfs_do_qemu() { fi } -BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}" -BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}" +BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst" +BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst" rootfs_prepare[weight] = "25" rootfs_prepare(){ - sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' + sudo tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console" + + # setup chroot + sudo "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" } ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" @@ -227,7 +230,7 @@ cache_deb_src() { # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and # relies on that through out the build. Copy that state information instead of apt-get update # which generates a new state from upstream. - sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/" + sudo tar -xf "${BOOTSTRAP_SRC}" ./var/lib/apt/lists --one-top-level="${ROOTFSDIR}" deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 0855875b..4e0ffc0d 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -162,7 +162,6 @@ do_bootstrap() { # setup chroot install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh" - "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" # update APT mount -o bind,private /dev ${ROOTFSDIR}/dev @@ -196,8 +195,12 @@ do_bootstrap() { mountpoint -q "${ROOTFSDIR}/base-apt" && \ umount "${ROOTFSDIR}/base-apt" + # Compress rootfs for compatibility + lopts="--one-file-system --exclude=var/cache/apt/archives" + tar --zstd -cf "${WORKDIR}/rootfs.tar.zst" $lopts -C "${ROOTFSDIR}" . + # Finalize debootstrap by setting the link in deploy - ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" + ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" EOSUDO deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" @@ -213,19 +216,17 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" bootstrap_sstate_prepare() { # this runs in SSTATE_BUILDDIR, which will be deleted automatically - lopts="--one-file-system --exclude=var/cache/apt/archives" - sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}") - sudo chown $(id -u):$(id -g) bootstrap.tar + sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst + sudo chown $(id -u):$(id -g) bootstrap.tar.zst } bootstrap_sstate_finalize() { # this runs in SSTATE_INSTDIR - # - after building the bootstrap, the tar won't be there, but we also don't need to unpack - # - after restoring from cache, there will be a tar which we unpack and then delete - if [ -f bootstrap.tar ]; then - sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar - sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" - rm bootstrap.tar + # we should restore symlinks after using tar + if [ -f bootstrap.tar.zst ]; then + mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst" + sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \ + "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" fi } diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc index c41dd3a1..f1ad891b 100644 --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc @@ -11,7 +11,6 @@ inherit deb-dl-dir FILESEXTRAPATHS:append = ":${LAYERDIR_core}/recipes-core/isar-bootstrap/files" -ROOTFSDIR = "${WORKDIR}/rootfs" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,apt,usrmerge" BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir" @@ -73,9 +72,6 @@ do_bootstrap() { fi E="${@ isar_export_proxies(d)}" - sudo rm -rf --one-file-system "${ROOTFSDIR}" - mkdir -p "${ROOTFSDIR}" - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)" bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list" @@ -181,18 +177,9 @@ do_bootstrap() { "${WORKDIR}/rootfs.tar.zst" \ "$bootstrap_list" - sudo -E -s <<'EOSUDO' - set -e - - tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console" - - # setup chroot - install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh" - "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" + # Finalize bootstrap by setting the link in deploy + sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" - # Finalize bootstrap by setting the link in deploy - ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" -EOSUDO if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" sudo rm -rf --one-file-system "${WORKDIR}/dl_dir" @@ -206,7 +193,7 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" bootstrap_sstate_prepare() { # this runs in SSTATE_BUILDDIR, which will be deleted automatically - sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst + sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst sudo chown $(id -u):$(id -g) bootstrap.tar.zst } @@ -214,8 +201,8 @@ bootstrap_sstate_finalize() { # this runs in SSTATE_INSTDIR # we should restore symlinks after using tar if [ -f bootstrap.tar.zst ]; then - mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" - sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \ + mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst" + sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \ "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" fi }