From patchwork Wed Nov 6 08:21:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 3899 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 06 Nov 2024 09:21:41 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f190.google.com (mail-lj1-f190.google.com [209.85.208.190]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 4A68LdGm017102 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 6 Nov 2024 09:21:40 +0100 Received: by mail-lj1-f190.google.com with SMTP id 38308e7fff4ca-2fb3e48fb66sf3743151fa.2 for ; Wed, 06 Nov 2024 00:21:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1730881294; cv=pass; d=google.com; s=arc-20240605; b=iXflTOcQy/QqteL4vMaYyrYoFSzSCGacelDK9V3XGNLwNeXpNrMczzDofOOL9K3SOo UIwHfPp+fdn8LOQx6nRjf1bAHO9AyJtxQ9A2tjG1oredvYoNgAcV0XPHI07a3MR2rQqz fxb9VX7aVqvZgHzC+owTsDY2GT3oNgwU5DcfQkxruS0HtHshOo/ioUoOx7s2wwF2co3X H5CuPXYpSd8R8eibUbgJd7FJHEr279Dh3p+uTRlsCqgTNSRcrP0gtWrW+sSMBKoXVbVY 93CmbB7iJ9WKwbv/ZbxJPFnuXseTFGKdHtfdk2FNd7S02zIZ0OjiCG3d9P4s+6B9Vu3Y wbMA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=fNbPD93Ni6IXi4nxpMBsz/N/ABtxqebG0Tfp1ktjE/M=; fh=bdNVqBCHITAaFdTaWFRpq8r+nugql3AX/3Gszv6CCZs=; b=MGHL2vGDRv7eh9NhyLDlKEU0Mb22ZqDMyPbw67UpxDPrBttu7AX1tiqwNTFvoOEi/l pFmEqxtRq4ATfCyAoRIqdp0lAACeuIDlODOQLWvunpMpsha6Y8WdDwsC58ElRZOl9NVw LPbI8qjoMVw+51d+r0ewb3RCF3NoouX5Ei6lQ9/fadBwbfUzt6QQbPBI6DAGCrymMoxV 2v+x76y0kGVgt7LHiCtxMCJQyI5CO0hFgnlaTw/87e7AOzdFVeJ39nirgy2TuMtL3fQY /xmSk+1L6PuRx2ccRJOIwLnWjiy4B3i7mG7NDnuAF7TowzgpoRW5bJ7nIMVwed0ZyLZZ fh0w==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1730881294; x=1731486094; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=fNbPD93Ni6IXi4nxpMBsz/N/ABtxqebG0Tfp1ktjE/M=; b=nc3EmsDGkJGLW7gPleATSDAJj0kvh0r+P9ehtu905PTbvFvCtsvgHQuUWPIJplyGRd XKe9fla+F3GMwK+vFoFPujKi4NtyVTry566SoeAYAs/qJlQRFlyHXxg2K2hSBPNL9kkM WWci8IqqwfG0jsqkRGgVD2YY08VUpIIHuzI0bQoOx6kVQB/mqQpsF+bXGZQ+8iA94bfp JyYNggJPHxg2f6Xb/jKa4g7hZymCmQ8UmVSt9EuxKRKdVZtO0n69k0leo4gDVwfoL9t/ Os39cCjqAPQ65OksPmlge1ykDLRpgobhXkTK3mjMtDgUS//kkSkTGFbxR8arlJi+VKMp XjMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730881294; x=1731486094; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to; bh=fNbPD93Ni6IXi4nxpMBsz/N/ABtxqebG0Tfp1ktjE/M=; b=ZG5iPS1XAttTg4AnWSUjJxi0dgCP5L1pDlFP1wkWyzgUVgo89pBwXW5HwNNMIQdo02 BaaychgI510NuKttz/tv/Z/SbP5NyuJvBPEUHAALjMx+dnQsH0+P+cn+9GJ6suhIe18r 8OkgZG1LAryC7u0kF94MdGetuWScAKf/R/rOPSAk4DOtinrJtNl3XGiDkdpHTm58okvs 8e5ExDxmbUNbicrpXREwvTbrNDDL1XWgyzUUg7L/UGeYCsIcPkftO0/zBQTWqMcJXZgi +KbQIjXn23pKCIdhAPx7XV76MhJe1ARghXpcO4ev+Qk3Dq2/Q1Kg3//tvEve3ynLrdlW EkuQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWlxdTWCygFpGJna+husvThgx4y/Iy4lypofyQ9VzpjvYnTuYYCv3xaw4vREfccp3k1AwzvMnk=@isar-build.org X-Gm-Message-State: AOJu0Yyac2raiqtGjRP7KOnX+fVp3QlucV6cIAaVq+tUeRncWegrACzF 1ftxH/pBmu0N7GnldUERhAgBMSq/A14X7dhmzcFwg7RV9N/xxHir X-Google-Smtp-Source: AGHT+IEjsq1CstabVEN64wMMpTbEpFgDkxkyoBbfSMvcBx4jphXoFqMkAYjgAqgmArfSAAtbW7S6Cw== X-Received: by 2002:a2e:be85:0:b0:2fb:56c1:bf with SMTP id 38308e7fff4ca-2fcbe04bb4dmr61645511fa.7.1730881293473; Wed, 06 Nov 2024 00:21:33 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:1f01:0:b0:2fb:358c:f74d with SMTP id 38308e7fff4ca-2fdeb667f6als13144761fa.2.-pod-prod-04-eu; Wed, 06 Nov 2024 00:21:31 -0800 (PST) X-Received: by 2002:a2e:b8c5:0:b0:2fc:9afe:1157 with SMTP id 38308e7fff4ca-2fedb794dc0mr87374451fa.2.1730881291091; Wed, 06 Nov 2024 00:21:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1730881291; cv=none; d=google.com; s=arc-20240605; b=Vldtgr24Fn9c8yJTA0niovUVIXpqRhvzEroZ3jAweNWkpZiDoHFvsEX1qfbn/oAdhf hFZpVVdSzyO/1ZhUUfZkxFZ83HeORDZs2c0BOyZFykNPNLn705oKWZSpTK1YQNmKw+Hn HKnxeyiQqLPimobc1mwp8e+QSqR9wjlAuzTT4iT72yLdV9Ttlx/qXzgUujal6fKbIiDz zYgEOJyokxvzoVCvVQZled3yuXAAA15k7t+f8r4UDR4aOXKCSc1mvpy/aTMCEUbPaN4D oiUWJk1/OG5qlWMcQgJ0DnjYD45jNKuZaQO4vV7gfnwzZ3JeOWhxiodvEMMjEGvTPnBs gcAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=FTzM+SpSU14IEU9VKNUCKCG3FornmhDhsewSCHJ+vT8=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=FLXGTupw0m7Pv7IbWcrWpDEHbD9D6GJNJfsjsxeDi9XvjhD+GathRlWt7djULV+wUN U9bMp897i+jqf97dPR6RB7tPhNqIaTCv94bxj4Z4+M0hcaWn0MuUZ/Z8dRuKhwv1qubW HKwsuH6ZwDNI5l9ZYfcDFtxHqfVQpeQBFlnOSEcKF/ydmquV0pEMSMl7k68ALcZThDV8 LWwBm96BxDQ1e0gP4Zp3UnK0Ea3JRYVeQhSuWhb3hiPEUzhZPIn6c2fmVl1ngpI8KRTf 2YBW6jV42fT1kAs/HbfEDItYmFoykUBYhVwXqE7OWcOTXn0Y2cChqAj1rQ+eKviYQFQx 9Tsg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5cee6b05269si128500a12.3.2024.11.06.00.21.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 06 Nov 2024 00:21:31 -0800 (PST) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from user-B660.promwad.corp ([159.148.83.114]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 4A68LOjw016956 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 6 Nov 2024 09:21:30 +0100 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH v11 7/8] isar-bootstrap: Use tar output instead of directory Date: Wed, 6 Nov 2024 10:21:16 +0200 Message-Id: <20241106082117.1089554-8-amikan@ilbers.de> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241106082117.1089554-1-amikan@ilbers.de> References: <20241106082117.1089554-1-amikan@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Do not unpack tar with rootfs content inside bootstrap recipe. This change can allow to remove sudo usage during the bootstrap later. Using tar output requires moving chroot prepare step to the later unpack step. Signed-off-by: Anton Mikanovich --- meta/classes/rootfs.bbclass | 11 +++++--- .../isar-bootstrap/isar-bootstrap.inc | 25 ++++++++++--------- .../isar-mmdebstrap/isar-mmdebstrap.inc | 22 ++++------------ 3 files changed, 25 insertions(+), 33 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index e339d24f..dee724a0 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -105,8 +105,8 @@ rootfs_do_qemu() { fi } -BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}" -BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}" +BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst" +BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst" def rootfs_extra_import(d): bb.utils._context["rootfs_progress"] = __import__("rootfs_progress") @@ -116,7 +116,10 @@ ROOTFS_EXTRA_IMPORTED := "${@rootfs_extra_import(d)}" rootfs_prepare[weight] = "25" rootfs_prepare(){ - sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' + sudo tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console" + + # setup chroot + sudo "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" } ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" @@ -275,7 +278,7 @@ cache_deb_src() { # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and # relies on that through out the build. Copy that state information instead of apt-get update # which generates a new state from upstream. - sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/" + sudo tar -xf "${BOOTSTRAP_SRC}" ./var/lib/apt/lists --one-top-level="${ROOTFSDIR}" deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 0e5ab57d..f9dae26a 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -183,9 +183,8 @@ do_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y dpkg - # setup chroot + # prepare setup chroot script install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh" - "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ @@ -199,8 +198,12 @@ do_bootstrap() { mountpoint -q "${ROOTFSDIR}/base-apt" && \ umount "${ROOTFSDIR}/base-apt" + # Compress rootfs for compatibility + lopts="--one-file-system --exclude=var/cache/apt/archives" + tar --zstd -cf "${WORKDIR}/rootfs.tar.zst" $lopts -C "${ROOTFSDIR}" . + # Finalize debootstrap by setting the link in deploy - ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" + ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" EOSUDO deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" @@ -216,19 +219,17 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" bootstrap_sstate_prepare() { # this runs in SSTATE_BUILDDIR, which will be deleted automatically - lopts="--one-file-system --exclude=var/cache/apt/archives" - sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}") - sudo chown $(id -u):$(id -g) bootstrap.tar + sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst + sudo chown $(id -u):$(id -g) bootstrap.tar.zst } bootstrap_sstate_finalize() { # this runs in SSTATE_INSTDIR - # - after building the bootstrap, the tar won't be there, but we also don't need to unpack - # - after restoring from cache, there will be a tar which we unpack and then delete - if [ -f bootstrap.tar ]; then - sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar - sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" - rm bootstrap.tar + # we should restore symlinks after using tar + if [ -f bootstrap.tar.zst ]; then + mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst" + sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \ + "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" fi } diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc index 4128ad80..1043f2d1 100644 --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc @@ -11,7 +11,6 @@ inherit deb-dl-dir FILESEXTRAPATHS:append = ":${LAYERDIR_core}/recipes-core/isar-bootstrap/files" -ROOTFSDIR = "${WORKDIR}/rootfs" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,apt,usrmerge" BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir" @@ -73,9 +72,6 @@ do_bootstrap() { fi E="${@ isar_export_proxies(d)}" - sudo rm -rf --one-file-system "${ROOTFSDIR}" - mkdir -p "${ROOTFSDIR}" - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)" bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list" @@ -182,17 +178,9 @@ do_bootstrap() { "${WORKDIR}/rootfs.tar.zst" \ "$bootstrap_list" - sudo -E -s <<'EOSUDO' - set -e - - tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console" - - # setup chroot - "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" + # Finalize bootstrap by setting the link in deploy + sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" - # Finalize bootstrap by setting the link in deploy - ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" -EOSUDO if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" sudo rm -rf --one-file-system "${WORKDIR}/dl_dir" @@ -206,7 +194,7 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" bootstrap_sstate_prepare() { # this runs in SSTATE_BUILDDIR, which will be deleted automatically - sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst + sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst sudo chown $(id -u):$(id -g) bootstrap.tar.zst } @@ -214,8 +202,8 @@ bootstrap_sstate_finalize() { # this runs in SSTATE_INSTDIR # we should restore symlinks after using tar if [ -f bootstrap.tar.zst ]; then - mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" - sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \ + mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst" + sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \ "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" fi }