From patchwork Tue Oct 12 04:04:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 929 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Tue, 12 Oct 2021 15:04:31 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f56.google.com (mail-lf1-f56.google.com [209.85.167.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 19CD4SCS003873 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 12 Oct 2021 15:04:29 +0200 Received: by mail-lf1-f56.google.com with SMTP id p42-20020a05651213aa00b003fd8935b8d6sf5310627lfa.10 for ; Tue, 12 Oct 2021 06:04:29 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1634043863; cv=pass; d=google.com; s=arc-20160816; b=pgVXIhg4gKJC2hsoL8OioAuF7PsOV+4jlV92UdDCsbvMAB9CrkSlBvU4tWE5quNd8W /IkbEwzJMhvoGOasCS9MtSRJPXgQzcooe+e+qzqEnl5x3ES9ScCVWijNxqb22Isix5ZW TdLuOEt7EjONoTUVYsyFZWzrTgZplQU5fRwCoBTdMZHXeqjFmJOk+vo9wejwJQmxmWSf +FbodqLaGkej1hQY57bD9IPOQjtbNgmIAJO6W/TdFXLbeWbAvOUJiiztmW5IDU7ldXOo 3BpKisRVay5vwGWoEmtwSenAAyAwAMrNSbnq5V+1J2BGgsAN116je+Cjp1QyXtlx/8Pa 2pBQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=ylc4EvjGQ7c/s7Yr49OLkHPIvKMNib1C4mXwzaXFh+w=; b=f6nG89T0i1WM6CMDZQEDCHl190C0es8l4TOp3jiW41KbhIVM+0YorlHqL33UsxFK67 cLyu8rlvAhcj2Km9ZmeQTVY1FZE6GRvIEHGS3cqnbf0xxzgq8KMEsSvXS050rsydkHF3 iTJUI/sGUYaRFdImwKj/Og66K4RDZXao9BD0cWWi/SZ4oUVRpVQK/qyI2FbSGRu6YPfY LxC6sZN3DNdktWlDMiAYMSrh3e9Wltu7RXkndSs1Mr68DsVE2p/+x6oBI7+iB5mshpNt smznz6H79h3cTMz/qWpjwjR62ozwQwawYj17LvLpsxv0QukEh5pBXpaSqDBBW4V+LGVO 0Zgg== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=ylc4EvjGQ7c/s7Yr49OLkHPIvKMNib1C4mXwzaXFh+w=; b=dO7Ii0iUPIROkrQuSVX/x1PFu+cOilusCWyiDBzsOdj9lGsOsaauMXEYuk8iowOmEO UEsm0/TQrybmdQSwjdoltgqf35l/Oc8F4puvFwNgMba5M7puHxwt+r+HyjGB0aDPgrjQ 02iZfl+3Vi2w3jQtusrYk9w4GYKk8M0C20wDgECg80OyOE1yGtU8oX+fl2xzy9hH1YPB 4B+GvkClBmszCVe/K2vYVqwNfLHJ/+lZiRdU8WEUbdd+aOZzwcRU9uqPrscTZvlBdXeh qO9wtGGPnSQgy20NuuRa6KkH5G9faSqJKrG4akFlWZykb82eqp0PChibnbQWn3LBA8sC SAvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=ylc4EvjGQ7c/s7Yr49OLkHPIvKMNib1C4mXwzaXFh+w=; b=LPbrknfpf30/dcD5U3sG3z6p4BicaO5vdNsSgB/dxvZQ9HmfEQ66QP0jRIAIZ+gMXz Eh6vBEsHfkOhWrwhXk2e6B60BIRmIDeY0TvUQTTKbMbXgbyD9e6b7VIyvdSbPgumtSgs W53dkR0eLO/FWqCt8hvKCYkb6VysepmYmi38uQ7WCZ5sEmoA15+d/vnvpuiKh10ZH5KU 3KgGCZvquNmkFRNHlJnXxtV9K2+wFw7iCx71p0aAOkuk7v4m6WyvI3QLWRyIm112iSoU W9EBdcXpn2hVjRiekudZiPHhQA6zTN207XyNN37/zvxS8IjaHZfobOC/By5KfU2jibzh +klg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM533cR9dCQ4iucLpIMNqF0Exfhi3fgIWBD8QTxTplIhddxQWvI5vJ +DxiJ07n6w1eGjv1H9skrD4= X-Google-Smtp-Source: ABdhPJx6uld7keo997o/RmTvJiyWJ5wPSCIivZBPTAIF70rdFan40cU7nAiLkyOZD/e/F/EibXJmiw== X-Received: by 2002:ac2:5a0c:: with SMTP id q12mr24027615lfn.133.1634043861465; Tue, 12 Oct 2021 06:04:21 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3994:: with SMTP id j20ls381216lfu.3.gmail; Tue, 12 Oct 2021 06:04:19 -0700 (PDT) X-Received: by 2002:ac2:508b:: with SMTP id f11mr33859492lfm.239.1634043859114; Tue, 12 Oct 2021 06:04:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634043859; cv=none; d=google.com; s=arc-20160816; b=zWnIjDQhctfPspS3gwkyfhz+rqOfdncT4ZWoJWAB1TL2qbi1qiyMxOMhhwzdoV0zVa qnC7EuGivIAVvEqVUJWFM2gDrQND9oDBwAoXCkBnJO0r5crs7s0EFawbBaok5zljxUW6 FWWGnnpbSd1ySEjxPFajRF1mlO9R1s9Bkqh0tZQ6WWeKfiOesHpvwSPAm997CiDhWyOo fiSD44H9UtmXHC31m1kJCa/9ubUuaG8rmB1XpQZiKqPAWlPycNwKN+NDVz++dsOBW+Iv GAAYHUYsdy+8u1xteobA1Dqp8B5psrviBaEm2LhZiU8+zv3sNh/B3sSqFPI0zj3fh5yO N/+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=sqPiDSQPMdLvKxuY2s1Cw1VDI80qcZKwjNzwhLAqFLQ=; b=Er3ozXDPUJM4oHRTSQ74T/XavEnADH5Jo1Fj4xI6P58294YgmbGJ+9d/fEbFtBBVNj mCLkilHNpOgOfrQf5E+94ZYpZCnyDlAgMX9I5bORKcPAoMFXG/XH/yxXUjugLTojKUn8 0TUl+qq3yd9jdQU0xXh8TfzzczuuKjFS79oqv7DZnTEz24D3AOB54/5XWIJtothuukac EkSwIW6lzVhSw+IVGdMV0/BRREwztLOce5LK47Rjd6Rf4J3QsORJjGT5/HnH36Bj+4TR cob+jOP0g/MDup8RNos9Fw7hDWeFwX+ckHzmiu7rz/dVnAm+yvITqCxhvfA7p6sorsKX yX2w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id z21si551797ljn.1.2021.10.12.06.04.18 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Oct 2021 06:04:19 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id 19CD4Itc002048 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 12 Oct 2021 15:04:18 +0200 Received: from random.ppmd.siemens.net (random.ppmd.siemens.net [139.25.68.25]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 19CD4HmN009739; Tue, 12 Oct 2021 15:04:18 +0200 From: Adriaan Schmidt To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: [RFC PATCH 5/5] meta: refactor to use the new mounting mechanism Date: Tue, 12 Oct 2021 15:04:13 +0200 Message-Id: <20211012130413.1719424-6-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211012130413.1719424-1-adriaan.schmidt@siemens.com> References: <20211012130413.1719424-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX Signed-off-by: Adriaan Schmidt --- meta/classes/buildchroot.bbclass | 52 ++++++++----------- meta/classes/cpiogz-img.bbclass | 3 +- meta/classes/dpkg-base.bbclass | 44 +++------------- meta/classes/dpkg-gbp.bbclass | 2 - meta/classes/dpkg.bbclass | 4 +- meta/classes/ext4-img.bbclass | 3 +- meta/classes/fit-img.bbclass | 4 +- meta/classes/image-tools-extension.bbclass | 4 +- meta/classes/image.bbclass | 30 +++-------- meta/classes/initramfs.bbclass | 2 +- meta/classes/isar-events.bbclass | 4 +- meta/classes/rootfs.bbclass | 50 ++++++------------ meta/classes/ubi-img.bbclass | 3 +- meta/classes/ubifs-img.bbclass | 3 +- meta/classes/vm-img.bbclass | 7 +-- meta/classes/wic-img.bbclass | 31 ++++------- .../isar-bootstrap/isar-bootstrap.inc | 43 +++++++++------ .../buildchroot/buildchroot.inc | 8 +-- 18 files changed, 104 insertions(+), 193 deletions(-) diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass index e9eb9af..7c34834 100644 --- a/meta/classes/buildchroot.bbclass +++ b/meta/classes/buildchroot.bbclass @@ -13,50 +13,42 @@ python __anonymous() { (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"): dep = "buildchroot-target:do_build" rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True) + mount_ctx = "buildchroot-target" else: dep = "buildchroot-host:do_build" rootfs = d.getVar('BUILDCHROOT_HOST_DIR', True) + mount_ctx = "buildchroot-host" d.setVar('BUILDCHROOT_DEP', dep) d.setVar('BUILDCHROOT_DIR', rootfs) + d.setVar('MOUNTS_CONTEXT', mount_ctx + "-" + d.getVar('DISTRO') + "-" + d.getVar('DISTRO_ARCH')) } -MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}.lock" +# mount settings +BUILDCHROOT_MOUNTS = " \ + bind:${REPO_ISAR_DIR}/${DISTRO}:${BUILDCHROOT_DIR}/isar-apt \ + bind:${DL_DIR}:${BUILDCHROOT_DIR}/downloads \ + rbind:/dev:${BUILDCHROOT_DIR}/dev \ + proc::${BUILDCHROOT_DIR}/proc \ + rbind:/sys:${BUILDCHROOT_DIR}/sys \ + ${@oe.utils.vartrue("ISAR_USE_CACHED_BASE_REPO", "bind:${REPO_BASE_DIR}:${BUILDCHROOT_DIR}/base-apt", "", d)} \ + " + +python () { + # find all tasks that want to use buildchroot + for task in [t for t in d.keys() if d.getVarFlag(t, 'task') and d.getVarFlag(t, 'buildchroot') == '1']: + d.prependVarFlag(task, 'prefuncs', "buildchroot_task_prefunc ") +} -buildchroot_do_mounts() { +buildchroot_task_prefunc() { sudo -s <<'EOSUDO' - ( flock 9 set -e - - mountpoint -q '${BUILDCHROOT_DIR}/isar-apt' || - mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${BUILDCHROOT_DIR}/isar-apt' - mountpoint -q '${BUILDCHROOT_DIR}/downloads' || - mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads' - mountpoint -q '${BUILDCHROOT_DIR}/dev' || - mount --rbind /dev '${BUILDCHROOT_DIR}/dev' - mount --make-rslave '${BUILDCHROOT_DIR}/dev' - mountpoint -q '${BUILDCHROOT_DIR}/proc' || - mount -t proc none '${BUILDCHROOT_DIR}/proc' - mountpoint -q '${BUILDCHROOT_DIR}/sys' || - mount --rbind /sys '${BUILDCHROOT_DIR}/sys' - mount --make-rslave '${BUILDCHROOT_DIR}/sys' - - # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set - if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] - then - mkdir -p '${BUILDCHROOT_DIR}/base-apt' - mountpoint -q '${BUILDCHROOT_DIR}/base-apt' || \ - mount --bind '${REPO_BASE_DIR}' '${BUILDCHROOT_DIR}/base-apt' - fi - - # Refresh or remove /etc/resolv.conf at this chance + # Refresh or remove /etc/resolv.conf if [ "${@repr(bb.utils.to_boolean(d.getVar('BB_NO_NETWORK')))}" = 'True' ] then - rm -rf '${BUILDCHROOT_DIR}/etc/resolv.conf' - else + rm -f '${BUILDCHROOT_DIR}/etc/resolv.conf' + elif [ -d '${BUILDCHROOT_DIR}/etc' ]; then cp -L /etc/resolv.conf '${BUILDCHROOT_DIR}/etc' fi - - ) 9>'${MOUNT_LOCKFILE}' EOSUDO } diff --git a/meta/classes/cpiogz-img.bbclass b/meta/classes/cpiogz-img.bbclass index 940e2fb..095e133 100644 --- a/meta/classes/cpiogz-img.bbclass +++ b/meta/classes/cpiogz-img.bbclass @@ -10,12 +10,11 @@ CPIO_IMAGE_FORMAT ?= "newc" do_cpiogz_image() { sudo rm -f ${CPIOGZ_IMAGE_FILE} - image_do_mounts sudo chroot ${BUILDCHROOT_DIR} \ sh -c "cd ${PP_ROOTFS}; /usr/bin/find . | \ /usr/bin/cpio -H ${CPIO_IMAGE_FORMAT} -o | /usr/bin/gzip -9 > \ ${PP_DEPLOY}/${CPIOGZ_FNAME}" sudo chown $(id -u):$(id -g) ${CPIOGZ_IMAGE_FILE} } - +do_cpiogz_image[mounts] = "${IMAGE_MOUNTS}" addtask cpiogz_image before do_image after do_image_tools diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 8a39a6d..71c1acd 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -96,7 +96,6 @@ python() { } do_apt_fetch() { - dpkg_do_mounts E="${@ isar_export_proxies(d)}" sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \ @@ -107,10 +106,9 @@ do_apt_fetch() { sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}" done - - dpkg_undo_mounts } - +do_apt_fetch[mounts] = "${DPKG_MOUNTS}" +do_apt_fetch[buildchroot] = "1" addtask apt_fetch after do_unpack before do_apt_unpack do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" @@ -119,7 +117,6 @@ do_apt_fetch[depends] = "${BUILDCHROOT_DEP}" do_apt_unpack() { rm -rf ${S} - dpkg_do_mounts E="${@ isar_export_proxies(d)}" for uri in "${SRC_APT}"; do @@ -132,10 +129,9 @@ do_apt_unpack() { dpkg-source -x "${dscfile}" "${PPS}"' \ my_script "${DISTRO}" "${uri}" done - - dpkg_undo_mounts } - +do_apt_unpack[mounts] = "${DPKG_MOUNTS}" +do_apt_unpack[buildchroot] = "1" addtask apt_unpack after do_apt_fetch before do_patch addtask cleanall_apt before do_cleanall @@ -174,27 +170,7 @@ do_prepare_build[deptask] = "do_deploy_deb" BUILDROOT = "${BUILDCHROOT_DIR}/${PP}" -dpkg_do_mounts() { - mkdir -p ${BUILDROOT} - sudo mount --bind ${WORKDIR} ${BUILDROOT} - - buildchroot_do_mounts -} - -dpkg_undo_mounts() { - i=0 - while ! sudo umount ${BUILDROOT}; do - sleep 0.1 - if [ `expr $i % 100` -eq 0 ] ; then - bbwarn "${BUILDROOT}: Couldn't unmount ($i), retrying..." - fi - if [ $i -ge 10000 ]; then - bbfatal "${BUILDROOT}: Couldn't unmount after timeout" - fi - i=`expr $i + 1` - done - sudo rmdir ${BUILDROOT} -} +DPKG_MOUNTS = "bind:${WORKDIR}:${BUILDROOT} ${BUILDCHROOT_MOUNTS}" # Placeholder for actual dpkg_runbuild() implementation dpkg_runbuild() { @@ -204,14 +180,13 @@ dpkg_runbuild() { python do_dpkg_build() { lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", shared=True) - bb.build.exec_func("dpkg_do_mounts", d) try: bb.build.exec_func("dpkg_runbuild", d) finally: - bb.build.exec_func("dpkg_undo_mounts", d) bb.utils.unlockfile(lock) } +do_dpkg_build[mounts] = "${DPKG_MOUNTS}" addtask dpkg_build before do_build KEEP_INSTALLED_ON_CLEAN ?= "0" @@ -248,18 +223,15 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" do_deploy_deb[dirs] = "${S}" python do_devshell() { - bb.build.exec_func('dpkg_do_mounts', d) - isar_export_proxies(d) buildchroot = d.getVar('BUILDCHROOT_DIR') pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS')) termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'" oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d) - - bb.build.exec_func('dpkg_undo_mounts', d) } - +do_devshell[mounts] = "${DPKG_MOUNTS}" +do_devshell[buildchroot] = "1" addtask devshell after do_prepare_build DEVSHELL_STARTDIR ?= "${S}" do_devshell[dirs] = "${DEVSHELL_STARTDIR}" diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass index d956e8c..e3bf305 100644 --- a/meta/classes/dpkg-gbp.bbclass +++ b/meta/classes/dpkg-gbp.bbclass @@ -13,7 +13,6 @@ GBP_DEPENDS ?= "git-buildpackage pristine-tar" GBP_EXTRA_OPTIONS ?= "--git-pristine-tar" do_install_builddeps_append() { - dpkg_do_mounts distro="${DISTRO}" if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_DISTRO}" @@ -26,7 +25,6 @@ do_install_builddeps_append() { sudo -E chroot ${BUILDCHROOT_DIR} \ apt-get install -y -o Debug::pkgProblemResolver=yes \ --no-install-recommends ${GBP_DEPENDS} - dpkg_undo_mounts } dpkg_runbuild_prepend() { diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 4e7c2f7..96aefba 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -7,7 +7,6 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}" # Install build dependencies for package do_install_builddeps() { - dpkg_do_mounts E="${@ isar_export_proxies(d)}" distro="${DISTRO}" if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then @@ -19,12 +18,13 @@ do_install_builddeps() { deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}" sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ ${PP}/${PPS} ${PACKAGE_ARCH} - dpkg_undo_mounts } addtask install_builddeps after do_prepare_build before do_dpkg_build # apt and reprepro may not run in parallel, acquire the Isar lock do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" +do_install_builddeps[mounts] = "${DPKG_MOUNTS}" +do_install_builddeps[buildchroot] = "1" addtask devshell after do_install_builddeps diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass index 334dc64..84dd4f2 100644 --- a/meta/classes/ext4-img.bbclass +++ b/meta/classes/ext4-img.bbclass @@ -13,11 +13,10 @@ do_ext4_image() { truncate -s ${ROOTFS_SIZE}K '${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}' - image_do_mounts - sudo chroot ${BUILDCHROOT_DIR} /sbin/mke2fs ${MKE2FS_ARGS} \ -F -d '${PP_ROOTFS}' '${PP_DEPLOY}/${EXT4_IMAGE_FILE}' } addtask ext4_image before do_image after do_image_tools do_ext4_image[prefuncs] = 'set_image_size' +do_ext4_image[mounts] = "${IMAGE_MOUNTS}" diff --git a/meta/classes/fit-img.bbclass b/meta/classes/fit-img.bbclass index 82b96d8..a34517a 100644 --- a/meta/classes/fit-img.bbclass +++ b/meta/classes/fit-img.bbclass @@ -18,11 +18,11 @@ do_fit_image() { rm -f '${DEPLOY_DIR_IMAGE}/${FIT_IMAGE_FILE}' - image_do_mounts - # Create fit image using buildchroot tools sudo chroot ${BUILDCHROOT_DIR} /usr/bin/mkimage ${MKIMAGE_ARGS} \ -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${PP_DEPLOY}/${FIT_IMAGE_FILE}' sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${FIT_IMAGE_FILE}' } + addtask fit_image before do_image after do_image_tools do_transform_template +do_fit_image[mounts] = "${IMAGE_MOUNTS}" diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 9f28800..265fb0c 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -17,13 +17,13 @@ DEPENDS += "${IMAGER_BUILD_DEPS}" do_install_imager_deps[depends] = "${BUILDCHROOT_DEP}" do_install_imager_deps[deptask] = "do_deploy_deb" do_install_imager_deps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" +do_install_imager_deps[mounts] = "${BUILDCHROOT_MOUNTS}" +do_install_imager_deps[buildchroot] = "1" do_install_imager_deps() { if [ -z "${@d.getVar("IMAGER_INSTALL", True).strip()}" ]; then exit fi - buildchroot_do_mounts - E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO} sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index ec93cab..dd934a7 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -52,15 +52,12 @@ DEPENDS += "${IMAGE_INSTALL}" ISAR_RELEASE_CMD_DEFAULT = "git -C ${LAYERDIR_core} describe --tags --dirty --match 'v[0-9].[0-9]*'" ISAR_RELEASE_CMD ?= "${ISAR_RELEASE_CMD_DEFAULT}" -image_do_mounts() { - sudo flock ${MOUNT_LOCKFILE} -c ' \ - mkdir -p "${BUILDROOT_DEPLOY}" "${BUILDROOT_ROOTFS}" "${BUILDROOT_WORK}" - mount --bind "${DEPLOY_DIR_IMAGE}" "${BUILDROOT_DEPLOY}" - mount --bind "${IMAGE_ROOTFS}" "${BUILDROOT_ROOTFS}" - mount --bind "${WORKDIR}" "${BUILDROOT_WORK}" - ' - buildchroot_do_mounts -} +IMAGE_MOUNTS = " \ + bind:${DEPLOY_DIR_IMAGE}:${BUILDROOT_DEPLOY} \ + bind:${IMAGE_ROOTFS}:${BUILDROOT_ROOTFS} \ + bind:${WORKDIR}:${BUILDROOT_WORK} \ + ${BUILDCHROOT_MOUNTS} \ + " ROOTFSDIR = "${IMAGE_ROOTFS}" ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status" @@ -190,21 +187,6 @@ do_rootfs_finalize() { find "${ROOTFSDIR}/usr/bin" \ -maxdepth 1 -name 'qemu-*-static' -type f -delete - mountpoint -q '${ROOTFSDIR}/isar-apt' && \ - umount -l ${ROOTFSDIR}/isar-apt - rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt - - mountpoint -q '${ROOTFSDIR}/base-apt' && \ - umount -l ${ROOTFSDIR}/base-apt - rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt - - mountpoint -q '${ROOTFSDIR}/dev' && \ - umount -l ${ROOTFSDIR}/dev - mountpoint -q '${ROOTFSDIR}/sys' && \ - umount -l ${ROOTFSDIR}/proc - mountpoint -q '${ROOTFSDIR}/sys' && \ - umount -l ${ROOTFSDIR}/sys - rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf" rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list" diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass index 10a642b..b5aba91 100644 --- a/meta/classes/initramfs.bbclass +++ b/meta/classes/initramfs.bbclass @@ -25,8 +25,8 @@ ROOTFS_PACKAGES = "initramfs-tools ${INITRAMFS_PREINSTALL} ${INITRAMFS_INSTALL}" inherit rootfs do_generate_initramfs[dirs] = "${DEPLOY_DIR_IMAGE}" +do_generate_initramfs[mounts] = "${ROOTFS_MOUNTS}" do_generate_initramfs() { - rootfs_do_mounts rootfs_do_qemu sudo -E chroot "${INITRAMFS_ROOTFS}" \ diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass index 92aff20..73419b4 100644 --- a/meta/classes/isar-events.bbclass +++ b/meta/classes/isar-events.bbclass @@ -8,8 +8,6 @@ addhandler build_started python build_started() { bb.utils.remove(d.getVar('TMPDIR') + "/work/*/*/*/temp/once.*") - bb.utils.remove(d.getVar('TMPDIR') + "/work/*/*/*/rootfs.mount") - bb.utils.remove(d.getVar('TMPDIR') + "/deploy/buildchroot-*/*.mount") } build_started[eventmask] = "bb.event.BuildStarted" @@ -54,7 +52,7 @@ python build_completed() { with open('/proc/mounts') as f: for line in f.readlines(): if basepath in line: - bb.debug(1, '%s left mounted, unmounting...' % line.split()[1]) + bb.warn('%s left mounted, unmounting...' % line.split()[1]) subprocess.call( ["sudo", "umount", "-l", line.split()[1]], stdout=subprocess.DEVNULL, diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index f9151c5..7b9fbb1 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -26,38 +26,17 @@ export LANG = "C" export LANGUAGE = "C" export LC_ALL = "C" -rootfs_do_mounts[weight] = "3" -rootfs_do_mounts() { - sudo -s <<'EOSUDO' - mountpoint -q '${ROOTFSDIR}/dev' || \ - mount --rbind /dev '${ROOTFSDIR}/dev' - mount --make-rslave '${ROOTFSDIR}/dev' - mountpoint -q '${ROOTFSDIR}/proc' || \ - mount -t proc none '${ROOTFSDIR}/proc' - mountpoint -q '${ROOTFSDIR}/sys' || \ - mount --rbind /sys '${ROOTFSDIR}/sys' - mount --make-rslave '${ROOTFSDIR}/sys' - - # Mount isar-apt if the directory does not exist or if it is empty - # This prevents overwriting something that was copied there - if [ ! -e '${ROOTFSDIR}/isar-apt' ] || \ - [ "$(find '${ROOTFSDIR}/isar-apt' -maxdepth 1 -mindepth 1 | wc -l)" = "0" ] - then - mkdir -p '${ROOTFSDIR}/isar-apt' - mountpoint -q '${ROOTFSDIR}/isar-apt' || \ - mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt' - fi - - # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set - if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] - then - mkdir -p '${ROOTFSDIR}/base-apt' - mountpoint -q '${ROOTFSDIR}/base-apt' || \ - mount --bind '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt' - fi - -EOSUDO -} +MOUNTS_CONTEXT = "${PN}-${DISTRO}-${DISTRO_ARCH}" +ROOTFS_MOUNTS = " \ + rbind:/dev:${ROOTFSDIR}/dev \ + proc::${ROOTFSDIR}/proc \ + rbind:/sys:${ROOTFSDIR}/sys \ + bind:${REPO_ISAR_DIR}/${DISTRO}:${ROOTFSDIR}/isar-apt \ + ${@oe.utils.vartrue("ISAR_USE_CACHED_BASE_REPO", "bind:${REPO_BASE_DIR}:${ROOTFSDIR}/base-apt", "", d)} \ + " + +mounts_task_prefunc[weight] = "3" +mounts_task_postfunc[weight] = "3" rootfs_do_qemu() { if [ '${@repr(d.getVar('ROOTFS_ARCH') == d.getVar('HOST_ARCH'))}' = 'False' ] @@ -153,13 +132,15 @@ do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}" do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build" do_rootfs_install[deptask] = "do_deploy_deb" +do_rootfs_install[mounts] = "${ROOTFS_MOUNTS}" +do_rootfs_install[mounts-noauto] = "1" python do_rootfs_install() { configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split() install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or "").split() # Mount after configure commands, so that they have time to copy # 'isar-apt' (sdkchroot): - cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds + cmds = ['rootfs_prepare'] + configure_cmds + ['mounts_task_prefunc'] + install_cmds + ['mounts_task_postfunc'] # NOTE: The weights specify how long each task takes in seconds and are used # by the MultiStageProgressReporter to render a progress bar for this task. @@ -230,9 +211,8 @@ rootfs_export_dpkg_status() { } do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" +do_rootfs_postprocess[mounts] = "${ROOTFS_MOUNTS}" python do_rootfs_postprocess() { - # Take care that its correctly mounted: - bb.build.exec_func('rootfs_do_mounts', d) # Take care that qemu-*-static is available, since it could have been # removed on a previous execution of this task: bb.build.exec_func('rootfs_do_qemu', d) diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass index c69ac4d..87f0187 100644 --- a/meta/classes/ubi-img.bbclass +++ b/meta/classes/ubi-img.bbclass @@ -21,11 +21,10 @@ do_ubi_image() { rm -f '${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}' - image_do_mounts - # Create ubi image using buildchroot tools sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \ -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' '${PP_WORK}/${UBINIZE_CFG}' sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}' } addtask ubi_image before do_image after do_image_tools do_transform_template +do_ubi_image[mounts] = "${IMAGE_MOUNTS}" diff --git a/meta/classes/ubifs-img.bbclass b/meta/classes/ubifs-img.bbclass index 5d48c1d..c6775d6 100644 --- a/meta/classes/ubifs-img.bbclass +++ b/meta/classes/ubifs-img.bbclass @@ -20,12 +20,11 @@ ISAR_CROSS_COMPILE_armhf = "1" do_ubifs_image() { rm -f '${DEPLOY_DIR_IMAGE}/${UBIFS_IMAGE_FILE}' - image_do_mounts - # Create ubifs image using buildchroot tools sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} \ -r '${PP_ROOTFS}' '${PP_DEPLOY}/${UBIFS_IMAGE_FILE}' sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${UBIFS_IMAGE_FILE}' } +do_ubifs_image[mounts] = "${IMAGE_MOUNTS}" addtask ubifs_image before do_image after do_image_tools diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass index b230af2..c6cfbf9 100644 --- a/meta/classes/vm-img.bbclass +++ b/meta/classes/vm-img.bbclass @@ -33,13 +33,12 @@ CONVERSION_OPTIONS = "${@set_convert_options(d)}" do_convert_wic() { rm -f '${DEPLOY_DIR_IMAGE}/${VIRTUAL_MACHINE_IMAGE_FILE}' - image_do_mounts bbnote "Creating ${VIRTUAL_MACHINE_IMAGE_FILE} from ${WIC_IMAGE_FILE}" sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ /usr/bin/qemu-img convert -f raw -O ${VIRTUAL_MACHINE_IMAGE_TYPE} ${CONVERSION_OPTIONS} \ '${PP_DEPLOY}/${SOURCE_IMAGE_FILE}' '${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE}' } - +do_convert_wic[mounts] = "${IMAGE_MOUNTS}" addtask convert_wic before do_build after do_wic_image do_copy_boot_files do_install_imager_deps do_transform_template # User settings for OVA @@ -92,8 +91,6 @@ do_create_ova() { export DISK_NAME=$(basename -s .vmdk ${VIRTUAL_MACHINE_DISK}) export LAST_CHANGE=$(date -u "+%Y-%m-%dT%H:%M:%SZ") - image_do_mounts - sudo -Es chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} <<'EOSUDO' export DISK_SIZE_BYTES=$(qemu-img info -f vmdk "${VIRTUAL_MACHINE_DISK}" \ | gawk 'match($0, /^virtual size:.*\(([0-9]+) bytes\)/, a) {print a[1]}') @@ -112,5 +109,5 @@ do_create_ova() { tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO } - +do_create_ova[mounts] = "${IMAGE_MOUNTS}" addtask do_create_ova after do_convert_wic before do_deploy diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index d849ad9..f8f9c0d 100644 --- a/meta/classes/wic-img.bbclass +++ b/meta/classes/wic-img.bbclass @@ -137,6 +137,17 @@ python check_for_wic_warnings() { } do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" +do_wic_image[mounts] = "${BUILDCHROOT_MOUNTS}" +python() { + buildchroot = d.getVar('BUILDCHROOT_DIR') + dirs = ((d.getVar('BBLAYERS') or '').split() + + (d.getVar('STAGING_DIR') or '').split() + + (d.getVar('SCRIPTSDIR') or '').split() + + (d.getVar('BITBAKEDIR') or '').split()) + for dir in dirs: + d.appendVarFlag('do_wic_image', 'mounts', f" pbind:{dir}:{buildchroot}{dir}") +} + python do_wic_image() { bb.build.exec_func("generate_wic_image", d) bb.build.exec_func("check_for_wic_warnings", d) @@ -144,17 +155,6 @@ python do_wic_image() { addtask wic_image before do_image after do_image_tools generate_wic_image() { - buildchroot_do_mounts - sudo -s <<'EOSUDO' - ( flock 9 - for dir in ${BBLAYERS} ${STAGING_DIR} ${SCRIPTSDIR} ${BITBAKEDIR}; do - mkdir -p ${BUILDCHROOT_DIR}/$dir - if ! mountpoint ${BUILDCHROOT_DIR}/$dir >/dev/null 2>&1; then - mount --bind --make-private $dir ${BUILDCHROOT_DIR}/$dir - fi - done - ) 9>${MOUNT_LOCKFILE} -EOSUDO export FAKEROOTCMD=${FAKEROOTCMD} export BUILDDIR=${BUILDDIR} export MTOOLS_SKIP_CHECK=1 @@ -200,13 +200,4 @@ EOSUDO done rm -rf ${BUILDCHROOT_DIR}/${WICTMP} rm -rf ${IMAGE_ROOTFS}/../pseudo - sudo -s <<'EOSUDO' - ( flock 9 - for dir in ${BBLAYERS} ${STAGING_DIR} ${SCRIPTSDIR} ${BITBAKEDIR}; do - if mountpoint -q ${BUILDCHROOT_DIR}/$dir; then - umount ${BUILDCHROOT_DIR}/$dir - fi - done - ) 9>${MOUNT_LOCKFILE} -EOSUDO } diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index b8af676..cc6c073 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -117,7 +117,6 @@ def get_apt_source_mirror(d, aptsources_entry_list): mirror_list = [entry.split() for entry in premirrors.split('\\n') if any(entry)] - for regex, replace in mirror_list: match = re.search(regex, aptsources_entry_list[2]) @@ -318,9 +317,6 @@ do_bootstrap() { fi echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" - - mkdir -p ${ROOTFSDIR}/base-apt - mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt else install -v -m644 "${APTSRCS}" \ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" @@ -364,13 +360,27 @@ do_bootstrap() { install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" - # update APT - mount --rbind /dev ${ROOTFSDIR}/dev - mount --make-rslave ${ROOTFSDIR}/dev - mount -t proc none ${ROOTFSDIR}/proc - mount --rbind /sys ${ROOTFSDIR}/sys - mount --make-rslave ${ROOTFSDIR}/sys +EOSUDO + deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_DISTRO}" +} +addtask bootstrap before do_build after do_generate_keyrings +do_bootstrap[vardeps] += " \ + DISTRO_APT_PREMIRRORS \ + ISAR_ENABLE_COMPAT_ARCH \ + ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ + " +do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" +do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config" + + +do_bootstrap_finalize() { + E="${@ isar_export_proxies(d)}" + export BOOTSTRAP_FOR_HOST E + + deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_DISTRO}" + + sudo -E -s <<'EOSUDO' export DEBIAN_FRONTEND=noninteractive if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then @@ -386,18 +396,19 @@ do_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ -o Debug::pkgProblemResolver=yes - umount -l "${ROOTFSDIR}/dev" - umount -l "${ROOTFSDIR}/proc" - umount -l "${ROOTFSDIR}/sys" - umount -l "${ROOTFSDIR}/base-apt" || true - # Finalize debootstrap by setting the link in deploy ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" EOSUDO deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_DISTRO}" } -addtask bootstrap before do_build after do_generate_keyrings +addtask bootstrap_finalize after do_bootstrap before do_build +do_bootstrap_finalize[mounts] = " \ + rbind:/dev:${ROOTFSDIR}/dev \ + proc::${ROOTFSDIR}/proc \ + rbind:/sys:${ROOTFSDIR}/sys \ + ${@oe.utils.vartrue("ISAR_USE_CACHED_BASE_REPO", "bind:${REPO_BASE_DIR}:${ROOTFSDIR}/base-apt", "", d)} \ + " CLEANFUNCS = "clean_deploy" clean_deploy() { diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc index 31524a1..ea4a3ba 100644 --- a/meta/recipes-devtools/buildchroot/buildchroot.inc +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc @@ -41,13 +41,7 @@ BUILDCHROOT_PREINSTALL_COMMON = " \ equivs \ adduser" -rootfs_do_mounts_append() { - sudo -s <<'EOSUDO' - mkdir -p '${BUILDCHROOT_DIR}/downloads' - mountpoint -q '${BUILDCHROOT_DIR}/downloads' || \ - mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads' -EOSUDO -} +ROOTFS_MOUNTS += "bind:${DL_DIR}:${BUILDCHROOT_DIR}/downloads" ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files" buildchroot_install_files() {