From patchwork Tue Oct 12 04:04:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 926 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Tue, 12 Oct 2021 15:04:27 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f58.google.com (mail-lf1-f58.google.com [209.85.167.58]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 19CD4P9x003862 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 12 Oct 2021 15:04:25 +0200 Received: by mail-lf1-f58.google.com with SMTP id k18-20020a05651210d200b003fd86616d39sf5419347lfg.2 for ; Tue, 12 Oct 2021 06:04:25 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1634043860; cv=pass; d=google.com; s=arc-20160816; b=BlNvhUMYo8fnQf/3H0mNyI2mr+ZHfIWRcT+sNWksN5c6+AMkIKMsXZ58wKN56ZSE+2 zN2/d9COu1vl/durkSs8Tmchm00LiUPnZhBJ7axr6jrDdU1DGyW3BuIH2IjNDSyB4DFf D/ulgTuN4CZVH4ID0ku0HSMETvGuaLHy4teZ8XrVZ2NTeSj5hVXIRKgGXQ2OlKyT1ZKP qv6y6JDXqol9KXxOrP8MLvV6nca1gjs/RJgceRqpJluAxJASV7uexYAI++CzkaM6kDy3 Kk9c/QMlTyzRQACS0JxEQX9Q+Kt2iX3t1a1kBDclzfa328+6MS3n0RdocM0y0Exf+4PS UPJA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=NswFgf56vfLRtdEyB8b50trkINk3lkncLFyswNffQJo=; b=bzdFw2fcIaWJflY6A5uQEWtSnZBNkr+Mr+S832reMXfwMEfn325ujKaAAkm29dDl5O RykI8AHDaql5xsY/iv/AFcuT8d9BdGWX3oDP/u1LGTeUIy/M2FlL1bYZr6i0Wj+nVs3Q M4Q5HXcqOC8lsW0lP0vKwMIUoiIRiZnZHlNKeNsphLn35VtI3ICuYiCJIj7qVWmX6dGx KDMqdNxHsim6Bcx+NWhC+azkwwO5A7URLfiz/1K14A5chsBME3f5v6BqO603cK7DtUBS X44tY/s/y+Ckh5s/31oMlcmC8+DadJy0NZW6lYkCxVDp8RrRt7THt/P3Oqm/DLo3mtTj ZfqQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=NswFgf56vfLRtdEyB8b50trkINk3lkncLFyswNffQJo=; b=Fy/gx2VybpzDC3lEXGdrtNjiBDW6/wRN0DbFBGo8ebelO/7beUyn711PWSl1VmLfXa oQ24wlssGdwuyWynKyP72mCEopdeP0Np+bhaPlVNoeRZ/JyO9L5E4ZfImkGnK9Yi9xpr WgaEsOm9+wKkVC7Mdy1y3mcP+GFKA3He8wySdRYI7jGnw1ro1LLE6UETm7r23jb2OyqI WQcWgH1MWGnf+iwP74HAFy2F6j4ngNcKexadcdQ4wnW+ZrPqh9DdPsK7JLPmAWARrJkr s8Xk/XBebXdB7CnW7QoN/ZyqRS/AX+1JYVnzc3zBCH8c6nF1Sl5LZGjRpQN6dOWhDk0d cIrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=NswFgf56vfLRtdEyB8b50trkINk3lkncLFyswNffQJo=; b=sS8DiwfykMN1Ib37vL7kWTWgu/IKEo/5MBIPcPT/tTn+OthkGb67AG3eqjz9XPz7y1 T3l7ka6ghdzfaQNXkRsbbG63BsU5Jn+icOC1by6PKchvSGu5asUPmGPnWQRyBVwDBLdC +3JeB4xMn51J0kcHiDGPrj6mnMdniSXWYMMzUMhORU6C10UGuCnprh9AuoTztbMMgZsQ BA4gtxBE5ZJL1ivUYXfgXCoJ4xSPZQeXu+xVJK+aU47n/wE1/igcxErEyt/SfYdR2mBx t1+DFmAEExFuWt2j0eYmu1dDjwLHU3tWN0whZejVwOOXJgxypYU7gaYrxddpodVD8+q3 zwZw== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM5306vNFtMS0mMs5t3Zbnf0kUMeSf3pJgT+rVlfocKp4DxUevsoCt 2xkle4uBTxFXFHlbaXDkAaM= X-Google-Smtp-Source: ABdhPJwtXv6YkJnSurWstdBPLy5WssjGZuJnucsiDWeg251/FkwGppqOq8bqHC+XNwu/TyDWCGA7sQ== X-Received: by 2002:a2e:a7d3:: with SMTP id x19mr15618301ljp.506.1634043860092; Tue, 12 Oct 2021 06:04:20 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3a83:: with SMTP id q3ls380726lfu.2.gmail; Tue, 12 Oct 2021 06:04:19 -0700 (PDT) X-Received: by 2002:ac2:5e32:: with SMTP id o18mr17151216lfg.556.1634043859059; Tue, 12 Oct 2021 06:04:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634043859; cv=none; d=google.com; s=arc-20160816; b=g2uGucYuA72221k0bFLpsIw4R/8deur5ZnD1pj1MkJwxJmjq5s8z3rC7lIps7c3lO4 Bv6CsJRkygO7H29yyltSpB+h7WfU0LH0pQKF6yyJGWuqWURbQOgMc1Bwwlk+UBjBvjWk H+/B1mUwxD+L2TWqONyYw1RF7WH4wAQbXYP+v+rnid9eYtwIuM1TatRjvqC+LaxLu5HD aUrTnppCJ1dBcChCcNr5Aa70qFmAz2D68CjsGC9JwhxYmf+2aDzwy3gU2ctVaDnXGTKm rN5IudzPGARbg/dYG/9XDdsUoGHK/GfMxgDEMmJ277WCKdEm2AXRq4EGXHxnOHl6JVFP O0+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=p8gCRFkbGI+SN61p6tRtU2xdRnDyHbdhgrNhdRS9iww=; b=mmThQ9AU21NJEuSdX4sHoIbnEKOlWjHZdmbHHqkppkgmHwngCyskhs0Ok3EgNQTZSw kcK3oG3rvRf+aN2ihSU5L26I6clsKPv3aZUb3InblFFfxN3UlNHHrDoh1a7VaeIgqhtF jUyL/cQKtTOELxVNur1aPTeJkcWwYx9EKLZ/LfQLj6UX0e20YaIAj8g563xQI41A2jPv KJ9TayHA2a5IrKC3rXFT/toX1zChZqPsU8q1H07EIb+4IWWey8ayV+26AGBuoncSYIwk 4vKp9YF+LGkA0BYp4buT1rR9noVllURTgCeplAhNNGKTrB/4TPa1pbLdqrQUH59qrVtN IC6A== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id o4si712492ljj.3.2021.10.12.06.04.18 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Oct 2021 06:04:19 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id 19CD4IUk010589 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 12 Oct 2021 15:04:18 +0200 Received: from random.ppmd.siemens.net (random.ppmd.siemens.net [139.25.68.25]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 19CD4HmM009739; Tue, 12 Oct 2021 15:04:18 +0200 From: Adriaan Schmidt To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: [RFC PATCH 4/5] meta: add mounts class Date: Tue, 12 Oct 2021 15:04:12 +0200 Message-Id: <20211012130413.1719424-5-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211012130413.1719424-1-adriaan.schmidt@siemens.com> References: <20211012130413.1719424-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX The new mounts.bbclass allows annotation of tasks, to describe which directories need to be mounted. All mounting and unmounting is then done automatically, and reference counting is used on a per-mountpoint basis to determine when umounts need to happen. Mounts are described as "cmd:src:dest", where cmd is * `bind` for a simple bind mount * `rbind` for a recursive bind mount * `pbind` for a "private" bind mount * `proc` for a "-t proc" mount A task is annotated using the varflag [mounts]. If mounting should not happen automatically before/after the task, you can set do_task[mounts-noauto] = "1", in which case you can manually call `mount_task_prefunc` and `mount_task_postfunc` at more convenient times. Signed-off-by: Adriaan Schmidt --- meta/classes/mounts.bbclass | 271 ++++++++++++++++++++++++++++++++++++ meta/conf/bitbake.conf | 2 +- 2 files changed, 272 insertions(+), 1 deletion(-) create mode 100644 meta/classes/mounts.bbclass diff --git a/meta/classes/mounts.bbclass b/meta/classes/mounts.bbclass new file mode 100644 index 0000000..de2375e --- /dev/null +++ b/meta/classes/mounts.bbclass @@ -0,0 +1,271 @@ + +python () { + # find all tasks that request [mounts], and hook up our functions + for task in [t for t in d.keys() if d.getVarFlag(t, 'task') and d.getVarFlag(t, 'mounts')]: + if d.getVarFlag(task, 'mounts-noauto') == "1": + continue + d.prependVarFlag(task, 'prefuncs', "mounts_task_prefunc ") + d.appendVarFlag(task, 'postfuncs', " mounts_task_postfunc") +} + +MOUNTS_DB = "${TMPDIR}/mounts" +MOUNTS_CONTEXT ?= "default" +MOUNTS_LOCK = "${MOUNTS_DB}/${MOUNTS_CONTEXT}.mountlock" +MOUNTS_TAB = "${MOUNTS_DB}/${MOUNTS_CONTEXT}.mounttab" + +def get_requested_mounts(d, task=None): + if task is None: + task = d.getVar('BB_CURRENTTASK') + if not task: + bb.fatal("mount code running without task context!?") + if task.startswith("do_"): + task = task[3:] + mounts = (d.getVarFlag("do_" + task, 'mounts') or "").split() + mounts_out = [] + for m in mounts: + ms = m.split(':') + if len(ms) == 3 and ms[0] in 'bind rbind pbind proc'.split(): + mounts_out.append(ms) + else: + bb.error(f"Invalid mount spec: {':'.join(ms)}") + return mounts_out + +def read_mtab(d, mtab_file=None): + from collections import namedtuple + Mount = namedtuple('Mount', 'cmd source target count') + if mtab_file is None: + mtab_file = d.getVar("MOUNTS_TAB", True) + # mtab format is "cmd:source:target:count" + try: + with open(mtab_file, 'r') as f: + data = [line.strip().split(':') for line in f.readlines()] + except FileNotFoundError: + return {} + mounts = {} + for m in data: + if not len(m) == 4: + bb.fatal("corrupt mtab!?") + mt = Mount._make(m) + mounts[mt.target] = mt._replace(count=int(mt.count)) + return mounts + +def write_mtab(d, mtab, mtab_file=None): + if mtab_file is None: + mtab_file = d.getVar("MOUNTS_TAB", True) + with open(mtab_file, 'w') as f: + for cmd, source, target, count in mtab.values(): + f.write(f"{cmd}:{source}:{target}:{count}\n") + +def shorten_path(x, n=3): + xs = x.split('/') + if len(xs) <= n: + return '/'.join(xs) + return '.../'+'/'.join(xs[-3:]) + +mount_bind() { + sudo -s <<'EOSUDO' + SOURCE="${@d.getVar('MOUNT_ARG_SOURCE')}" + TARGET="${@d.getVar('MOUNT_ARG_TARGET')}" + mkdir -p "$TARGET" + mountpoint -q "$TARGET" || mount --bind "$SOURCE" "$TARGET" +EOSUDO +} + +umount_bind() { + sudo -s <<'EOSUDO' + TARGET="${@d.getVar('UMOUNT_ARG_TARGET')}" + mountpoint -q "$TARGET" && umount "$TARGET" +EOSUDO +} + +mount_rbind() { + sudo -s <<'EOSUDO' + SOURCE="${@d.getVar('MOUNT_ARG_SOURCE')}" + TARGET="${@d.getVar('MOUNT_ARG_TARGET')}" + mkdir -p "$TARGET" + mountpoint -q "$TARGET" || mount --rbind "$SOURCE" "$TARGET" + mount --make-rslave "$TARGET" +EOSUDO +} + +umount_rbind() { + sudo -s <<'EOSUDO' + TARGET="${@d.getVar('UMOUNT_ARG_TARGET')}" + mountpoint -q "$TARGET" && umount -R "$TARGET" +EOSUDO +} + +mount_pbind() { + sudo -s <<'EOSUDO' + SOURCE="${@d.getVar('MOUNT_ARG_SOURCE')}" + TARGET="${@d.getVar('MOUNT_ARG_TARGET')}" + mkdir -p "$TARGET" + mountpoint -q "$TARGET" || mount --bind --make-private "$SOURCE" "$TARGET" +EOSUDO +} + +umount_pbind() { + sudo -s <<'EOSUDO' + TARGET="${@d.getVar('UMOUNT_ARG_TARGET')}" + mountpoint -q "$TARGET" && umount "$TARGET" +EOSUDO +} + +mount_proc() { + sudo -s <<'EOSUDO' + TARGET="${@d.getVar('MOUNT_ARG_TARGET')}" + mkdir -p "$TARGET" + mountpoint -q "$TARGET" || mount -t proc none "$TARGET" +EOSUDO +} + +umount_proc() { + sudo -s <<'EOSUDO' + TARGET="${@d.getVar('UMOUNT_ARG_TARGET')}" + mountpoint -q "$TARGET" && umount "$TARGET" +EOSUDO +} + +python mounts_task_prefunc () { + from collections import namedtuple + Mount = namedtuple('Mount', 'cmd source target count') + task = d.getVar('PN') + ':' + d.getVar('BB_CURRENTTASK') + lock = bb.utils.lockfile(d.getVar("MOUNTS_LOCK")) + mounts = get_requested_mounts(d) + mtab = read_mtab(d) + for cmd, source, target in mounts: + mt = mtab.get(target) + if mt: + count = mt.count + 1 + bb.debug(1, f"mount({task}): already mounted {shorten_path(mt.source)} at {shorten_path(mt.target)}, cnt={count}") + mtab[target] = mt._replace(count=count) + continue + bb.debug(1, f"mount({task}): mounting {shorten_path(source)} at {shorten_path(target)}, cnt=1") + d.setVar('MOUNT_ARG_SOURCE', source) + d.setVar('MOUNT_ARG_TARGET', target) + bb.build.exec_func('mount_' + cmd, d) + mtab[target] = Mount(cmd, source, target, 1) + write_mtab(d, mtab) + bb.utils.unlockfile(lock) +} + +python mounts_task_postfunc () { + task = d.getVar('PN') + ':' + d.getVar('BB_CURRENTTASK') + lock = bb.utils.lockfile(d.getVar("MOUNTS_LOCK")) + mounts = get_requested_mounts(d) + mtab = read_mtab(d) + + # release mounts + for cmd, source, target in mounts: + mt = mtab.get(target) + if mt is None: + bb.error(f"{target} not mounted. inconsistent mtab!?") + continue + count = mt.count - 1 + bb.debug(1, f"umount({task}): releasing {shorten_path(target)}, cnt={count}") + mtab[target] = mt._replace(count=count) + + # collect targets to unmount, in reverse order + umounts = [] + for cmd, source, target in reversed(mounts): + mt = mtab.get(target) + if mt and mt.count == 0: + umounts.append(target) + for target, mt in mtab.items(): + if mt.count < 0: + bb.error("count on {target} < 0. BUG!?!") + elif mt.count == 0 and not target in umounts: + umounts.append(target) + + # now do the unmounting + for target in umounts: + try: + bb.debug(1, f"umount({task}): unmounting {shorten_path(target)}") + d.setVar('UMOUNT_ARG_TARGET', target) + bb.build.exec_func('umount_' + mt.cmd, d) + del mtab[target] + except bb.process.ExecutionError as e: + if e.exitcode == 32: + # target busy + bb.debug(1, f"umount({task}): target busy, moving on...") + else: + bb.warn(f"umount({task}): failed to unmount {target}: {str(e)}") + + write_mtab(d, mtab) + bb.utils.unlockfile(lock) +} + +# call postfunc explicitly in case a failing task has [mounts] +addhandler mounts_taskfail +python mounts_taskfail() { + task = d.getVar('BB_CURRENTTASK') + if not task: + bb.fatal("mount code running without task context!?") + if task.startswith("do_"): + task = task[3:] + if d.getVarFlag("do_" + task, 'mounts') and not d.getVarFlag("do_" + task, 'mounts-noauto') == "1": + bb.build.exec_func('mounts_task_postfunc', d) +} +mounts_taskfail[eventmask] = "bb.build.TaskFailed" + +# bb.event.Build* handlers don't have a task context. +# Don't access MOUNTS_CONTEXT from here! +addhandler mounts_init +python mounts_init() { + bb.utils.remove(d.getVar('MOUNTS_DB') + "/*.mounttab") + bb.utils.remove(d.getVar('MOUNTS_DB') + "/*.mountlock") +} +mounts_init[eventmask] = "bb.event.BuildStarted" + +addhandler mounts_cleanup +python mounts_cleanup() { + # look through MOUNTS_DB for contexts + import glob + import time + base = d.getVar('MOUNTS_DB') + locks = glob.glob(base + "/*.mountlock") + tabs = glob.glob(base + "/*.mounttab") + + # there should not be any locks? + if len(locks) > 0: + bb.error(f"mounts_cleanup: someone still holding lock? ({str(locks)})") + + # cleanup any existing contexts + for mtab_file in tabs: + mtab = read_mtab(d, mtab_file) + if len(mtab) > 0: + bb.note(f"mounts_cleanup: {mtab_file.split('/')[-1]}") + + done = [] + for target, mt in mtab.items(): + if mt.count < 0: + bb.error("count on {target} < 0. BUG!?!") + continue + if mt.count > 0: + bb.error(f"cound on {target} > 0. BUG!?!") + + bb.note(f"mounts_cleanup: unmounting {target}") + for i in range(10): + try: + d.setVar('UMOUNT_ARG_TARGET', target) + bb.build.exec_func('umount_' + mt.cmd, d) + done.append(target) + break + except bb.process.ExecutionError as e: + if e.exitcode == 32: + # target busy + time.sleep(1) + continue + else: + bb.error(f"umount({task}): {str(e)}") + done.append(target) + break + bb.warn(f"mounts_cleanup: failed to umount {target}") + done.append(target) + + for target in done: + del mtab[target] + write_mtab(d, mtab, mtab_file) +} + +mounts_cleanup[eventmask] = "bb.event.BuildCompleted" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 7f5901d..4726eaf 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -113,7 +113,7 @@ PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}" BBINCLUDELOGS ??= "yes" # Add event handlers for bitbake -INHERIT += "isar-events" +INHERIT += "mounts isar-events" include conf/local.conf include conf/multiconfig/${BB_CURRENT_MC}.conf