From patchwork Fri Jan 13 14:31:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2489 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 13 Jan 2023 15:31:47 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-vs1-f56.google.com (mail-vs1-f56.google.com [209.85.217.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 30DEVjkj029487 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 13 Jan 2023 15:31:46 +0100 Received: by mail-vs1-f56.google.com with SMTP id a186-20020a6766c3000000b003cf5c5b9d9dsf4444889vsc.15; Fri, 13 Jan 2023 06:31:46 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673620300; cv=pass; d=google.com; s=arc-20160816; b=wAMDC0w/sBXTKnc9DtEoVh0cvpSfJkxXi6H92KHA3oWEmZSSZLxE+bzmf69lHzMze0 AXcxT2la1/H1FoMqXr8bmaO0ExLMJF84Mh32nqIzGu6jRlWcsXb5g2cFS4lh8Tsl9BeH JCfewZkd31QPQauZmkJHzNHHI4WbSCQLf+3YYufYEn5hyVTR0qpDkIdDqxG2yPxy3Ado ALs7YBrlkhOWjvhLJGqQ9W82rPh3/cpgnDGpJXiH4xGXpXsN8BjpZUjZTDMkWYraG4mK /9gfBCymu/icamamUaOY5kUHcXjkXTuoGZnxw/LUfB3s8L7vZ79xWpcIWE/tglg+GfX3 NEbA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=uCVKzT6NUnVnXfU7c8RlSXH3/uk+yFBdkWoxwuS11SY=; b=OzMk/iVONgrD8UEKWuDEzmMBsWrMsL91DF3jTDRpBKWUW8rsUnrR1viPrhjalkdekr E7y56Vx2J5n4UzQJU8ocmUHEvh7y78PQyJSR8/bGUNwGLXghDASM6+pH/dOFZd5CFt/7 80BkdE8jCGo24tUOkSOq+cz9QqHiN2whYAZfG7Wl7q7UPvt4rba3Fon1jjXvSoQjQnWl Pml/f+/E4skjlTombIf5AjVcmAy9YZUe8k7lHAjsLxiqlfSBjngO2usZ5zzPyOd0wukn fRzoGan5UM1VkgNrHfFFGSXei6T20O3ycry6YiEb9GInFzVUmGORAljMxMHzy/1fqMRa 6g5g== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=uCVKzT6NUnVnXfU7c8RlSXH3/uk+yFBdkWoxwuS11SY=; b=slxSM1Fzc/mTEPaN4cokKsHb+8usGVgwQdLmiXpSLSvl/U8PP2pRTnPpyXOQZQl1UY JFZWTyQwTGGI5XuSAX03Xr4l3xwGUgt/eMJ5rg0qrxDugD+dh/KDxN4bQYdnQsyY+1Kx 5I5MnlSmaA9E3ZNq8t7gZagu2GPXY+YyKQMONsIi2XGCaJE31sXg7Hq4yzKAPfbeiRqd qpiOXbYGFuI4BZtgnUWdu8qj8bBBjk3fIs+kaFGXaWDzOZ/TyF29zYyagcwHVhJQWDPw 00voxjCDgG+EiNCI6Xe12L0iIQheb+AxWQ9zG4aVyQYI5LzmTlL5rfhTfcSpbPYgbabF 1XbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=uCVKzT6NUnVnXfU7c8RlSXH3/uk+yFBdkWoxwuS11SY=; b=oyMrGI8yanl+pnbkB3Ki2wcCC3xtYgxuAZ0PDbMfTgV8alJQ37TAAwj5qdYwTXCFk3 rpmx0xys+pqEvObIeiiDdp0Sffb4pP7B839MUydxHjUEv0/uQDM1J6gkqwjR+/45LP1E qhgxCYOwlgWQBoLXCYJT6LszHv4qiE9metYhEYkpSmJHnf7Y1INH7cfwlpbZhSNlLszl WSXCZBY16TlpGr20WCWA1lnva8ETYRwViZhB1kDVU8xQsUiQeaYBPJbjZIdAcWgH6d7v b/E5LhnoN/g/QETpSkupLZ3tcD1dD6fRx8h8H5AYoKRQfQZpqRWqMfRdRw3Bmz2/Fk4+ /7aA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AFqh2krOthvldsTYYPxkXXx/3opTxHDejzWgQL8AhBHaj8GW7kUci9xi IRe/ZfRvW7nsyCZyvUwKewA= X-Google-Smtp-Source: AMrXdXsD9lhcUAV+J1y//5hlSXyKLIS8DNmSk2mtu8JYaQ+euuii/cbgodlX+73F5nBscKXu7203CQ== X-Received: by 2002:a05:6122:179d:b0:3d9:15da:d9f6 with SMTP id o29-20020a056122179d00b003d915dad9f6mr4604693vkf.21.1673620300065; Fri, 13 Jan 2023 06:31:40 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1f:2f91:0:b0:3b4:4e41:3c89 with SMTP id v139-20020a1f2f91000000b003b44e413c89ls927352vkv.6.-pod-prod-gmail; Fri, 13 Jan 2023 06:31:39 -0800 (PST) X-Received: by 2002:a1f:19c5:0:b0:3da:f477:3df with SMTP id 188-20020a1f19c5000000b003daf47703dfmr6772267vkz.9.1673620296089; Fri, 13 Jan 2023 06:31:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673620295; cv=none; d=google.com; s=arc-20160816; b=rDiPiTKKnMLPonS2nCArp4tgPoeM4uYOP/OWbz4vN0PTeZx8xL3o/mw6qKvkGAen8J dbmocCtw88Y9d2L0ukio+TyYoUTJIFP/ikfyQBlKZHQtUYpv2ma2si+T29uKTkOXqF3h dq33y64xiMxyA4lZ8bUZtXXIabRvXno8WAQHt7NuZI72H6wpogFGRgUGYjDf1YQJs/hM VPQGg1xX9p7cDRseKuvcpc3nsufiIsaX46kz2u+aO8CPD22IhZv6fllQzdLD/Sol5vlm WSj7/jLBGhUIXqbI/qczSnoMqwMvJGjakjPEUvTUrClZgnzeKeSgfTQkrGB3Gd51CBzu zZdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=8PepkxLAwouv2LCXuZNhQUfZKciiR74IP07JeNs6hZg=; b=uqyNfsLOkR2QIw6Fe8/LcyVQvLEk7/vTvzhe2a8mVnQMUl+0XCxyxkViYgvuWTcVtt ZiivHwwY7va0hJwO8JuR5bpPoCwgnBTdwo0wfuVP4f838mprUULRxtp3VhJNcOabeCR6 xDiKuOtFKGlVCgbz/vpPFEwSyHcD8ldd4r+9tW8ds7+V4x9+FciWGIGOO8gFoA1RipNH jTyieYOzZXJjHneVrMY+QJjWnkkYLQyT2dL4eP7pYFrbYPqxdWJPuL9zGDCVLEz9S3WJ QwPCB9zJFXU+yIUULu3pzFjCy3ZSa+bgQ4hFB288moVH/nBgomyGLi8PUMiOQOGMXAXT PD7w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from relay02.th.seeweb.it (relay02.th.seeweb.it. [2001:4b7a:2000:18::163]) by gmr-mx.google.com with ESMTPS id n14-20020a1fa40e000000b003d53e3ed270si1402705vke.0.2023.01.13.06.31.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:31:35 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) client-ip=2001:4b7a:2000:18::163; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:cd05:a8a7:67a9:5f16]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id 698C51F66B; Fri, 13 Jan 2023 15:31:33 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v8] image tools ext.: start_imager_session not break the rebuild, v8 Date: Fri, 13 Jan 2023 15:31:27 +0100 Message-Id: <20230113143127.4121031-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::163 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" image tools extension, start_imager_session, bugfix and improvment The original Ilbers ISAR code was buggy but the 1st fix do not improve the performances nore solved completely the problem which has been defintely addressed in part.2 - then this p.3 aims to improve the performance brutally deleting the closing schroot session without relies on the slower and sometimes failing schroot -e python script. p.1: bugfix, initial p.2: bugfix completed, code rationalisation p.3: performance improvement bypassing schroot -e and Co. v.4: the squash of the three parts v.5: deep reworking v.6: bugfixing v.7: removed the code left behind but not used v.8: bb.error(2, "...") bugfixed Signed-off-by: Roberto A. Foglietta meta/classes/isar-events.bbclass Signed-off-by: Roberto A. Foglietta --- meta/classes/image-tools-extension.bbclass | 39 ++++++++-------------- meta/classes/isar-events.bbclass | 32 ++++++++++++------ meta/classes/sbuild.bbclass | 3 +- 3 files changed, 37 insertions(+), 37 deletions(-) diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 48a93c8..9c46b65 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -60,18 +60,20 @@ python do_start_imager_session() { sbuild_chroot = d.getVar("SBUILD_CHROOT", True) session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) try: - if subprocess.run("schroot -l --all-sessions | grep %s" % session_id, shell=True).returncode: - subprocess.run("schroot -b -c %s -n %s" % (sbuild_chroot, session_id), shell=True, check=True) - bb.debug(2, "Open schroot session %s" % session_id) - else: - subprocess.run("schroot --recover-session -c %s" % session_id, shell=True, check=True) - bb.debug(2, "Reuse schroot session %s" % session_id) - d.setVar("SCHROOT_OPEN_SESSION_ID", session_id) + bb.debug(2, "Opening schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -b -c %s -n %s -- printenv -0 SCHROOT_ALIAS_NAME" + % (sbuild_chroot, session_id), shell=True, check=True) except subprocess.CalledProcessError as err: - subprocess.run("schroot -e -c %s" % session_id, shell=True) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) - bb.fatal("Could not create schroot session: %s" % err.output.decode('utf-8') if err.output else "") + try: + bb.debug(2, "Reusing schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" + % session_id, shell=True, check=True) + except subprocess.CalledProcessError as err: + bb.debug(2, "Closing schroot session %s (%s)" % (sbuild_chroot, session_id)) +# bb.build.exec_func("stop_imager_session", d) + bb.build.exec_func("stop_schroot_session", d) + if 'id' in locals(): + d.setVar("SBUILD_CHROOT", id) } addtask start_imager_session before do_stop_imager_session after do_rootfs_finalize @@ -79,20 +81,7 @@ do_stop_imager_session[depends] = "${SCHROOT_DEP}" do_stop_imager_session[nostamp] = "1" do_stop_imager_session[network] = "${TASK_USE_SUDO}" python do_stop_imager_session() { - import subprocess - session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) - try: - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) - subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) - except subprocess.CalledProcessError as err: - bb.error("Could not close schroot session %s: %s" % (session_id, err.output.decode('utf-8')) if err.output else "") - finally: - if 'id' in locals(): - d.setVar("SBUILD_CHROOT", id) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) + bb.build.exec_func("stop_schroot_session", d) } addtask stop_imager_session before do_deploy after do_image diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass index 5343d4b..ce5a2de 100644 --- a/meta/classes/isar-events.bbclass +++ b/meta/classes/isar-events.bbclass @@ -32,7 +32,6 @@ python task_started() { task_started[eventmask] = "bb.build.TaskStarted" addhandler task_failed - python task_failed() { # Avoid false positives if a second target depends on this task and retries # the execution after the first failure. @@ -42,6 +41,25 @@ task_failed[eventmask] = "bb.build.TaskFailed" addhandler build_completed +python stop_schroot_session() { + session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) + import subprocess + try: + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, + shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') + bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) + subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) + d.setVar("SBUILD_CHROOT", id) + bb.build.exec_func("remove_mounts", d) + bb.build.exec_func("schroot_delete_configs", d) + except subprocess.CalledProcessError as err: + bb.error("Could not close schroot session %s: %s" + % (session_id, err.output.decode('utf-8').str if err.output else "")) + udir = d.getVar("SCHROOT_OVERLAY_DIR", True) + subprocess.run("sudo rm -rf --one-file-system %s/%s /var/lib/schroot/session/%s" + % (udir, session_id, session_id), shell=True) +} + python build_completed() { import subprocess @@ -56,16 +74,8 @@ python build_completed() { shell=True, stdout=subprocess.PIPE).stdout.decode('utf-8') for line in sessions.splitlines(): session_id = line.split(':', 1)[1] - bb.debug(1, 'Closing imager session %s' % session_id) - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - if id: - subprocess.run('schroot --recover-session -c %s' % session_id, shell=True, check=True) - subprocess.run('schroot -e -c %s' % session_id, shell=True, check=True) - if 'id' in locals(): - d.setVar('SBUILD_CHROOT', id) - bb.build.exec_func('remove_mounts', d) - bb.build.exec_func('schroot_delete_configs', d) + bb.debug(2, 'Closing imager session %s' % session_id) + bb.build.exec_func("stop_schroot_session", d) with open('/proc/mounts') as f: for line in f.readlines(): diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass index 06c01d6..25c3e08 100644 --- a/meta/classes/sbuild.bbclass +++ b/meta/classes/sbuild.bbclass @@ -26,6 +26,7 @@ SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getp SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}" SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}" +SCHROOT_OVERLAY_DIR ?= "${TMPDIR}/schroot-overlay" SBUILD_CONFIG="${WORKDIR}/sbuild.conf" @@ -54,7 +55,7 @@ root-groups=root,sbuild source-root-users=${SCHROOT_USER} source-root-groups=root,sbuild union-type=overlay -union-overlay-directory=${TMPDIR}/schroot-overlay +union-overlay-directory=${SCHROOT_OVERLAY_DIR} preserve-environment=true EOF