From patchwork Tue Jan 10 04:24:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2437 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 10 Jan 2023 05:24:50 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-qt1-f183.google.com (mail-qt1-f183.google.com [209.85.160.183]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 30A4OndD008021 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 10 Jan 2023 05:24:50 +0100 Received: by mail-qt1-f183.google.com with SMTP id ay40-20020a05622a22a800b003a978b3019csf4922081qtb.16; Mon, 09 Jan 2023 20:24:50 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673324684; cv=pass; d=google.com; s=arc-20160816; b=CffnLl75uu5IOJcWSXdZGtvzyuKEhLHHh6b6F5gr6tNabECWLqvt8U5DPIvNThiscK 3ziSH5QRjYSJyss9pNZJ3FRQUCqKYvDu7DrnHIm7yyNP5Sv5lxBj8ZjxrOGy9EIU4OIn dpDO7RK5d+kv/VVuNCA+bXCa5dTIYL6RMtoA7DYD6t1uUv3x8OkjtlRb2RJxmLsC7IDP j+eyCQApK3P88hqQ3gn1Q1o5soBygh9lbCqCZKaWAsnQr7o27dqODi0zibbbmvOAIb5f X2GyNnJycStUoJuUlhjyHrQmpT+wEDIkXgiqhCkwnNSkl05tu+GCvNALCg3VeP20rtfS Ei4w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=FVKWBSwKYFJAiPNjwdbTqOSB6T/BT/dlcia2ojoj8hY=; b=KAnGbHypSornqgyYUsJ1rZW63ozUlHtDjlKgCS61j0g4OykvojnVsHbLZzFydsPvpB 1QyVtbnhtAzSfYCOrPH5SgD7avYN3rPSZWB3yNp8PBnqyZU++EM2Kjope54HWciJ6b2C EB7bcMks8rEjnUwdplqiqbfMUXnCwroT437i1x/bdPxbe6RxkeAwH0I2FmZY3bTgxRAG 5Esp852Tih8Af8HPSP5s2MLb6uICBtEhE60MEJiHPlCfUSc+eNIxQMTmRVpvNnjcm8Y0 VXs9LxG9bt/XKUD3uOgPJ8hRJsuWIpK7ef1/btiIsBDF/7Aj/JToFhnn3AhOvdAJALzq 7gaQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=FVKWBSwKYFJAiPNjwdbTqOSB6T/BT/dlcia2ojoj8hY=; b=Wme84Z1XA87qY0+gvWe6Q181cFXzU3NH72fSrPbbGoYVZVIgeYkQ3EMWP8GKhHwRw6 G6EedcNMIYWXbImqj5q5CjwDEc97+uZolMJWZtaoJs1DoYjna4yld/uCHsUC6eseU3qg EP3KoBGMLLe07KyYu2YgK8h+ND/wTzW2YdwsDN1IoTs+bckHx5GQGnuMx+1m+E1rKo01 nAAeDpDXR+w+GjpNdFAhWF2KTfsaa1F/NzvjBN11CvYYk7Kzc4JRI2VSls1diVqzag87 mtxGq1beL9ypXN94a+NK+aI/MqkVltHqxGqhGaMLrABTdjlsXNGqL2BXOx9smTlOLU4C uYdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=FVKWBSwKYFJAiPNjwdbTqOSB6T/BT/dlcia2ojoj8hY=; b=fJk7kW5yQlSP+rhhlBpOlR0uHOdbq51NFJ7YHmPvLURGe9Ac3HUtXcPhZgeG3bfU/H IsYsKhXXB6sqmwbIsJyLeX61mcoN7PjALZZpE2mnSO6y+IA/Nj9Xqnlwa9Gol0QmqXUu BHoTpBr3Z9cQYQjk/Z9wD6Nt1FZo93MKLvB8wBxE51VHiIDhnv2aSmelb0I5RhPf2a/M N7AC/m11nTLGbgSw8vp3PlGzloojPcSNdpdn3/wu61li4ORlCq+sRrcER5/6pC3q18Hy 08YHYPO50hPKyZq5b0UZIxbGmrYrsmPvr/98UQoG0jwDwUkYuQtiOl23uqvk1CF+ggM/ Rz1A== Sender: isar-users@googlegroups.com X-Gm-Message-State: AFqh2koyJ/ta2Rwufq7tOAe7u6JDoXCkWFqdPOLhEnFHsnGnlEXPa8UM sGbRO6tE1y+k/Uq9Wtz3K4g= X-Google-Smtp-Source: AMrXdXuoKs2YtDz5yo9MJlny+I0r6qv11cRgddEkIdThZG3l8wubIjeuMl+FE0qbngWKcjern0XADw== X-Received: by 2002:a37:ae05:0:b0:702:4126:2a24 with SMTP id x5-20020a37ae05000000b0070241262a24mr2682983qke.715.1673324683889; Mon, 09 Jan 2023 20:24:43 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac8:4891:0:b0:3a6:9e8c:1506 with SMTP id i17-20020ac84891000000b003a69e8c1506ls682516qtq.4.-pod-prod-gmail; Mon, 09 Jan 2023 20:24:43 -0800 (PST) X-Received: by 2002:a05:622a:410d:b0:3ae:2272:e43c with SMTP id cc13-20020a05622a410d00b003ae2272e43cmr5437909qtb.29.1673324683284; Mon, 09 Jan 2023 20:24:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673324683; cv=none; d=google.com; s=arc-20160816; b=QzoJAAXXqBjJ+YWGVujXRR+Yop9f7RF86NFmSgkMOUo7qMXYNkNsesGkjZ7qWTzIHP Wqk03L8JfdynWMaZ3Jpgr9APle0xJHl4IZZlMzLql4uhodvN2AzwfCbUByxrLI9nJeTL tor4YSZUBJiiEbFSZc5gklgTq6Z66tNUKOFLiNwemVDVmDPzpDzbWkD9w5G6Sb7M2YBs FAmAoPZHHAGievS2U9kibbx2r5F7evI5EY6tV3aTIUbKE5LGr4RP5hPYr/ZFNKB2nTlA MuoanhLh/tlp4vWC7OMcjF1NRyDrQdw+pmeeUhISK5QYk3wlt3eedhCTwxsmUJtV8Qo2 skig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=wZ8ktzUjrIuL/iGInllODbIpfI6MkLc6aeFg90FuTT0=; b=FgOG9vbHvT7S/x3Zw2jQVklzfJ1XKC+nKTHKGO/hqTK7x/MaqQEJWA7EcRVmsn1SDz vPrm25GiNIXtwuTsvPUQ8f/xlIt/wMZ08X+syFbb8x3JPvgSdkEiiDDhQvQc45GSpezo TdsNtsRxAdwnfkIq+hX0tva6DpmOySsh4ojsPw0m0wHHQSWOmbRI7hrJJrSP0OCwoF4t 1Ne+IgpujACwiwKp0NpL2SWoetYBVulwiBDGZBLIC/Mdw5ZNblft8p9N87/c2f+wCuBR K0Jtkgy9HXAa/mA7ltkgXNBOzYEpP7vUdV/39FEXbHl8FjbtTb4pk9PMOMyi8gB8/USf 7AfA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from relay04.th.seeweb.it (relay04.th.seeweb.it. [2001:4b7a:2000:18::165]) by gmr-mx.google.com with ESMTPS id h4-20020ac87764000000b0039a9b745f0asi530242qtu.5.2023.01.09.20.24.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 20:24:42 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) client-ip=2001:4b7a:2000:18::165; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:2373:f2dd:a867:c892]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id D75A71F8BE; Tue, 10 Jan 2023 05:24:40 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v7] image tools ext.: start_imager_session not break the rebuild, v7 Date: Tue, 10 Jan 2023 05:24:38 +0100 Message-Id: <20230110042438.1226639-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" image tools extension, start_imager_session, bugfix and improvment The original Ilbers ISAR code was buggy but the 1st fix do not improve the performances nore solved completely the problem which has been defintely addressed in part.2 - then this p.3 aims to improve the performance brutally deleting the closing schroot session without relies on the slower and sometimes failing schroot -e python script. p.1: bugfix, initial p.2: bugfix completed, code rationalisation p.3: performance improvement bypassing schroot -e and Co. v.4: the squash of the three parts v.5: deep reworking v.6: bugfixing v.7: removed the code left behind but not used Signed-off-by: Roberto A. Foglietta --- meta/classes/image-tools-extension.bbclass | 39 ++++++++-------------- meta/classes/isar-events.bbclass | 31 +++++++++++------ meta/classes/sbuild.bbclass | 3 +- 3 files changed, 37 insertions(+), 36 deletions(-) diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 48a93c8..9c46b65 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -60,18 +60,20 @@ python do_start_imager_session() { sbuild_chroot = d.getVar("SBUILD_CHROOT", True) session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) try: - if subprocess.run("schroot -l --all-sessions | grep %s" % session_id, shell=True).returncode: - subprocess.run("schroot -b -c %s -n %s" % (sbuild_chroot, session_id), shell=True, check=True) - bb.debug(2, "Open schroot session %s" % session_id) - else: - subprocess.run("schroot --recover-session -c %s" % session_id, shell=True, check=True) - bb.debug(2, "Reuse schroot session %s" % session_id) - d.setVar("SCHROOT_OPEN_SESSION_ID", session_id) + bb.debug(2, "Opening schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -b -c %s -n %s -- printenv -0 SCHROOT_ALIAS_NAME" + % (sbuild_chroot, session_id), shell=True, check=True) except subprocess.CalledProcessError as err: - subprocess.run("schroot -e -c %s" % session_id, shell=True) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) - bb.fatal("Could not create schroot session: %s" % err.output.decode('utf-8') if err.output else "") + try: + bb.debug(2, "Reusing schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" + % session_id, shell=True, check=True) + except subprocess.CalledProcessError as err: + bb.debug(2, "Closing schroot session %s (%s)" % (sbuild_chroot, session_id)) +# bb.build.exec_func("stop_imager_session", d) + bb.build.exec_func("stop_schroot_session", d) + if 'id' in locals(): + d.setVar("SBUILD_CHROOT", id) } addtask start_imager_session before do_stop_imager_session after do_rootfs_finalize @@ -79,20 +81,7 @@ do_stop_imager_session[depends] = "${SCHROOT_DEP}" do_stop_imager_session[nostamp] = "1" do_stop_imager_session[network] = "${TASK_USE_SUDO}" python do_stop_imager_session() { - import subprocess - session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) - try: - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) - subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) - except subprocess.CalledProcessError as err: - bb.error("Could not close schroot session %s: %s" % (session_id, err.output.decode('utf-8')) if err.output else "") - finally: - if 'id' in locals(): - d.setVar("SBUILD_CHROOT", id) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) + bb.build.exec_func("stop_schroot_session", d) } addtask stop_imager_session before do_deploy after do_image diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass index 5343d4b..7f68ffa 100644 --- a/meta/classes/isar-events.bbclass +++ b/meta/classes/isar-events.bbclass @@ -42,6 +42,25 @@ task_failed[eventmask] = "bb.build.TaskFailed" addhandler build_completed +python stop_schroot_session() { + session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) + import subprocess + try: + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, + shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') + bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) + subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) + d.setVar("SBUILD_CHROOT", id) + bb.build.exec_func("remove_mounts", d) + bb.build.exec_func("schroot_delete_configs", d) + except subprocess.CalledProcessError as err: + bb.error(2, "Could not close schroot session %s: %s" + % (session_id, err.output.decode('utf-8')) if err.output else "") + udir = d.getVar("SCHROOT_OVERLAY_DIR", True) + subprocess.run("sudo rm -rf --one-file-system %s/%s /var/lib/schroot/session/%s" + % (udir, session_id, session_id), shell=True) +} + python build_completed() { import subprocess @@ -56,16 +75,8 @@ python build_completed() { shell=True, stdout=subprocess.PIPE).stdout.decode('utf-8') for line in sessions.splitlines(): session_id = line.split(':', 1)[1] - bb.debug(1, 'Closing imager session %s' % session_id) - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - if id: - subprocess.run('schroot --recover-session -c %s' % session_id, shell=True, check=True) - subprocess.run('schroot -e -c %s' % session_id, shell=True, check=True) - if 'id' in locals(): - d.setVar('SBUILD_CHROOT', id) - bb.build.exec_func('remove_mounts', d) - bb.build.exec_func('schroot_delete_configs', d) + bb.debug(2, 'Closing imager session %s' % session_id) + bb.build.exec_func("stop_schroot_session", d) with open('/proc/mounts') as f: for line in f.readlines(): diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass index 06c01d6..25c3e08 100644 --- a/meta/classes/sbuild.bbclass +++ b/meta/classes/sbuild.bbclass @@ -26,6 +26,7 @@ SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getp SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}" SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}" +SCHROOT_OVERLAY_DIR ?= "${TMPDIR}/schroot-overlay" SBUILD_CONFIG="${WORKDIR}/sbuild.conf" @@ -54,7 +55,7 @@ root-groups=root,sbuild source-root-users=${SCHROOT_USER} source-root-groups=root,sbuild union-type=overlay -union-overlay-directory=${TMPDIR}/schroot-overlay +union-overlay-directory=${SCHROOT_OVERLAY_DIR} preserve-environment=true EOF