From patchwork Fri Jan 13 18:50:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2491 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 13 Jan 2023 19:50:59 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-yw1-f184.google.com (mail-yw1-f184.google.com [209.85.128.184]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 30DIowBU030521 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 13 Jan 2023 19:50:58 +0100 Received: by mail-yw1-f184.google.com with SMTP id 00721157ae682-4c8e781bc0asf188357877b3.22; Fri, 13 Jan 2023 10:50:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673635852; cv=pass; d=google.com; s=arc-20160816; b=VhAssUSpsjCFARgH4miIjtvWTM0DfLzZesvqvt0ZAJGfCgR2aALMGTNmKAsKVT8pfV q8DeJv60MYfh4Pw4A6nozbg3eVzXCu26KP9IG3/CbL2BB1U6aLJVHJf2Hyl5EOxlpGsU vENzZ96vzXdIYM0lve1wngggbSYywvIxgFD05qXsqxhTGeimikrRwRQkDkHnXgtsTc1h yqqLJIJ1wm1mP+2xtKpg307qZZwXl5MUMCgzimJjr7mbuARXsf2deSwknrQdlusXsslZ jCmaO/ak+ov7gBuxClift5mA+TqBnrJkECohP+x54Sj4pG4sWtFshrUUb6/s90xojCPZ 3ehA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=I7cZcXMhzk7vM06i0IHlwOebTJXe5pV5CieeKq7EKFQ=; b=GyRr1+sBF6M4S5U6pgnPFOuHi2rSMjBuwmGaj5IZPzqgW7ubTdiPZ9O6vddY96kIyd Uo4DywrETRx7m4BVQJUbLJKmxjA60GB7p8dX++LqGhirD8lu65Z+NYvulq6+tp1/ksGK pR6Vlif2gI7WDyn5oD0iC0ZCqK2ClyQF6AugR2KtblFsvlCH2+9/pLcxX3R9zlA4ozLT muRqqVHf20dXe03NcB4ntgLtI/GkMkvEMgTSGoiTecEpmzm2y3upbfEDXxT3AOc6WA54 i6YYTB0tckDNSzj4H65JITC9m+zW7YrFc94wM92CFoA6ezBOriGsr/Dp5HdJfE63bESP UT0Q== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.170 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=I7cZcXMhzk7vM06i0IHlwOebTJXe5pV5CieeKq7EKFQ=; b=EgmjlGs5PFSQ6OGD/DO+kRe4O3eHToXH3hSM9RvwObPVJEiBboisuh/0dFQUoUMG+K TscpPhKdX4MqgeYnbloPMs4jY0y0wNWwxBpwUi3gIl/hQ43IxGoVbH3sk+hfhyxan1j5 pHAz86wuOIpaRsoDiIY5GyCZTroGflAyk3V8x57pz03tt5Rh6GigpKO77z9oLKJe4dtK YnVDULDXS5Va3G+Nxa/XnsCsCkg1qwiJsxynr7w4JhxDX7qt4X8cysU63T8PvHORfOiV Ih8afc9G0zLxa9aAd8OMACxc29Glu+w4EeW3+QBRBPnPostVOjQhV/tRQVG+zD2hJhsu 66dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=I7cZcXMhzk7vM06i0IHlwOebTJXe5pV5CieeKq7EKFQ=; b=dm7abZJQYnJxLaB6sAy6ThLHEhq6lbwFR/enepszfs4ttIYBAb6sc2lhnXBXCcfAKt UgvP5vHApmBN9XqZUgN34eIRjUfMUAoP9fKaMqH3Kg+IDJXln/SH8tE3y6Jtw4UxK4RA f1YQEN13gLPJ9B4tkFEh9+4PphPE0PULbdkgq8msG8CD0mi1O1G0XtHJSMHVXeoy+wv4 UoYhGTSYGwg8QA/cpJQFn5aYBwBU1Y3j8HMBpNKGuUl7pSlBSTQagPW6UTKcdl+AhqE4 BnExMSKFRlwwMFvxRudy7lCdiunSJkwj6kd5yQr7QOy6PDmK/751ktYuQecSnMO1h7a7 08gg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AFqh2kqNxin8quX7yHRTSmtIckKDi5orxg2+eM/N2XgQ92lr7JF/aO8g TYbkNE4xmi7IZZohw3dZjLg= X-Google-Smtp-Source: AMrXdXvsuFmgMRuN/O0yrdSXv+/niMFvL7MJQUAesG3YSoxRmqo1CqeQKr9nR8a5tC7kFPosfaTPNA== X-Received: by 2002:a25:fc6:0:b0:6d5:a304:b9cf with SMTP id 189-20020a250fc6000000b006d5a304b9cfmr10347238ybp.293.1673635852525; Fri, 13 Jan 2023 10:50:52 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a25:a466:0:b0:7d1:9d8f:ed96 with SMTP id f93-20020a25a466000000b007d19d8fed96ls854827ybi.4.-pod-prod-gmail; Fri, 13 Jan 2023 10:50:51 -0800 (PST) X-Received: by 2002:a5b:a45:0:b0:7c3:b23e:f9b9 with SMTP id z5-20020a5b0a45000000b007c3b23ef9b9mr7816339ybq.53.1673635851898; Fri, 13 Jan 2023 10:50:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673635851; cv=none; d=google.com; s=arc-20160816; b=brDOkh/TxTzxnRjCv2f1mvGruL6DMMDhSm0z66/5o3rfr4vTZB/euxmyM8NkyRIBnq +UrhlBbePKO2QaHVGrvxljrUbMhCrLor8aI+1heNhYm0xVDjwwF02Vtb3EiLTDMiF4W4 LNUMqGduRsQnmrL8vr+RVsHtTt2s1pGRvrV1I/MVjddneXGoI/qmFNCKEHC8Z9pRa16s S57X3KrdfgEuUhlLopYeNis6reHGwKwToGf53q+gcHYswXSY9cbit9Bc/h6OfXtOjEA7 jAhBtF5FNxJmk3so4NCPN22L+KKjvueH2upXW1XqvvzHjNYv9nLqHkkIPqkQY4/KaUzb LZXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=fno8iMX4qpTSdCn7iDltXf3k93xGNRVjNzNnrXiQsa4=; b=mNPHyhuZKWbCedqqwI0PiCz2zbzbKxYaGNYWRnfJDCMq67+jAOOhHutZMXFODYjCdY 0nwqc1KaRznzSCkJcAVMnG0p9DgHI0mXzW5d9mX0UW7vqRzS55SLVyjTczMomwgtptBI 42Na1ByLnh9bcGM6YkODNKlJBBoimsqnNnWdjUmipPyi5mfllf8jdsQvdHt3558Dr/qq lMdEq+mqexftjMejJNAzvp0lWxIEg8ZlWF+lm7Pq7UEZnHr/mP4my+BlLfwwRUfeIeGl 5ELxJWgZMeVh/hOz+cHbR9yRbgv6c7fVSmlQCguWro1qTdI4arvqIOHYVgezl593JT0M oljw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.170 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from m-r1.th.seeweb.it (m-r1.th.seeweb.it. [5.144.164.170]) by gmr-mx.google.com with ESMTPS id f2-20020a25cf02000000b006ddea715dd2si2369983ybg.0.2023.01.13.10.50.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 10:50:51 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.170 as permitted sender) client-ip=5.144.164.170; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:cd05:a8a7:67a9:5f16]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id 994911F899; Fri, 13 Jan 2023 19:50:49 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v9] image tools ext.: start_imager_session not break the rebuild, v9 Date: Fri, 13 Jan 2023 19:50:47 +0100 Message-Id: <20230113185047.334809-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 5.144.164.170 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" image tools extension, start_imager_session, bugfix and improvment The original Ilbers ISAR code was buggy but the 1st fix do not improve the performances nore solved completely the problem which has been defintely addressed in part.2 - then this p.3 aims to improve the performance brutally deleting the closing schroot session without relies on the slower and sometimes failing schroot -e python script. p.1: bugfix, initial p.2: bugfix completed, code rationalisation p.3: performance improvement bypassing schroot -e and Co. v.4: the squash of the three parts v.5: deep reworking v.6: bugfixing v.7: removed the code left behind but not used v.8: bb.error(2, "...") bugfixed v.9: two attempts: current rebuild saved, not only the next one Signed-off-by: Roberto A. Foglietta --- meta/classes/image-tools-extension.bbclass | 54 +++++++++------------- meta/classes/isar-events.bbclass | 32 ++++++++----- meta/classes/sbuild.bbclass | 3 +- 3 files changed, 46 insertions(+), 43 deletions(-) diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 48a93c8..3cc33cf 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -55,23 +55,28 @@ do_start_imager_session[nostamp] = "1" do_start_imager_session[network] = "${TASK_USE_SUDO}" python do_start_imager_session() { import subprocess - bb.build.exec_func("schroot_create_configs", d) - bb.build.exec_func("insert_mounts", d) - sbuild_chroot = d.getVar("SBUILD_CHROOT", True) - session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) - try: - if subprocess.run("schroot -l --all-sessions | grep %s" % session_id, shell=True).returncode: - subprocess.run("schroot -b -c %s -n %s" % (sbuild_chroot, session_id), shell=True, check=True) - bb.debug(2, "Open schroot session %s" % session_id) - else: - subprocess.run("schroot --recover-session -c %s" % session_id, shell=True, check=True) - bb.debug(2, "Reuse schroot session %s" % session_id) - d.setVar("SCHROOT_OPEN_SESSION_ID", session_id) - except subprocess.CalledProcessError as err: - subprocess.run("schroot -e -c %s" % session_id, shell=True) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) - bb.fatal("Could not create schroot session: %s" % err.output.decode('utf-8') if err.output else "") + attempts=0 + while attempts < 2: + attempts+=1 + bb.build.exec_func("schroot_create_configs", d) + bb.build.exec_func("insert_mounts", d) + sbuild_chroot = d.getVar("SBUILD_CHROOT", True) + session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) + try: + bb.debug(2, "Opening schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -b -c %s -n %s -- printenv -0 SCHROOT_ALIAS_NAME" + % (sbuild_chroot, session_id), shell=True, check=True) + attempts=2 + except subprocess.CalledProcessError as err: + try: + bb.debug(2, "Reusing schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" + % session_id, shell=True, check=True) + except subprocess.CalledProcessError as err: + bb.debug(2, "Closing schroot session %s (%s)" % (sbuild_chroot, session_id)) + bb.build.exec_func("stop_schroot_session", d) + if 'id' in locals(): + d.setVar("SBUILD_CHROOT", id) } addtask start_imager_session before do_stop_imager_session after do_rootfs_finalize @@ -79,20 +84,7 @@ do_stop_imager_session[depends] = "${SCHROOT_DEP}" do_stop_imager_session[nostamp] = "1" do_stop_imager_session[network] = "${TASK_USE_SUDO}" python do_stop_imager_session() { - import subprocess - session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) - try: - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) - subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) - except subprocess.CalledProcessError as err: - bb.error("Could not close schroot session %s: %s" % (session_id, err.output.decode('utf-8')) if err.output else "") - finally: - if 'id' in locals(): - d.setVar("SBUILD_CHROOT", id) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) + bb.build.exec_func("stop_schroot_session", d) } addtask stop_imager_session before do_deploy after do_image diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass index 5343d4b..ae42b63 100644 --- a/meta/classes/isar-events.bbclass +++ b/meta/classes/isar-events.bbclass @@ -32,7 +32,6 @@ python task_started() { task_started[eventmask] = "bb.build.TaskStarted" addhandler task_failed - python task_failed() { # Avoid false positives if a second target depends on this task and retries # the execution after the first failure. @@ -42,6 +41,25 @@ task_failed[eventmask] = "bb.build.TaskFailed" addhandler build_completed +python stop_schroot_session() { + session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) + import subprocess + try: + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, + shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') + bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) + subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) + d.setVar("SBUILD_CHROOT", id) + bb.build.exec_func("remove_mounts", d) + bb.build.exec_func("schroot_delete_configs", d) + except subprocess.CalledProcessError as err: + bb.warn("Failed to close schroot session, removing %s: %s" + % (session_id, err.output.decode('utf-8').str if err.output else "")) + udir = d.getVar("SCHROOT_OVERLAY_DIR", True) + subprocess.run("sudo rm -rf --one-file-system %s/%s /var/lib/schroot/session/%s" + % (udir, session_id, session_id), shell=True) +} + python build_completed() { import subprocess @@ -56,16 +74,8 @@ python build_completed() { shell=True, stdout=subprocess.PIPE).stdout.decode('utf-8') for line in sessions.splitlines(): session_id = line.split(':', 1)[1] - bb.debug(1, 'Closing imager session %s' % session_id) - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - if id: - subprocess.run('schroot --recover-session -c %s' % session_id, shell=True, check=True) - subprocess.run('schroot -e -c %s' % session_id, shell=True, check=True) - if 'id' in locals(): - d.setVar('SBUILD_CHROOT', id) - bb.build.exec_func('remove_mounts', d) - bb.build.exec_func('schroot_delete_configs', d) + bb.debug(2, 'Closing imager session %s' % session_id) + bb.build.exec_func("stop_schroot_session", d) with open('/proc/mounts') as f: for line in f.readlines(): diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass index 06c01d6..25c3e08 100644 --- a/meta/classes/sbuild.bbclass +++ b/meta/classes/sbuild.bbclass @@ -26,6 +26,7 @@ SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getp SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}" SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}" +SCHROOT_OVERLAY_DIR ?= "${TMPDIR}/schroot-overlay" SBUILD_CONFIG="${WORKDIR}/sbuild.conf" @@ -54,7 +55,7 @@ root-groups=root,sbuild source-root-users=${SCHROOT_USER} source-root-groups=root,sbuild union-type=overlay -union-overlay-directory=${TMPDIR}/schroot-overlay +union-overlay-directory=${SCHROOT_OVERLAY_DIR} preserve-environment=true EOF