From patchwork Sun Jun 12 23:07:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 1874 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 13 Jun 2022 09:08:09 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ua1-f55.google.com (mail-ua1-f55.google.com [209.85.222.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 25D788KP027790 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 13 Jun 2022 09:08:09 +0200 Received: by mail-ua1-f55.google.com with SMTP id y15-20020ab0638f000000b00368a2d9b075sf2219713uao.13 for ; Mon, 13 Jun 2022 00:08:09 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1655104083; cv=pass; d=google.com; s=arc-20160816; b=CgFatmnyTtgm0OyvJ4CVR+E0/fm465XL/zsrfxwZ+7uk6pmDzbMz90mWtgmZmlmBeG S1ARMsTTdPBk0cncWdf4RlSoqSXL+fJ8GGkZNCslzlQQreHdQb7jLUfSNQnVSD9qKR7l MLM0c5XtHSw52YVcO3JruW7qSbcx6gnka1rtxjTJDvG1QrwHoid4h8TO87Y6UKqv1Vjm tQMLxrmTU+6gFOoxaPHFZY+jGkoF9mVOYPOqq1oSLRsh1XkzlL2clmNvMBEBocq0Sm9H h2Hyxggg70w3I5sFQ/0JusbtZI5cmH/dyAvU4ceH3nLH9WWmO4WMoTTUrx8t6x8ITFaJ 6nSA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=agN7Y5wccoHvR8uwA/0wP22aqiPPM3H25lMMaxGZ6Qg=; b=Y3kmgWJBcgRbj600vt25IF3xt7ZzbChXqtg58l3iM+UHMxrHpZg1pIG4qy0cbKrbR5 JL29O3TmZi9TZqW2msfJPZjEA1E+HewyyB1kdzo+LjsR/iJo4Obk01exC6dkEuGWCHhZ b8WkCxmM+ouVHVpBfPI/gIoQ+McNHbi4XyeqGgF1zlflnRFVnXdW5H8h4bDBSsebYHA2 J/NN8DlGeB5USxC02NYAahxGQ7ffKO8V856pXEJPEPgawN7XwbFdtoCDlgfLSaQXt2BE LGIt8fAvLa7v1AQ8CNRBoJO6G8OptHShxcGvr2Hb5ElUvO6+fdUr7SBsIo5z5qYLnRxv noAQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=agN7Y5wccoHvR8uwA/0wP22aqiPPM3H25lMMaxGZ6Qg=; b=hf6w3lXphgUE40/76CEw+4MjURz0hC9VTDVmE8t12acoOgzGTRTRHbChLOOfKW2cAK LHgY6nw+OZv8Ff61LdnV6D6t0QrVG4YWTZKGYVECigL6GtP98r5yFOsedr6fjUmHkAkd kAYaDj3kvq2RfmquBV2iSV9Scn0sYnwZInt/rKpY2WaVERHINalPA/PPKEoQ/wfwiePq QFo4zjsWNqGF0If2FZoYJpbQwtH0pvev9SI0FvzBs4pPIFGgegE4PS9RUWcqZw3og9OO KEFnRaa7tN5ipBU38WBHPnput4mQ6+LpGOA0s8zDgFIRYfAAntmdQEdGNgqpC+oSMVuJ XyMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=agN7Y5wccoHvR8uwA/0wP22aqiPPM3H25lMMaxGZ6Qg=; b=B+HT6m2VdOUBPppx7MKUVjw43KiGYMeSud1OSaHOiPJ7vKnBZiAPxmfG7C4Z1y9qF+ LevC6WkwD2IcWkmRbBEwJIOv/j2HfFNI0aqCdgAulOXguQoL5H8sZ1C+x3iE7TJnJU4q jyxFESgjaL+8ePpDHq2FuwhsYR77Ua8il0ezL7yrgR2F34p2BZTbNQKWVXK1vorQBhJf b9iX9JMxlU1T9a4shfVDGX2DcrLVSbbTgU/qXOC5hbzbBxJi7MUcnI56BYUAkucuZDFO kXTGnYSKT074vuOeKVznBWAkfFlUJ3Cm1WUMbCgYfoPIfZI+LI1dGfdcQLTd8Yfk10RU 76nA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM530BsBC8XdP+dsB5RH6VPK0b0waAYbx6/Mp96vte1MExiw1QMDly GMfvnfp+fArzlx8KF7tgnsA= X-Google-Smtp-Source: ABdhPJx3DEDFqR54ULnjUUHq2JqO3HRFHUim79d+aRreusfENWQYYVpNRu05kZdD9XOV2o5VwbGLeQ== X-Received: by 2002:a67:6f85:0:b0:34c:12b2:1325 with SMTP id k127-20020a676f85000000b0034c12b21325mr7701689vsc.78.1655104082798; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a67:60c7:0:b0:34b:bc0a:4dd3 with SMTP id u190-20020a6760c7000000b0034bbc0a4dd3ls275477vsb.0.gmail; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) X-Received: by 2002:a05:6102:34c3:b0:349:f12c:ccf4 with SMTP id a3-20020a05610234c300b00349f12cccf4mr24399848vst.64.1655104082262; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1655104082; cv=none; d=google.com; s=arc-20160816; b=Jp2qvEfR+jFWqiJScIKS16hBUq1MxTIUF8AJSQ7xtvVV5qPlHHDEhaau1IWczjyu0B gi7BnSBaH1wb34stGjF87Luz7rFupOFrEHVUItovWrIPKAH1yRJaZtIEIRYDrwCs5Lzw QtBS2VTmpWw81w1gTr+VZ4eb5Pk9bAQXfw5CgdQ+lp+Dd2aNFVvGPpSTkw7f3jJsRODu 3XbBWZbAv+MYeE0R3uUEGonehbDcMt4i0Bt+ML4ja1gKfe4fk+sHLpchqdidWTuC+YI6 YYx9rkQ9mbQWlXI4b3csj4sfbdJHWiOU9Bfn40aswuNPBehP2RFBde9JdS0L9qQZn8Wj FClA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=7R5jWwTKHsvyUYF0oojR492fH6vB3A1JHvBEarx8NKc=; b=ZQEfu39IK085+qf4v+YTGbryEFQQAm9Nv7q1Xme1Itzz6NeHS58gG1BEoGzzgLYJsR axDkQenyYiVjHepUL3tSyeYONczVkF8oz7q46PxzvLZBOJgXWBLSWMUbuL6wsn9qmrSB Oz0l2GTKIPJo16sAZkopLyLLAfX2hI0sd7QunqVGLUHMFN6Me8yhbQ0AXP/ueJQMgNUW LnDl4mEmsL160Ys/IrIh37aR4dS2Kqnno/akHDNdd2sjUL0vuH+2NxQkgfNt5Bmlvva6 pNBzf0dtQB/YWFD8/5v/Z1UEbLLdkKhXSO6d0xdAr/U7tnMYCRB9kBuk7seQ3YzpleR5 6Tkg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id h14-20020ab0334e000000b00378e80fec38si357283uap.2.2022.06.13.00.08.01 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 13 Jun 2022 00:08:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 25D77xZ8027766 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 13 Jun 2022 09:08:00 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [RFC v2 1/3] meta: Reuse existing local isar-apt repo in new builds Date: Mon, 13 Jun 2022 09:07:57 +0200 Message-Id: <20220613070759.16949-2-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220613070759.16949-1-ubely@ilbers.de> References: <20220613070759.16949-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX The idea of reusing isar-apt repo at second build is to early exit from build-related tasks if the package with the same version already exists in current isar-apt repo. Also, redefining default values of REPO_ISAR_DIR and REPO_ISAR_DB_DIR maybe useful to share isar-apt between different builds and not to keep it under ${TMPDIR}: REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt" REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db" The variable ISAR_APT_REUSE in local.conf controls the behvaiour: 0: Don't use isar-apt cache, always rebuilt packages (default) 1: Use local isar-apt cache, rebuilt packages only if needed Signed-off-by: Uladzimir Bely --- meta-isar/conf/local.conf.sample | 4 ++ meta/classes/dpkg-base.bbclass | 1 + meta/classes/isar-apt-cache.bbclass | 69 +++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+) create mode 100644 meta/classes/isar-apt-cache.bbclass diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample index 58f3e1a2..523da5c3 100644 --- a/meta-isar/conf/local.conf.sample +++ b/meta-isar/conf/local.conf.sample @@ -221,6 +221,10 @@ ISAR_CROSS_COMPILE ?= "0" # does not access the network #BB_NO_NETWORK ?= "1" +# +# Uncomment to use previously built isar_apt repo if package version hasn't changed +#ISAR_APT_REUSE ?= "1" + # Set root password to 'root' # Password was encrypted using following command: # mkpasswd -m sha512crypt -R 10000 diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 7e032ba3..a9697fbd 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -9,6 +9,7 @@ inherit debianize inherit terminal inherit repository inherit deb-dl-dir +inherit isar-apt-cache DEPENDS ?= "" diff --git a/meta/classes/isar-apt-cache.bbclass b/meta/classes/isar-apt-cache.bbclass new file mode 100644 index 00000000..4550886d --- /dev/null +++ b/meta/classes/isar-apt-cache.bbclass @@ -0,0 +1,69 @@ +# This software is a part of ISAR. +# Copyright (C) 2017-2019 Siemens AG +# Copyright (C) 2019 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +iac_check_package_in_repo() { + DISTS_ISAR="${REPO_ISAR_DIR}/${DISTRO}/dists/${DEBDISTRONAME}" + PACKAGES="${DISTS_ISAR}/main/binary-${PACKAGE_ARCH}/Packages" + + if [ ! -f "${PACKAGES}" ]; then + return 1 + fi + + if ! grep -q "^\(Package\|Source\): ${PN}$" "${PACKAGES}"; then + return 2 + fi + + local repo_version=$(sed -n -e '/^\(Package\|Source\): ${PN}$/,/Version/p' ${PACKAGES} | grep '^Version' | sed -e 's/.* //' | head -n1) + local new_version=$(head -n1 ${S}/debian/changelog | sed -e 's/.* (\(.*\)) .*/\1/' | head -n1) + + if [ "${repo_version}" != "${new_version}" ]; then + return 3 + fi + + bbnote "Package ${PN}-${PV} is already in isar-apt repo, reuse it" + return 0 +} + +iac_check_need_rebuild() { + if [ "${ISAR_APT_REUSE}" != "1" ]; then + return 1 + fi + + iac_check_package_in_repo +} + +def iac_check_task_early_exit(d): + try: + bb.build.exec_func("iac_check_need_rebuild", d) + return True + except: + return False + + +do_apt_fetch_prepend() { + iac_check_need_rebuild && return +} + +do_apt_unpack_prepend() { + iac_check_need_rebuild && return +} + +do_prepare_build_prepend() { + iac_check_need_rebuild && return +} + +do_deploy_deb_prepend() { + iac_check_need_rebuild && return +} + +do_install_builddeps_prepend() { + iac_check_need_rebuild && return +} + +python do_dpkg_build_prepend() { + if iac_check_task_early_exit(d): + return 0 +} From patchwork Sun Jun 12 23:07:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 1873 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 13 Jun 2022 09:08:09 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ua1-f60.google.com (mail-ua1-f60.google.com [209.85.222.60]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 25D7889U027789 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 13 Jun 2022 09:08:09 +0200 Received: by mail-ua1-f60.google.com with SMTP id k40-20020ab059eb000000b00364ef6b085bsf2259840uad.0 for ; Mon, 13 Jun 2022 00:08:09 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1655104083; cv=pass; d=google.com; s=arc-20160816; b=R6e/SLzyGX4UybNDWRqV6zPEvwvU4gWEwTTvexAxqFuGCcpDvE9j/8JWDV5IDeCmEu ketbH66+40az2qk9XXLHiNt16AryBu+Hx9qOcKeJTN8yDhyj9iT48cFssbHUzuBAHi19 NaFi43MBHLIC4JCbybBKBgviztUKUCbQlXnCAcNRdg+nwYdBDwYNKP7/r7okgHfNpzMv etdZxm29nESYWpvMI3VnOrtLPzPa6jGO3rHRgbehkLFfwsXnszpG+ew7icHrLrb/UMiu 4pCt/M3rRfxdnw5FuXvjuCb2jqnzY6pvYrAjdTDy3Wp+a0fbyFWfguKOVD6r9KKuDsWB 0c7Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=qot3vNsWBMoe+JdIBIo11wkgMGQiWyELgu1PU4VpSXo=; b=Rt4OTd9gSz++DajpaDCon7crdmI1QubqOKC8hn7Brr/AstqdaVph0UD+w/IoWkDxvQ 06cDaJ1vn+xuTuOnL3SBuDqVSxIHA/CceMksV4YAsFIfoKIF8ZBQ3rPGI2qpYmk1kgLZ qe+Q/i/tTgpZ8tzbtJ/DWQWeZh2JcgJGMbP08gQQeTf52v5C77dY3tKqsrAJHjLl38o0 h0AW6XnuOCsnnbor07Q7cffpVzSkA0Z/Np1UL+8zHeJ4TL3qxuSrn+xYYDFfZh92BZBi wDB3K3jSIicEb4PwJ/MvpQCYrJNHpcQn/qwuVoZ7hHnuAj4o4NZESNfHVV8eQjfacdYM qx7g== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=qot3vNsWBMoe+JdIBIo11wkgMGQiWyELgu1PU4VpSXo=; b=lqG1BUSFzlzyi6JSdcsZXfvwZlTkzf1RJISRD/0O0LpaTheiEwATtaC3ObuLFj8sM+ mXW4upeUaL8/PutYplODR3EMDccUP+opxcTc+DXIu9rrWeruIQKv5JqS3YbwnE+7BNUZ NnOTvULBsgxnOoJelnT+YS1rFiUJyx8NF39rvUBFwhDsnQiq7yt3yXqTv30JHW64tXpq rbbDghFOq6ft7lpOoK/2xJds38wz8ddKJSPljNTxovXJ7SAXLVOYLXfoI/JKP2g9fOIj UmMoLoy82iAwGOhr8ToE1wEGtRtHAGhVRjPgBHMbyTL8uH8gwBYy4Xqkk2TYXZkzMueY iOHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=qot3vNsWBMoe+JdIBIo11wkgMGQiWyELgu1PU4VpSXo=; b=0SGf2lEgAcAWOdVwyyoxHU4x78Ix3J0BK2qJ3TFdlqg2b7Zf1iuOJkEnfmvXPN9TYH otKq0rVPMzNnPcOVLrSaqWCZ/MLkEAXy+zoXj0N+dnijztiTlMp9YlePyBdeON431qDy EbPWlDTk+HyOL80LY118anGNddiDziD16Wo6k7+pviw0deF4k7fn1Tdokg6928TrNBh0 bo43n5uczGeUV5CqmHU/w+320wkt3c2hpVJVyyBDtHkPmEZ43LOi5vhDoj3WsedAXvTt PlUK8+wtf3eQkSJYfO7e5VGsmIPs0YWoKMwFOdXPJ193qSN+JWVk1Fmytq6/60+CtgFz v5BA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM5320NsjQnTdOu6uqpHoie5+OxPSH95+gCla+pkgiabqs5soGn1Hx F1fPLBpWb+Y4OwFQUqFbOYc= X-Google-Smtp-Source: ABdhPJyLjB+Zth442mSdVhf1qW5D6Rc6ruOU4nnPEjmFX0toQJWrWxTpYB0S7lXu31y9WKGrrl8xww== X-Received: by 2002:ab0:5a85:0:b0:35d:20c3:5463 with SMTP id w5-20020ab05a85000000b0035d20c35463mr44308574uae.103.1655104082798; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6122:1396:b0:35d:950b:5117 with SMTP id m22-20020a056122139600b0035d950b5117ls122234vkp.1.gmail; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) X-Received: by 2002:a05:6122:a28:b0:366:d97e:1bd1 with SMTP id 40-20020a0561220a2800b00366d97e1bd1mr4294579vkn.7.1655104082267; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1655104082; cv=none; d=google.com; s=arc-20160816; b=vrfMZ+GTZ80yG72CO7UFF/9IC/Wq+27jOYlUj8GMoGYMyrlGAxnA1dootBQgfhg+jZ 80yeyo5oFJX2YgoCCxwrv8z/RIVzvPhCHl5+hUwni/2EKG4s08Ka4Y7k8RjmeR9U2F04 PS27GniS4Z6123dEkXgaJioX7uVOETR3D+p0FVk7f9Jc7+sL203XhAzfmQnspt5DPwXi 0uh+bl01twN83CQZtz4mWlcqKkkgSsF6mlJLyZPnk53OKfGKugsYL7aYa+NMzQsTH4IP ZC+4MivYY7X6HJiYhJqrOTw5ppmkGgGpIvbNR3zMdN2mX6xVrTFcFMQQKpl+RENoEsWD a6gQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=U/ynoNiZaEt7+4oJK/PBxYENfiQ7Igg1yiX5IX/zUG8=; b=jyMvus6wD+ScjoZZtYch7RMfYK2bYz1kYg8FisTn8AM0sta0CzO3fmJwULmO4Hgnux dw/RMWenjjFUhO/BLmaLInVEwYVQoXP8kIpxs++iPzqLwQBVH7z3PSZdAmOXG1+5rEiy B2pGG+ANjUe7sdW50luUJi+OUUk8g8hlH4BP9n4cUka1QNHKTTSPBTniClZsecdwbZHq QbssUcXiXw40Sun9akGsIHFBREmQa5IWqvvp9y8nbsVKmbIQJjtw82mbE6yw6pkXjjRE jRw00nxYKr27Id5YyX5JY6kt38CHO7cAvoS+/fWYv2KdWbhlM257cH8wyvgh8+x4QOG1 J3mA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 11-20020a05612208cb00b00360358bed08si321488vkg.0.2022.06.13.00.08.01 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 13 Jun 2022 00:08:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 25D77xZ9027766 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 13 Jun 2022 09:08:00 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [RFC v2 2/3] doc: Add section for isar-apt reuse functionality Date: Mon, 13 Jun 2022 09:07:58 +0200 Message-Id: <20220613070759.16949-3-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220613070759.16949-1-ubely@ilbers.de> References: <20220613070759.16949-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX Signed-off-by: Uladzimir Bely --- doc/user_manual.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/doc/user_manual.md b/doc/user_manual.md index 02874b6d..80f79ddd 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -24,6 +24,7 @@ Copyright (C) 2016-2019, ilbers GmbH - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem) - [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem) - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages) + - [Reusing isar-apt repository](#reusing-isar-apt-repository) ## Introduction @@ -1254,6 +1255,28 @@ sudo rm -rf tmp bitbake mc:qemuarm-buster:isar-image-base ``` +## Reusing isar-apt repository + +### Motivation + +Building custom recipes from sources may take a lot of time to download the sources and build the package. Also, reusing isar-apt makes it possible to work offline. + +### Solution + +After a custom package is built, it's placed into isar-apt repository. It's possible to reuse this repository for further builds. The behavior is controlled by `ISAR_APT_REUSE` variable in `conf/local.conf`: + +- `ISAR_APT_REUSE = "0"` - always rebuild custom packages from sources, even if they already exist in the repository +- `ISAR_APT_REUSE = "1"` - reuse previously built binary if it exists in repository with the same version + +If `ISAR_APT_REUSE` is not set it behaves like `"0"` + +It also may be useful to move `isar-apt` directory out from `tmp/deploy` so that it could be easily shared between builds. To do that, modify `conf/local.conf` by adding lines similar to the following ones: + +``` +REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt" +REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db" +``` + ## Add foreign packages from other repositories to the generated image ### Motivation From patchwork Sun Jun 12 23:07:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 1875 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 13 Jun 2022 09:08:10 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-oa1-f58.google.com (mail-oa1-f58.google.com [209.85.160.58]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 25D788N6027794 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 13 Jun 2022 09:08:09 +0200 Received: by mail-oa1-f58.google.com with SMTP id 586e51a60fabf-f1d7f9dc02sf4399080fac.17 for ; Mon, 13 Jun 2022 00:08:09 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1655104083; cv=pass; d=google.com; s=arc-20160816; b=j+hrHfQIe0Szp/VV1uvuHo4FoVWH4SYXlndJ3kLe5DXxR+DLwhwec7nlGfU2hM+vtj s9MQJspO/WuJaAjGIzqEHzxIVlO78cGf97taSOQUr9OdUdCMoOZgKU4cdvJM19YL4oNY L0BelY/yl5G8raguaqQivgJr3aFAj0LuHdEqR138om2VxsVQJYg8GC0dONnueXuHn6aS cX9Ew+GZU+d3BbD0P49YhBjJwWi67xar0WAYCzyVuphkiXHCRGpg95159FGUpfLbhj0s jtbXHOzDY1DoEpA7ewh11qMOwq79FW8Jf4QDAchPJ6NlvX8cda6FqzQ2WQAVVftT44OC 8yWA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=Zj+Gn7QvEp6kIkSvfMlTI5sZfuIi8USPqz6yX7fvjyc=; b=eUPgS9Y9cB+NcQJJF9w5YXM6hw9CSrxSyQcWJgQTnWcDKRX84iEHNPDhJsiy/DCQZh BD0/ESXiXoo+oFpe/7zkhyAgbcoUur5htP6JjAYtLp5euokTZEkdEhMu98x+NY58RXQ4 yDSSj8sfUBWUt0ivn1NqtjvK83cqPhaj0Cgnr64p9u3wLO9BYdkI+wIMzuk9HQ54ESUP g/8qpsq1LvoJvgFYGfqDzmMznYz+KkQGMQtMvsR5ZHcRZ3CdZaAK+ka6jIe5LpMRgsMh gtaHtxqcPYUaVBGm5hfLXWwHxuyS+F79pz7WKj0K1nO53S1n4u18GD5ovewO0uOR4mn+ HF1g== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Zj+Gn7QvEp6kIkSvfMlTI5sZfuIi8USPqz6yX7fvjyc=; b=NMLIocy3QnKypu/63Iz9VoBEi8rTOwjJQrkkdEAdsYgd5TblIEb2WMwwJdXAvZ+7RG 9aUhTSbIRuDoyuJnRUaA5APYHB3Nu3bxrsrgJAWMs4cazEZ2qsHan6s7v12tnd3yZ6gk ZMoDjEkRPd0s36jndmG3XxVOp/UXfsZPyqcspLz0TlMZCmYhi1g5nBgTabVX/KL9ewJq aMEI6+kGiXXUTgfPAko2uf/aoRGSXExhe/AY2dMH+/JbEHqSGJfJCVGwtOglBxQ0xzOO H6UpLme71ZVUEBDVNvryrKmFJEIQCsX5/W84s1ix3jO3mqctDW68dAvH58bfOeGo4yra yC0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Zj+Gn7QvEp6kIkSvfMlTI5sZfuIi8USPqz6yX7fvjyc=; b=eJ5aWXR1EUI+rTuPXAvH/i5sNkR79Ie+RIqnlwq1avxn+UE0HzSNKn+RwuUuHQVzZv t6E/VyL2iYqXUbz+3Yw/1JjP4aYEky4y1yGnYKMVtmayQVClw7KfO3iQ9ceZkNqI30a0 Q0Yrwv22DpRuVUbGZ5pegAp/O9AMQMQ3KkSwhh/I/zRckBA8PGLl0ZBgulQCT0mlNXqa uZfervv2zNnvsSbCIBXkTuqbLH8bwGeu1dn8Xacjk2YpCTvIU8nhZ/uuqRYyao/OF/Q7 f979+Gxh/SiNMX7R9sFgCF6kNxOPz0qljASjhTV5ar214wBdW7nXwcubZFAIiMVJMy/5 Eofw== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM530PDyVPfZEryaBvdZM6/YJJLoaEcQ850ujIxiCYi4yGJ0BjCWuz Lej5svnAuwkU993iwXzgJ5g= X-Google-Smtp-Source: ABdhPJwKoCirZvRua5IUDtP7vUmYMKmQmf/KVmQZ8w+jzfQDd0vApRHHP8AuUkXcr68o2Nx6QJZZJg== X-Received: by 2002:a05:6808:1794:b0:32f:c68:c3db with SMTP id bg20-20020a056808179400b0032f0c68c3dbmr5673768oib.209.1655104083145; Mon, 13 Jun 2022 00:08:03 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6870:3c06:b0:fa:8e9d:2fea with SMTP id gk6-20020a0568703c0600b000fa8e9d2feals806693oab.6.gmail; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) X-Received: by 2002:a05:6870:d355:b0:101:2030:268c with SMTP id h21-20020a056870d35500b001012030268cmr4215239oag.264.1655104082639; Mon, 13 Jun 2022 00:08:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1655104082; cv=none; d=google.com; s=arc-20160816; b=ZlibrrnCuuighdAvYeieBi43XLin5O599Wv0y6HcJ3XhDUUVzmqAHkK1BWIJJS59+J wadyJlhH8uhz+sO5t04TVqNfWyeGj03qwLpJts7KBEHTkSZf1wx+85bJDr7kQYvj9w3U 8ss6uYoqpXYdpTr5mSPGqruNKMOg55eXaDv4vtj9Yo+p6L9KP1MaliiIhpFtjH4FRUTp XlJZJaa5zTmwMNCQl7kcAiEfSlqU9Wpb13zA4t1pCo0EtGhVOStA1657rVbMeWpd7Zxl CTX43dstj1kU+pYRcqiOXktEwaUfbKPMuREBuBMp1Wb7yCD4n72kS1IOivjA7AXhaawQ R7fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=SyO7+QZIvFKLoiOxURkyFXh0RCKqXkologR65BtUtkE=; b=MT/8d+sf4P85tczozXlxP85MQj+a+r2g83zi6KvYOe1ASrqveluwUddnT0L2EQkby+ kd9QVp/p1lGGF20ESnSVMYh+ziR97tTyomfL/IGI1Q6G11ot7m9iQfoyWsx2rjjcPRCD hLATUpvhlQZE2CaQXOnIgvu4sc+QkEutzNEOs5yTSNdVNZ0mi6n/5Fzvu0AvQoc+Z1S4 mLcviOyby++2WpqiQa6wznt6cAxMDqVa8t7h1qu7bHrdYcmlA/xNeiHtJGsjodJ6oJ2i 9dVq8Sz5cAs/ww2Kj2AF05TbzSiyTKORJOt7E37zjLIm3g7yW+7PY50nvsw9tYVVDtuu qpRw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id q23-20020a9d4b17000000b0060bade020f3si304481otf.5.2022.06.13.00.08.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 13 Jun 2022 00:08:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 25D77xZA027766 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 13 Jun 2022 09:08:00 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [RFC v2 3/3] ci: Added test for isar-apt reuse functionality Date: Mon, 13 Jun 2022 09:07:59 +0200 Message-Id: <20220613070759.16949-4-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220613070759.16949-1-ubely@ilbers.de> References: <20220613070759.16949-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX Test makes clean build of "libhello" package that is placed to local isar-apt repo outside of 'tmp'. Second build with cleaned 'tmp' should use previously built package that is detected by missing Makefile in source directory. Signed-off-by: Uladzimir Bely --- testsuite/cibase.py | 42 ++++++++++++++++++++++++++++++++++++++++++ testsuite/cibuilder.py | 7 ++++++- testsuite/citest.py | 12 ++++++++++++ 3 files changed, 60 insertions(+), 1 deletion(-) diff --git a/testsuite/cibase.py b/testsuite/cibase.py index 2ffb8191..9ef9d247 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -73,6 +73,48 @@ class CIBaseTest(CIBuilder): self.delete_from_build_dir('tmp') self.delete_from_build_dir('ccache') + def perform_isarapt_test(self, package_target, **kwargs): + def check_compiled(package_name): + # Makefile should exist if there was compilation + makefile = glob.glob(f'{self.build_dir}/tmp/work/*/{package_name}/*/{package_name}-*/Makefile') + if makefile: + self.log.info("Makefile was generated") + return True + + self.log.info("Makefile was not generated") + return False + + self.configure(isar_apt_reuse=True, **kwargs) + + package_name = package_target.split(':')[-1] + + # Cleanup sstate and tmp before test` + self.delete_from_build_dir('tmp') + self.delete_from_build_dir('sstate-cache') + self.delete_from_build_dir('isar-apt') + + self.log.info('Start build and populate isar-apt...') + self.bitbake(package_target, **kwargs) + compiled1 = check_compiled(package_name) + + if not compiled1: + self.fail('isar-apt repo used at first build') + + # Cleanup build stuff + self.delete_from_build_dir('tmp') + self.delete_from_build_dir('sstate-cache') + + self.log.info('Starting build and reuse isar-apt repo...') + self.bitbake(package_target, **kwargs) + compiled2 = check_compiled(package_name) + + if compiled2: + self.fail('isar-apt repo not used at second build') + + # Cleanup + self.delete_from_build_dir('tmp') + self.delete_from_build_dir('sstate-cache') + def perform_sstate_test(self, image_target, package_target, **kwargs): def check_executed_tasks(target, expected): taskorder_file = glob.glob(f'{self.build_dir}/tmp/work/*/{target}/*/temp/log.task_order') diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py index bc48d47f..42e344a3 100755 --- a/testsuite/cibuilder.py +++ b/testsuite/cibuilder.py @@ -54,7 +54,7 @@ class CIBuilder(Test): def configure(self, compat_arch=True, cross=None, debsrc_cache=False, container=False, ccache=False, sstate=False, offline=False, - gpg_pub_key=None, **kwargs): + isar_apt_reuse=False, gpg_pub_key=None, **kwargs): # write configuration file and set bitbake_args # can run multiple times per test case self.check_init() @@ -76,6 +76,7 @@ class CIBuilder(Test): f' container = {container}\n' f' ccache = {ccache}\n' f' sstate = {sstate}\n' + f' isar_apt_reuse = {isar_apt_reuse}\n' f' gpg_pub_key = {gpg_pub_key}\n' f'===================================================') @@ -107,6 +108,10 @@ class CIBuilder(Test): f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n') if distro_apt_premir: f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir) + if isar_apt_reuse: + f.write('ISAR_APT_REUSE = "1"\n') + f.write('REPO_ISAR_DIR = "${TOPDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"\n') + f.write('REPO_ISAR_DB_DIR = "${TOPDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"\n') # include ci_build.conf in local.conf with open(self.build_dir + '/conf/local.conf', 'r+') as f: diff --git a/testsuite/citest.py b/testsuite/citest.py index 16e38d07..7545d9e6 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -60,6 +60,18 @@ class CcacheTest(CIBaseTest): self.init() self.perform_ccache_test(targets) +class IsarAptTest(CIBaseTest): + + """ + Test rebuild speed improve with isar-apt reuse enabled + + :avocado: tags=isarapt,full + """ + def test_isarapt_reuse(self): + target = 'mc:qemuamd64-bullseye:libhello' + self.init() + self.perform_isarapt_test(target) + class CrossTest(CIBaseTest): """