From patchwork Fri May 26 07:00:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 93 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 26 May 2023 09:00:43 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-qk1-f183.google.com (mail-qk1-f183.google.com [209.85.222.183]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 34Q70eBb013361 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 26 May 2023 09:00:41 +0200 Received: by mail-qk1-f183.google.com with SMTP id af79cd13be357-75b06a31daasf81417885a.2; Fri, 26 May 2023 00:00:41 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1685084435; cv=pass; d=google.com; s=arc-20160816; b=j6/Qb5IV1HnuJtpAkunKBIH0YIXO1/E4Raud1kG5MepX8RD7oSbQiKjL8BYf+pXI06 IgW9ovGm1hgo7gow+TDF5EnK9hhPrYNLWJ4Vx5vnomqtVFwfqOYEM2BxlGSFg1yJ1ObM EMYOF6RaNkBOhFRPXKYxx0+FgtZGnpacnQMXtWEdQHyj36JKR2NBtWCReNscFR5t0/J4 MQm7C02SxCq661FguMA4z1hzlC71+Jw45jfR/TXhl6IlCaN6ae+D0lzIjlzuTwj5atb5 J6H1ygqlG6j4IkRfyhymX13rsn6TFspN94zIvK9HozNXRXJldNLMh+mPdDbNOHvLiASx wJdg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:to:from:sender:dkim-signature; bh=9szrsfdgXEIvIPYJWSatBYSKEhqstAobdrFlVM319zk=; b=vx/3D41BuAxA5YBP9crtgfvGBG0bwoR35iRBkUFTApoaqQ7ZKu1Hyuh2Fi6zt5VS4f wzxuV6HS1TS5KmqTnz+ysOM8TfJVdcn6Rn7AnNtSpcZ2eRFlLb8NEWYGaiZHGHkaXver VsJCx0G3oUe/0ODfIVz84LQhklUJt6wrzv84TGEKWhooJVzRehtfyLbH3EMSOPCCTeC2 DX36JlizmWRIMzLKIRzjz4qh9SEOkekR0KRq4rHRHXUpTSOnWqUS2na52/v0affM03ks eIakOywqwzLz5xhS9xw9xmU/aM+fJwBEzt9pVopHe2LZTB+WvaYC2z+uB7HlM0E0NHZo A5OA== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20221208; t=1685084435; x=1687676435; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=9szrsfdgXEIvIPYJWSatBYSKEhqstAobdrFlVM319zk=; b=dLcQBqClUS174bwRNLvygBLrGb9Ls5G9PDGtaQHUzHbXRrkIlxQzZzAAZgbOOhv8aX kBqETKkkrGIS4PDK1723quFQZknmVkJoDmf9JI5xj381x0mQ0qDJ2zXGsoelKByKY1yy OBRs8gEcHzKR9hKnmK3B6XND1Kq7YOOc3+I50XsRgtVdFFNZtuh6yeGi+sTZjdeUu+sK bdf8RuBPpJnH4Id5C1dRviq2XbcdAO4bx9QzVDAysiIUW3MdzDqYBq3cE0NMSJ41Kd4Y GCraXcb2EPg5buiQgBjtYFWgzNISN3if4qRbAsWRRyUZhQQcnjNWALWVrvDPIP++d6FI iNGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685084435; x=1687676435; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=9szrsfdgXEIvIPYJWSatBYSKEhqstAobdrFlVM319zk=; b=ECylahJN4HJGoqTSd8O/NTiM1a2200VHXVXrlRBLnq0rbwgZrYQ6wilwE46w21mCzU Gcc8gntBqmZeO7Uh9IYJdzAC6yGPLOK9aAvUAYkEb3VaJObzmGPr0Mf4PXNsPWpy8Fov bS6UwYndeKHQNf+zL8T83D/6IRRDUphTLV1v6abiQdTSSpkz1kIApMeqzSb7nmqiq/6f iSIzs3Ku4nbBvRppj8IowpnNIOkR7GxKPkBanA8mc885Xfi3+8xcD2maJ+MOBVw2fZn3 9e4Hk4yjC87nwBpUY3eh0YHxBUv45mJAGTQKwdIWxBeI9ccJYsxvCW4MwF4QtsDcpyCt 7Tsw== Sender: isar-users@googlegroups.com X-Gm-Message-State: AC+VfDzQ9sCTgo2BmtohzNpD4L7v0zj8TZ0CyTGlmLD6L/6uyt24M7x+ wDlMsgdf1AU79L1j4nIjLF4= X-Google-Smtp-Source: ACHHUZ7SLPx3KOGMJlslMc9Wo3dbcyPJite+rUmeVARlO6OaBgKfH051M0msUoIlsVQcCJ87Gt3vQA== X-Received: by 2002:a05:620a:1909:b0:75c:9b66:d01d with SMTP id bj9-20020a05620a190900b0075c9b66d01dmr387704qkb.9.1685084435187; Fri, 26 May 2023 00:00:35 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac8:75d8:0:b0:3f5:3b17:86f3 with SMTP id z24-20020ac875d8000000b003f53b1786f3ls2582182qtq.1.-pod-prod-01-us; Fri, 26 May 2023 00:00:34 -0700 (PDT) X-Received: by 2002:a05:622a:651:b0:3f5:1ff7:db72 with SMTP id a17-20020a05622a065100b003f51ff7db72mr755069qtb.22.1685084434581; Fri, 26 May 2023 00:00:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685084434; cv=none; d=google.com; s=arc-20160816; b=Xaz+YY5wdhXwAf9c5Ykj+a8TELF4/qWpsE4tClxNJL+nrQ1/6tFMn4l6l+DZeLdhRX 6b/ZfyD7LZPfL5xyCADeQQi6e4qSkzSe0u6n2GSw17HDb++xr9lW6KyvlTGQQoPSuylc EGdhiXlzvNaBd6BFM2yaMP0996cqdfC6XEydHP4vuPVW5ZnbYIBxa08DMJTD5KgDXrn6 E7EbCTKEFvujXDj2zYp2Gh2z7DmJJoNAD7wKoAo+tdi9KolUhp4oObrfUQoTI9F3Uxzl yoRJC1cUt7XEgWkOE/s1Xw5w5ooMKWwY7fwVWssiKI/ZCvTqjnGXdIMzntxnqnQ3rFZe Cu+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=9IlyKdq/6/GyL+WTq1aDXpzThLnVvPO9vx6q/AaXua0=; b=fYJ4bjB7zaKE9Hl2u9xsCXD0Vfh2cpxi8mjyBd/noRNy0CW+BwH4R+ELfAbgvZ/+Os v3WcyxgPku8AnqMvXxpkDbO60NU9Cy693XZWeX4l7Ems0QQCmQN9qwjWA0VK950U7w3h L9f383BfaCHK7oBSpHb4Layi5gOyRcvkTmQmc9CjSpW7YEIrHD41qP0KB5rfG289QQp1 mkax4UAbbicyLRb6DZEsbk2++UadtcOfFy53rVjJKVTfK+KLpnUBA1GYUF3h4Rq7AZwp 4eq3sopMYGQYMMmyTww0yGp1Y1yFXO1TiZDsGV6UTY/xhDMp8PkRQ5HluWk8jZVEMbxF 7BBw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id v8-20020a05622a188800b003ef46e5143dsi306243qtc.3.2023.05.26.00.00.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 26 May 2023 00:00:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 34Q70R3U013306 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 26 May 2023 09:00:28 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v5 00/12] Improving base-apt usage PoC Date: Fri, 26 May 2023 09:00:15 +0200 Message-Id: <20230526070027.16890-1-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Currently, base-apt is used in the following way: - After the first build, all .deb files that took part in the process are is cached in the ${DL_DIR}/deb/. All the packages are downloaded from the remote repositories. - At the second build, `base-apt` repo is first created from the previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set). Futher debootstrap and installing packages are done from this local repo The idea of this patchset is using local `base-apt` repo even at the first build. Before any build step that requires additional packages, these packages are predownloaded with `debrepo` script to the `base-apt` repo and the debootstrapping / installation are always done from it. Potentialy, such an approach allows to stop using deb-del-dir import and export functionality in favour of debian-like repositories. Currently, separate `debrepo` script is used for prefetching packages to the local 'base-apt' repo. It requires python3-apt to be installed on the build host. Potentially, the functionality it provides could be directly integrated to the `debrepo.bbclass`. Changes since v4: - rebased on latest next; - rearranged patches since some of them are already in next; - added possibility to select between new "prefetch" base-apt mode and old behaviour when it's populated on 2nd build with packages downloaded during 1st build. New behaviour is disabled by default, but enabled in local.conf.example for testing purposes. - code passes both full and fast CI in both "old" and "new" modes. Changes since v3: - rebased on latest next; - cross-building for raspberry supported; - code passes both full and fast CI. Changes since v2: - populate base-apt before using at all steps of native build. Changes since v1: - rebased on latest next; - updated patchset description. Uladzimir Bely (12): meta: move base-apt from deploy directory to the top ci_build.sh: Install python3-apt if not installed scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to debootstrap meta: Consider global debrepo context base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode Set ISAR_PREFETCH_BASE_APT by default meta: Specify grub-efi packages arch Disable deb-dl-dir in base-apt prefetch mode RECIPE-API-CHANGELOG.md | 9 + meta-isar/conf/local.conf.sample | 4 + meta/classes/buildchroot.bbclass | 5 +- meta/classes/deb-dl-dir.bbclass | 25 + meta/classes/debrepo.bbclass | 86 ++++ meta/classes/dpkg-base.bbclass | 47 +- meta/classes/dpkg.bbclass | 8 + meta/classes/image-locales-extension.bbclass | 5 + meta/classes/image-tools-extension.bbclass | 9 + meta/classes/rootfs.bbclass | 13 +- meta/conf/bitbake.conf | 9 +- meta/conf/distro/debian-common.conf | 8 +- .../isar-bootstrap/isar-bootstrap.inc | 111 ++++- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../buildchroot/buildchroot.inc | 11 + .../sbuild-chroot/sbuild-chroot.inc | 11 + scripts/ci_build.sh | 8 +- scripts/debrepo | 443 ++++++++++++++++++ testsuite/citest.py | 3 + 19 files changed, 804 insertions(+), 32 deletions(-) create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo