From patchwork Thu Mar 14 07:27:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 161 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:31:08 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f58.google.com (mail-lf1-f58.google.com [209.85.167.58]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V1HK021545 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:01 +0100 Received: by mail-lf1-f58.google.com with SMTP id 2adb3069b0e04-50e91f9d422sf662756e87.2 for ; Thu, 14 Mar 2024 00:31:01 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401456; cv=pass; d=google.com; s=arc-20160816; b=TYpYRyDEpXPeI2+/WXKv49FS0+a7onX1UV+0LVccXarT4W2uuAOTjyJN6TdTya0Vu8 upahviuSZ2eQqWXugZOk+PtJBTcIs7LC4z3iBztLi1zylX/cWLOUAsITASGo2Ab0z6jc AZcf3BasCGaN+RlMQ7Taj0+DkpS00jVUn6vhj+xpH5QsiuzS6APbUVN8eHm8nHO0YtW4 Wp1X3bJv3wT8bfJoW1I7yAj1EdyAlcxCksL7fPW6/jFJNEVGwZFmIuMNBVkpSMmJ3wOV 7kaSt5hyictce1XEIUjooOGcpkqsgTtyubBxjcEiR6z9LwIfU3j3Mg4ijUgdSXnEhnQA SWPw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:to:from:sender:dkim-signature; bh=ZOGkF9Bf/PYk5/293uSwphZMkyiLayzS7PursAZBR2o=; fh=lB6NHN6WU1v1dyOh0tYlRCNlLBVbc4noZyE+c4HIRhg=; b=c5bDq8fDTBV1ImXQ0lQUMJKXxi2wZaUP54OVwmz85LKTbblHYHLMdLBe8wo0guh0uw up6pWZbrjrpg/20Uo6l4UfAGBg3GajoNBCruYt7HbZ9hrbC3Aeks/6cDZCB0+BJy/CBH QRuT6qhGRu7Wl+w0rNAPiNSbw9LzuVqNeOPxKDOixQsSII7K6FPKWFVaWu+IIiNStMah /P1WWghe5gTEsre1auxxYkLYqT526plKrfIxO0K6o8lAzqjGoCIEXEgXna5LYuke35x/ X9H583OQF9JZZrIAF1XusF67GZv9aeMcuIuBwdyRglH1DBOoZJGiaf4O3jYLtYGoELE7 2g0w==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401456; x=1711006256; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=ZOGkF9Bf/PYk5/293uSwphZMkyiLayzS7PursAZBR2o=; b=HXO/cK7tlMsFshEaJi4waANLiENnMD464rctiITYxPvnZ2irmyB6mHPhCmevrwOFKP dmMmv7LSSXWyFzcckjyv6gqtaXcZNHCzDmisGSqbnayopYBq5A3E1HljaNFS8qUzOHRO ayMNplgLpkQzS7CMWQ4ZtARn5KEG40/Uu8RD9NxDunCLek/JIptXprU0ocDOkcXswDxD gm35UizUSwmKQaULgA3heReIre3/45J29331hVxG4rEOsoDQrHKSVpNVFD7IUaptWD6L DHEe6m7qGDmVI7OypOzgTlkWkAtInz/jTMfe+GRc8EuqssTg1zn3jSHHAOHPf4sZYdT/ 7AsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401456; x=1711006256; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=ZOGkF9Bf/PYk5/293uSwphZMkyiLayzS7PursAZBR2o=; b=sv+wo2NsSH6f+qV0nSTAPKbrLBv222kDF2XHoWyJ8h16ylnVh7xRa+VuOMtOh+KRvS TfNc7N4BnQYAzcu1dygltwLCIW0h3Eob5MQVSuum6GQwTePQAQ/YIanylotb9Jm6VMV9 7s8vM5jFR86ynBn3Qyv5cM9IbtLd263hxBEeSU56OtXTij1iv2vvMNWXHNGb+cy/gnM3 N8B21UfWZ2e1xK9g7Z7B1q3obkugAtrShU+kKIUqC8ueOO+Xy3aTpDROctLpAYH9arDx KGEpZQjFnFDcYNekW34LkiKpOKD/QxwmYBX/EW5PClaYTkAG4N8iMmjzvAqTrLr9K2al FhBg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUMwBOexHclr8qjaqht1AS4A6g2LVWx6UmdBfqjfnYJTjMSwt15Pfb4Lb6jVpIxdDSuI4wYMWw62SMIGQvgA09meSodEMY= X-Gm-Message-State: AOJu0Yz2n9C14tplsQuKqVXSaXX5Go+DMWTIT6XawNiHS8T8txU5WU+w hgi//4XnL2/7K/q9FvXmF697oAc0SLITT22fxBXtcA8cTXh2pqqM X-Google-Smtp-Source: AGHT+IEPW6EU1vgZKYOuGkZsCSd3bjx1cBWf0YTJIJdKozPap3EdHpleBtNMyvBQUzXhFUQxTteY7w== X-Received: by 2002:a05:6512:2f4:b0:513:cbbb:3a46 with SMTP id m20-20020a05651202f400b00513cbbb3a46mr554151lfq.2.1710401455485; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:5e73:0:b0:513:b1c4:e1be with SMTP id a19-20020ac25e73000000b00513b1c4e1bels242078lfr.2.-pod-prod-04-eu; Thu, 14 Mar 2024 00:30:53 -0700 (PDT) X-Received: by 2002:ac2:5b03:0:b0:513:c596:713 with SMTP id v3-20020ac25b03000000b00513c5960713mr573481lfn.28.1710401452676; Thu, 14 Mar 2024 00:30:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401452; cv=none; d=google.com; s=arc-20160816; b=tgtBpJ3yror70sykGVz2LXULjrxhCRpEnHySX+3fwPPEn19m9bWA21XRCm/u8uttm5 9bXzCM3Ftlgo2goqdj7pkTyFOHmzqEagrN+8Sgswv3a4Z8rQpt5FsKg0aNsQQFBwmUxI EQJa42MzoJ7Op5utKt8RVW4i4+upZoq6yDI1nzW15ILyT5a4h17D6zNbgDIdfAI6P8mO ZtF2Fzo2bjBIjFOe/ViythFtqP2PE3QJMbyjTZoNHSAZUGd8N3bW9y33egdVNP5AF5BG 6LBa82n/HCHpbaFfjfgQWHf0GlvimIq+Ssk3z+Ewo0oD3PbGYZ+waxgP650y00aiR4ge cR7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=d6kYZbQPXmngIQ+XKKTntp1aZkkM52Ezildn/UEVybs=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=crzipkmOX+b94qOFHK938GerH9IPY7i8U2PwdFEvsdr/Ay4PVtreg1rzhctogZ/ZJE h3v1ySoVwOrLSNDQVg9CTIUR9M94rhbmLt3M5GTHvrMVTPiUF1WMT4iRvIdHFFuPkYlN p+PeBq5FLpAPKkTGAcBGH3RshKhTYUfXW4v6WOuk10UpynBpWPKcqX//x7Udhb+RyILk Zn6DtBZz8kiJor9YTJ/4hZIIBSoJzO8j4leYYfagOihlw5UpvazmaOd0ihGdtKTqzxCw +S3HbrnEPUEHXUM6/frIMToLnAd+QGmISqFJ+X3ps63wvWijK/CDJeuzB60e7HC5W41q NBew==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id m8-20020ac24288000000b00513b9961bbfsi48192lfh.4.2024.03.14.00.30.52 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7Uo9t021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:51 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 00/11] Improving base-apt usage Date: Thu, 14 Mar 2024 10:27:17 +0300 Message-ID: <20240314073047.29465-1-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Currently, base-apt is used in the following way: - After the first build, all .deb files that took part in the process are is cached in the ${DL_DIR}/deb/. All the packages are downloaded from the remote repositories. - At the second build, `base-apt` repo is first created from the previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set). Futher debootstrap and installing packages are done from this local repo. The idea of this patchset is to use local `base-apt` repo even during the first build. Before running any task that requires additional packages, these packages are predownloaded using `debrepo` script to the `base-apt` repo and the debootstrapping / installation steps are always done from it. Such approach allows to stop using deb-del-dir import/export functionality in favour of using local 'base-apt' repository. New approach requires 'apt-get update' to be run after every call of `debrepo` script. It differs from the old approach, when it was run only once after debootstrapping and Debian package database considered unchanged during the build. But, potentially, new approach in combination with the patchset "Introduce Debian dependencies and provides helper", could allow to prepopulate 'base-apt' repo before running any bootstrap/build tasks, even at the first "clean" build. Currently, separate `debrepo` script is used for prefetching packages to the local 'base-apt' repo. It requires python3-apt to be installed on the build host. Potentially, the functionality it provides could be directly integrated to the `debrepo.bbclass`. Some examples of using `debrepo` script in standalone mode: ``` # Create local `/build/ba` repository sufficient to debootstrap Debian # base system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` This patchset v6 can be considered as the first "non-POC" version since it was heavily tested in CI and looks stable enough. Known issues: - May stuck when 'BASE_REPO_FEATURES = "cache-deb-src"' set. Changes since v5: - Rebased on latest next. - Changed order of the patches. - Fixes in `debrepo` script that allow to use it outside Isar in 'standalone' mode. Changes since v4: - Rebased on latest next. - Rearranged patches since some of them are already in next. - Added possibility to select between new "prefetch" base-apt mode and old behaviour when it's populated on 2nd build with packages downloaded during 1st build. New behaviour is disabled by default, but enabled in local.conf.example for testing purposes. - Code passes both full and fast CI in both "old" and "new" modes. Changes since v3: - Rebased on latest next. - Cross-building for raspberry supported. - Code passes both full and fast CI. Changes since v2: - Populate base-apt before using at all steps of native build. Changes since v1: - Rebased on latest next. - Updated patchset description. Uladzimir Bely (11): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to debootstrap meta: Consider global debrepo context base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 9 + kas/opt/Kconfig | 13 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes/deb-dl-dir.bbclass | 19 + meta/classes/debrepo.bbclass | 81 +++ meta/classes/dpkg-base.bbclass | 38 +- meta/classes/dpkg.bbclass | 8 + meta/classes/image-locales-extension.bbclass | 5 + meta/classes/image-tools-extension.bbclass | 9 + meta/classes/rootfs.bbclass | 12 +- meta/conf/bitbake.conf | 5 + .../isar-bootstrap/isar-bootstrap.inc | 106 +++- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbuild-chroot/sbuild-chroot.inc | 11 + scripts/ci_build.sh | 8 +- scripts/debrepo | 571 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 8 +- 19 files changed, 916 insertions(+), 24 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo