From patchwork Mon Jul 27 11:26:18 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aliaksei Karpovich X-Patchwork-Id: 394 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 27 Jul 2026 13:29:13 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f184.google.com (mail-lj1-f184.google.com [209.85.208.184]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 66RBTCf1001011 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Jul 2026 13:29:12 +0200 Received: by mail-lj1-f184.google.com with SMTP id 38308e7fff4ca-39c79b2836dsf11061671fa.3 for ; Mon, 27 Jul 2026 04:29:12 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1785151746; cv=pass; d=google.com; s=arc-20260327; b=oeLpU2EXjAP/J+WKG9N7MMCZ1IQrIxVzQgGsbC6C/W50pb3HZmY52HazAtXr5do7r/ 5W0157hmf6lC9fqDmSbzSffm6Aj9QrhQNhizUxJO/UQmvAlXauX52XJV1/elFWjnOM0x Nwn0SETdgOCV6wFYhTQlzltJClx+JxFGYDTNjleNMUcAuTTroDlUR+Nsdd6wd0lyK46W VxMFvu9syHRqBJy86YlVmaTA44tUMFEUOqbKxa9VK+CRKARuCr4wW8u32i2o3vxYTEcB 42Cd2DCQOzCz/uJGjQVr0oHF6hitLdbc8zwthsptjsqr69JY7I/CC2UTXsgEYs4VVJrx QwFQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=wCR6v0QP6sDHRNth0bg3FtZ3bWDQW2D1YAeTLYUD2Xw=; fh=ojdXPTBW0DAGqokbvspy4a4FDSoUksLh1oBF3xGeEQw=; b=b4IOso9GN/dTqR8+zka9eKYq/jgrHZpX8d2c+KStJb7UYp1RvFaEQu5Gg6pSehjmCr XCMLhTRB0kHtjSM3dLlCXGQTiMfuoxcRdLByKBm7RyTZFsv/5J44vUBTpvxyFW2YneAL eS2Hf9P6RPmiNz1O3dRiKUAJdib8AWcmi9oyWU1iSn4R6Mi39r5UOicAT1ZD6bvWFVI3 hw9YeJnqjOB0gsWtvbCajUdnNMxxR3nIVVuw4uE33vFIjkMV6jMw2tMzOqq6sssSyVtQ H0X1ZFfluDYvxlZ/FsgoL61xtpxfNLzzNsjPKQn2zDh6E5A03facrbLE+8ZrEC48g/7f U4Rw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1785151746; x=1785756546; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wCR6v0QP6sDHRNth0bg3FtZ3bWDQW2D1YAeTLYUD2Xw=; b=PTj0wdqGVXtaew5P4yueTJirY9nzKfHnFD+u6PK2OZ7x2Vcey2RD1OAGSk+RRVP5aG XeCcmOTyVjM0CuGilBEHbGkxnYGd2n22kXseHrf+n7xSdZagK/hlVzvtt+m8OrXfRtKY IVTjaYkDi/XPc6yTjZnnBvb9bVPxaTepPrbhE+v2cyeEf7Tmw+COhOSs5QJNaywfGsRL tfBdT/au0kDwjrg8TVfaa3ToVLA0h/Arxd0zF7yD++IX3DrMLpBYwfnoCUpNjEygk5Xu FUOnwbUBUQw7DDx3UepDVMYgXgckoLTESiT3fBHfVtLdcRzB+sJI8qd9l/0eeqLFRBp7 QOmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785151746; x=1785756546; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to:content-type; bh=wCR6v0QP6sDHRNth0bg3FtZ3bWDQW2D1YAeTLYUD2Xw=; b=dhCb2fqe+KuJ97EeARY+hoU7uCL9SSX/eqCk/2/HSXMMChN/7iINrhyCUVBMtk2FDG OfLxm4++x0o6P4Px/0mivLgjdzVCeNtj28mlzfKGaqNcI2QVpaFci+VmcdvkGSeOAtsF IZ8BNLS8Jrbui1GDZ3HmWhI+avdVKM99tGS67gy0JiKvyZw70SC7lxT5NTG0XSHekovD 1VlUHJDXMxdHBk3kxGzWd/FR3UD5WYw6CJJu0xiui1N+W6Ds1snGnQP9sNw3Hc2UhYPF OKs+ZEMhemrJi6n9Q9QwK27ShJgrriuNyXdKE7t83DvE4GAJulq5Ba/+2jPY8HYLt8js Q/Xw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AHgh+RoKh5/cs6QeURsdcZooTxniptAGzh9mSPi7sXNawcB5DGpQ02b2PEX4m3dfvimokax9OMR9PBc=@isar-build.org X-Gm-Message-State: AOJu0Yw9KAq6EsiPpTSlSwKKLPx7Z6QZWpEnRqUDz5Eairf0ePGtq7Bk QlmXI387lozR7m5qfR9d9x1i7NrWFShonQlFJ/7BYE23nV1af6gKDhRY X-Received: by 2002:a05:651c:210e:b0:38e:8760:2922 with SMTP id 38308e7fff4ca-39f287dfc2bmr12507051fa.18.1785151745574; Mon, 27 Jul 2026 04:29:05 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="Aa7YSPRfA7GUrxRCJB0/CgMgANlEotMHA7F5vrvrtnLtXZvmyw==" Received: by 2002:a05:651c:20cd:20b0:39e:e8d1:b786 with SMTP id 38308e7fff4ca-39f1925cba8ls8519551fa.0.-pod-prod-04-eu; Mon, 27 Jul 2026 04:29:01 -0700 (PDT) X-Received: by 2002:a2e:a592:0:b0:39b:1b6d:5027 with SMTP id 38308e7fff4ca-39f287bae1emr13949811fa.11.1785151741193; Mon, 27 Jul 2026 04:29:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1785151741; cv=none; d=google.com; s=arc-20260327; b=bY+n+BiQ9BulcLO52M1ahCEmCnIs/OoSBB/ncN6GQNheDIuI1hJ7HI9xACwb0jYJTR dIXapRG+KeND3NulBPeAnWgzW+QNI4nR7pBqfpEXCZa/ez7YzJ3T7Q6aBhGMMRJPa/oY a2D2tEd79yBlTkH2vvPOvtz0Gl+Ohb5ADoaiKY1MPogXObJkQjpcOLFDJ2YKwQhwMeXa 5/9iWQcp6x9oYnH4p68v+jTIc+jLDzpYc9lKZCF0KrgGVSd/ggA5KVCY7oWLJ/doWvJA 2y0JoNsqG1dO8Ve8NvRNREuozcBwUiX0vosDBmHqwxj18syRPQmwgqACeRCwWffHgtx6 tG8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=uYHQ8HZQsA1ND+rhZetA0I8qqB8bQo9bpBtAMO/8mAs=; fh=u+3kXEMzY4uVhDBiujW2QUo7NnpMLrGEgMQBYpCTRN4=; b=KlxvJO3l54K2/Bie7Kno8RQ20LNdqMAlFdutGNyvJIDe8Hv1nfIdXL775p4gv1T1f2 N5E6CyVsUKAYf9WcIK54aUP4n4Ixdxn6Z60p9La9UF7RAnn2DolX0guxa3W43Jlfl6BW nia+X9yVQXzaiFKm3F6CPd/Vn8tHv1xJkt25WNCEIdtNb5MNssejhdu47R+mf9fvtZIj e3o7GGxe759Wyyf5qVHnYmSMBFkAQqXYUgcv9spkPlEG2J5j0RHL1F1XWbWQFfVDPjgq A/v1EbRC4IINKwCpdpX5ONPyyXlOIazRgwa2iZJtevpyQW5lMhV2Myy6cWfZWQzb+WJE wOEQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-39f2223d6e9si1290581fa.4.2026.07.27.04.29.00 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Jul 2026 04:29:01 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from akarpovich-Victus-by-HP-Laptop-16-e0xxx.promwad.corp (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 66RBSjdQ000930 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Jul 2026 13:28:59 +0200 From: Aliaksei Karpovich To: isar-users@googlegroups.com Cc: Aliaksei Karpovich Subject: [v8 00/14] Improving base-apt usage Date: Mon, 27 Jul 2026 13:26:18 +0200 Message-ID: <20260727112812.2255297-1-akarpovich@ilbers.de> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, mmdebstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving and reusing the packages. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to bootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v7: - Rebased on latest next (70661c40) - Fixed url parsing in rootfs_install_pkgs_isar_download() - Added version and arch for downloaded package in rootfs_install_pkgs_isar_download() - Workaround for base-apt to force to install python3-apt amd64 - Added using override for reprepro to fix offline build Known issues: - Didn't pass the testsuite CI - 50% of fast tests are failed Aliaksei Karpovich (4): rootfs: Fix URL parsing rootfs: Add version and arch for downloaded package sbom-chroot: Fix python3-apt dependency debrepo: Add using override for reprepro Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to bootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 10 + doc/user_manual.md | 1 + kas/opt/Kconfig | 12 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes-recipe/bootstrap.bbclass | 50 +- meta/classes-recipe/crossvars.bbclass | 1 + meta/classes-recipe/deb-dl-dir.bbclass | 47 ++ meta/classes-recipe/dpkg-base.bbclass | 1 + meta/classes-recipe/dpkg.bbclass | 8 + .../image-locales-extension.bbclass | 5 + .../image-tools-extension.bbclass | 13 + meta/classes-recipe/rootfs.bbclass | 21 +- meta/classes-recipe/sbuild.bbclass | 3 +- meta/classes/debrepo.bbclass | 90 +++ meta/conf/bitbake.conf | 5 + meta/lib/aptsrc_fetcher.py | 22 + .../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 10 +- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbom-chroot/sbom-chroot.bb | 2 +- .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/debrepo | 659 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 7 + 26 files changed, 998 insertions(+), 18 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo