From patchwork Fri Sep 4 07:52:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aliaksei Karpovich X-Patchwork-Id: 410 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 04 Sep 2026 09:56:01 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f56.google.com (mail-lf1-f56.google.com [209.85.167.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 6847tuXg004153 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 4 Sep 2026 09:55:56 +0200 Received: by mail-lf1-f56.google.com with SMTP id 2adb3069b0e04-5b2a9f6b0b2sf402361e87.2 for ; Fri, 04 Sep 2026 00:55:56 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1788508551; cv=pass; d=google.com; s=arc-20260327; b=YPU72dM4WQDRCwc3LjWWOsbAVa5cuXG7BgrxJiT5CS84I9+KuFHXeG3gx42MKGBm8S bZ+vaX6qy4T9pghaV9PSJwypHDcltQr5A2COqhAmhvsFD1yimjcuu/yC/4A+VBsAYtcZ MkCGH5pF49tnZsrlggyHraGtDgrUxzmhjLb1sfQBaYzu/YXUaTyQO1sO+dgGD5KMsY59 N6gIaTHtlZwnIGaMYRTePcEFxLV0Us5ljFm1VNRA+QQabz0fxLaKuLDkP3u9UdSOybM4 vBmKcRub0EE2++owIvwjrSmFu7F3XQnwgQlbfgWBBoNyjvNuEkktWkAWHpAPKdT6BpHI IsSA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=ORrOAqSwN4VapxMGHnJnASCKwJWEoRKmUeOGaD0H+V4=; fh=bVtts9eDOnJbPBeFPkR1u7jmqRvwNMXv/x+ki2ldh/w=; b=eifSe+ssA137R9F3+KrzE+BgEdv+08MeZdRmk6N8BuBynDEcsiCFuhrjp5dKfg8YGg ZrnmBI0l946kpIVk8xTkjDI/35fsZ+Au3YtMiPmNRopK+p+JeM1Auca2SG3zFQM+w/rm wNhwbRvrGQ1BeZDaGuLg7hsfnC4n+Nd+yKiD7rAcHcQ9WYHbhpO3gP90+UImaHlXc/jk mJ+DuK/Hin+0mmHjDs44eW7ecyZCpa3PdU8Qd8jO7XPjePdendBaKdbevvQ0QpOpwt1s z0z1k8Hsi90vk8MHEjmvMfoV1dukjzBkCz/JXDc4n1aWcWUEqSmJgH8T7Iawr+5/xbjO xJ+g==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1788508551; x=1789113351; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ORrOAqSwN4VapxMGHnJnASCKwJWEoRKmUeOGaD0H+V4=; b=WF44cRNwAjc5GZY71mf6plYF++IX7kEJiBKJVwqarM72Zx+QpNZJN8ErSrjrp/ibJb Yn6hc6sJQ7FnauKN8LXhBs9BlrMjgSyQ0aMctvFdvL+KfYCB6sv0fXdB1moBtaXN8oft sSF9hsrx+tqE3rHP56Sle+1+KIL+gpd349F/PTpVrTjSvWoOpR7dBBg7mdZkRpmJJkLU QEnO3gHjkht8iTuQBhC4SttJ4iGYXmeFQXOU83A7tYuHhyi2BmO9nDbFX0oDbIeWSP28 o3+308pk95l3CXjMdyP9e6nMEtInUNjHrZkHZuXguljlOWO2UXqLxVL5ml/rxQB3o4G2 Ty7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788508551; x=1789113351; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to:content-type; bh=ORrOAqSwN4VapxMGHnJnASCKwJWEoRKmUeOGaD0H+V4=; b=YTnPGS72jSnuLGT9qKSgF7a2ZWTpud5AgyJrqlm81V1QjBbx89R6j8Sta/IhlNhVTJ a6DyQXY1nClsDMJfTdsRdrWyM/yDJoth8qwV7tZMwWPPAo/OitbH5iNYX2DC65eh9yJk dStPh3PHa/PhhKTNgq2EbW+dX+Er5Xh6teUzW034sv7qlFdHISbnaYjIXVNt2t24+MAY b7nUHEIH03xtGaT7/gICbygq5pHNME94R2rrWaAVpKRc84Gx48LdEPu+86aU09RTiDSN bpKJHAaJyjjpBenygOfVHfou3KRpWU5I//1qr70Wa901/7NTjKxXY+ZIoNb3XRkFUI0X 7t2Q== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AKwUvByjaWpI4wbHk1VmzEwwAbGpprXzxwKDS8Ug5IN094Q9lqvJSp/vNNR5K1p9txCLWumoA+rfFWU=@isar-build.org X-Gm-Message-State: AFuF++lawOB20QcNOOvcU/MUQNG+kmxQ4H/THKq3FSo2yaJ6om68pH6g wPZmyUWaAPHHvdIAZIctpC/NlFfK1qIhxiDYxwMaWWcQtKsB9mj0DbBg X-Received: by 2002:a05:6512:ad1:b0:5b6:1a7c:35 with SMTP id 2adb3069b0e04-5b61a7c023cmr271232e87.56.1788508550374; Fri, 04 Sep 2026 00:55:50 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="ATskLdd2BKyXIojWh+qaTXDH5yoVyKdA2c7KWIbE/ygM5cX3BA==" Received: by 2002:a05:6512:638b:10b0:5b6:1a72:3d10 with SMTP id 2adb3069b0e04-5b61a723e32ls78320e87.2.-pod-prod-06-eu; Fri, 04 Sep 2026 00:55:46 -0700 (PDT) X-Received: by 2002:a05:6512:a34f:b0:5b6:1a7c:3c with SMTP id 2adb3069b0e04-5b61afa3cdfmr118285e87.63.1788508545732; Fri, 04 Sep 2026 00:55:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1788508545; cv=none; d=google.com; s=arc-20260327; b=HQbOwfIc+TKJLIG5znPZx7V0P22SMpnmjPD7FA2hQ1rxL+fTo1CEDs63JpveQ5Exgy HrTzCXxlxvuYTiROsA+/F2K6V1WnafRdOUKzxP3RUtC4nAmJjQqtJ1bLKUh1i3+CvJEM +m63rwZbFg/0jOwSo6o3sHctzkVS4eBcO66PgGLRucPFe1U40XMosQzuFxAug5FAMMI4 qCaoZcHKZs3LpRNbexDt3ifkEydb//TPxtaGj5vzt5/cyTv4zaHiEOAPLJ4Hc1NJqm1N mP7as+aEd5YUubnObmoZxcWEjQ9nitkLS1uJobza9NgBoZNuJRFew+IsZAgYvyJHE1Ib wgUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=3HY4iqBmnhtBAAJT0ep+LICqLIdu1ES88CPrbes1MBo=; fh=u+3kXEMzY4uVhDBiujW2QUo7NnpMLrGEgMQBYpCTRN4=; b=Ws3FfC5MKcw4kuzHfV+hwTWgUq1PJh1L3kUBKQhdFtTD234FTZC5eb/BDUSh+ZON9s 6Ovn7pc+gHS9aLLgP4cw2F8fy1+e/Vy/jqN3dUq/dUvWbID9ArfU5Zq37uChRLCRwuva fAVy/cLH5Us9ygezXGTl9RYTS5bLendixhBPpJUdeymLNn4bwbCrfQJN9EwAcE/kLbG6 q6df7bo9mKXREokclu5p6vJECEgQdfl+Mv1F+QpquWhgQw/WiilP0EVu4/oKht/Wq5CS LkCCWPQHGzaOP4ZPWFzzJCQPOa18FuDkS534aV4zJgPBTiuhaUCcmz+iQ2edHxqLFQ0i z48g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-5b6166046cbsi35500e87.0.2026.09.04.00.55.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 04 Sep 2026 00:55:45 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from akarpovich-Victus-by-HP-Laptop-16-e0xxx.promwad.corp (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 6847tRSA004124 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 4 Sep 2026 09:55:42 +0200 From: Aliaksei Karpovich To: isar-users@googlegroups.com Cc: Aliaksei Karpovich Subject: [v10 00/12] Improving base-apt usage Date: Fri, 4 Sep 2026 09:52:42 +0200 Message-ID: <20260904075515.25962-1-akarpovich@ilbers.de> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS, T_SPF_HELO_TEMPERROR autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, mmdebstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving and reusing the packages. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to bootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v9: - Rebased on fresh next (436bc96f) - Removed "Fix python3-apt dependency" Changes since v8: - Rebased on fresh next (86c761e4) - Removed fixes for isar-apt download in rootfs (use Felix's fix) - Renamed debrepo -> deb-repo - Fixed failed tests on Full and Fast CI testsets Aliaksei Karpovich (2): debrepo: Add override for reprepro base-apt: Move bullseye-mipsel in separate dir Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to bootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 9 + doc/user_manual.md | 1 + kas/opt/Kconfig | 12 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes-recipe/bootstrap.bbclass | 50 +- meta/classes-recipe/crossvars.bbclass | 1 + meta/classes-recipe/deb-dl-dir.bbclass | 43 ++ meta/classes-recipe/dpkg-base.bbclass | 1 + meta/classes-recipe/dpkg.bbclass | 8 + .../image-locales-extension.bbclass | 5 + .../image-tools-extension.bbclass | 13 + meta/classes-recipe/rootfs.bbclass | 15 +- meta/classes-recipe/sbuild.bbclass | 3 +- meta/classes/debrepo.bbclass | 89 +++ meta/conf/bitbake.conf | 10 +- meta/conf/distro/debian-bullseye.conf | 1 + meta/lib/aptsrc_fetcher.py | 46 +- .../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 22 +- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbom-chroot/sbom-chroot.bb | 2 + .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/deb-repo | 689 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 7 + 27 files changed, 1047 insertions(+), 29 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/deb-repo