From patchwork Thu Aug 20 13:29:15 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aliaksei Karpovich X-Patchwork-Id: 408 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 20 Aug 2026 15:41:45 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f62.google.com (mail-wm1-f62.google.com [209.85.128.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 67KDfitL013534 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Aug 2026 15:41:45 +0200 Received: by mail-wm1-f62.google.com with SMTP id 5b1f17b1804b1-4955fd77c18sf16215845e9.2 for ; Thu, 20 Aug 2026 06:41:44 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1787233299; cv=pass; d=google.com; s=arc-20260327; b=foHuob+nS6rM5WqAt1Bp092tFJN7Zno60DDxJuYyvAMjw8gLMf4E1EQYD58QNPPcpY XMwsdiVClJKnK7wFn5aRc1VdI4LWdrvzYCoexkq9jM/C7bgo7ZKGiiAQD06O61rzm46w cbEcBYLC71WfJGSCSyQUczkO8yW67BMA/+zcqNs9CgYsMsCUZFkBcGx+9kibFdVJG0D6 Y2a7OK4v8EeVN0nmLuf4WG7u3rObgfMxQZwF9kTpde2zL4GyvEiHjtyxZ2AXXqtb/Zlo bnFaoSCBz5mpBz7wjrqEB+lrFBuc6riOFOkNP8POyHNX/+UeNdZbjxFtEuFVhxw+r9fD X5qg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=HmF0SpahJEi+WSLJCmVTOkMU+w+cYeyuG3ol4Puw/HA=; fh=GUW4oKPWvMbX4XrWvtdCwVkliA2CSJ9PwKTh1Kjhwzg=; b=JeyObPf5CxW/ol3K693V1uRERHSEFM+cPKexdLaL56c/kopUOXaWAsyg0bc8FKAlUK VnBCFxRn8HioSQV4b1cITwSf+xW2o0ledHlroqTcOi1CZ2kBg8Lg0NsmAzIuIsO/x9s0 3H4NQlZO7FqOyRzdxgCY2fGAZ37jRiZbXXMmG3to88XfWJNlQjonwRzuwQ7wbrL6J1vN jzzR35pnWR9oOcjViGjy/xmDf/fj87QEl2p9swPXvohDABHVFc+LNKuK6llT8L/LvYhg bgg/LJJs6j3jroMltxN7xz+b5QxFYgdiAwfoI1eP/yyl6qb6LswZTD5CWJUt5iAZP5j3 h31A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1787233299; x=1787838099; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=HmF0SpahJEi+WSLJCmVTOkMU+w+cYeyuG3ol4Puw/HA=; b=oe6FiUeqABZ43ZFlmPL1KJZrXLlwg0Weu7i+rOViNAvTYOCASuuI2cXlHndjLN/o+b se8spr32G2MO66xVlncc7WnaxHV+suO6JB/aop4bUjw5BpDM0lWHXhp9blyZ3GGpBit9 J5Ln9dicD0gu7p7M8nUfL/HzQF0fgyc+CN78X1GzjIfB/sfavOpCJlFMF+ylaB6n2Qxl TeOMAer1QR/mJAIc/T5RYf1jbbhnog9EvXFSSHLZ/ziDmRq6QoJlvC+1gtFpjMw2iZY7 ihr70EMPmJJro7bd6Q7SfHFP/NRkwh8WzaO/TlXZwYzA40hR73zjjJnOVmu5G8aTh+T2 3zTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787233299; x=1787838099; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to:content-type; bh=HmF0SpahJEi+WSLJCmVTOkMU+w+cYeyuG3ol4Puw/HA=; b=qpEMfRETTizKsbRxbXpRltms6W8VxHX/UbUGDrhsRk704NcGZ73yk/td/gnq5pS6hZ gKNS708iFxqfmqzhLS2guv9x5yBBPCSTaabDLpBAwS+KRKQy6nqcy/MuwOw93UicBKKy BJhDcMSoS2HoesS8isV/Fwk7X1g+ySEuH8kSXL9LQjU6MQLBwSVVWBQF1dpssmm/cjAT kyXSFvodjk9dH14xETbBinvHP0pPJkGZLCV4RHriyPiMrDd/kgAIVcXgr9UeFff9fa+L 32WEBe8/hX6xRHB4E5VqaogpgSmJsA7+7awgZdwQhFHYZyGj4LFX1wVUp4/GkYtGHsXd CdMQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AHgh+RpKrrAZcmaFUCw3A1w+rLNBCRtUXMhXdIREvFfDuUBvIhdwz8DDi2Rjfn4YxsanZvVBbFAovso=@isar-build.org X-Gm-Message-State: AOJu0YxoUg7UdFgNOS+0o2jua2Ii8ufAFkgxJdXt3j/ntUb+9Ax+FFNy iq1Y1QpX4d84bsjejq1Ed8w05uTGg+yrUT+64S02QedfE0YRVssTbXGb X-Received: by 2002:a05:600c:4ec9:b0:495:4d5c:903e with SMTP id 5b1f17b1804b1-499aa167256mr212973395e9.7.1787233299124; Thu, 20 Aug 2026 06:41:39 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="ATskLdec/GDpKxEIwaCV5pHoAcjYf0GpXG5fnEhyZnh7aSSuWA==" Received: by 2002:a05:600c:1391:b0:495:55df:a958 with SMTP id 5b1f17b1804b1-499b059e1a3ls10008345e9.1.-pod-prod-01-eu; Thu, 20 Aug 2026 06:41:36 -0700 (PDT) X-Received: by 2002:a05:600c:3487:b0:499:53c4:1daa with SMTP id 5b1f17b1804b1-499aa1c2764mr248077945e9.15.1787233296076; Thu, 20 Aug 2026 06:41:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1787233296; cv=none; d=google.com; s=arc-20260327; b=fjmZcU5yOP+BLf7S13/rtheeAbDFu2PRj3w/Li6ggEZrUN16axjce7APCjJfVArqXq Ybd9Te7KobUpUpHSZVhz3BZ/ItVDI/eE1enOxDQ7GGPqvKALow8enJaJyJQuAnISzOp5 3BgnURHHUe1gAZ7YMPP1WEh60TVIBo3Hlfox2ok/5ilFGH0HVEzqkBiB+LPEYO1req8L pTrRIs/j3GNwb4o8HPWJA1oNbucY5ofJjiShXk3cHjRI9WwKbK4dXIDcH7YyqsKeuinK is56Cz2lsWAtNp9Cqyym6YcM90mP6yOEFM9khrdqwwBfZuPWnKkjZqV/Dv+gpflFgBJv 5v9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=8iQpt/ZidIqc4IPbicA5T4j9NpgTpOdF5o+sPrNguPM=; fh=u+3kXEMzY4uVhDBiujW2QUo7NnpMLrGEgMQBYpCTRN4=; b=Zw2F0y/Ddsi/oCup86C3AWMlwKsQjcyG/GnEbmDKrtoZN7iDRE/o/Lvrh+1vZGGgiM OBEPK5xhDFeZtjWRM+Q44AXbhG+HMBGSDy4V+4qL72DDG/XR7UXqw4Z41kn+Mmy2M2Rc VKVYY+ZW4XtD2ReBrWKdj4mZdCsAkH9FySDNSp9kCvrWOLAyqKAHtPRdssUyZACVmZ9X DrUpwu6R0rIjMiKYFwELhOZj8wFRJCacUuU07wUQXzn4Mhek9n4vqXO8GbCQ3CpTruuM WQNux12da+0SUd83P8petJCADBQKVriswkHrPXEM171j7iIRHIU21WQ6UEpRwpfIHnZI d1Jw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-499b21466d6si350315e9.1.2026.08.20.06.41.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 20 Aug 2026 06:41:36 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from akarpovich-Victus-by-HP-Laptop-16-e0xxx.promwad.corp (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 67KDf8lN013513 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 20 Aug 2026 15:41:31 +0200 From: Aliaksei Karpovich To: isar-users@googlegroups.com Cc: Aliaksei Karpovich Subject: [v9 00/13] Improving base-apt usage Date: Thu, 20 Aug 2026 15:29:15 +0200 Message-ID: <20260820133934.383231-1-akarpovich@ilbers.de> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, mmdebstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving and reusing the packages. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to bootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v8: - Rebased on fresh next (86c761e4) - Removed fixes for isar-apt download in rootfs (use Felix's fix) - Renamed debrepo -> deb-repo - Fixed failed tests on Full and Fast CI testsets Changes since v7: - Rebased on latest next (70661c40) - Fixed url parsing in rootfs_install_pkgs_isar_download() - Added version and arch for downloaded package in rootfs_install_pkgs_isar_download() - Workaround for base-apt to force to install python3-apt amd64 - Added using override for reprepro to fix offline build Aliaksei Karpovich (3): sbom-chroot: Fix python3-apt dependency debrepo: Add override for reprepro base-apt: Move bullseye-mipsel in separate dir Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to bootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 9 + doc/user_manual.md | 1 + kas/opt/Kconfig | 12 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes-recipe/bootstrap.bbclass | 50 +- meta/classes-recipe/crossvars.bbclass | 1 + meta/classes-recipe/deb-dl-dir.bbclass | 43 ++ meta/classes-recipe/dpkg-base.bbclass | 1 + meta/classes-recipe/dpkg.bbclass | 8 + .../image-locales-extension.bbclass | 5 + .../image-tools-extension.bbclass | 13 + meta/classes-recipe/rootfs.bbclass | 15 +- meta/classes-recipe/sbuild.bbclass | 3 +- meta/classes/debrepo.bbclass | 89 +++ meta/conf/bitbake.conf | 10 +- meta/conf/distro/debian-bullseye.conf | 1 + meta/lib/aptsrc_fetcher.py | 46 +- .../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 22 +- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbom-chroot/sbom-chroot.bb | 2 +- .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/deb-repo | 689 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 7 + 27 files changed, 1046 insertions(+), 30 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/deb-repo