mbox series

[v8,00/14] Improving base-apt usage

Message ID 20260727112812.2255297-1-akarpovich@ilbers.de
Headers show
Series Improving base-apt usage | expand

Message

Aliaksei Karpovich July 27, 2026, 11:26 a.m. UTC 
`base-apt` is a local apt repository containing all upstream (Debian, Raspberry
Pi OS, Ubuntu...) packages needed for a particular build. This series implements
upfront repository downloading. This is the first step towards local partial
mirror management.

The current approach in `next`:

- On the first build, mmdebstrap and sbuild are used for building Isar
artifacts. The packages downloaded from the Internet are cached in local
directories.
- On the next build:
  - Analyze the logs from the previous build, save packages downloaded by the
  bootstraps, sbuilds and imagers into `base-apt`.
  - Use `base-apt` for bootstrapping, building and image creation.

Some issues with the current approach:

1. Different policies must be followed for the first and the subsequent builds.
2. As we have multiple versions of the same package from the main and security
repositories and rely on build logs and `find` for populating `base-apt`, extra
care must be taken to ensure that the right package version lands in `base-apt`.
3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving
and reusing the packages.
4. Source packages are stored in a different flat directory, `apt-get source`
for upstream packages is not possible.
5. At the moment of `base-apt` creation all we have is the package name. The
knowledge about the upstream repositories is lost and no local repository
policy implementation is possible (e.g., for the "multiple products, multiple
distros" use case).
6. For implementing further use cases like "fetch all sources necessary for
bootstrapping the base system itself", additional logic is necessary.

The new approach:

- On the first build:
  - All packages necessary for bootstrapping and building are identified and
  downloaded upfront.
  - `base-apt` is used for bootstrapping, building and image creation.
- On the next build:
  - `base-apt` is used for bootstrapping, building and image creation.

This series addresses issues 1-5 and provides the architecture for
implementing further use cases.

The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to
zero falls back to the old approach.

The implementation uses `debrepo` script which can also be called manually for
pre-fetching packages to the local 'base-apt' repository. It requires
`python3-apt` to be installed on the build host. Some examples of its usage in
standalone mode:

```
# Create local `/build/ba` repository sufficient to bootstrap Debian system with
`armhf` architecture:
debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf

# Add some packages to this repo (e.g., build deps for some recipe):
debrepo --workdir=/build/dr locales gnupg

# Add srcpackages for some package to the repo:
debrepo --workdir=/build/dr --srcmode tzdata
```

Changes since v7:
 - Rebased on latest next (70661c40)
 - Fixed url parsing in rootfs_install_pkgs_isar_download()
 - Added version and arch for downloaded package in
 rootfs_install_pkgs_isar_download()
 - Workaround for base-apt to force to install python3-apt amd64
 - Added using override for reprepro to fix offline build

Known issues:
 - Didn't pass the testsuite CI - 50% of fast tests are failed

Aliaksei Karpovich (4):
  rootfs: Fix URL parsing
  rootfs: Add version and arch for downloaded package
  sbom-chroot: Fix python3-apt dependency
  debrepo: Add using override for reprepro

Uladzimir Bely (10):
  scripts: Add debrepo python script handling base-apt
  meta: Add debrepo bbclass handling base-apt prefetching
  meta: Always use base-apt repo in local mode
  meta: Use cached base-apt repo to bootstrap
  base-apt: Predownload packages to base-apt before install
  meta: Add cache-deb-src functionality in base-apt mode
  testsuite: Set ISAR_PREFETCH_BASE_APT by default
  Disable deb-dl-dir in base-apt prefetch mode
  kas: Add PREFETCH_BASE_APT config entry
  ci_build.sh: Install python3-apt if not installed

 RECIPE-API-CHANGELOG.md                       |  10 +
 doc/user_manual.md                            |   1 +
 kas/opt/Kconfig                               |  12 +
 kas/opt/prefetch-base-apt.yaml                |   9 +
 meta-test/conf/local.conf.sample              |   3 +
 meta/classes-recipe/bootstrap.bbclass         |  50 +-
 meta/classes-recipe/crossvars.bbclass         |   1 +
 meta/classes-recipe/deb-dl-dir.bbclass        |  47 ++
 meta/classes-recipe/dpkg-base.bbclass         |   1 +
 meta/classes-recipe/dpkg.bbclass              |   8 +
 .../image-locales-extension.bbclass           |   5 +
 .../image-tools-extension.bbclass             |  13 +
 meta/classes-recipe/rootfs.bbclass            |  21 +-
 meta/classes-recipe/sbuild.bbclass            |   3 +-
 meta/classes/debrepo.bbclass                  |  90 +++
 meta/conf/bitbake.conf                        |   5 +
 meta/lib/aptsrc_fetcher.py                    |  22 +
 .../isar-mmdebstrap/isar-mmdebstrap-host.bb   |   2 +
 .../isar-mmdebstrap/isar-mmdebstrap.inc       |  10 +-
 meta/recipes-devtools/base-apt/base-apt.bb    |  21 +-
 .../sbom-chroot/sbom-chroot.bb                |   2 +-
 .../sbuild-chroot/sbuild-chroot-host.bb       |   2 +
 scripts/ci_build.sh                           |   8 +-
 scripts/debrepo                               | 659 ++++++++++++++++++
 testsuite/cibase.py                           |   4 +
 testsuite/cibuilder.py                        |   7 +
 26 files changed, 998 insertions(+), 18 deletions(-)
 create mode 100644 kas/opt/prefetch-base-apt.yaml
 create mode 100644 meta/classes/debrepo.bbclass
 create mode 100755 scripts/debrepo