mbox series

[v5,00/12] Improving base-apt usage PoC

Message ID 20230526070027.16890-1-ubely@ilbers.de
Headers show
Series Improving base-apt usage PoC | expand

Message

Uladzimir Bely May 26, 2023, 7 a.m. UTC 
Currently, base-apt is used in the following way:

 - After the first build, all .deb files that took part in the process
are is cached in the ${DL_DIR}/deb/. All the packages are downloaded
from the remote repositories.

 - At the second build, `base-apt` repo is first created from the
previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set).
Futher debootstrap and installing packages are done from this local repo

The idea of this patchset is using local `base-apt` repo even at
the first build. Before any build step that requires additional
packages, these packages are predownloaded with `debrepo` script to
the `base-apt` repo and the debootstrapping / installation are always
done from it.

Potentialy, such an approach allows to stop using deb-del-dir import
and export functionality in favour of debian-like repositories.

Currently, separate `debrepo` script is used for prefetching packages
to the local 'base-apt' repo. It requires python3-apt to be installed
on the build host. Potentially, the functionality it provides could be
directly integrated to the `debrepo.bbclass`.

Changes since v4:
 - rebased on latest next;
 - rearranged patches since some of them are already in next;
 - added possibility to select between new "prefetch" base-apt mode and
old behaviour when it's populated on 2nd build with packages downloaded
during 1st build. New behaviour is disabled by default, but enabled in
local.conf.example for testing purposes.
 - code passes both full and fast CI in both "old" and "new" modes.

Changes since v3:
 - rebased on latest next;
 - cross-building for raspberry supported;
 - code passes both full and fast CI.

Changes since v2:
 - populate base-apt before using at all steps of native build.

Changes since v1:
 - rebased on latest next;
 - updated patchset description.

Uladzimir Bely (12):
  meta: move base-apt from deploy directory to the top
  ci_build.sh: Install python3-apt if not installed
  scripts: Add debrepo python script handling base-apt
  meta: Add debrepo bbclass handling base-apt prefetching
  meta: Always use base-apt repo in local mode
  meta: Use cached base-apt repo to debootstrap
  meta: Consider global debrepo context
  base-apt: Predownload packages to base-apt before install
  meta: Add cache-deb-src functionality in base-apt mode
  Set ISAR_PREFETCH_BASE_APT by default
  meta: Specify grub-efi packages arch
  Disable deb-dl-dir in base-apt prefetch mode

 RECIPE-API-CHANGELOG.md                       |   9 +
 meta-isar/conf/local.conf.sample              |   4 +
 meta/classes/buildchroot.bbclass              |   5 +-
 meta/classes/deb-dl-dir.bbclass               |  25 +
 meta/classes/debrepo.bbclass                  |  86 ++++
 meta/classes/dpkg-base.bbclass                |  47 +-
 meta/classes/dpkg.bbclass                     |   8 +
 meta/classes/image-locales-extension.bbclass  |   5 +
 meta/classes/image-tools-extension.bbclass    |   9 +
 meta/classes/rootfs.bbclass                   |  13 +-
 meta/conf/bitbake.conf                        |   9 +-
 meta/conf/distro/debian-common.conf           |   8 +-
 .../isar-bootstrap/isar-bootstrap.inc         | 111 ++++-
 meta/recipes-devtools/base-apt/base-apt.bb    |  21 +-
 .../buildchroot/buildchroot.inc               |  11 +
 .../sbuild-chroot/sbuild-chroot.inc           |  11 +
 scripts/ci_build.sh                           |   8 +-
 scripts/debrepo                               | 443 ++++++++++++++++++
 testsuite/citest.py                           |   3 +
 19 files changed, 804 insertions(+), 32 deletions(-)
 create mode 100644 meta/classes/debrepo.bbclass
 create mode 100755 scripts/debrepo