| Message ID | 20240725151006.2129-1-ubely@ilbers.de |
|---|---|
| Headers | show |
| Series | Improving base-apt usage | expand |
On Thu, 2024-07-25 at 18:07 +0300, Uladzimir Bely wrote: > `base-apt` is a local apt repository containing all upstream (Debian, > Raspberry Pi OS, Ubuntu...) packages needed for a particular build. > This series implements upfront repository downloading. This is the > first step towards local partial mirror management. > > The current approach in `next`: > > - On the first build, debootstrap and sbuild are used for building > Isar artifacts. The packages downloaded from the Internet are cached > in local directories. > - On the next build: > - Analyze the logs from the previous build, save packages > downloaded by the bootstraps, sbuilds and imagers into `base-apt`. > - Use `base-apt` for bootstrapping, building and image creation. > > Some issues with the current approach: > > 1. Different policies must be followed for the first and the > subsequent builds. > 2. As we have multiple versions of the same package from the main and > security repositories and rely on build logs and `find` for > populating `base-apt`, extra care must be taken to ensure that the > right package version lands in `base-apt`. > 3. We rely on internal implementation of `debootstrap` and `sbuild` > for saving and reusing the packages. Changing to e.g. `mmdebstrap` > breaks the unrelated `base-apt` functionality. Hi, what is the overall plan w.r.t. this patch series? My understanding was that we want to switch to mmdebstrap rather sooner than later. But now you write that this will be incompatible with mmdebstrap. For me, this series still looks pretty experimental. Felix > 4. Source packages are stored in a different flat directory, `apt-get > source` for upstream packages is not possible. > 5. At the moment of `base-apt` creation all we have is the package > name. The knowledge about the upstream repositories is lost and no > local repository policy implementation is possible (e.g., for the > "multiple products, multiple distros" use case). > 6. For implementing further use cases like "fetch all sources > necessary for bootstrapping the base system itself", additional logic > is necessary. >
On Thu, 2024-08-01 at 08:57 +0000, MOESSBAUER, Felix wrote: > On Thu, 2024-07-25 at 18:07 +0300, Uladzimir Bely wrote: > > `base-apt` is a local apt repository containing all upstream > > (Debian, > > Raspberry Pi OS, Ubuntu...) packages needed for a particular build. > > This series implements upfront repository downloading. This is the > > first step towards local partial mirror management. > > > > The current approach in `next`: > > > > - On the first build, debootstrap and sbuild are used for building > > Isar artifacts. The packages downloaded from the Internet are > > cached > > in local directories. > > - On the next build: > > - Analyze the logs from the previous build, save packages > > downloaded by the bootstraps, sbuilds and imagers into `base-apt`. > > - Use `base-apt` for bootstrapping, building and image creation. > > > > Some issues with the current approach: > > > > 1. Different policies must be followed for the first and the > > subsequent builds. > > 2. As we have multiple versions of the same package from the main > > and > > security repositories and rely on build logs and `find` for > > populating `base-apt`, extra care must be taken to ensure that the > > right package version lands in `base-apt`. > > 3. We rely on internal implementation of `debootstrap` and `sbuild` > > for saving and reusing the packages. Changing to e.g. `mmdebstrap` > > breaks the unrelated `base-apt` functionality. > > Hi, what is the overall plan w.r.t. this patch series? My > understanding > was that we want to switch to mmdebstrap rather sooner than later. > But > now you write that this will be incompatible with mmdebstrap. > Hello. "Rely on internal implementation" means that we currently workaround some things related to deb-dl-import / deb-dl-export : - sbuild: 2-stage deb cache import/export: (downloads/deb => import to workdir/rootfs => symlinking to apt cache in sbuild chroot => build => export to workdir/rootfs => export to downloads/deb): - mmdebstrap: since it is not possible to mmdebstrap into non-empty dir, we can't simply use deb-dl-import before mmdebstrap called and have to use some hooks to workaround it. With new base-apt approach, deb-dl import/export (and all related workarounds) becomes redundant. When new approach becomes stable enough we could completely remove fallback mode with deb-dl import/export functionality. > For me, this series still looks pretty experimental. That's why currently we saved fallback mode when things should work like before (ISAR_PREFETCH_BASE_APT = "0") > > Felix > > > 4. Source packages are stored in a different flat directory, `apt- > > get > > source` for upstream packages is not possible. > > 5. At the moment of `base-apt` creation all we have is the package > > name. The knowledge about the upstream repositories is lost and no > > local repository policy implementation is possible (e.g., for the > > "multiple products, multiple distros" use case). > > 6. For implementing further use cases like "fetch all sources > > necessary for bootstrapping the base system itself", additional > > logic > > is necessary. > > > > -- > Siemens AG, Technology > Linux Expert Center > >