mbox series

[RFC,0/2] Buildchroot non-determinism

Message ID 20211129145241.2932943-1-Vijaikumar_Kanagarajan@mentor.com
Headers show
Series Buildchroot non-determinism | expand

Message

Vijai Kumar K Nov. 29, 2021, 4:52 a.m. UTC 
Hello All,

Sometimes buildchroot-host completes long before isar-bootstrap-target
and proceeds with building packages and populates it in isar-apt.

By the time buildchroot-target is triggered, some packages are available
via isar-apt and as per the default preference[1], is installed instead
of the one from other mirrors.

This is not always the case, sometimes there is no such race and the
packages from DISTRO_APT_SOURCES mirrors are installed.

This makes buildchroot non-deterministic and affects caching.

One solution is to provide a mechanism to set preferences for
buildchroot. This way, the user would have flexibility to control the
mirrors from which certain packages are installed.

This RFC introduces 2 new variables
1. BUILDCHROOT_HOST_APT_PREFERENCES
2. BUILDCHROOT_TARGET_APT_PREFERENCES
to set the preferences for buildchroot-host/target respectively.

Ofcourse, there might be more ways to solve this problem. Happy to
discuss them here.

Thanks,
Vijai Kumar K

[1]: https://github.com/ilbers/isar/blob/919fc995fc8ecb675f8bf639ee8628f90728b1ea/meta/classes/rootfs.bbclass#L90

Vijai Kumar K (2):
  meta: Move aggregate_files to base class
  buildchroot: Add provision to specify apt preferences

 meta/classes/base.bbclass                     | 10 +++++++
 .../isar-bootstrap/isar-bootstrap.inc         | 10 -------
 .../buildchroot/buildchroot.inc               | 30 +++++++++++++++++++
 3 files changed, 40 insertions(+), 10 deletions(-)

Comments

Jan Kiszka Nov. 30, 2021, 6:45 a.m. UTC | #1 
On 29.11.21 15:52, Vijai Kumar K wrote:
> Hello All,
> 
> Sometimes buildchroot-host completes long before isar-bootstrap-target
> and proceeds with building packages and populates it in isar-apt.
> 
> By the time buildchroot-target is triggered, some packages are available
> via isar-apt and as per the default preference[1], is installed instead
> of the one from other mirrors.
> 
> This is not always the case, sometimes there is no such race and the
> packages from DISTRO_APT_SOURCES mirrors are installed.
> 
> This makes buildchroot non-deterministic and affects caching.

Just to make the scenario even clearer: This is about packages needed
for the buildchroot-target which happen to be rebuilt by the
buildchroot-host (the other way around is not possible), practically
essential packages that - for whatever reason - requires a rebuild.

> 
> One solution is to provide a mechanism to set preferences for
> buildchroot. This way, the user would have flexibility to control the
> mirrors from which certain packages are installed.
> 
> This RFC introduces 2 new variables
> 1. BUILDCHROOT_HOST_APT_PREFERENCES
> 2. BUILDCHROOT_TARGET_APT_PREFERENCES
> to set the preferences for buildchroot-host/target respectively.

What is the difference to HOST_DISTRO_APT_PREFERENCES vs.
DISTRO_APT_PREFERENCES? That must be made clear I think.

Jan

> 
> Ofcourse, there might be more ways to solve this problem. Happy to
> discuss them here.
> 
> Thanks,
> Vijai Kumar K
> 
> [1]: https://github.com/ilbers/isar/blob/919fc995fc8ecb675f8bf639ee8628f90728b1ea/meta/classes/rootfs.bbclass#L90
> 
> Vijai Kumar K (2):
>   meta: Move aggregate_files to base class
>   buildchroot: Add provision to specify apt preferences
> 
>  meta/classes/base.bbclass                     | 10 +++++++
>  .../isar-bootstrap/isar-bootstrap.inc         | 10 -------
>  .../buildchroot/buildchroot.inc               | 30 +++++++++++++++++++
>  3 files changed, 40 insertions(+), 10 deletions(-)
>