add support for rebuilding essential packages

There are cases when we need to rebuild a package that is installed during
bootstrap (we call those "essential package" here). This patch introduces
`ISAR_REBUILD_ESSENTIAL_PKGS` which can be set in (distro/layer) config, and
lists all essential packages.

During build, Isar ensures that essential packages are built before any others,
so that their locally built versions are available in isar-apt, and will
be used in any subsequent package builds.

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>

---
This has interdependencies with the proposed multiarch feature,
and builds on v2 of that series.

Adriaan
---
 meta/classes/dpkg-base.bbclass |  1 +
 meta/classes/essential.bbclass | 39 ++++++++++++++++++++++++++++++++++
 meta/classes/image.bbclass     |  1 +
 3 files changed, 41 insertions(+)
 create mode 100644 meta/classes/essential.bbclass

On 20.02.23 11:32, Adriaan Schmidt wrote:
> There are cases when we need to rebuild a package that is installed during
> bootstrap (we call those "essential package" here). This patch introduces
> `ISAR_REBUILD_ESSENTIAL_PKGS` which can be set in (distro/layer) config, and
> lists all essential packages.
> 
> During build, Isar ensures that essential packages are built before any others,
> so that their locally built versions are available in isar-apt, and will
> be used in any subsequent package builds.
> 
> Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> 
> ---
> This has interdependencies with the proposed multiarch feature,
> and builds on v2 of that series.
> 
> Adriaan
> ---
>  meta/classes/dpkg-base.bbclass |  1 +
>  meta/classes/essential.bbclass | 39 ++++++++++++++++++++++++++++++++++
>  meta/classes/image.bbclass     |  1 +
>  3 files changed, 41 insertions(+)
>  create mode 100644 meta/classes/essential.bbclass
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 55cc6655..ce301346 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -10,6 +10,7 @@ inherit debianize
>  inherit terminal
>  inherit repository
>  inherit deb-dl-dir
> +inherit essential
>  
>  DEPENDS ?= ""
>  RPROVIDES ?= "${PROVIDES}"
> diff --git a/meta/classes/essential.bbclass b/meta/classes/essential.bbclass
> new file mode 100644
> index 00000000..cb444674
> --- /dev/null
> +++ b/meta/classes/essential.bbclass
> @@ -0,0 +1,39 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2023 Siemens AG
> +
> +ISAR_REBUILD_ESSENTIAL_PKGS ?= ""
> +
> +python() {
> +    isar_rebuild_essential_pkgs = (d.getVar('ISAR_REBUILD_ESSENTIAL_PKGS', True) or '').split()
> +    build_compat = d.getVar('ISAR_ENABLE_COMPAT_ARCH', True) == "1"
> +    build_native = not d.getVar('DISTRO_ARCH', True) == d.getVar('HOST_ARCH')
> +
> +    # construct list of essential packages that should be rebuilt:
> +    # if we can't build compat, don't include any -compat packages
> +    # if we don't need native (because DISTRO_ARCH == HOST_ARCH), don't build native
> +    # otherwise, automatically include compat/native when we can build them
> +    essential_packages = []
> +    for p in isar_rebuild_essential_pkgs:
> +        if p.endswith('-compat') and build_compat:
> +            essential_packages.append(p)
> +        elif p.endswith('-native') and build_native:
> +            essential_packages.append(p)
> +        else:
> +            essential_packages.append(p)
> +            if build_compat:
> +                essential_packages.append(f'{p}-compat')
> +            if build_native:
> +                essential_packages.append(f'{p}-native')
> +
> +    # bail out if this recipe is in the essential list
> +    if d.getVar('PN') in essential_packages:
> +        return
> +
> +    # add dependencies to all packages from the essential list
> +    for p in essential_packages:
> +        if d.getVar('do_prepare_build'):
> +            d.appendVarFlag('do_prepare_build', 'depends', f' {p}:do_deploy_deb')
> +        if d.getVar('do_install_rootfs'):
> +            d.appendVarFlag('do_install_rootfs', 'depends', f' {p}:do_deploy_deb')
> +}
> +

.git/rebase-apply/patch:69: new blank line at EOF.

> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index ce7c549c..550785ea 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -80,6 +80,7 @@ image_do_mounts() {
>  }
>  
>  inherit multiarch
> +inherit essential
>  
>  ROOTFSDIR = "${IMAGE_ROOTFS}"
>  ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache"

Under test...

Jan

On Monday, February 20, 2023 at 11:32:22 AM UTC+1 Adriaan Schmidt wrote:

There are cases when we need to rebuild a package that is installed during 
bootstrap (we call those "essential package" here). This patch introduces 
`ISAR_REBUILD_ESSENTIAL_PKGS` which can be set in (distro/layer) config, 
and 
lists all essential packages. 

During build, Isar ensures that essential packages are built before any 
others, 
so that their locally built versions are available in isar-apt, and will 
be used in any subsequent package builds. 

Signed-off-by: Adriaan Schmidt <adriaan...@siemens.com> 

--- 
This has interdependencies with the proposed multiarch feature, 
and builds on v2 of that series. 

Adriaan 
--- 
meta/classes/dpkg-base.bbclass | 1 + 
meta/classes/essential.bbclass | 39 ++++++++++++++++++++++++++++++++++ 
meta/classes/image.bbclass | 1 + 
3 files changed, 41 insertions(+) 
create mode 100644 meta/classes/essential.bbclass 

diff --git a/meta/classes/dpkg-base.bbclass 
b/meta/classes/dpkg-base.bbclass 
index 55cc6655..ce301346 100644 
--- a/meta/classes/dpkg-base.bbclass 
+++ b/meta/classes/dpkg-base.bbclass 
@@ -10,6 +10,7 @@ inherit debianize 
inherit terminal 
inherit repository 
inherit deb-dl-dir 
+inherit essential 

DEPENDS ?= "" 
RPROVIDES ?= "${PROVIDES}" 
diff --git a/meta/classes/essential.bbclass 
b/meta/classes/essential.bbclass 
new file mode 100644 
index 00000000..cb444674 
--- /dev/null 
+++ b/meta/classes/essential.bbclass 
@@ -0,0 +1,39 @@ 
+# This software is a part of ISAR. 
+# Copyright (C) 2023 Siemens AG 
+ 
+ISAR_REBUILD_ESSENTIAL_PKGS ?= "" 
+ 
+python() { 
+ isar_rebuild_essential_pkgs = (d.getVar('ISAR_REBUILD_ESSENTIAL_PKGS', 
True) or '').split()


",True" should not be necessary
here's getVar()'s signature:

def getVar(self, var, expand=True, noweakdefault=False, parsing=False)
 


+ build_compat = d.getVar('ISAR_ENABLE_COMPAT_ARCH', True) == "1" 
+ build_native = not d.getVar('DISTRO_ARCH', True) == d.getVar('HOST_ARCH') 
+ 
+ # construct list of essential packages that should be rebuilt: 
+ # if we can't build compat, don't include any -compat packages 
+ # if we don't need native (because DISTRO_ARCH == HOST_ARCH), don't build 
native 
+ # otherwise, automatically include compat/native when we can build them 
+ essential_packages = [] 
+ for p in isar_rebuild_essential_pkgs: 
+ if p.endswith('-compat') and build_compat: 
+ essential_packages.append(p) 
+ elif p.endswith('-native') and build_native: 
+ essential_packages.append(p) 
+ else: 
+ essential_packages.append(p) 
+ if build_compat: 
+ essential_packages.append(f'{p}-compat') 
+ if build_native: 
+ essential_packages.append(f'{p}-native') 
+ 
+ # bail out if this recipe is in the essential list 
+ if d.getVar('PN') in essential_packages: 
+ return 
+ 
+ # add dependencies to all packages from the essential list 
+ for p in essential_packages: 
+ if d.getVar('do_prepare_build'): 
+ d.appendVarFlag('do_prepare_build', 'depends', f' {p}:do_deploy_deb') 
+ if d.getVar('do_install_rootfs'): 
+ d.appendVarFlag('do_install_rootfs', 'depends', f' {p}:do_deploy_deb') 
+} 
+ 
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass 
index ce7c549c..550785ea 100644 
--- a/meta/classes/image.bbclass 
+++ b/meta/classes/image.bbclass 
@@ -80,6 +80,7 @@ image_do_mounts() { 
} 

inherit multiarch 
+inherit essential 

ROOTFSDIR = "${IMAGE_ROOTFS}" 
ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest 
export-dpkg-status clean-log-files clean-debconf-cache"

On 28.02.23 08:06, Cedric Hombourger wrote:
> 
> 
> On Monday, February 20, 2023 at 11:32:22 AM UTC+1 Adriaan Schmidt wrote:
> 
>     There are cases when we need to rebuild a package that is installed
>     during
>     bootstrap (we call those "essential package" here). This patch
>     introduces
>     `ISAR_REBUILD_ESSENTIAL_PKGS` which can be set in (distro/layer)
>     config, and
>     lists all essential packages.
> 
>     During build, Isar ensures that essential packages are built before
>     any others,
>     so that their locally built versions are available in isar-apt, and
>     will
>     be used in any subsequent package builds.
> 
>     Signed-off-by: Adriaan Schmidt <adriaan...@siemens.com>
> 
>     ---
>     This has interdependencies with the proposed multiarch feature,
>     and builds on v2 of that series.
> 
>     Adriaan
>     ---
>     meta/classes/dpkg-base.bbclass | 1 +
>     meta/classes/essential.bbclass | 39 ++++++++++++++++++++++++++++++++++
>     meta/classes/image.bbclass | 1 +
>     3 files changed, 41 insertions(+)
>     create mode 100644 meta/classes/essential.bbclass
> 
>     diff --git a/meta/classes/dpkg-base.bbclass
>     b/meta/classes/dpkg-base.bbclass
>     index 55cc6655..ce301346 100644
>     --- a/meta/classes/dpkg-base.bbclass
>     +++ b/meta/classes/dpkg-base.bbclass
>     @@ -10,6 +10,7 @@ inherit debianize
>     inherit terminal
>     inherit repository
>     inherit deb-dl-dir
>     +inherit essential
> 
>     DEPENDS ?= ""
>     RPROVIDES ?= "${PROVIDES}"
>     diff --git a/meta/classes/essential.bbclass
>     b/meta/classes/essential.bbclass
>     new file mode 100644
>     index 00000000..cb444674
>     --- /dev/null
>     +++ b/meta/classes/essential.bbclass
>     @@ -0,0 +1,39 @@
>     +# This software is a part of ISAR.
>     +# Copyright (C) 2023 Siemens AG
>     +
>     +ISAR_REBUILD_ESSENTIAL_PKGS ?= ""
>     +
>     +python() {
>     + isar_rebuild_essential_pkgs =
>     (d.getVar('ISAR_REBUILD_ESSENTIAL_PKGS', True) or '').split()
> 
> 
> ",True" should not be necessary
> here's getVar()'s signature:
> 
> def getVar(self, var, expand=True, noweakdefault=False, parsing=False)
>  

Yep, we still see this pattern popping up.

I fact, we have 85(!) left-overs of this legacy pattern in Isar. Let's
we write a quick patch...

Jan

In the email from Monday, 20 February 2023 13:32:14 +03 user Adriaan Schmidt wrote:
> There are cases when we need to rebuild a package that is installed during
> bootstrap (we call those "essential package" here). This patch introduces
> `ISAR_REBUILD_ESSENTIAL_PKGS` which can be set in (distro/layer) config, and
> lists all essential packages.
> 
> During build, Isar ensures that essential packages are built before any others,
> so that their locally built versions are available in isar-apt, and will
> be used in any subsequent package builds.
> 
> Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> 
> ---
> This has interdependencies with the proposed multiarch feature,
> and builds on v2 of that series.
> 
> Adriaan
> ---
>  meta/classes/dpkg-base.bbclass |  1 +
>  meta/classes/essential.bbclass | 39 ++++++++++++++++++++++++++++++++++
>  meta/classes/image.bbclass     |  1 +
>  3 files changed, 41 insertions(+)
>  create mode 100644 meta/classes/essential.bbclass
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 55cc6655..ce301346 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -10,6 +10,7 @@ inherit debianize
>  inherit terminal
>  inherit repository
>  inherit deb-dl-dir
> +inherit essential
>  
>  DEPENDS ?= ""
>  RPROVIDES ?= "${PROVIDES}"
> diff --git a/meta/classes/essential.bbclass b/meta/classes/essential.bbclass
> new file mode 100644
> index 00000000..cb444674
> --- /dev/null
> +++ b/meta/classes/essential.bbclass
> @@ -0,0 +1,39 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2023 Siemens AG
> +
> +ISAR_REBUILD_ESSENTIAL_PKGS ?= ""
> +
> +python() {
> +    isar_rebuild_essential_pkgs = (d.getVar('ISAR_REBUILD_ESSENTIAL_PKGS', True) or '').split()
> +    build_compat = d.getVar('ISAR_ENABLE_COMPAT_ARCH', True) == "1"
> +    build_native = not d.getVar('DISTRO_ARCH', True) == d.getVar('HOST_ARCH')
> +
> +    # construct list of essential packages that should be rebuilt:
> +    # if we can't build compat, don't include any -compat packages
> +    # if we don't need native (because DISTRO_ARCH == HOST_ARCH), don't build native
> +    # otherwise, automatically include compat/native when we can build them
> +    essential_packages = []
> +    for p in isar_rebuild_essential_pkgs:
> +        if p.endswith('-compat') and build_compat:
> +            essential_packages.append(p)
> +        elif p.endswith('-native') and build_native:
> +            essential_packages.append(p)
> +        else:
> +            essential_packages.append(p)
> +            if build_compat:
> +                essential_packages.append(f'{p}-compat')
> +            if build_native:
> +                essential_packages.append(f'{p}-native')
> +
> +    # bail out if this recipe is in the essential list
> +    if d.getVar('PN') in essential_packages:
> +        return
> +
> +    # add dependencies to all packages from the essential list
> +    for p in essential_packages:
> +        if d.getVar('do_prepare_build'):
> +            d.appendVarFlag('do_prepare_build', 'depends', f' {p}:do_deploy_deb')
> +        if d.getVar('do_install_rootfs'):
> +            d.appendVarFlag('do_install_rootfs', 'depends', f' {p}:do_deploy_deb')
> +}
> +
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index ce7c549c..550785ea 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -80,6 +80,7 @@ image_do_mounts() {
>  }
>  
>  inherit multiarch
> +inherit essential
>  
>  ROOTFSDIR = "${IMAGE_ROOTFS}"
>  ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache"
> 

Applied (with removing redundant expand=True) to next, thanks.