isar-mmdebstrap: reduce rootfs size as much as possible

From: srinuvasan <srinuvasan.a@siemens.com>

reduce the rootfs image size by excluding man and doc, this
changes significantly reduce size from (~80MB) to (~64MB).

Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
---
 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 9 +++++++++
 1 file changed, 9 insertions(+)

Le jeudi 9 janvier 2025 à 13:17:57 UTC+1, srinuv...@siemens.com a écrit :

From: srinuvasan <srinuv...@siemens.com> 

reduce the rootfs image size by excluding man and doc, this 
changes significantly reduce size from (~80MB) to (~64MB). 

Signed-off-by: srinuvasan <srinuv...@siemens.com> 
--- 
meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 9 +++++++++ 
1 file changed, 9 insertions(+) 

diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
index 931f6f13..1e2ad40f 100644 
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
@@ -24,6 +24,14 @@ DPKG_HOOKS ?= 
"${@'--hook-dir='+d.getVar('MM_HOOK_JESSIE') \ 
MMHOOKS:focal ?= "${DPKG_HOOKS}" 
MMHOOKS:buster ?= "${DPKG_HOOKS}" 

+CLEANUP_DOCS = "--dpkgopt='path-exclude=/usr/share/man/*' \ 
+ --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \ 
+ --dpkgopt='path-exclude=/usr/share/locale/*' \ 
+ --dpkgopt='path-include=/usr/share/locale/locale.alias' \ 
+ --dpkgopt='path-exclude=/usr/share/doc/*' \ 
+ --dpkgopt='path-include=/usr/share/doc/*/copyright' \ 
+ --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*'" 
+ 


Not.a good idea IMO - some users may want to build images with man pages / 
documentation
At best this could be an image extension. It could take informed decisions: 
if /usr/bin/man isn’t there then remove man pages. Users should be able to 
enable/disable that customisation

DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg" 

def get_distro_have_https_source(d): 
@@ -199,6 +207,7 @@ do_bootstrap() { 
--customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g" "$1/etc/locale.gen"' 
\ 
--customize-hook='chroot "$1" /usr/sbin/locale-gen' \ 
--customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \ 
+ ${CLEANUP_DOCS} \ 
--skip=cleanup/apt \ 
--skip=download/empty \ 
${@get_apt_opts(d, '--aptopt')} \

09/01/2025 14:41, Cedric Hombourger wrote:
>
>
> Le jeudi 9 janvier 2025 à 13:17:57 UTC+1, srinuv...@siemens.com a écrit :
>
>     From: srinuvasan <srinuv...@siemens.com>
>
>     reduce the rootfs image size by excluding man and doc, this
>     changes significantly reduce size from (~80MB) to (~64MB).
>
>     Signed-off-by: srinuvasan <srinuv...@siemens.com>
>     ---
>     meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 9 +++++++++
>     1 file changed, 9 insertions(+)
>
>     diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     index 931f6f13..1e2ad40f 100644
>     --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     @@ -24,6 +24,14 @@ DPKG_HOOKS ?=
>     "${@'--hook-dir='+d.getVar('MM_HOOK_JESSIE') \
>     MMHOOKS:focal ?= "${DPKG_HOOKS}"
>     MMHOOKS:buster ?= "${DPKG_HOOKS}"
>
>     +CLEANUP_DOCS = "--dpkgopt='path-exclude=/usr/share/man/*' \
>     + --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \
>     + --dpkgopt='path-exclude=/usr/share/locale/*' \
>     + --dpkgopt='path-include=/usr/share/locale/locale.alias' \
>     + --dpkgopt='path-exclude=/usr/share/doc/*' \
>     + --dpkgopt='path-include=/usr/share/doc/*/copyright' \
>     + --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*'"
>     +
>
>
> Not.a good idea IMO - some users may want to build images with man 
> pages / documentation
> At best this could be an image extension. It could take informed 
> decisions: if /usr/bin/man isn’t there then remove man pages. Users 
> should be able to enable/disable that customisation
Agree, but maybe we can introduce some kind of global switch (features
implementation from Chris can be helpful here) to enable such kind of hacks.
And of course it should be disabled by default.
>
>     DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"
>
>     def get_distro_have_https_source(d):
>     @@ -199,6 +207,7 @@ do_bootstrap() {
>     --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g"
>     "$1/etc/locale.gen"' \
>     --customize-hook='chroot "$1" /usr/sbin/locale-gen' \
>     --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \
>     + ${CLEANUP_DOCS} \
>     --skip=cleanup/apt \
>     --skip=download/empty \
>     ${@get_apt_opts(d, '--aptopt')} \
>     -- 
>     2.39.5
>

Le jeudi 9 janvier 2025 à 13:58:37 UTC+1, Anton Mikanovich a écrit :

09/01/2025 14:41, Cedric Hombourger wrote: 
> 
> 
> Le jeudi 9 janvier 2025 à 13:17:57 UTC+1, srinuv...@siemens.com a écrit : 
> 
> From: srinuvasan <srinuv...@siemens.com> 
> 
> reduce the rootfs image size by excluding man and doc, this 
> changes significantly reduce size from (~80MB) to (~64MB). 
> 
> Signed-off-by: srinuvasan <srinuv...@siemens.com> 
> --- 
> meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 9 +++++++++ 
> 1 file changed, 9 insertions(+) 
> 
> diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
> b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
> index 931f6f13..1e2ad40f 100644 
> --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
> +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc 
> @@ -24,6 +24,14 @@ DPKG_HOOKS ?= 
> "${@'--hook-dir='+d.getVar('MM_HOOK_JESSIE') \ 
> MMHOOKS:focal ?= "${DPKG_HOOKS}" 
> MMHOOKS:buster ?= "${DPKG_HOOKS}" 
> 
> +CLEANUP_DOCS = "--dpkgopt='path-exclude=/usr/share/man/*' \ 
> + --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \ 
> + --dpkgopt='path-exclude=/usr/share/locale/*' \ 
> + --dpkgopt='path-include=/usr/share/locale/locale.alias' \ 
> + --dpkgopt='path-exclude=/usr/share/doc/*' \ 
> + --dpkgopt='path-include=/usr/share/doc/*/copyright' \ 
> + --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*'" 
> + 
> 
> 
> Not.a good idea IMO - some users may want to build images with man 
> pages / documentation 
> At best this could be an image extension. It could take informed 
> decisions: if /usr/bin/man isn’t there then remove man pages. Users 
> should be able to enable/disable that customisation 
Agree, but maybe we can introduce some kind of global switch (features 
implementation from Chris can be helpful here) to enable such kind of 
hacks. 
And of course it should be disabled by default. 


I would surely vote for Chris’ implementation since it is something he has 
implemented at my request :). Glad to see that we have a use-case for it 
(within this layer itself). Good feedback / suggestions Anton.
 

> 
> DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg" 
> 
> def get_distro_have_https_source(d): 
> @@ -199,6 +207,7 @@ do_bootstrap() { 
> --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g" 
> "$1/etc/locale.gen"' \ 
> --customize-hook='chroot "$1" /usr/sbin/locale-gen' \ 
> --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \ 
> + ${CLEANUP_DOCS} \ 
> --skip=cleanup/apt \ 
> --skip=download/empty \ 
> ${@get_apt_opts(d, '--aptopt')} \ 
> -- 
> 2.39.5 
>

Thanks cedric and anton for the review. Let's wait for Chris's changes and try to use this one as a switch.

Best Regards,
Srinu

-----Original Message-----
From: Anton Mikanovich <amikan@ilbers.de> 
Sent: 09 January 2025 18:29
To: Cedric Hombourger <chombourger@gmail.com>; isar-users <isar-users@googlegroups.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>
Subject: Re: [PATCH] isar-mmdebstrap: reduce rootfs size as much as possible

09/01/2025 14:41, Cedric Hombourger wrote:
>
>
> Le jeudi 9 janvier 2025 à 13:17:57 UTC+1, srinuv...@siemens.com a écrit :
>
>     From: srinuvasan <srinuv...@siemens.com>
>
>     reduce the rootfs image size by excluding man and doc, this
>     changes significantly reduce size from (~80MB) to (~64MB).
>
>     Signed-off-by: srinuvasan <srinuv...@siemens.com>
>     ---
>     meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 9 +++++++++
>     1 file changed, 9 insertions(+)
>
>     diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     index 931f6f13..1e2ad40f 100644
>     --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
>     @@ -24,6 +24,14 @@ DPKG_HOOKS ?=
>     "${@'--hook-dir='+d.getVar('MM_HOOK_JESSIE') \
>     MMHOOKS:focal ?= "${DPKG_HOOKS}"
>     MMHOOKS:buster ?= "${DPKG_HOOKS}"
>
>     +CLEANUP_DOCS = "--dpkgopt='path-exclude=/usr/share/man/*' \
>     + --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \
>     + --dpkgopt='path-exclude=/usr/share/locale/*' \
>     + --dpkgopt='path-include=/usr/share/locale/locale.alias' \
>     + --dpkgopt='path-exclude=/usr/share/doc/*' \
>     + --dpkgopt='path-include=/usr/share/doc/*/copyright' \
>     + --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*'"
>     +
>
>
> Not.a good idea IMO - some users may want to build images with man 
> pages / documentation At best this could be an image extension. It 
> could take informed
> decisions: if /usr/bin/man isn’t there then remove man pages. Users 
> should be able to enable/disable that customisation
Agree, but maybe we can introduce some kind of global switch (features implementation from Chris can be helpful here) to enable such kind of hacks.
And of course it should be disabled by default.
>
>     DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"
>
>     def get_distro_have_https_source(d):
>     @@ -199,6 +207,7 @@ do_bootstrap() {
>     --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g"
>     "$1/etc/locale.gen"' \
>     --customize-hook='chroot "$1" /usr/sbin/locale-gen' \
>     --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \
>     + ${CLEANUP_DOCS} \
>     --skip=cleanup/apt \
>     --skip=download/empty \
>     ${@get_apt_opts(d, '--aptopt')} \
>     -- 
>     2.39.5
>

On 09.01.25 13:17, srinuvasan.a via isar-users wrote:
> From: srinuvasan <srinuvasan.a@siemens.com>
> 
> reduce the rootfs image size by excluding man and doc, this
> changes significantly reduce size from (~80MB) to (~64MB).
> 

What's the (final) difference to isar-exclude-docs?

Jan

I validated the isar-mmdebstrap-target rootfs with and without docs, I could able to see approx  17MB differences.

Note: compared the compressed rootfs (rootfs.tar.zst) file

Many thanks,
Srinu

-----Original Message-----
From: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com> 
Sent: 09 January 2025 20:50
To: Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>; isar-users@googlegroups.com
Cc: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com>; amikan@ilbers.de
Subject: Re: [PATCH] isar-mmdebstrap: reduce rootfs size as much as possible

On 09.01.25 13:17, srinuvasan.a via isar-users wrote:
> From: srinuvasan <srinuvasan.a@siemens.com>
> 
> reduce the rootfs image size by excluding man and doc, this changes 
> significantly reduce size from (~80MB) to (~64MB).
> 

What's the (final) difference to isar-exclude-docs?

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

On 09.01.25 16:27, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote:
> I validated the isar-mmdebstrap-target rootfs with and without docs, I could able to see approx  17MB differences.
> 

That is not my point.

Are you trying to optimize the intermediate artifact sizes or the final
image? The latter should also be addressed by using isar-exclude-docs.
The former would need some stronger motivation, I would say...

Jan

> Note: compared the compressed rootfs (rootfs.tar.zst) file
> 
> Many thanks,
> Srinu
> 
> -----Original Message-----
> From: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com> 
> Sent: 09 January 2025 20:50
> To: Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>; isar-users@googlegroups.com
> Cc: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com>; amikan@ilbers.de
> Subject: Re: [PATCH] isar-mmdebstrap: reduce rootfs size as much as possible
> 
> On 09.01.25 13:17, srinuvasan.a via isar-users wrote:
>> From: srinuvasan <srinuvasan.a@siemens.com>
>>
>> reduce the rootfs image size by excluding man and doc, this changes 
>> significantly reduce size from (~80MB) to (~64MB).
>>
> 
> What's the (final) difference to isar-exclude-docs?
> 
> Jan
> 
> --
> Siemens AG, Foundational Technologies
> Linux Expert Center

Hi Jan,

            I hope this isar-exclude-docs package can be used to exclude the documentation in the image rootfs, but here I introduced this customization for the bootstrap rootfs.

Additionally we are cleaning the locales under /usr/share in the bootstrap rootfs.

With this reduced size of the bootstrap rootfs, we achieve the sbuildchroot-<host/target> size also reduced.

Many thanks,
Srinu
-----Original Message-----
From: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com> 
Sent: 09 January 2025 21:00
To: Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>; isar-users@googlegroups.com
Cc: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com>; amikan@ilbers.de
Subject: Re: [PATCH] isar-mmdebstrap: reduce rootfs size as much as possible

On 09.01.25 16:27, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote:
> I validated the isar-mmdebstrap-target rootfs with and without docs, I could able to see approx  17MB differences.
> 

That is not my point.

Are you trying to optimize the intermediate artifact sizes or the final image? The latter should also be addressed by using isar-exclude-docs.
The former would need some stronger motivation, I would say...

Jan

> Note: compared the compressed rootfs (rootfs.tar.zst) file
> 
> Many thanks,
> Srinu
> 
> -----Original Message-----
> From: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>
> Sent: 09 January 2025 20:50
> To: Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>; 
> isar-users@googlegroups.com
> Cc: Hombourger, Cedric (FT FDS CES LX) 
> <cedric.hombourger@siemens.com>; amikan@ilbers.de
> Subject: Re: [PATCH] isar-mmdebstrap: reduce rootfs size as much as 
> possible
> 
> On 09.01.25 13:17, srinuvasan.a via isar-users wrote:
>> From: srinuvasan <srinuvasan.a@siemens.com>
>>
>> reduce the rootfs image size by excluding man and doc, this changes 
>> significantly reduce size from (~80MB) to (~64MB).
>>
> 
> What's the (final) difference to isar-exclude-docs?
> 
> Jan
> 
> --
> Siemens AG, Foundational Technologies
> Linux Expert Center


--
Siemens AG, Foundational Technologies
Linux Expert Center

On 10.01.25 07:02, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote:
> Hi Jan,
> 
>             I hope this isar-exclude-docs package can be used to exclude the documentation in the image rootfs, but here I introduced this customization for the bootstrap rootfs.
> 
> Additionally we are cleaning the locales under /usr/share in the bootstrap rootfs.
> 
> With this reduced size of the bootstrap rootfs, we achieve the sbuildchroot-<host/target> size also reduced.
> 

Still no reason to not use that existing package, maybe extending it. It
already comes with the configurability that your approach is missing and
cannot introduce that nicely.

Jan

> Many thanks,
> Srinu
> -----Original Message-----
> From: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com> 
> Sent: 09 January 2025 21:00
> To: Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>; isar-users@googlegroups.com
> Cc: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com>; amikan@ilbers.de
> Subject: Re: [PATCH] isar-mmdebstrap: reduce rootfs size as much as possible
> 
> On 09.01.25 16:27, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote:
>> I validated the isar-mmdebstrap-target rootfs with and without docs, I could able to see approx  17MB differences.
>>
> 
> That is not my point.
> 
> Are you trying to optimize the intermediate artifact sizes or the final image? The latter should also be addressed by using isar-exclude-docs.
> The former would need some stronger motivation, I would say...
> 
> Jan
> 
>> Note: compared the compressed rootfs (rootfs.tar.zst) file
>>
>> Many thanks,
>> Srinu
>>
>> -----Original Message-----
>> From: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>
>> Sent: 09 January 2025 20:50
>> To: Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>; 
>> isar-users@googlegroups.com
>> Cc: Hombourger, Cedric (FT FDS CES LX) 
>> <cedric.hombourger@siemens.com>; amikan@ilbers.de
>> Subject: Re: [PATCH] isar-mmdebstrap: reduce rootfs size as much as 
>> possible
>>
>> On 09.01.25 13:17, srinuvasan.a via isar-users wrote:
>>> From: srinuvasan <srinuvasan.a@siemens.com>
>>>
>>> reduce the rootfs image size by excluding man and doc, this changes 
>>> significantly reduce size from (~80MB) to (~64MB).
>>>
>>
>> What's the (final) difference to isar-exclude-docs?
>>
>> Jan
>>
>> --
>> Siemens AG, Foundational Technologies
>> Linux Expert Center
> 
> 
> --
> Siemens AG, Foundational Technologies
> Linux Expert Center

On 10.01.25 07:13, Jan Kiszka wrote:
> On 10.01.25 07:02, Arjunan, Srinu (FT FDS CES LX PBU 2) wrote:
>> Hi Jan,
>>
>>             I hope this isar-exclude-docs package can be used to exclude the documentation in the image rootfs, but here I introduced this customization for the bootstrap rootfs.
>>
>> Additionally we are cleaning the locales under /usr/share in the bootstrap rootfs.
>>
>> With this reduced size of the bootstrap rootfs, we achieve the sbuildchroot-<host/target> size also reduced.
>>
> 
> Still no reason to not use that existing package, maybe extending it. It
> already comes with the configurability that your approach is missing and
> cannot introduce that nicely.
> 

Plus, your approach cleans up before all packages have been install,
isar-exclude-docs does that afterwards. So, yours is architecturally
inferior. I really see no reason to go this way.

Jan

10/01/2025 08:13, Jan Kiszka wrote:
> Still no reason to not use that existing package, maybe extending it. It
> already comes with the configurability that your approach is missing and
> cannot introduce that nicely.
>
> Jan

Hello Jan,

The main reason not to use isar-exclude-docs is that this package 
doesn't exist
during bootstrap rootfs preparations. Moreover to build it we need bootstrap
recipe building already finished.

Le vendredi 10 janvier 2025 à 08:19:23 UTC+1, Anton Mikanovich a écrit :

10/01/2025 08:13, Jan Kiszka wrote: 
> Still no reason to not use that existing package, maybe extending it. It 
> already comes with the configurability that your approach is missing and 
> cannot introduce that nicely. 
> 
> Jan 

Hello Jan, 

The main reason not to use isar-exclude-docs is that this package 
doesn't exist 
during bootstrap rootfs preparations. Moreover to build it we need 
bootstrap 
recipe building already finished. 


Too be  honest, I am not sure why we would want to apply any optimizations 
at the bootstrap phase as needs for images or sbuild environments are not 
known at that stage (eg one may want man pages in the SDK). Am I missing 
something?

On 10.01.25 08:19, Anton Mikanovich wrote:
> 10/01/2025 08:13, Jan Kiszka wrote:
>> Still no reason to not use that existing package, maybe extending it. It
>> already comes with the configurability that your approach is missing and
>> cannot introduce that nicely.
>>
>> Jan
> 
> Hello Jan,
> 
> The main reason not to use isar-exclude-docs is that this package
> doesn't exist
> during bootstrap rootfs preparations. Moreover to build it we need
> bootstrap
> recipe building already finished.
> 

What matters is the final image, not the bootstrapped one. And this is
what the postinst script of isar-exclude-docs addresses. The hook of
this patch only addresses what the essential packages bring in form of
docs & Co., thus fails to achieve its declared goal.

Jan