Message ID | 20230123140016.71522-1-michael.adler@siemens.com |
---|---|
State | Superseded, archived |
Headers | show |
Series | feat: add zstd support to image conversions | expand |
On 23.01.23 15:00, Michael Adler wrote: > This patch enables ISAR to build zstd compressed images. > I was briefly thinking you had solved the dpkg vs. zstd issue - just a dream :) Jan > Signed-off-by: Michael Adler <michael.adler@siemens.com> > --- > meta/classes/imagetypes.bbclass | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass > index 1e8e223d..6950da7a 100644 > --- a/meta/classes/imagetypes.bbclass > +++ b/meta/classes/imagetypes.bbclass > @@ -93,7 +93,7 @@ IMAGE_CMD_ubi() { > IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template" > > # image conversions > -IMAGE_CONVERSIONS = "gz xz" > +IMAGE_CONVERSIONS = "gz xz zst" > > CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" > CONVERSION_DEPS_gz = "gzip" > @@ -105,3 +105,11 @@ XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" > XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" > CONVERSION_DEPS_xz = "xz-utils" > + > +ZST_LEVEL ?= "19" > +ZST_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +ZST_THREADS[vardepvalue] = "1" > +ZST_OPTIONS ?= "-${ZST_LEVEL} --threads=${ZST_THREADS}" > +ZST_OPTIONS[vardepsexclude] += "ZST_LEVEL ZST_THREADS" > +CONVERSION_CMD_zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZST_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" > +CONVERSION_DEPS_zst = "zstd"
Il Lun 23 Gen 2023, 20:22 Jan Kiszka <jan.kiszka@siemens.com> ha scritto: > On 23.01.23 15:00, Michael Adler wrote: > > This patch enables ISAR to build zstd compressed images. > > > > I was briefly thinking you had solved the dpkg vs. zstd issue - just a > dream :) > Hi Jan. I think you are speaking about zstd compression for the debian package that is 1% bigger but 13x times faster in expanding. [1] Ubuntu and Arch adopted zstd but not debian which is still using xz and as far as I have seen neither in parallel using option -T. I think you should ask them to produce all package with export XZ_DEFAULTS='-T 8' and enabling dpkg to set internally XZ_DEFAULTS='-T 0' unless it is differently set. What Do you think about changing ISAR to use zstd or xz -T without repackaging everything at fetch time? It would be a solution: fetch, repackage with zstd and the downloads remain for a long time like a permanent cache and also the benefits. The counter effect is that those debian packages and those produced by ISAR will be not anymore compatible with a standard debian and this could be a nasty surprise in some cases for some users. The alternative is to repackage everything with xz parallel. Usually a modern laptop has 8 pipes but I have the sensation that with -T 0 it counts just the cores because on my machine spawn 4 not 8. Best regards, R- [1] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/
On Tue, 24 Jan 2023 at 04:17, Roberto A. Foglietta <roberto.foglietta@gmail.com> wrote: > > Il Lun 23 Gen 2023, 20:22 Jan Kiszka <jan.kiszka@siemens.com> ha scritto: >> >> On 23.01.23 15:00, Michael Adler wrote: >> > This patch enables ISAR to build zstd compressed images. >> > >> >> I was briefly thinking you had solved the dpkg vs. zstd issue - just a >> dream :) > > > Hi Jan. > > I think you are speaking about zstd compression for the debian package that is 1% bigger but 13x times faster in expanding. [1] [...] > > The alternative is to repackage everything with xz parallel. Usually a modern laptop has 8 pipes but I have the sensation that with -T 0 it counts just the cores because on my machine spawn 4 not 8. Forget about the last 4 instead of 8 about xz. Every compressor is doing the same on my laptop and it is a complete novelty. Possibly the newer versions spawn with limitation to not clog the machine. Yes, all 8 CPUs (4 cores X 2 threads) are online and account for every command that gives access to them (lscpu, proc, nproc, htop, etc.). Something similar on your machines? Best regards, R-
In the email from Monday, 23 January 2023 17:00:16 +03 user Michael Adler wrote: > This patch enables ISAR to build zstd compressed images. > > Signed-off-by: Michael Adler <michael.adler@siemens.com> > --- > meta/classes/imagetypes.bbclass | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/imagetypes.bbclass > b/meta/classes/imagetypes.bbclass index 1e8e223d..6950da7a 100644 > --- a/meta/classes/imagetypes.bbclass > +++ b/meta/classes/imagetypes.bbclass > @@ -93,7 +93,7 @@ IMAGE_CMD_ubi() { > IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template" > > # image conversions > -IMAGE_CONVERSIONS = "gz xz" > +IMAGE_CONVERSIONS = "gz xz zst" > > CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS_gz = > "gzip" > @@ -105,3 +105,11 @@ XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} > --threads=${XZ_THREADS}" XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT > XZ_THREADS" > CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS_xz = > "xz-utils" > + > +ZST_LEVEL ?= "19" > +ZST_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +ZST_THREADS[vardepvalue] = "1" > +ZST_OPTIONS ?= "-${ZST_LEVEL} --threads=${ZST_THREADS}" > +ZST_OPTIONS[vardepsexclude] += "ZST_LEVEL ZST_THREADS" > +CONVERSION_CMD_zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZST_OPTIONS} Since we switched to Bitbake 2.0, we need v2 with using new syntax. Of course, in case we need this feature at all... > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS_zst = > "zstd"
diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass index 1e8e223d..6950da7a 100644 --- a/meta/classes/imagetypes.bbclass +++ b/meta/classes/imagetypes.bbclass @@ -93,7 +93,7 @@ IMAGE_CMD_ubi() { IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template" # image conversions -IMAGE_CONVERSIONS = "gz xz" +IMAGE_CONVERSIONS = "gz xz zst" CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS_gz = "gzip" @@ -105,3 +105,11 @@ XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS_xz = "xz-utils" + +ZST_LEVEL ?= "19" +ZST_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" +ZST_THREADS[vardepvalue] = "1" +ZST_OPTIONS ?= "-${ZST_LEVEL} --threads=${ZST_THREADS}" +ZST_OPTIONS[vardepsexclude] += "ZST_LEVEL ZST_THREADS" +CONVERSION_CMD_zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZST_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS_zst = "zstd"
This patch enables ISAR to build zstd compressed images. Signed-off-by: Michael Adler <michael.adler@siemens.com> --- meta/classes/imagetypes.bbclass | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)