| Message ID | 20230210134535.241845-1-michael.adler@siemens.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | [v2] feat: add zstd support to image conversions | expand |
On Fri, 2023-02-10 at 14:45 +0100, Michael Adler wrote: > This patch enables ISAR to build zstd-compressed images. > > ISAR already supports building xz-compressed images but there are > several key > differences between the two that may make one more suitable than the > other, > depending on the specific use case: > > 1. Compression speed: zstd is generally faster than xz when it comes > to both > compression and decompression times. > > 2. Compression ratio: "zstd and xz trade blows in their compression > ratio. > Recompressing all [ArchLinux] packages to zstd with our options > yields a total > ~0.8% increase in package size on all of our packages combined, but > the > decompression time for all packages saw a ~1300% speedup." [1] > > 3. Memory usage: zstd uses a smaller amount of memory compared to xz. > > [1] > https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ > > Signed-off-by: Michael Adler <michael.adler@siemens.com> > > --- > meta/classes/imagetypes.bbclass | 5 ++++- > meta/conf/bitbake.conf | 3 +++ > 2 files changed, 7 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/imagetypes.bbclass > b/meta/classes/imagetypes.bbclass > index 0cff440..f11ddc6 100644 > --- a/meta/classes/imagetypes.bbclass > +++ b/meta/classes/imagetypes.bbclass > @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() { > IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" > > # image conversions > -IMAGE_CONVERSIONS = "gz xz" > +IMAGE_CONVERSIONS = "gz xz zst" > > CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c -- > rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" > CONVERSION_DEPS:gz = "gzip" > > CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" > CONVERSION_DEPS:xz = "xz-utils" > + > +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_OPTIONS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" > +CONVERSION_DEPS:zst = "zstd" > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 522241a..a1bad71 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT > XZ_THREADS" > # Default parallelism for zstd > ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > ZSTD_THREADS[vardepvalue] = "1" > +ZSTD_LEVEL ?= "19" > +ZSTD_OPTIONS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" > +ZSTD_OPTIONS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" This probably also needs to be renamed to ZSTD_DEFAULTS to be consistent with XZ_DEFAULTS. The details have been discussed in "imagetypes XZ_OPTIONS replaced by XZ_DEFAULTS defined in bitbake.comf". Please double check. Felix > > BBINCLUDELOGS ??= "yes" > > -- > 2.39.1 >
Hi Felix, > This probably also needs to be renamed to ZSTD_DEFAULTS to be > consistent with XZ_DEFAULTS. I agree that it probably makes sense to use the same naming scheme (<algo>_DEFAULTS) for both xz and zstd. Thanks for the heads up, I will send v3 in a moment. > The details have been discussed in "imagetypes XZ_OPTIONS replaced by > XZ_DEFAULTS defined in bitbake.comf". well, but in my case it's correct because I use the same variable name (ZSTD_OPTIONS) in both files :) Kind Regards, Michael
diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass index 0cff440..f11ddc6 100644 --- a/meta/classes/imagetypes.bbclass +++ b/meta/classes/imagetypes.bbclass @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() { IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" # image conversions -IMAGE_CONVERSIONS = "gz xz" +IMAGE_CONVERSIONS = "gz xz zst" CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS:gz = "gzip" CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz = "xz-utils" + +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS:zst = "zstd" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 522241a..a1bad71 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" # Default parallelism for zstd ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" ZSTD_THREADS[vardepvalue] = "1" +ZSTD_LEVEL ?= "19" +ZSTD_OPTIONS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" +ZSTD_OPTIONS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" BBINCLUDELOGS ??= "yes"
This patch enables ISAR to build zstd-compressed images. ISAR already supports building xz-compressed images but there are several key differences between the two that may make one more suitable than the other, depending on the specific use case: 1. Compression speed: zstd is generally faster than xz when it comes to both compression and decompression times. 2. Compression ratio: "zstd and xz trade blows in their compression ratio. Recompressing all [ArchLinux] packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup." [1] 3. Memory usage: zstd uses a smaller amount of memory compared to xz. [1] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ Signed-off-by: Michael Adler <michael.adler@siemens.com> --- meta/classes/imagetypes.bbclass | 5 ++++- meta/conf/bitbake.conf | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-)