Message ID | 20230210143326.267033-1-michael.adler@siemens.com |
---|---|
State | Accepted, archived |
Headers | show |
Series | [v3] feat: add zstd support to image conversions | expand |
In the email from Friday, 10 February 2023 17:33:26 +03 user Michael Adler wrote: > This patch enables ISAR to build zstd-compressed images. > > ISAR already supports building xz-compressed images but there are several > key differences between the two that may make one more suitable than the > other, depending on the specific use case: > > 1. Compression speed: zstd is generally faster than xz when it comes to both > compression and decompression times. > > 2. Compression ratio: "zstd and xz trade blows in their compression ratio. > Recompressing all [ArchLinux] packages to zstd with our options yields a > total ~0.8% increase in package size on all of our packages combined, but > the decompression time for all packages saw a ~1300% speedup." [1] > > 3. Memory usage: zstd uses a smaller amount of memory compared to xz. > > [1] > https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-co > mpression/ > > Signed-off-by: Michael Adler <michael.adler@siemens.com> > --- > meta/classes/imagetypes.bbclass | 5 ++++- > meta/conf/bitbake.conf | 3 +++ > 2 files changed, 7 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/imagetypes.bbclass > b/meta/classes/imagetypes.bbclass index 0cff440..c2033fd 100644 > --- a/meta/classes/imagetypes.bbclass > +++ b/meta/classes/imagetypes.bbclass > @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() { > IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" > > # image conversions > -IMAGE_CONVERSIONS = "gz xz" > +IMAGE_CONVERSIONS = "gz xz zst" > > CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS:gz = > "gzip" > > CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz = > "xz-utils" > + > +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS:zst = > "zstd" > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 522241a..c9f52a8 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > # Default parallelism for zstd > ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > ZSTD_THREADS[vardepvalue] = "1" > +ZSTD_LEVEL ?= "19" > +ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" > +ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" > > BBINCLUDELOGS ??= "yes" Applied to next, thanks.
Thanks. After seeing this i asked myself if etcher can handle that, because that is the tool we often tell Windows/Mac and even Linux users to use for flashing. Seems it does not yet https://github.com/balena-io/etcher/issues/2856 Henning Am Fri, 10 Feb 2023 15:33:26 +0100 schrieb Michael Adler <michael.adler@siemens.com>: > This patch enables ISAR to build zstd-compressed images. > > ISAR already supports building xz-compressed images but there are > several key differences between the two that may make one more > suitable than the other, depending on the specific use case: > > 1. Compression speed: zstd is generally faster than xz when it comes > to both compression and decompression times. > > 2. Compression ratio: "zstd and xz trade blows in their compression > ratio. Recompressing all [ArchLinux] packages to zstd with our > options yields a total ~0.8% increase in package size on all of our > packages combined, but the decompression time for all packages saw a > ~1300% speedup." [1] > > 3. Memory usage: zstd uses a smaller amount of memory compared to xz. > > [1] > https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ > > Signed-off-by: Michael Adler <michael.adler@siemens.com> > --- > meta/classes/imagetypes.bbclass | 5 ++++- > meta/conf/bitbake.conf | 3 +++ > 2 files changed, 7 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/imagetypes.bbclass > b/meta/classes/imagetypes.bbclass index 0cff440..c2033fd 100644 > --- a/meta/classes/imagetypes.bbclass > +++ b/meta/classes/imagetypes.bbclass > @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() { > IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" > > # image conversions > -IMAGE_CONVERSIONS = "gz xz" > +IMAGE_CONVERSIONS = "gz xz zst" > > CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c > --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" > CONVERSION_DEPS:gz = "gzip" > CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz = > "xz-utils" + > +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" > +CONVERSION_DEPS:zst = "zstd" diff --git a/meta/conf/bitbake.conf > b/meta/conf/bitbake.conf index 522241a..c9f52a8 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT > XZ_THREADS" # Default parallelism for zstd > ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > ZSTD_THREADS[vardepvalue] = "1" > +ZSTD_LEVEL ?= "19" > +ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" > +ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" > > BBINCLUDELOGS ??= "yes" >
diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass index 0cff440..c2033fd 100644 --- a/meta/classes/imagetypes.bbclass +++ b/meta/classes/imagetypes.bbclass @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() { IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" # image conversions -IMAGE_CONVERSIONS = "gz xz" +IMAGE_CONVERSIONS = "gz xz zst" CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS:gz = "gzip" CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz = "xz-utils" + +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS:zst = "zstd" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 522241a..c9f52a8 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" # Default parallelism for zstd ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" ZSTD_THREADS[vardepvalue] = "1" +ZSTD_LEVEL ?= "19" +ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" +ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" BBINCLUDELOGS ??= "yes"
This patch enables ISAR to build zstd-compressed images. ISAR already supports building xz-compressed images but there are several key differences between the two that may make one more suitable than the other, depending on the specific use case: 1. Compression speed: zstd is generally faster than xz when it comes to both compression and decompression times. 2. Compression ratio: "zstd and xz trade blows in their compression ratio. Recompressing all [ArchLinux] packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup." [1] 3. Memory usage: zstd uses a smaller amount of memory compared to xz. [1] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ Signed-off-by: Michael Adler <michael.adler@siemens.com> --- meta/classes/imagetypes.bbclass | 5 ++++- meta/conf/bitbake.conf | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-)