From patchwork Thu Apr 27 05:51:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 2777 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 27 Apr 2023 07:51:44 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f192.google.com (mail-lj1-f192.google.com [209.85.208.192]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 33R5ph1L004024 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 27 Apr 2023 07:51:43 +0200 Received: by mail-lj1-f192.google.com with SMTP id 38308e7fff4ca-2a83a0b7be1sf33398131fa.1; Wed, 26 Apr 2023 22:51:43 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1682574698; cv=pass; d=google.com; s=arc-20160816; b=he4C44/grpOUZBD5L2gxDu/u5CStBL4fW5/OZPolPKy0yksXM/Vvn6W88sKP6UCCVr /SbEwsxOFfOvuQ2zKkZqGMVaLHLQDlmQdDcjPWfkzsFLiMqZRzux9TLDLUMf99KKEzr+ KrOKQRHKtwVI59DLnvF5jQ+O2BjtkTYzy7W7KBhBBDx+kt7ZCy9BmtVq9VAGxWjHE69F RQ0y8eUL0Dnzg/O5+Hk+7noQk6fEFnwArIVjiEhHnVs0ipG8fnT084tjfDTrSR82k1cz hr2EfXgPz8EmUTkZvc1J9BIZgyUQD2P40O33N/H3UgKTJExFm3XGwdW+ughKj2BwU8J0 xn5A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=r9dWBb+GkFnO4VC/8ZH06jAC/4zCurQcjdColR31FXo=; b=jmxhcYxfk2PIbsXKAghLFgKbbGh6DP+LUfYcyow1EpntVH/lfJdPT68QDbivGhsl63 niYokFpG3mVMbwbyact8SPJsCY+9UnBfJR9foyRO0FsXhL8HWNhaN0+U0TZsAJprbGmS ciUrb7wVP5AwfH/5NVjULON9NGIGdK0cRE/qaD7b2xSXIsp1wb+/hxbVRgsmvZPM+M8v YVjxf26aNd2HOt0dFyFHi9wHkYJRCBYUfsW9/bU1jY+P345LmOfprDNRm8LmtcqPMbCC VLyr+mB3DyT8EFyJ/SHnuGYlQk2a1KZ0wlrrx3Uh8CunlhGh6d8Zc9aoZcQCpCt92ZoO 9n6w== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=HJuLfP2f; spf=pass (google.com: domain of fm-72506-202304270551349d19be964259d636a6-mlwhf2@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-72506-202304270551349d19be964259d636a6-MLWhf2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20221208; t=1682574698; x=1685166698; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=r9dWBb+GkFnO4VC/8ZH06jAC/4zCurQcjdColR31FXo=; b=OUvCVnLaz0t1UnT4P4YksmJU6qvNyGV1ll1U5kAHCDYliPZ9YXbOMLHpFoH1NT4asc F6GkZSFQUjE7kHyR3wooE865Pjp/x2FTvOXMakgjOZG8yxR9Z14eS5QDf+1ean6ivQ2r FQpV3eLY5ajSJCY+koOnCpJkwioPqVAgHdv0HcP83Zic8j5DpXKG7t7qDLGlkuqZGzuV coR65L1NkZ1Lhx4AedVtuMTLBPs0K+IBEbBTLxuCoFDIf0n6wpUPYEd0kORW+JiiCPHM pFmbwHIy75iY8sb85eewKeXDFGSMjAeSFz6GqoQy4liCxrCvMw6VIReiLWjwkUExgw0/ Z2wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682574698; x=1685166698; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r9dWBb+GkFnO4VC/8ZH06jAC/4zCurQcjdColR31FXo=; b=MW4EhLqjAgm/Dq83Ep3bk9X6MQuDkX8Hf/ClPPkG+LlnCP9Z9tm/cpqdKhjip8G+M3 hTgArBP0E3cVaYky8+/vg1bHxJnaYMM55tqMH3n73Z6YWSpc+XoESk41erotp8zcenWR m1MTgQaJHb+xzngdzFwz8i+Uxzp/y7vUL7jHCxHESwlGHe0S3kTZZXvnlNSQhvA6mG5X wVXcy9+0FDxF1rBRDPBCkMRFSS/5Ao47kfSY4SbDV3f8gLJgDt8/Rdx/EnBKMXI6Z5of v/q7X2mX4gB4g8rcTwLpKt1vdFnSnc7EvB/Kv9UN8ihEhmFdkXiB8tf4Nkqx8GE2iweO peDQ== X-Gm-Message-State: AC+VfDw2AjLoXfxl+4MZCTbroie9aAARVS/bv0CiGxCu0GtruVrEUx17 Wj2foxT7Ay7CWh+BDhg9O4E= X-Google-Smtp-Source: ACHHUZ4RkAvZGyirQhscfJX2vqs0aGQJwGGm4trpVnwKRps2PF6Kt6P5au8FO/pWQAqd5Faoh5x99A== X-Received: by 2002:a2e:9099:0:b0:2a7:8607:2e64 with SMTP id l25-20020a2e9099000000b002a786072e64mr192083ljg.3.1682574697851; Wed, 26 Apr 2023 22:51:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:2117:b0:2aa:14a2:14bd with SMTP id a23-20020a05651c211700b002aa14a214bdls158277ljq.6.-pod-prod-gmail; Wed, 26 Apr 2023 22:51:36 -0700 (PDT) X-Received: by 2002:a05:651c:104a:b0:2a8:c7e0:7764 with SMTP id x10-20020a05651c104a00b002a8c7e07764mr294360ljm.18.1682574695972; Wed, 26 Apr 2023 22:51:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682574695; cv=none; d=google.com; s=arc-20160816; b=b9RO3PrYfU/khXb5gIY0IU9BEX4XgshK7kYiF6I9dEFgOgKqwjTxLCq8HnaOooTGUU 8KTNowzW0aJeaLudpyfbQnqGHQ9ojyTfmo+34HwrjPb3u3H+kXsZFWLwLi2NvDCDqxfQ G7K7cJt9/VM2b7obxEUgPX9kPDRxR7YvAWk0S3UbZOzVIBuhrd78Faeneb0yEu0U7GuQ 9gtckto1VNsQUqSAXhNNcFPc32QtC5AzC7h0pek4lSYVOdfUO5SbNxA+by3zRvyngrPf 8WUn3apjxQotkiCA+Ipg7PVCEAAPN79QDm5z2ik0uROZ49vYVpbeLSw7y4rL0NHEuFgC h69g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=NCeK0Wsfjwl25BAj/0xUdczw20Sv+MYEzEC3P8tU4OI=; b=LANJK57hxNa8Su/iAEMbF9exE0TiJPIYSS+iav528cpvx4f1EUhhIVLoqXxtBgCXde bN7yrB010jar/QxZ01mJjCKcoVxfxLg4Z1OHDcSJZbRjCXngrAun6avWa0FxkCgyJhL9 P3mWr2KjUYsc/T0inycjx1eS9PE/fRVQPZiRoLi7qqC4GJR9mmp+6dgrn3tdelUaGfcQ gyxbqXLmODiA7UccR5KF1J0SSkf8cri/39GuWG39vkth92nJwmJK55OFmHbBpZDBKcCd 7ys4VGI21wsPM+rGfWoaeYH6MouL21kfZzEg2SegLZPOIZ/Dms8sYFtDWbYpnJnlScKF /UCw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=HJuLfP2f; spf=pass (google.com: domain of fm-72506-202304270551349d19be964259d636a6-mlwhf2@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-72506-202304270551349d19be964259d636a6-MLWhf2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id bz18-20020a05651c0c9200b002a8b4c3cf27si965737ljb.0.2023.04.26.22.51.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 Apr 2023 22:51:35 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-72506-202304270551349d19be964259d636a6-mlwhf2@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 202304270551349d19be964259d636a6 for ; Thu, 27 Apr 2023 07:51:35 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: henning.schild@siemens.com, Felix Moessbauer Subject: [PATCH 1/1] add zchunk compression support Date: Thu, 27 Apr 2023 05:51:18 +0000 Message-Id: <20230427055118.1993072-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-72506:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=HJuLfP2f; spf=pass (google.com: domain of fm-72506-202304270551349d19be964259d636a6-mlwhf2@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-72506-202304270551349d19be964259d636a6-MLWhf2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch adds support to compress images with zchunk. The compression format is optimized for delta transfers, by ensuring that small changes on the input also only invalidate some blocks of the output. There are not yet many configuration options available, but once available, they can be added to the ZCK_DEFAULTS variable. Signed-off-by: Felix Moessbauer --- meta/classes/imagetypes.bbclass | 5 ++++- meta/conf/bitbake.conf | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass index a80a6da5..3639662b 100644 --- a/meta/classes/imagetypes.bbclass +++ b/meta/classes/imagetypes.bbclass @@ -93,7 +93,7 @@ IMAGE_CMD:ubi() { IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" # image conversions -IMAGE_CONVERSIONS = "gz xz zst" +IMAGE_CONVERSIONS = "gz xz zst zck" CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS:gz = "gzip" @@ -103,3 +103,6 @@ CONVERSION_DEPS:xz = "xz-utils" CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c --sparse ${ZSTD_DEFAULTS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" CONVERSION_DEPS:zst = "zstd" + +CONVERSION_CMD:zck = "${SUDO_CHROOT} sh -c 'cd $(dirname ${IMAGE_FILE_CHROOT}); zck ${ZCK_DEFAULTS} ${IMAGE_FILE_CHROOT}'" +CONVERSION_DEPS:zck = "zchunk" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index c660660a..f73e4838 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -144,6 +144,9 @@ ZSTD_LEVEL ?= "19" ZSTD_DEFAULTS ?= "--rsyncable -${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" +# Default compression settings for zchunk +ZCK_DEFAULTS ?= "" + BBINCLUDELOGS ??= "yes" # Add event handlers for bitbake