From patchwork Tue Jul 16 14:18:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3698 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:20 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f57.google.com (mail-ed1-f57.google.com [209.85.208.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIJEJ013324 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:20 +0200 Received: by mail-ed1-f57.google.com with SMTP id 4fb4d7f45d1cf-58c4f94b57csf4332391a12.0 for ; Tue, 16 Jul 2024 07:18:20 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139494; cv=pass; d=google.com; s=arc-20160816; b=ykzF9KwW/y1RESn9zjn2E64tIrjEDbh39FDjj0ud8Nr/844HTw8F8FWnyABT2cdZzX HWhVasUOVF6JKHsFFLCj1yVyWEiwiA16FnoRJrz2tlQH7a+XrdrSOrJD7hc52iPxKwfV 7s8aRBynkE0i78WV1cl7+N9bLk79NSpxuGxoZoSwLmTLDWvMz88VN64+IqqXHIPhEpaL V4AEuqE4VD/z5MKfHjT2x1O7/a/k2e2d/CoZh3j43tfMuX9zdVf/eS18mg5nt7fMNPpW wfvj++7PZdQwq1kDmUho89PoZtjsA61nOaGVYr4tPFKCl19eVkSMo2yvDyT8ZH/drhGs TwMA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; fh=1BDurDXYBFmfUnMK/x5QtycmSEnKbZv9ZRUufnZDkc4=; b=PRJnzz/xTWdHYMiV1rJ7knMN/5x9PmqSn57qnRhVOHZCLS3bBxts/LH2teKjrGwkjk Pww3Qf9GvP1RXiFP5h1PnkzcmiOZLaEvdpaDyzNDwn53tyJg3pI+p0ReuO+PZyZDxLGO MWMnNMG70jfOsdHjMRAOMEiPVQ5q0592lHDt5hf7i5VW6vXcsFlRsVNB8Qut57phweKi +UyUksDaAxfW+JSwr6PU65RWJISN0suZ1A47KXd7AD1eXvJWHYvl78CQ+92XyMUzyLdP h0vPdfdIIvxnpRdns2vOL0Xa447kvZU6xv8/AkI4VtDO75Bd6tq8tndWdgb7m8bEgiFV n1eA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139494; x=1721744294; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; b=o0MQQjPmBLtOu7FoXGheP53bMPGc80xV8/YkU5673wBRkKiZdODr70TVLo+SZkrLL5 tks74xDyqdpR6BWZ0Q0Z/Z0aQYNk+gE59dXszpjgrVufB+6gpDir0hxFvGjp/t4Hf0Jo l0gtOTQpIalycA4HKOb2xiebnLw9wjRsFkoixTRQqQ8UifcI2SfFj2GebHj72RoAodDx a6XFlSejnYJQ3hs4OuIrRDzA67xeXm0boqDq3l0/cfb0OfWAi5gbpYSV/wWMaGP5MRcz rB5nbZ1qw3Ry6qmg+5a2/by6+hLxk/n3YKjoI531gPpBjbSxVu0HYonNtF6CGfcj1x5S TN5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139494; x=1721744294; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; b=Ds/ztH07NsNA0t8ACkDv8lAT2ct9mI4cGBejQrt/AUXk1CQVIuOVSgiq1/tCyfDpQE /ItF58dEAbEkxjW9NUkNDQ1uQeFm2cOicG6bDxSIgwIybr0OkgkCbe+um3f0SArIbNnY dVfhxa9a/30V4pYhVqX4WU/mQOaofb/0rhZ0cGqdGY9+UH3086cj/+e1xQfZ2scJKVim yQ5TEqc1FKlvOrpk9Eft9GoEEg9Jn8up57xn988g2QbzObeQnzHNGsSNJZUF8FYAJMyZ GPq1xU7ZrVBEIiltcKcqqPnpCRV78EtUWBXGuhrM2uJFNwke7oS0APyS+9dziKTTHFMf OHbQ== X-Forwarded-Encrypted: i=2; AJvYcCUzqqrlcChiyJmINmmSk1DfKYB/BvThWEcWzZx1mfqO73Gak4CL28jHTg7RCO5+tCHAy2WxrRqJYEQqdpUtu5vcGDVkKZc= X-Gm-Message-State: AOJu0YyKpIjEaArPDduQ3UUCXd/rxYkrlliHMab5uZqT9f743FPvvK5e FDTO5BGlcaFMG8D3B54LiEhzQdUqwEvnsA70g5U/ZL8BS19ajm2n X-Google-Smtp-Source: AGHT+IGQjdjJUbhBeyxEgnBt2zTvIYjytX/SASyWiUCoTHSBsvGU0DTYBRHOxMuFj8HGtCKlYSqP0g== X-Received: by 2002:a50:aa93:0:b0:59f:9fc7:1e66 with SMTP id 4fb4d7f45d1cf-59f9fc721bcmr523379a12.9.1721139494100; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:4310:b0:599:9c73:c392 with SMTP id 4fb4d7f45d1cf-5999c73c484ls2421931a12.2.-pod-prod-04-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a05:6402:210b:b0:57c:672b:ca34 with SMTP id 4fb4d7f45d1cf-59ef01be830mr1701555a12.28.1721139491565; Tue, 16 Jul 2024 07:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=ooOFwoJhiLPMsL6IbIMRDku1pClUnZXOMGM9T8x6PcY969J4Ws7+Yzbszcckm3IpoU KA7cHJesZViRCtCkWKTk40TakIvrjAKA1yPd647+EfJtyWbsPQwQPZH3csTmk/NtyN02 QzFv+bFQbxDy51LbR8lOLrNEaXCyMrnuc5CdKhYB/XfC77aIMIxTJX9vouDcKYZPlqNH ACpa8QmnTe+NenQBf8P6m2Pfjme/GO4I2NSIxbZLs5Vo4K1og1l63BuA6b2/Filt5B5U yCMZPP/C/3enjqip4v+recPWjqR3EJD3bmkACXw/Wpc4ueD3Hj2aTAkmP9fQTteeBzxa +p1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=O6OIIUZOAeri4B2tkgLzjLGvlfizpgiO81DeH8YO/nY=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=ySOzk2+RuPxdXReAVqTjgewCNC6rXJRK0XvmFqzVv69EHLcPBn3AcpPRR1oiMqQGey QUdThfL++z/PzTRTbCZFZuiojUbgF+JO7zE5BXJeuo9Nd6DGx/sSt/31U3AltyvRBbEw GrkC9uOPz2LLWZMlZW5uWsi77m8aER5rY+HLwgEvVDGNkDGBw3oj09J5wseNXdgrtfA9 8aPLfM2IRe9a/jzTZ1CYjAoniKJZWjkAHPFpyAeXIwGA6fV8mbfC1ldG2T2pfTI3jDhi KUibKrgwetqY+u6bfaP5qjz4rGTgoYjt1nTOjd/4G7YPDCJWVcuMXD4IHOsfWx9vQFgZ Iydg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-59b27a0af16si219724a12.5.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 202407161418115ce2812f9612c0cc2e for ; Tue, 16 Jul 2024 16:18:11 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry Date: Tue, 16 Jul 2024 16:18:06 +0200 Message-ID: <3b7b8dbdde7fa3a4184daa3f8d567e72c8b50d2d.1721139489.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed, by default with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 101 ++++++++++++++++++ .../container-loader/docker-loader.inc | 10 ++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 ++ 5 files changed, 151 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..e97e829b --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,101 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_COMPRESSION ?= "zst" +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += " \ + ${CONTAINER_ENGINE_PACKAGES} \ + ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + ''}" + +CONTAINER_COMPRESSOR_CMD = "${@ \ + 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +CONTAINER_DECOMPRESSOR_CMD = "${@ \ + 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DECOMPRESSOR_CMD \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + import os + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + image_name = host + (path if path != "/" else "") + unpacked_image = workdir + "/" + image_name.replace('/', '.') + dest_dir = D + "/usr/share/" + BPN + "/images" + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar" + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest" + + bb.utils.remove(tar_image) + cmd = f"skopeo copy dir:{unpacked_image} " \ + f"docker-archive:{tar_image}:{image_name}{docker_ref}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \ + image_name + docker_ref + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service + + # Do not compress the package, most of its payload is already, and trying + # nevertheless will only cost time without any gain. + cat <> ${S}/debian/rules +override_dh_builddeb: + dh_builddeb -- -Znone +EOF +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..b6abec92 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman"