From patchwork Fri Jul 19 16:38:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3709 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:56 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f56.google.com (mail-lf1-f56.google.com [209.85.167.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcrZ2002644 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:54 +0200 Received: by mail-lf1-f56.google.com with SMTP id 2adb3069b0e04-52e994d8e26sf1621149e87.2 for ; Fri, 19 Jul 2024 09:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407127; cv=pass; d=google.com; s=arc-20160816; b=kagLEtdFYGH6On2zThQhiZ4VZ/AFJqCFOeFmRRttYf4ELzk4i+RBkXoUTYtis+ClBo VzqxbaIU0KuD66pTWyXE0Xz3Ba37pf1eJ5tHQzLqs8q97IBYJ2Adswsfo7BQ1kPdKXNP k1Iy30GflH8+INwOC96nhDxerlIQ37L5NR0tpcdFPlHIxPpfLdtBKz+p9fPfNsT+BblW R1pdqAIl6eG1mum2eEEXLBdFzJ/Ux/wj7lbHTVO4YgVhab+7nPayuzzrTUBccOM+POCj IEM+U4Qe5Zrv88iI3dFw79LAr690O1BnTviP9yudm9mB+o/fRaUHKPYGgOZw3V6h1eND RPkA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; fh=hvgAxZX/N832QdoQH4xrYJAH0heQh+gSAr91Dg1B9YY=; b=sZArVWjjk8C58mxyx9goZ6friGYdTQYNSthRi89zPyJwmzUjVIyEcVm5Jek8iNyIwA KopzeZGz6a/VPH9OAvIo8xRdhV3zDP140ae8Z2Uls+K7PRjv/yos3aHK8b8e5J2krXsD h7FXrHkJX/OQiVqAx2iDMPloi/y2XVnB4Uc99WzCJ0E+3cg+m5gaNL+xVFfkws8nNg+M GnEdD3AL/vgMD2sPpdvJg8tLaqQMewM/9Yu0ERNUYSWzPHH47Bf+/uugtaYsT5pEImBY 7pVy4YoOunclsy+LSV941VCrRo8IhpxTqedYL3gOH/iClzXvmJG++h5cfQkc84KTBW02 kblQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407127; x=1722011927; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; b=hefXgHDDK5wv+BHN3aUGj1Y9On63AsCSABG0JfluIFprXPyXOrRTxjm6IcaQJiWil/ 6SYg085s+4TME5Z1cBvdx+BNLn83ZVOBm/0KGwNko4BMBQgZy6V2ePaITbSMQT6csVi+ C8NgpN+fzUZkGIVmKcTNO1bbUKMw9Aw4kM2Pcthn2O59MYx1fmxrZpxg7q2CVrGkdcNj hUR54yg7FX68OBRQey71DxAKiC8NRl4f44jBSajKkiUG6FbnTgTJYnPs/Pcm2uC4k6cq usg2C5tht46/DtCkeaU/mzsGwA3sYhtlqwK+MezhmHvAXsDHhz0erghQZMBkBBaDLaIr RS1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407127; x=1722011927; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; b=Wfj5AncYzaakOF/lXFBuC/94KsNYTFn2UOVt1TH4XwNC09Kq3GpqfTOerB8xBD165g uNB9hsjOW4RUyk8gCD16kkHq/PFpNaw0YhC92FKh0isk/2Mm+FApd6GhVXq8/wccdRjN d+MrGL1ZevfIlhI9A39lavkGzUd0dDRb0R1D/gMrDNztStBoUUbgvYYraA4/E4WfQgX5 +caT22MDXW6gFdXri5l0HGFvkfOreAces/g4M+w746z26BrlBqWA/UwGIrb0dwaTTg3G hJET/gQRB8obHhmXP0X8Z7m4OTrj4YlyzD75winaoc0YHW+N6Rd7//m6cZgYIYW5E9Hy U/og== X-Forwarded-Encrypted: i=2; AJvYcCVqR3hsd2D/dPAXqkD49MB1mSTrVxdB1Q5q5idsI1tr2tEWT7QivYmb3TAMJSMmerniS4dUL0ie3YTxTaYWM8nX4+YlPnU= X-Gm-Message-State: AOJu0Ywkl9TtGiSWCo2IvHyOogZNN9+wxrtQB/CIS21nMzHIkYMQcmHt vcwxjFJAoHdEEXYY8db3Ijj0NFvWojCqArE96QZywcQkrKsqfbea X-Google-Smtp-Source: AGHT+IGuNM6BLXiKYzgLddi66zjBoQN/AKFBVlqMqSXh4BoIcQSG03vKSyb01CqwQCbMBkQKmNfOvA== X-Received: by 2002:a2e:a404:0:b0:2ee:8c8d:d9dd with SMTP id 38308e7fff4ca-2ef16840f94mr2137721fa.36.1721407127003; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a9a4:0:b0:2ee:605b:3d02 with SMTP id 38308e7fff4ca-2ef04dda89cls12094081fa.2.-pod-prod-01-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a2e:9101:0:b0:2ec:440c:4e1c with SMTP id 38308e7fff4ca-2ef167824f8mr2177381fa.11.1721407124619; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=HFDMVULEFGPvIHdxCPEcDfqh3wlsa0ws1/6O5+k9Vj+r+jLZRC33QeSDrwJ2o53Jvp XJ8tuPKI2v1CDyHgpEcFo2YoFEsJG5IivYIn1KSicPHl2nkHAgRVyI5jWDn0ZHRmKAEN QhvWkHyJPk8REogjJnpvJeCM6Sw5XPvfq7pkgC+FkpDiIHnES3naaEDHa1pw0lGJZMkY qK6vE/zHHz5f4a0urOUhNdwwA82fp/Cek4yP/qJ3tA6OEZSSCTjv+xCoj+QO768ES5kX b3SmGsOJLtNMwWSA3cu2Psu2k6GHGQWv+rU9PDS5zY5rPXvaKD27V/xWQ9CdppZxLilC OORg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=YTKlvg2yRZslJX8lHMaY1WIwEOdoVww+2lzK9eEeB0c=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=JoYzezORfijh41zLWwr86FL+yCPEYGzs2/ELKXGOsIYMmpS8o5H+ul5S9axDXRFdP8 xuBykVBXjMycM/sJ9B7NSQfoA794kNbf1oE5I86x7FkecqQgkKpz/T7V9X6kObhq1EzM VZmX6MCQjcG+tce2w9f9qX9LY97SHe2vm+/cO1uNDUuODeqpj8x6embN+jIXtzRKEfEq kSbFcl6hy8UYr7MsBpIaqgm62FaqtZvzfwR0Z0PdF9Ik33/6INCsbnGYMkiH9upMIqVJ AwO6VkP6aENlxUQCnsbL/7I8o+0XkxpvMWr7U5f4BkSt4laIVhWw1MWmY9rX+oK5SxfF LLiQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427d2911f57si1635545e9.0.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20240719163844b8f4655d42b9dc012d for ; Fri, 19 Jul 2024 18:38:44 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 2/5] container-loader: Introduce helper to load container images into local registry Date: Fri, 19 Jul 2024 18:38:40 +0200 Message-ID: <5af6163750f7ae0cb186e52727afe3ced1db2ce2.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 73 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 +++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 +++++ .../container-loader/podman-loader.inc | 10 +++ 5 files changed, 123 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..5fd8d23c --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,73 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += "${CONTAINER_ENGINE_PACKAGES}, zstd" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + from oe.path import copyhardlink + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + tag = parm["tag"] if "tag" in parm else "latest" + image_name = host + (path if path != "/" else "") + image_file = image_name.replace('/', '.') + \ + ":" + tag + ".zst" + dest_dir = D + "/usr/share/" + BPN + "/images" + + copyhardlink(workdir + "/" + image_file, dest_dir + "/" + image_file) + + line = f"{image_file} {image_name}:{tag}" + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service + + # Do not compress the package, most of its payload is already, and trying + # nevertheless will only cost time without any gain. + cat <> ${S}/debian/rules +override_dh_builddeb: + dh_builddeb -- -Znone +EOF +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..2356e31c --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + pzstd -c -d /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman"