From patchwork Mon Jul 15 10:08:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3693 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:20:54 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f62.google.com (mail-wm1-f62.google.com [209.85.128.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FAKr7I006227 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:20:53 +0200 Received: by mail-wm1-f62.google.com with SMTP id 5b1f17b1804b1-4266fbae4c6sf30823445e9.0 for ; Mon, 15 Jul 2024 03:20:53 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038848; cv=pass; d=google.com; s=arc-20160816; b=c4xi2j+9wU3gCr9eMLlLzNW4+3Jcw4GQ5TV2P1/Ix8HTcKhaApNIp7GTA8EuTwoPO/ IOKn3sELB2EX6YDumcyVn1IatHjDaD4SdZMLrxhxJ6lhqtubH9mHzqee40lfXJy7YAOD pZvRDVoM3gzGF521rwQ/7tGPB4BCiVFuaVxCPD7Uih69ttnJP36ZcB5a9a6lwuX0WLU5 x4HUHw4NFyaHQaVEJKsvEcU35qCKTPaXOwMRmjvxGqjytow7KJ9mcuNLXQk/NWjnXIE9 S/L+ZarjkpxeFEpr4duJJBB93EMPPNAmDaaKXRZ+1bB0q6838LvZJ6bB4oLgkGHA+0ps zU0Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EoDxlovoIdtwN9tjI7QFj8OPZS9oPqzi2cyN04XoWwI=; fh=rpyyCBOJCoDbgVEPkBXBUGe1wc13ws3DpS7A/fMmuDw=; b=T1mJP/vLrmgfeuB4cvHAwjfp8eu5NOfkUo0uzYSB2/6QAK0tq6obeZRDhcFFvAEo23 zmqRq4wkNqg3cLoJ4bE0ICbwv2xmgmIOaH3xL33bsJxtIi1M0GOq9b3V9U7FfKarrSJa ecZfJOWVpYzHOsyNsnYqsEUffIE/B51O6T2n41K0907ZWRoEdlsEekaskwMNvJExauFr vkTgnKDXd70w8Ba+HqzPzzTGiS76MV+42cKWWrUId8AQEfJ79Fe36W9+rrpOZIYKyCYC htOLq5gUYfgoDP7qCWaOsvj66fnKhxu5Nqxg2u4a6aoXkrTp4rBTFS8VrFdg737eRKhG FQgQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fxvQ3eLh; spf=pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-202407151008339faedc6c53b9c7b06d-XIRAWL@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038848; x=1721643648; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=EoDxlovoIdtwN9tjI7QFj8OPZS9oPqzi2cyN04XoWwI=; b=v4pJjg3c9adkSiTIcVDmCnIHuQ6b2flznYYLcn6+L6bZuUMQF+o21TuosNXwGf+Dlq 5Ey0MKCkx546BUESxrqbJiT8ew3WQCVmXkqXwzwZArcuEvFHPYY5BHKJDNMRPvpXzVnE EWOu3DWOtgThY0400ofdvqgfcLsW1BwLbJMeHTEPNtvDZga5N/lJ2v9K5iPrRGthHdEJ JBW8hP39Hqd2lsZ6dvFIE1nSIMKV/EwTJd+go1YUHnkMdfyB1PXiovaGCBjfP5jjpnFd w6smPJaUk6bzlGhAMSmyGsgdenpThGS3pBQp39jGMdrtw5cqXyBa5mBejGHXU5KKPjzU Tvsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038848; x=1721643648; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=EoDxlovoIdtwN9tjI7QFj8OPZS9oPqzi2cyN04XoWwI=; b=ZzIZXy1g+RTpbYGwYkrl8Z++FQ2plYjEMoO5HOOmDyG5JqclA4ximrHZq2GfB3bpSZ V5/BH/REzkupYukrgHK12GpUu4xFjELDuclWYRM7F+3N+GTnM62wVYr6oyJ60wLiLTlm YR6NxsLr7Tek91WtU6noHWCBQ4LgMn+16G707oEIw1ZkVCsESi59dLrurBm+7S6hHJru fsDI7RzyDmeERPL1BDC/kOMyZzXOzxxVloCa82mB1FLdoMX2KRgy9fEMgKq2pzYVOsap 8kPqdZP/ywA8PEteQ/Jgp9t3Mmqhloi7aStcxttQgE3csVQajV4HMeVWrPCqWQYGHLLl xr/g== X-Forwarded-Encrypted: i=2; AJvYcCX3fCQ4YjSLkXR8jvns/aIA5HU/6u/wr8TNHxE8ms79GYWF45pbhOfODbgEPZVbmU9nIqq3hb+En5thIqI7N7ST3/pQr0E= X-Gm-Message-State: AOJu0YxeFxEGmfkL+pLFAfGAkvryRDn4k+pH1cC2EQkgUhUbHl9pYvtj sstWXfxNAuz7jZ5opxrUQ34aaHDag4SzWUh34YMONtq/2DCRcW28 X-Google-Smtp-Source: AGHT+IFrDVuqNyvfv/BfGz2eEeD/eQh9ZAR0TSBqBobBbDOIxJ1n4JE7bX+OtQJvOXhl4Zd22isxrw== X-Received: by 2002:a05:6512:2350:b0:52c:987f:b355 with SMTP id 2adb3069b0e04-52eb99cc6abmr13515129e87.42.1721038116818; Mon, 15 Jul 2024 03:08:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3b02:b0:427:96b1:a684 with SMTP id 5b1f17b1804b1-42798478ec6ls21375715e9.0.-pod-prod-07-eu; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) X-Received: by 2002:a05:600c:4982:b0:426:6314:3336 with SMTP id 5b1f17b1804b1-426708fa8bbmr121731375e9.36.1721038114681; Mon, 15 Jul 2024 03:08:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038114; cv=none; d=google.com; s=arc-20160816; b=0RwfKGB8XAHuVu4ncfaXqXA8j+iyCUV5Kic2u2YtFG/lsWFj67yzmz7WNBTQlmVhMW sE663AdEBGapcQW8Knf7wfaeg48iBhOYjar9StUdUCKN2ht7/Rmjjk78G2I1jEeaOCLF e3KbjvAB3sKtzpV3zw8xAzZji/vlROXKcBE7+wXXYa+dlq62YNOy6dMrbMNR+OeJUzbh gWXDBXhjr4MgC6LM/0wsJsfrMPBvjGIHxSSPfjflpabAw6PtIx0f2HrtlK5HZyX0iOaq VreCIP0I8OYAauSKjh7ppvHR4P0TaEtMrF61UE9c3EG1FB9VckUB6Wk+UkjvWEx+2BOi PtqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=Z3WBY2NDqDJJkDez3e+RMxMNkjaN1jw0I79TYdvWnCo=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=B+/1jwLKtekR5U0/o/DOTYaLC3dCZBmha65FdzL3pbeks2nVleZj23tFwi5q8usgwR V2T+VGBNvFaLTrs3v4BEjwQRs2ojKvaNV1pYZnPB4XmxxOYkz7eZukkALlCQ3fyBczTd ZkffDuRkSC0tTaHANas03pYBz64ttFOcnCQrnLeJVdcmyGMm4+up/nHm6IYtfNxVwX7/ GQqFFa0EArVBJVYCRkqqLKuxCgvh6kvOe8env4iDOa2soVe+Udgi8T0ikdJ8+bWDAIGr WvgxE2gs4i5TdT8Lg/HzZmY/m2rrpIuqjOS83olfIoYb5Vxf3zhgkeqPnmEznOb06wTl Pc1Q==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fxvQ3eLh; spf=pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-202407151008339faedc6c53b9c7b06d-XIRAWL@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427a2db519esi841125e9.0.2024.07.15.03.08.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:34 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 202407151008339faedc6c53b9c7b06d for ; Mon, 15 Jul 2024 12:08:34 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 2/5] container-loader: Introduce helper to load container images into local registry Date: Mon, 15 Jul 2024 12:08:29 +0200 Message-ID: <27306a537db0da8d51a0ca709b9113248331d340.1721038111.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fxvQ3eLh; spf=pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-202407151008339faedc6c53b9c7b06d-XIRAWL@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed, by default with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 94 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 ++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 ++ 5 files changed, 144 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..a0c2ddb3 --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,94 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_COMPRESSION ?= "zst" +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += " \ + ${CONTAINER_ENGINE_PACKAGES} \ + ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + ''}" + +CONTAINER_COMPRESSOR_CMD = "${@ \ + 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +CONTAINER_DECOMPRESSOR_CMD = "${@ \ + 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DECOMPRESSOR_CMD \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + import os + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + image_name = host + (path if path != "/" else "") + unpacked_image = workdir + "/" + image_name.replace('/', '.') + dest_dir = D + "/usr/share/" + BPN + "/images" + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar" + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest" + + bb.utils.remove(tar_image) + cmd = f"skopeo copy dir:{unpacked_image} " \ + f"docker-archive:{tar_image}:{image_name}{docker_ref}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \ + image_name + docker_ref + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..b6abec92 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman"