From patchwork Tue Jul 9 17:31:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3672 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 09 Jul 2024 19:31:50 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f57.google.com (mail-wm1-f57.google.com [209.85.128.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 469HVnLc006522 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 9 Jul 2024 19:31:49 +0200 Received: by mail-wm1-f57.google.com with SMTP id 5b1f17b1804b1-42725ef39e2sf4178925e9.0 for ; Tue, 09 Jul 2024 10:31:49 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1720546304; cv=pass; d=google.com; s=arc-20160816; b=wpiwVzkNpQYezsCZLkqHDOuoHRUMn6RtremqLM6y2+dNhXX1lkEMPyZBsN6NimTMOF FvfNmBQ128oniI5Cod7Qe8JpeYcGwdKSgA4XGGkoCtuKNrb7Ydz9736JJahOzPPO2o4Z KfU4h+2TG6RQl/2sCU21mmAPCmooykzvXGe+PVDrURvS+SmOFPXEmKXnXhxpsrWNscgP 1lrufylL4WFLcIu+r9k2pai8RetOKHExTO8vvi3bPLgTRQRaSObfxFgMiWb1tywcZCjM dyz2Mlc4q0lIrJhxvYqzGSQE3LvkIGxv2j8wYVpbKTR+0uliMP1MlcTwXhnDaUuQB093 OJ5w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=9KFqqe0+zUNuf2b3S+LuEbkWCkj4tHfVFrRH80sjhzs=; fh=Hcc+btNqJwyKxqPaL7+rDw/HEUNK1itHI9e7YqVOdlY=; b=vemFuBxqsRWDNjLAab5XLotRKQs9Or5/K1AOq7DUdpxAobG1q0aAroTz0VPawM4T/G TAef+TWq5eQHt8qyWE0us3jMVch6yOndgwCH3UEvT+sDwMzIhzW3VxQjA5zysJoCgLlv tJD81jukbgkLyNkGm7ln+FS7ee+aGdG+LdjfAXUEtAdklHsDnPrso3cEURdvtsQBbG4q 9yPEPPztcNdv+yX5W2qPa9WyOT0c7p9WHYjC2AYpjILIeBjXMzylUaPkw68ZHlJOkIZL OSsIk0fH/PG183Q1nvp+YreGJ9RpJx6jAHjwPtb8zjRzhOgXAv03BwIWiZPtz2dE/cmE XU5Q==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BxUVJ49q; spf=pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-294854-20240709173140ddaf302001721ef599-8LvdHi@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1720546304; x=1721151104; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :from:to:cc:subject:date:message-id:reply-to; bh=9KFqqe0+zUNuf2b3S+LuEbkWCkj4tHfVFrRH80sjhzs=; b=tVRl5DKuA1wS+VYDmaC8SI0e2OUgDt8rLnlCGfz/GKlggq6K778nyPbRA565n9mXDf UsNfhdkDZj6l8vy0ujCwDI80urvULZ0kYtdGx/SORzOSsupqD++glc8C+eiKgxoSjkG+ ITtD7NxTIgoIJr/twmbPrjUNMDZmOQ9hxL2f/MFPvabLCNxCV0+ywXh7a4gNw15VzjQt VkqRnV+mObDNSNSDPgcY65JYuwYJKVqrL6cb55WAVEjYLUD/FNbIcockSIjUR4kIFtER Vz8HcVTaln1Y8HTQTXUS6dvZsPhfBAGGxY2o0BRF4F4PwymUXPuAglMOCUH/IrmEx9/9 QVLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720546304; x=1721151104; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :x-beenthere:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9KFqqe0+zUNuf2b3S+LuEbkWCkj4tHfVFrRH80sjhzs=; b=lTy3zLJr2wZKanJAawCajI1XYbn0tK/1JlACaI6qn5t7xmGzhSI+5gxtZ5I/aW+rG+ BmFaM5oj4V2SWcbf5I2Jnsc5roq+2v9GNfnzCa+t7H3uFMOhWR/LF5ygkGd0VBGCSPo4 j4XL4LjxIeB/vxhnjiaWZDtd44ISWIsecCraVJcHTTiD+fBenJhVk9tl2aEG1/Iqs5SR fEICkEymiOd9lytQJebjNDA1WuUmaXhaO0s4MyPIphApYJ/ipDfZR1N/kwY8K/ZJDY6e QK2mqADH9boyGvqEpGGxMmu8SOXHHz1aq1DdD4PL33g4JxSAOM/qb2aXe6k4iZZN0An2 DocQ== X-Forwarded-Encrypted: i=2; AJvYcCWO63QwHxHhDTS0QzqDQzFMg/vy/ygMArwA9HaippKKhxcIEIEzqaF3SYp+rl+h4XqaeyqVkJ9oUEcZ6LRyEUbF0Dzgee8= X-Gm-Message-State: AOJu0YyezILJ/Bwq1WO9MwhP08k+QqsfSCSK+feWQwfdVIfRXXEmp5xH 2WS06XoTKmYLkGXJLn567yLCZjv/sG6t5+z5rEtuZ99sqIeUJzck X-Google-Smtp-Source: AGHT+IEqrfNvHf47zObvKMmE00TDrqxXlXjjSEudiBz2D49RPGGE50DE4+Kp8e1PgImprDSE9/ah3g== X-Received: by 2002:a05:600c:4ba2:b0:426:6ead:5709 with SMTP id 5b1f17b1804b1-426707cf736mr19336035e9.9.1720546303397; Tue, 09 Jul 2024 10:31:43 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d0f:b0:427:9377:3cc4 with SMTP id 5b1f17b1804b1-42793774063ls94135e9.1.-pod-prod-06-eu; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) X-Received: by 2002:a05:600c:4341:b0:424:ad14:6b79 with SMTP id 5b1f17b1804b1-426706c9159mr23700795e9.8.1720546301252; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1720546301; cv=none; d=google.com; s=arc-20160816; b=X3kRsZe08+yj2m3sBtOSLuq6/h4kMGaXEW4p/DnJ2P+ydDuPLyh+RFieEeh9WoJLx9 o/IfGEIuCx1YI76XC/l48p0pOd2DYe3RD0hVCZBzsZ1jvXK/7ruMS/hYS7NA/6XMfhxT q9zqsZG1Ps4HZT3uzuWw3A0gLNnwoWoHhHyRsoODOiqZnVlBz0WJOwhFnKwX683sF+/D 52cQzr/Sp4hSEuYfU/MrCwiomHIO+KesC7Xh73cBsglbnTjVXOy7jBcI4DkjyN02B5C8 6zdMvH67urCUCCvtQ1JzTbg2Ee6V7Po4BjgUHJ5f6OeKhoc3aM6WV/k2OX30rlNWcfW7 ccWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:to:from:dkim-signature; bh=L8JXHy4gkDOvNTnuYGp+qJaSa8/+nONEOLnUEdxdg5k=; fh=hOBXq6nisExkIRIwN8IDOmOCLKwX8PM0ANlxQbNMEqA=; b=E6KNMnoCk5v+UzF2/oao2KvYbQkKCxy4wuuBGMT/55O2fnfOvMLsxdv09ZtfktZMsz sj2o1/KYykRlkz5TFhF/I4gWfhKIfBMxLHmBrFUEmLptKXgjdbthcEJugNqjZPQ6TIRN d7SyrNoCZwApcQ5ntQ8bVLBw682O/h87cK6cfCMHUKLj3XS7RYj6oF5RSyeAfJZ92vPo ZfpO8FZrheewZ+9j/wc0pa3wHZ/Zbt4W+GPXv3Tk4qA6UrANKNo2BEc1rc72btK6sYaj /GcXkhYpRMjXisw86pOhzETs+WRJ2Mbr13sxZklo8gf4zxtUtiu6i/1a+4sjOAgzNNk9 53iw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BxUVJ49q; spf=pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-294854-20240709173140ddaf302001721ef599-8LvdHi@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net. [185.136.64.228]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-426725584d2si626425e9.0.2024.07.09.10.31.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Jul 2024 10:31:41 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) client-ip=185.136.64.228; Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20240709173140ddaf302001721ef599 for ; Tue, 09 Jul 2024 19:31:40 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Subject: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry Date: Tue, 9 Jul 2024 19:31:38 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BxUVJ49q; spf=pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-294854-20240709173140ddaf302001721ef599-8LvdHi@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed, by default with xz. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 76 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 +++ .../files/container-loader.service.tmpl | 11 +++ .../files/container-loader.sh.tmpl | 13 ++++ .../container-loader/podman-loader.inc | 10 +++ 5 files changed, 120 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..8e352214 --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,76 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += "CONTAINER_ENGINE" + +CONTAINER_COMPRESSION ?= "xz" + +DEBIAN_DEPENDS += " \ + ${CONTAINER_ENGINE_PACKAGES} \ + ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ''}" + +CONTAINER_COMPRESSOR = "${@ \ + 'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ''}" + +python do_install() { + import os + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + PN= d.getVar('PN') + + image_list = open(D + "/usr/share/" + PN +"/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + image_name = host + (path if path != "/" else "") + unpacked_image = workdir + "/" + image_name.replace('/', '.') + dest_dir = D + "/usr/share/" + PN + "/images" + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar" + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest" + + cmd = f"skopeo copy dir:{unpacked_image} " \ + f"docker-archive:{tar_image}:{image_name}{docker_ref}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \ + image_name + docker_ref + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() + + bb.utils.copyfile(workdir + "/container-loader.sh", + D + "/usr/share/" + PN + "/container-loader.sh") +} +do_install[cleandirs] += "${D}/usr/share/${PN}/images" + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..afde55d3 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,11 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${PN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..31d27865 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,13 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image" + fi +done < /usr/share/${PN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman"