From patchwork Fri Jul 19 16:38:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3708 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:56 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f57.google.com (mail-lf1-f57.google.com [209.85.167.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcrBR002645 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:54 +0200 Received: by mail-lf1-f57.google.com with SMTP id 2adb3069b0e04-52e993e31a9sf1705029e87.1 for ; Fri, 19 Jul 2024 09:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407128; cv=pass; d=google.com; s=arc-20160816; b=flwalhknr47IFNsMfs9ns+jSOlHZh0eD5erDnH2PCll/YR/KMuxySlMSPLcdIs7/I0 NQ5yWH6mUOJ4j98/5UmAZmqnjgnhoDZKUyzx6qkhXvjZbLX8x31yflfOQU3OdxeS9Yxx AG0V8hWeIDcpiYbt59rRINc0up83J2F47fY75wXGaolGN/70y/GPvVALN9GgBXUCeuCY AlkBBD7gVXZ05pdZlFLZa6sd1DyJGDFR+J4hVi/xTwTYv/8kA8OfQJQ2XJCjv5HyeCdK h4R1Lxxh9ISBwMUM/Bt7zw/hIr48kPZ9VXfz6U3uX66dBBkbozXKRB2reXGFBY8w4de0 ub/A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; fh=IHp74hTUh0ItTXC1vRjKc78GIVciWXPlmP5xzMrXiKE=; b=zdIM03IvLsV4cPWBjnTUSlY/dEupLR6+diaVFgceebw3Taj0XcFxc/ZKdhvBm8aslC 214BxH9moKtw4Pdq/zHqwSMV1GmCKw7TkJeZYng1oq0Els+yF9FgK02XTGV2yYgBswRh 3+gpwN5YSCYl0z5gkmFApr+XMfgMItG9EizIWdeTo1CkpeLk33hBajT0qlXhtyws3qnT 5zIuPoJlvmWs2XbZZIInP67KZOtCa05ndUg9W/YuGOHxyrRGWl1bp/jMBMuAQHBvGHut N7fnCOOTiTlgJYQEQnAYVz8USVyeye7NKDyR8BGalNjZEYyFCG47UaqvLhpr3uollzcK BAnQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407127; x=1722011927; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; b=NT1V/skf7BCP8JE+H531rKubo+j99CP/ir61W0gGI+KYsu6vIfqjf417Sm0KLYz1an VxJFv+q6w0cxAn2jwk86DK0AqVaEy0GEAQqTlmOgbImwuIu8eIOe4y1FhrDFNancKiwp dPGwjeTm7xzIZK9W74Q0avBKnfQZs2KGcO1InPMJyn/qe9RX9EjJyqSOxiDQswDaiEfk JNL0b+K93Ix2qx2cu+u7xXuSMVVjvSbmnp4vK5GWW0fFHFKsM4t38fOm+eiCm78Mryw4 WCVWWS86hoafie7kz7Qp5FMHEh+uUt410k16GXMBD/ipBARZpjxXzohYx2l2eix+eWZ0 Q3Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407127; x=1722011927; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; b=u+I3YYPt5Kad6ZKkcljboxpHoUXmEn75qOdmup4TLPjCiM24NCMwTDqWQNYGFQN/Y3 J9ROcfw9kUxMMwRWAYRZJ03MSGB6oP34aMkh/LGhoEqwTMohjAGhNDc+sZSFQ/C7VQJO S4zNl9h6Wucj+DUcd/202PQBpaFMkbtGoByWrA2GtiNeuFAG2FNAUwoXY0YweZLWXPZZ OoYLww9do3aXVdWf8e217Du+lPMIhHqWJtyyydwuo42AlqDPdfUTn5eaeDgc0NmOBCdN HdFnoEZB7cyOIVRz98QGznDgjT3/aG38xg09/0+Zgp9Fep4JRyOnEbUe4kszeIqDIojY XrOg== X-Forwarded-Encrypted: i=2; AJvYcCVHxYUXQo0fgeIBhtEigrMwxUU3DyHafOkLoZyY6qdyUCZ6uz3nR+mWuFl9gOnrL+29GlUjEgnhFXQy3WBkaCQ7ZT2EI9s= X-Gm-Message-State: AOJu0Yz36cHFoR0UssThoYD90PkkqcfO+xO5we7m9PbDpG0pi+0HIoO0 AvaoilTDyZF9x3po/hLxmnMx2zm9Txsx6i9Wlv/GGTemKkH+PVXh X-Google-Smtp-Source: AGHT+IHaC2gceeeJQruDY5QEsjbXZig7LE2ETEMxod6+WZ1HCBdh1KwzWImRgSvASowScyo9GBPaFQ== X-Received: by 2002:a05:6512:3f13:b0:52c:dc57:868b with SMTP id 2adb3069b0e04-52eeaeb5d8amr4001575e87.13.1721407127305; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:b0b:b0:52d:259b:377c with SMTP id 2adb3069b0e04-52eeacb4a2fls874116e87.1.-pod-prod-07-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a2e:9893:0:b0:2ee:bd1b:84c with SMTP id 38308e7fff4ca-2ef166214f5mr2378601fa.0.1721407124979; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=pamy7Fi4ghv/6AVkTTFSEYS1ALfuN/z09ikMhW7U3SRiTNB2O2S/w+zjsKSj2CCZ7k SGmWQWcGopxelCvI5rH69L+5z1+xc7TgWem1mV/4Acd5WOE+QG3AwJTAm6Fe/ZfgwBSo cbTtpf49xZSb+08on7Mk3gV1kzsWh/+LH5Uz2ysdcI3E6hiY9KPh4Vf+TVDm2qPhhFp1 vYNH8jn4Dv1FUnPHcGEtRKweNWgRgI73iy5S7nZwLnoGSOZg0GdaHIqT3Hpa6uUEFZal PG1c1Uvu2b3Fg2COFES1pxFe0WVY910iXSUa88mchSWzsiT1epXYOXr/27ZA6b8DmrMw E/8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=tur0DYltz4BuUK9fghA1yjJMijPlyfHlWAkmvVWKtn4=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=YR1f/i+xrRysNZxKzcftpZT6n1rsjBJJV9uuOTcUqL65nf4msVH82P4ODZKeoyuYzO l0Ya/99XleAQ8GGzNSOmQPqonIUTSoujk4A8s8GpJ2ypPZvBmXJ8cH3y0946nCrdi0G8 1R/DcQegGDsosB4FdUDUtxdLZLouduLCdfIfWZYj0jjx2vXSbra2goP/AOY9gLxDgdJB ZoffPBbiFwHNLSuxCHo7Ylu0TSOtTMKmM+ZeqwHLH6NUiHNGSQe3GNm21GLIDdhCsz4x bTA0ZcHUIuInbs+7SkqQxXazaBSXqW19/d7weGgqWrc3BGANKhekwzexC9u0kvwl5wG8 bZyg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2ef0fd30618si385501fa.8.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 20240719163844e28a1b2b4382c7a18d for ; Fri, 19 Jul 2024 18:38:44 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 1/5] Introduce fetcher from container registries Date: Fri, 19 Jul 2024 18:38:39 +0200 Message-ID: <82da88bf02bf928d8807bc93bfb5fcdeece1f558.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. In case a multi-arch image is specified, the fetcher will only pull for the package architecture of the requesting recipe. The image is stored compressed in docker-archive format and, wherever possible, hard-linked from DL_DIR to WORKDIR. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..0d659154 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,86 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import oe.path +import os +import tempfile +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + ud.arch = d.getVar('PACKAGE_ARCH') + ud.variant = None + if ud.arch == "armhf": + ud.arch = "arm" + ud.variant = "v7" + elif ud.arch == "armel": + ud.arch = "arm" + ud.variant = "v6" + + ud.container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = ud.container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = ud.container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.arch + "/" + \ + (ud.variant + "/" if ud.variant else "") + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \ + ".zst" + + def download(self, ud, d): + tarball = ud.localfile[:-len('.zst')] + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: + # Take a two steps for downloading into a docker archive because + # not all source may have the required Docker schema 2 manifest. + runfetchcmd("skopeo copy --preserve-digests " + \ + f"--override-arch {ud.arch} " + \ + (f"--override-variant {ud.variant} " if ud.variant else "") + \ + f"docker://{ud.container_src} dir:{tmpdir}", d) + runfetchcmd(f"skopeo copy dir:{tmpdir} " + \ + f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d) + zstd_defaults = d.getVar('ZSTD_DEFAULTS') + runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + image_file = ud.localname + ":" + ud.tag + ".zst" + oe.path.remove(rootdir + "/" + image_file) + oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file) From patchwork Fri Jul 19 16:38:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3709 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:56 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f56.google.com (mail-lf1-f56.google.com [209.85.167.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcrZ2002644 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:54 +0200 Received: by mail-lf1-f56.google.com with SMTP id 2adb3069b0e04-52e994d8e26sf1621149e87.2 for ; Fri, 19 Jul 2024 09:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407127; cv=pass; d=google.com; s=arc-20160816; b=kagLEtdFYGH6On2zThQhiZ4VZ/AFJqCFOeFmRRttYf4ELzk4i+RBkXoUTYtis+ClBo VzqxbaIU0KuD66pTWyXE0Xz3Ba37pf1eJ5tHQzLqs8q97IBYJ2Adswsfo7BQ1kPdKXNP k1Iy30GflH8+INwOC96nhDxerlIQ37L5NR0tpcdFPlHIxPpfLdtBKz+p9fPfNsT+BblW R1pdqAIl6eG1mum2eEEXLBdFzJ/Ux/wj7lbHTVO4YgVhab+7nPayuzzrTUBccOM+POCj IEM+U4Qe5Zrv88iI3dFw79LAr690O1BnTviP9yudm9mB+o/fRaUHKPYGgOZw3V6h1eND RPkA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; fh=hvgAxZX/N832QdoQH4xrYJAH0heQh+gSAr91Dg1B9YY=; b=sZArVWjjk8C58mxyx9goZ6friGYdTQYNSthRi89zPyJwmzUjVIyEcVm5Jek8iNyIwA KopzeZGz6a/VPH9OAvIo8xRdhV3zDP140ae8Z2Uls+K7PRjv/yos3aHK8b8e5J2krXsD h7FXrHkJX/OQiVqAx2iDMPloi/y2XVnB4Uc99WzCJ0E+3cg+m5gaNL+xVFfkws8nNg+M GnEdD3AL/vgMD2sPpdvJg8tLaqQMewM/9Yu0ERNUYSWzPHH47Bf+/uugtaYsT5pEImBY 7pVy4YoOunclsy+LSV941VCrRo8IhpxTqedYL3gOH/iClzXvmJG++h5cfQkc84KTBW02 kblQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407127; x=1722011927; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; b=hefXgHDDK5wv+BHN3aUGj1Y9On63AsCSABG0JfluIFprXPyXOrRTxjm6IcaQJiWil/ 6SYg085s+4TME5Z1cBvdx+BNLn83ZVOBm/0KGwNko4BMBQgZy6V2ePaITbSMQT6csVi+ C8NgpN+fzUZkGIVmKcTNO1bbUKMw9Aw4kM2Pcthn2O59MYx1fmxrZpxg7q2CVrGkdcNj hUR54yg7FX68OBRQey71DxAKiC8NRl4f44jBSajKkiUG6FbnTgTJYnPs/Pcm2uC4k6cq usg2C5tht46/DtCkeaU/mzsGwA3sYhtlqwK+MezhmHvAXsDHhz0erghQZMBkBBaDLaIr RS1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407127; x=1722011927; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; b=Wfj5AncYzaakOF/lXFBuC/94KsNYTFn2UOVt1TH4XwNC09Kq3GpqfTOerB8xBD165g uNB9hsjOW4RUyk8gCD16kkHq/PFpNaw0YhC92FKh0isk/2Mm+FApd6GhVXq8/wccdRjN d+MrGL1ZevfIlhI9A39lavkGzUd0dDRb0R1D/gMrDNztStBoUUbgvYYraA4/E4WfQgX5 +caT22MDXW6gFdXri5l0HGFvkfOreAces/g4M+w746z26BrlBqWA/UwGIrb0dwaTTg3G hJET/gQRB8obHhmXP0X8Z7m4OTrj4YlyzD75winaoc0YHW+N6Rd7//m6cZgYIYW5E9Hy U/og== X-Forwarded-Encrypted: i=2; AJvYcCVqR3hsd2D/dPAXqkD49MB1mSTrVxdB1Q5q5idsI1tr2tEWT7QivYmb3TAMJSMmerniS4dUL0ie3YTxTaYWM8nX4+YlPnU= X-Gm-Message-State: AOJu0Ywkl9TtGiSWCo2IvHyOogZNN9+wxrtQB/CIS21nMzHIkYMQcmHt vcwxjFJAoHdEEXYY8db3Ijj0NFvWojCqArE96QZywcQkrKsqfbea X-Google-Smtp-Source: AGHT+IGuNM6BLXiKYzgLddi66zjBoQN/AKFBVlqMqSXh4BoIcQSG03vKSyb01CqwQCbMBkQKmNfOvA== X-Received: by 2002:a2e:a404:0:b0:2ee:8c8d:d9dd with SMTP id 38308e7fff4ca-2ef16840f94mr2137721fa.36.1721407127003; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a9a4:0:b0:2ee:605b:3d02 with SMTP id 38308e7fff4ca-2ef04dda89cls12094081fa.2.-pod-prod-01-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a2e:9101:0:b0:2ec:440c:4e1c with SMTP id 38308e7fff4ca-2ef167824f8mr2177381fa.11.1721407124619; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=HFDMVULEFGPvIHdxCPEcDfqh3wlsa0ws1/6O5+k9Vj+r+jLZRC33QeSDrwJ2o53Jvp XJ8tuPKI2v1CDyHgpEcFo2YoFEsJG5IivYIn1KSicPHl2nkHAgRVyI5jWDn0ZHRmKAEN QhvWkHyJPk8REogjJnpvJeCM6Sw5XPvfq7pkgC+FkpDiIHnES3naaEDHa1pw0lGJZMkY qK6vE/zHHz5f4a0urOUhNdwwA82fp/Cek4yP/qJ3tA6OEZSSCTjv+xCoj+QO768ES5kX b3SmGsOJLtNMwWSA3cu2Psu2k6GHGQWv+rU9PDS5zY5rPXvaKD27V/xWQ9CdppZxLilC OORg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=YTKlvg2yRZslJX8lHMaY1WIwEOdoVww+2lzK9eEeB0c=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=JoYzezORfijh41zLWwr86FL+yCPEYGzs2/ELKXGOsIYMmpS8o5H+ul5S9axDXRFdP8 xuBykVBXjMycM/sJ9B7NSQfoA794kNbf1oE5I86x7FkecqQgkKpz/T7V9X6kObhq1EzM VZmX6MCQjcG+tce2w9f9qX9LY97SHe2vm+/cO1uNDUuODeqpj8x6embN+jIXtzRKEfEq kSbFcl6hy8UYr7MsBpIaqgm62FaqtZvzfwR0Z0PdF9Ik33/6INCsbnGYMkiH9upMIqVJ AwO6VkP6aENlxUQCnsbL/7I8o+0XkxpvMWr7U5f4BkSt4laIVhWw1MWmY9rX+oK5SxfF LLiQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427d2911f57si1635545e9.0.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20240719163844b8f4655d42b9dc012d for ; Fri, 19 Jul 2024 18:38:44 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 2/5] container-loader: Introduce helper to load container images into local registry Date: Fri, 19 Jul 2024 18:38:40 +0200 Message-ID: <5af6163750f7ae0cb186e52727afe3ced1db2ce2.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 73 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 +++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 +++++ .../container-loader/podman-loader.inc | 10 +++ 5 files changed, 123 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..5fd8d23c --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,73 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += "${CONTAINER_ENGINE_PACKAGES}, zstd" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + from oe.path import copyhardlink + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + tag = parm["tag"] if "tag" in parm else "latest" + image_name = host + (path if path != "/" else "") + image_file = image_name.replace('/', '.') + \ + ":" + tag + ".zst" + dest_dir = D + "/usr/share/" + BPN + "/images" + + copyhardlink(workdir + "/" + image_file, dest_dir + "/" + image_file) + + line = f"{image_file} {image_name}:{tag}" + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service + + # Do not compress the package, most of its payload is already, and trying + # nevertheless will only cost time without any gain. + cat <> ${S}/debian/rules +override_dh_builddeb: + dh_builddeb -- -Znone +EOF +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..2356e31c --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + pzstd -c -d /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman" From patchwork Fri Jul 19 16:38:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3707 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:55 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f62.google.com (mail-wm1-f62.google.com [209.85.128.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcq61002631 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:52 +0200 Received: by mail-wm1-f62.google.com with SMTP id 5b1f17b1804b1-4279b07cd45sf11963635e9.3 for ; Fri, 19 Jul 2024 09:38:52 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407127; cv=pass; d=google.com; s=arc-20160816; b=kl3etsSoOOPbO+IxVJMbntI42IlRYPul4YAaryQ35w2tuXu6t4Z5Vy66/TxtSme2lu 2+d9pfnNpOi+TY52TmcobBzXrP7R5zaixgm9pTc/EOlK0JnpbZYXz5QDzWjEgflEUVhg WRsfiEr9wWcYE/ird/8VfbuPHHXaFJTHhsXL/XIRYhi3ZP/pP0t9sjBXs7r3suXe3Zra 2dUbaqj6ARu8Y97rdU0lNlSnhwS3IP1yFEDA5wjZvkaA3kOzxycEaVoDal+byNAam4Vx 4OAO3McMdfYRz2yJdznf1ZtCDR4SMIlTJsIY6OVJsZyXharj8cxDTSho+VUXzd3BfcWM CKAA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/JEG2eHfVP6ffAkI6SG/Nc/dT/hWPXmNP73sPdZaXU4=; fh=kjnRHrchAD3ZqB3p1auudOMwh999eqixyFDZf/HFgOg=; b=mko+YRdAAK6cRLs6PsjWtUT8W0t4uQXqJGbvYEeSfqEaj22lURWiekiQQS/Op/uua6 0DEwK7pzNQw/mUqClB/HjzFtK4CZjT3kAq4+CnrUzyZeg0qU3AkZ/yd4MJVxV+GiRl2S GiJXpawINQziuymMyVsE90IQaOiYCPKuNZ7aeVWJ6gicGWAG+G0rXXAtNVw96G4tAyAe aS0pnHcHn1WftZhinKDdUPJFA1XXeKQNvwQa7HfgFS+nCNMIT2KsPMLOZ03fQryuV0Gs M2npIm2txNlv2ZQBw9pDnGC4Lh2v04RglXf+pSevQcC40Y/6JFA3YWfQ/JFk0XED0wHU D1Ig==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=Ck3i0xit; spf=pass (google.com: domain of fm-294854-202407191638443105857d543eccebaa-suvd_b@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-202407191638443105857d543eccebaa-suvd_B@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407127; x=1722011927; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=/JEG2eHfVP6ffAkI6SG/Nc/dT/hWPXmNP73sPdZaXU4=; b=YFigCRPA8TWnVVTuJf5GJAOvom4FAjCzl5Eh2+wGhhBnfygnwBATPvyBQQU9LNvO2o yyrjIctD8Pyq13TxyiY/ow9qRrZ5tiX4cAj54WSzV4KtRReP1H92mcydJAtWHKmYh3Wa Y0EeepMiIceljXnF99CuZ42jXLBRjV6/PMIK3rWxk7hDjupzfRGgen3vnd2qtwrS4Znc p6/C1FFBccU9y/J/5WvIqrqSNBWnMvrkDLuWRfvI1ex8KapfbKzqmfs74hbJroFQFgjI Z943V6B4W3UmdvlFL5oJKquPB+dLWQ1998Llq1N9detgF3xkcs/vOzW1t7LCsljcB9WJ PSxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407127; x=1722011927; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=/JEG2eHfVP6ffAkI6SG/Nc/dT/hWPXmNP73sPdZaXU4=; b=OQINejIiCpCxHRcguRcYdtoAvLcdVUTGUi4IpXdFQbr4Lv/wsrhNoOQzVcBszmJdmS KjHJLHhcnHn0Izd1EI58pRPjW/yKY4FjjQKGPxqVMH4z8bAapfCzsPZtpzHycXrScVuP yCk4cx3qbGFSiAvc4RuG807baz6d3ZP86RFqRn/gN3SwQNqrapp4fBHsHanS966ymA7U wxurVkdHTOhjwps418LDI8XccaQt/qYBRKiBKwkQuqGCI83qMCgrq+ypdf94CyVx7uTD OCihya24wZZh3l7iK1xcmaxodK/Fxf34yK4DpQgslaW4mNPWcxLza8ReMGySFUbDn6oA IrNQ== X-Forwarded-Encrypted: i=2; AJvYcCVwOalsAb3X26Iyi1ZcsNRH7xXlBwhPwGB7J6t28QvzLkeXWwtdl2ehA9wOc/NPdQpIE30vr4eapBlIylum9uChhV4MlUk= X-Gm-Message-State: AOJu0Yz4Wt0hGy2nFNPzL4fakOwvSk1LlKI0MZ3RFtDI6whlLw8jIhHU Te/BsPiqPgZU5lfoJXkU3Q6x2py0tmUZ60ZwTVPSG9B3iTga5Erf X-Google-Smtp-Source: AGHT+IEi62A+bhnarMQXDhA0dkbnDmCeIb0qmEfzkcINv3jlvieW2785fA6FjEulWNbTwYg+faoewQ== X-Received: by 2002:a05:600c:4fd2:b0:427:d8f2:332 with SMTP id 5b1f17b1804b1-427d8f206eemr10808335e9.7.1721407127149; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:4f52:b0:426:6982:f5c6 with SMTP id 5b1f17b1804b1-427c83992bfls14650555e9.1.-pod-prod-02-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a05:600c:3591:b0:426:6353:4b7c with SMTP id 5b1f17b1804b1-427c2caa7cbmr62338525e9.8.1721407124960; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=Q3gwH24s9VCH+dKuLRBnc/EvfJgEfgdsfphVoGKQKORKRdAIWBdXdignb51cbfHEA7 qYwde0U9SLm/6xrUpsKcVzsDwAZFcPTMeSvJPkXscBCKHypwfwk22qZlXABAyZxlT4xy 0ISTa6Z5Ab1IMsp3foX4FxHtEdcrInVA2BFVE9eBJ6D1RLDVjFDQmHgF4ilf7bEOJNh+ wCScxZn5xs90/PV63FCFHEmpxnlrPvDaCT2ntZhWhkvGFy2c0OZOC2zEDnQsXAGOJYi4 KodL5KBXPH8YvB10XgpyItsCxFYBwCTg9PieY5DG0sd+rgGbmkZlgW5oT7cSxBgjOrOX 95AQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=PZNATY6IoSLm+0l8pnfs+jH3Y4sl0XJy+TjIS1m4FCs=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=0iW3kBl0S+BgOt20J16edLQWXTBjMzZMCv/zBd661vOvKTghwmWyBJZ1GZxAEWY115 vvcaE7l+B063ijkaUp3JeqYS60xOJnCSCvcmWSUxostCEOLibJRbDnPzR9vPIXhZeHnA MKb9l3F2KCmRoTqnZx+5J8TWgP7Ph9MnOV05VC+iPs3uU1H7Z86EL5Hxe9MobkB+NRKn wLSUSYzRQ0HHz7yKnheiNS/UZmz3Cw2+2scPV/p5m5p2qsv87si6YXg5OvdN1vFsJFjW SHypbFOGDHIJHN4F5ICzKUUA00sAne2T0YsgMhJwxCBQBsv2def1ppiLK79MVQPMly1B AOdQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=Ck3i0xit; spf=pass (google.com: domain of fm-294854-202407191638443105857d543eccebaa-suvd_b@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-202407191638443105857d543eccebaa-suvd_B@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427d2911f57si1635545e9.0.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407191638443105857d543eccebaa-suvd_b@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 202407191638443105857d543eccebaa for ; Fri, 19 Jul 2024 18:38:44 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 3/5] meta-isar: Add demo packages for installing prebuilt containers Date: Fri, 19 Jul 2024 18:38:41 +0200 Message-ID: <27ef7f39f79a72f188f78637f25ac8a616f0947a.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=Ck3i0xit; spf=pass (google.com: domain of fm-294854-202407191638443105857d543eccebaa-suvd_b@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-202407191638443105857d543eccebaa-suvd_B@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka One recipe for docker, one for podman. Both pull from a registry that, in contrast to infamous dockerhub, should not throttle CI jobs running these frequently for testing purposes. The podman variant of the recipe is intentionally leaving out the digest to trigger the related warning of the container fetcher. These demos also come with kas integration. Signed-off-by: Jan Kiszka --- kas/package/Kconfig | 19 +++++++++++++++++++ kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++ kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++ .../prebuilt-docker-img_0.1.bb | 12 ++++++++++++ .../prebuilt-podman-img_0.1.bb | 10 ++++++++++ 5 files changed, 59 insertions(+) create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb diff --git a/kas/package/Kconfig b/kas/package/Kconfig index 35ba7cf1..395c3a08 100644 --- a/kas/package/Kconfig +++ b/kas/package/Kconfig @@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP default "kas/package/pkg_sshd-regen-keys.yaml" depends on PACKAGE_ISAR_CI_SSH_SETUP + +config PACKAGE_PREBUILT_DOCKER_IMG + bool "prebuilt-docker-img" + default y + +config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG + string + default "kas/package/pkg_prebuilt-docker-img.yaml" + depends on PACKAGE_PREBUILT_DOCKER_IMG + +config PACKAGE_PREBUILT_PODMAN_IMG + bool "prebuilt-podman-img" + default y + +config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG + string + default "kas/package/pkg_prebuilt-podman-img.yaml" + depends on PACKAGE_PREBUILT_PODMAN_IMG + endmenu config KAS_IMAGE_PREINSTALL diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml new file mode 100644 index 00000000..df96a484 --- /dev/null +++ b/kas/package/pkg_prebuilt-docker-img.yaml @@ -0,0 +1,9 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 + +header: + version: 14 + +local_conf_header: + package-prebuilt-docker-img: | + IMAGE_INSTALL:append = " prebuilt-docker-img" diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml new file mode 100644 index 00000000..d0b8da1c --- /dev/null +++ b/kas/package/pkg_prebuilt-podman-img.yaml @@ -0,0 +1,9 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 + +header: + version: 14 + +local_conf_header: + package-prebuilt-podman-img: | + IMAGE_INSTALL:append = " prebuilt-podman-img" diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb new file mode 100644 index 00000000..0dfc9b8f --- /dev/null +++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb @@ -0,0 +1,12 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require recipes-support/container-loader/docker-loader.inc + +CONTAINER_DELETE_AFTER_LOAD = "1" + +SRC_URI += "\ + docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \ + " diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb new file mode 100644 index 00000000..e671a494 --- /dev/null +++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require recipes-support/container-loader/podman-loader.inc + +SRC_URI += "\ + docker://quay.io/libpod/alpine;tag=latest \ + " From patchwork Fri Jul 19 16:38:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3711 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:57 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGctMx002668 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:55 +0200 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-2eee96e7dc2sf23564211fa.3 for ; Fri, 19 Jul 2024 09:38:55 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407129; cv=pass; d=google.com; s=arc-20160816; b=U2Toxa3MBfbGnB8kysY6SCRmrhdFq+slJyhU6oDJLbw4XuLpVs+5WCrOhjcMhKjy+X tuuEOpAlikcCoiWSW6D+iQqA8inWYXzzmjYkcV/5905iYZWufv6vjm+c45YNpmmskhY4 +W2FOGDQ9M9nzycnjjhYmNVQEJvgeWuASN5+SQLpaVlBMaWB7hPFg4HJfNyUxgVg+cVy PnnyE8ft81WJLisKax9+cuGaB0MzjajlxjinULtX/B29EgZInT94vutLhhJWhb3memi4 LFcFXsVq6QOeZSqjIwTP1Ia/+XPckv+gF7fDSneqCb7h/uV501mU801Yt1k3AHHiC2bK ORSg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iVamLCibBbm+jLmezxD1KFNpiI4T8oUm0dAJIGt7jPY=; fh=8/amcpiP5Csn49FhTE9eM/7XgPYhOR+I89/n8VtgeFc=; b=pgGxqJHh692Qg9exZ4TEG/52CWuxGhj2dsK/HMhRA9RntLCxWqMR5Px/yJHnWo33Ve 0Xu27FmWT3TTcCABXF8onD7bu4KRBp6K3tKfig97KVB42rS/dvsOs3lTX32gsazC9won giVKEdwpUX4h0akrippiv4hvyMWW8hfAzN95vURQIeQn1Snbik1xUHVFxfomoVHN1kaZ Epojh0vviGvI5c3Gr0Xie5Y1L36u2vu/apEDUdfsPrKF2Uzlz7Wlk/JGcJkyHiexEcd8 KohbEwRfi9mEto2DGYn3MtiQgZwmomzBOy2mV+S8ZxPob+da+l1eda3mjd/oVkQ3cVuB EEzw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=IoRGKb1x; spf=pass (google.com: domain of fm-294854-2024071916384469d3cd1665441a659f-o8sqr5@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024071916384469d3cd1665441a659f-o8sQR5@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407129; x=1722011929; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=iVamLCibBbm+jLmezxD1KFNpiI4T8oUm0dAJIGt7jPY=; b=a71ewnZXCnEjE7Rvu8MZhwyhSXH7dUh1s+gULB5uhOBuA/jtmqwOm4OVne8M8ptjVK xAcr4xvCJeeQfIofKzw4aWHHb5PApiSQLBg+QW3Fsd0DPKkxEwX4TLSPdO3JBfQ30pob QUmz941NTyEC2I/DFwhTCJP2be1G1zwBvMrI+xtEwfJewjys8JCYpEORhYZ6kxXMT83s FSQXZQAnWdWswDFw0lQBKJUNlXIqNcOImldbBM/1FDWkWp4F1wslzy0QZK4Ahl5r4FzP xBfI7FFtv4JmjvJ5b6V4t17pCh0+ATlJouRicTh2OxBiJqNjd5rRlD6mcJbbhp2LoteV Mh6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407129; x=1722011929; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=iVamLCibBbm+jLmezxD1KFNpiI4T8oUm0dAJIGt7jPY=; b=FZNUPR3DwI2z4ICSnhTY3XfGnc1zGE05Jm6bvaxTNqrpiN+ja7ac270q0jgeNh/hAc 466oc3lpvGc6Fm0YLI5sT1aVFE4cVXpLCpmPuM3yPMlezqJNzmtgt1EP79NZH34EPtS1 PDAHKkRquHogu/E0WyN9EOO31d/UWicozL8NwUxDl+FAsACM6SKlZlPw3w3emXCJOEDl C44QdTC7BbCZdh1ZIJP9NMm2fe2RZh3yIyqNfp/oqDsx2vbP4RxoTRe3mt8scTAl9Dto 3TUpeSehxPAzUzNm+bz6trZ4XMyQGLmcmFDa22GcKLE7aFfdzCaeNGjuL0GdfdLzu26T gUQA== X-Forwarded-Encrypted: i=2; AJvYcCXe5MVYE06w6yLlDqkLQ8unfeWjybHiW+gd6IkXNpVuxsDlfkaKiW8do7aoK5ELWG/RVLUhnacMEGjt3lj9g1vuvqLsr0A= X-Gm-Message-State: AOJu0Yw4FMeduyYyrob+DGEguaTyRVt8+Gh20MM/jJ2PTp8wDqhXPHng 3bTF65QWYjtZY9TPzpOBZp7XGibVyTSZUckwnhUn+/pCUpMxAreI X-Google-Smtp-Source: AGHT+IHMInWCuUS/Zxho/YCLYAF4TYjM355Cg9gSf6lRmDeAkJLKYe4LjKMtiCFYZ/ynJ89ZgR978g== X-Received: by 2002:a2e:92c5:0:b0:2ee:7a3e:4721 with SMTP id 38308e7fff4ca-2ef16826a21mr1679131fa.39.1721407128307; Fri, 19 Jul 2024 09:38:48 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:bc18:0:b0:2ee:8c6b:1209 with SMTP id 38308e7fff4ca-2ef04cd2b5bls11699941fa.0.-pod-prod-04-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a2e:7003:0:b0:2ee:8dce:2f94 with SMTP id 38308e7fff4ca-2ef1679c9d1mr2075561fa.27.1721407125280; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407125; cv=none; d=google.com; s=arc-20160816; b=pcJYzQZWlDXvNl2298ttSH74SX9CFcXJD32gQZsqBCTtfoHjh3udZKj3yajVnT7Hff dJSJt6OYGejrHxtOuRoQy43m41z7RPkYw2DImWhLQEfiVUrJyoX8qwoSm/TMQlPefJuR ZwnGTxtgvWXNfyaAZ8/Mek5UhsZFrYIwebQDBi+ZXPzIyh4zmskr0RH7Or/2mmx9PE3f Gx3jDi87k7th+Z4VFACpwICFV5JJNwczpN5yob+DuB2SjkjcH4+TqZm1vOxS1oqukEIa AP/tnd2DP+4+KCOGdZDLjVil3fsiFxA/UhPEnMQjDb4Zl2eAvJBcmutJJ1N6fNuF82OY ZgEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=5Q8nkkkjzpXg64IKufplF2I9id/rT8Gipiy3rjOvdQU=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=XbRE/NaR8PXquXbKL+t0BcL7zPDljGhljxjZkt+a53/Q3SY/fR/yCAk5jCu3Yd+Ma9 z53gW1vplS1VHu5ZNIRS4UdL9DhFXJBqKigkEL4xjBsWnJ40cayq3OokhHgIrg2RQgMJ l1lJzehr9kFAM+ckr+WRksl17UnYqPfL6H/HW5X0ZET3qAErfMcS1W/tQcosw8RbV9kL 8p6K+oujItJDzKNtGvZWOKuunm8FLODqMU4wFn//cxexLsB2P7Sect7wT0KB7fWE8pbq W/mXqr6zrWyW4VhaNzKl0p8dpj3CPdlFFUS11K6cb5C426PHcZCLv2a6r31er3xqJUf/ 6Jnw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=IoRGKb1x; spf=pass (google.com: domain of fm-294854-2024071916384469d3cd1665441a659f-o8sqr5@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024071916384469d3cd1665441a659f-o8sQR5@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2ef0fd30618si385501fa.8.2024.07.19.09.38.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:45 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-2024071916384469d3cd1665441a659f-o8sqr5@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 2024071916384469d3cd1665441a659f for ; Fri, 19 Jul 2024 18:38:45 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 4/5] ci: Add test cases for container fetching and loading Date: Fri, 19 Jul 2024 18:38:42 +0200 Message-ID: <3a0b76b54aca9cee33d249a2df73d233edf08708.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=IoRGKb1x; spf=pass (google.com: domain of fm-294854-2024071916384469d3cd1665441a659f-o8sqr5@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024071916384469d3cd1665441a659f-o8sQR5@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This plugs the two example recipes for loading container images into VM-based testing. The test consists of running 'true' in the installed alpine images. Rather than enabling the ci user to do password-less sudo, this uses su with the piped-in password. Another trick needed is to poll for the images because loading is performed asynchronously. Signed-off-by: Jan Kiszka --- .../recipes-core/images/isar-image-ci.bb | 2 ++ testsuite/citest.py | 21 +++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb index e5d51e6e..9133da74 100644 --- a/meta-test/recipes-core/images/isar-image-ci.bb +++ b/meta-test/recipes-core/images/isar-image-ci.bb @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys" # qemuamd64-bookworm WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks" +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img" # qemuamd64-bullseye IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz" @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL} # qemuarm64-bookworm IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz" IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}" +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img" diff --git a/testsuite/citest.py b/testsuite/citest.py index 8dd907d0..539c9440 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest): self.init() self.vm_start('mipsel','bookworm', image='isar-image-ci', script='test_kernel_module.sh example_module') + + + def test_amd64_bookworm_prebuilt_containers(self): + self.init() + self.vm_start('amd64', 'bookworm', image='isar-image-ci', + cmd='echo root | su -c \'' \ + 'PATH=\$PATH:/usr/sbin;' \ + 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \ + 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \ + 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \ + 'podman run --rm quay.io/libpod/alpine:latest true\'') + + def test_arm64_bookworm_prebuilt_containers(self): + self.init() + self.vm_start('arm64', 'bookworm', image='isar-image-ci', + cmd='echo root | su -c \'' \ + 'PATH=\$PATH:/usr/sbin;' \ + 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \ + 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \ + 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \ + 'podman run --rm quay.io/libpod/alpine:latest true\'') From patchwork Fri Jul 19 16:38:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3710 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:57 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f56.google.com (mail-wm1-f56.google.com [209.85.128.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcsb3002664 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:54 +0200 Received: by mail-wm1-f56.google.com with SMTP id 5b1f17b1804b1-42674318a4esf17640785e9.1 for ; Fri, 19 Jul 2024 09:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407128; cv=pass; d=google.com; s=arc-20160816; b=ysx2nNgLQGn3BaPTba+soI7309nBAM+r2fEfvCH4GDFKxeG8SqUvy8vrKWJl8FQ4Mq JZ956yGu6WWQ9+uzlWncyWTXza94EpvD951syzOJKRur5E2VYmgmvvwuXbi1caZsd7Fm xl+x/o1Loe/vWZE/92AvBpcW9JRlpuEyW5LkBBqwbsmVEtqon4NIOL1idthifCScxFba 49X1VgM8LdCYXd0sk39+847WlBvndlskZW3gysTP1sE7WsM6m+/q5v1LoycsdggqIXPK RT9kQAnMnFapB8xoO7NS/vfdfBkZmfsMA2GCy7S50ZGIVqvrIPWqImKf3lpYs38Sbwsi c78A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kIOJcMC6x4IWUMbuxMej4F19IxCf1XU//W3pVYaFPZY=; fh=I8niZciwQIgaBobYet8LxBT82Tv3W3lk+dSa0wPGKDI=; b=oCt24BkHf0nv6Zi6B7EOCLr1onR+wL1E7A8sGx8VH1forp7AOx+cbsvU1GMszlG86Y F3eJsIGY55YSVH4bMi9Y+bewJ3jRq8UpBgcXvPTSmuY/kZg75Alq+cRTq5ncme54c4fm MPWpPbhFUDRYiZzUsMQpJVWRcdE7jRlo07MiYCK2ORdxDW3KFq2ZZu6FWGBJQimguF2/ GIzMqomtHcssyMYqX8WZ4YNys2kJKU+cAdqTV1gKaMOU2Q6wfvH+zG2BxYgpklLoQrJM FG/cDuQ3VlmVNnuBJ9fdSzATvdwzpSY8uRiUgltmwSxFN0ZPnW0rIgnfHa6uONHF60Jz ha+Q==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=D84FDjb5; spf=pass (google.com: domain of fm-294854-2024071916384593b534507bf250d812-2vt89o@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-2024071916384593b534507bf250d812-2Vt89O@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407128; x=1722011928; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=kIOJcMC6x4IWUMbuxMej4F19IxCf1XU//W3pVYaFPZY=; b=g8reSTe4SLH1ZrBrBHFK2pOKJFzPPnGdqAzPXrT74jdsRbB8ZEd0kZGM8+uEDWEY9w clQWiXl4Y2v6OsAVZXgs2ZfU7zpqLPoGIrkdOwQm9iGpmXKugSds9WlXtZnPLpqvMc6j ux1yLFP+YY8bOpCfbF9EfT6JiZXKx359h2dfaKdKXh6gRB19bPYrnjGIF2U/XTYtnAcI u2BRnVlrpX1/oaVmk9TJ+hd/4XBWdpAapvddvpCqYZhgeqAjTTDoxlQGQHRrzR4Lh0E+ JDk71D75XLX2U9w69+lEkXUnSPMJByfGNpYoIHaIOmOFq6mkgZYEhoQWzFHETywlOYNe zzSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407128; x=1722011928; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=kIOJcMC6x4IWUMbuxMej4F19IxCf1XU//W3pVYaFPZY=; b=kfjHPcvlA7CbQVD+3Nio9PIWTgUCkx7WMfnuY8RDFGPDGwGIxRMU5UzlMr5WSscmoA luaL2TNEMWxoUof2obQDouv7lo2UKxE4zqN9FViWyxiQeWodGdB8DyBIi09oy+unZ+so q0E4NJTGlkoiHDS301FiOCc3OvLTdzEMO9AOI0o2goaX2rzug/8Tdtpx7PxBoALXyLGC Kh2J54OJJPv9znxnD24aYbl3hK9+n0X6Yp6prdLH0PAaptH5M1tcmenuXN3ylgN7RuFn wyT7zslyq6wc+gHnuzIIIpSZyw2eMkIorsTsZ3yfzk6rJ3vPgEHGpx/YMkK3l1eGbBpP 5SJg== X-Forwarded-Encrypted: i=2; AJvYcCXoIbWsUkEoE/9QyktBFnHydg0SVkroCnAnQ7usqKrwkoKF6BdQeTX/1GqY4rl6+4vqG/7B/onc3QmK9teYb5sO+2KDwY0= X-Gm-Message-State: AOJu0Yya7qyGRWgCcXdvloaVIFffGGdfa+dJ8d4rdZm+MEeAaN5voOqO BcPp6U4T/z8HtBY959bsoa0EyfPw0R1kjQiDwgzo1dJ+EXgkygDk X-Google-Smtp-Source: AGHT+IHuJmqaS5TiErZmy7geCYluS2gxM6X9T5FXRyMqtbjfVE0ZaGzTvScMYkdQB+zryVyYEwYxDw== X-Received: by 2002:a05:600c:4fc6:b0:426:5e1c:1ac2 with SMTP id 5b1f17b1804b1-427c2cacf29mr76892525e9.8.1721407127609; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:510a:b0:426:7318:c5a0 with SMTP id 5b1f17b1804b1-427c83a94e1ls15034325e9.2.-pod-prod-05-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a05:600c:1f07:b0:426:5b44:2be7 with SMTP id 5b1f17b1804b1-427c2cb865emr76412785e9.10.1721407125406; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407125; cv=none; d=google.com; s=arc-20160816; b=KpsfubUp8pB3qiO4V2s+yjwTd4gqhSMEdMkf+9mIVrpjJ4yALyoFz7KIbvl6Hc6uZb j5zHd9s635uVipFsV0mCyhNj6GWwODcf/A24QpMXaB1fOzF3BrLxFK29eIcopt5U4/XW hOgNtg7F80ehxuwdSKxYUduEe6Yj4NGUYAijJMBUjKOfzwM7v+tHjY5/g7u1eQj8g1zW 5zBbbTh4ccKQRG/sltCI7RheJ8b0+CCUX6nOAdZbg/jvI/PueZvzoK5749Y/1RDj5Tw1 GEFxLKbC9rLkT8YLz3KMA7Wstiwvn/zNav53mMISbzVn+qbzEj7JFqVxgxCxAwglTV7J 3I2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=BVG8ZbQIbKYYng1kSStiENS2JGVTDaSoFvKjxSoBf+Q=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=R5kIF5Y24g57iZiWqIqiQhYs7GKaIQYrDUn20oHhfeE60+aBNdZKvG1Uu9K9dzJbBv FDxdlWqNZd1N4vQw/edHqtoMjXubHZQtEcPTtosVAB0e/I3nN15rcSrxyS2nlP8KOQhb kdlyDtEVGuk8patVfSnZadQBqAg/hYdwuvNKytFuI3gkGOeh2AnLF9vWdpquYLynfQqY fRbNv4QVEyGhCq41aWV4A9V8vh16wAognjPGJ1BJ5KUwjelAXiCDVmMxyIhNL3NH591/ 4EiLw7vxuYIyCl5IQwi2Rm0kF6vjlmXnKG3rs7EpUgLGvcoIHBZC29MKfg5qS5wFtiZe d3UA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=D84FDjb5; spf=pass (google.com: domain of fm-294854-2024071916384593b534507bf250d812-2vt89o@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-2024071916384593b534507bf250d812-2Vt89O@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427d2911f57si1635545e9.0.2024.07.19.09.38.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:45 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-2024071916384593b534507bf250d812-2vt89o@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 2024071916384593b534507bf250d812 for ; Fri, 19 Jul 2024 18:38:45 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 5/5] doc: Describe how to use the container fetcher and loader Date: Fri, 19 Jul 2024 18:38:43 +0200 Message-ID: <049353103858d43105d45603619a7548f2a29579.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=D84FDjb5; spf=pass (google.com: domain of fm-294854-2024071916384593b534507bf250d812-2vt89o@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-2024071916384593b534507bf250d812-2Vt89O@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka Signed-off-by: Jan Kiszka --- doc/user_manual.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/doc/user_manual.md b/doc/user_manual.md index 776ae52c..2bdacbec 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -1519,3 +1519,61 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "" Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = ""`. To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`. + +## Pre-install container images + +If an isar-generated image shall provide a container runtime, it may also be +desirable to pre-install container images to avoid having to download them on +first boot or because they may not be accessible outside of the build +environment. Isar supports this scenario via two services, a container fetcher +and a container loader. + +### Bitbake fetcher for containers + +The bitbake fetching protocol "docker://" allows to download pre-built images +from container registries. The URL consists of the image path, followed by +a recommended digest in the form `digest=sha256:` and an optional +tag in the form `tag=`. A digest is preferred over a tag to identify an +image when fetching because it also allows to validate its integrity. If a tag +is not specified, `latest` is used as tag name. + +In case a multi-arch image is specified, the fetcher will only pull for the +package architecture of the requesting recipe (`PACKAGE_ARCH`). The fetched +images are stored as zstd-compressed in docker-archive format in the +`WORKDIR` of the recipe. The name of the image is derived from the container +image name, replacing all `/` with `.` and appending `:.zst`. Example: +`docker://debian;tag=bookworm` will be saved as `debian:bookworm.zst`. + +### Container loader helpers + +To create a Debian package which can carry container images and load them into +local storage of docker or podman, there is a set of helpers available. To use +them in an own recipe, add +`require recipes-support/container-loader/docker-loader.inc` when using docker +and `require recipes-support/container-loader/podman-loader.inc` when using +podman. The loader will try to transfer the packaged image into the container +runtime storage on boot, but only if no container image of the same name and +tag is present already. + +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images +remain by default available and may be used again for loading the storage after +it may have been emptied later on (factory reset). + +Source container images may either be fetched as binaries from a registry, see +above, or built via isar as well. + +### Example + +This creates a debian package which will download, package and then load the +`debian:bookworm-20240701-slim` container image into the docker container +storage. The package will depend on `docker.io`, insuring that that basic +runtime services are installed on the target as well. The packaged image will +be deleted from the target device's rootfs after successful import. + +``` +require recipes-support/container-loader/docker-loader.inc + +CONTAINER_DELETE_AFTER_LOAD = "1" + +SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim" +```