From patchwork Tue Jul 16 14:18:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3701 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:22 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIL5S013354 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:21 +0200 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-2ee8ceb0852sf52097191fa.0 for ; Tue, 16 Jul 2024 07:18:21 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139495; cv=pass; d=google.com; s=arc-20160816; b=BL6oGkUNJgT7BwhJSCdHgNdHI1EfdRM7OWUIgAcz4ddcO/8KFWX/MuA2F1Q6/JRfPg HQ1RN44zxsypvnMerD5F07Fn+ZfwMEF3tXfDfmHmX3G7D9jLNyMTeEJ8GSwkTWNthqJe fUitBq+mKpOn1okyoGe+RBw61iwLGr/nwQHesH1t51fdTVK6A/IuwzhoCUwizMBN/xKj XLShuE22OYd7u17NeZ6sRrSgTOkjWjeJDc5teZT3d88Izrhv7cYxKNssyRK3cdHq2OU9 X0G8nX9CPqJddT5FJLJ+pwLo/DzjPbw1k7oNZDzFgtGsS9Sa/3tRNuF4xMby0ZDASsMJ 2q3g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xdIictTAYnYgwEr3bkpnkim8XzLLI9JCpIo5FXw+m+Y=; fh=TcHzqIaBlqsn6gYEnG2DFOCPZkIFC7b96dPD7HupqJQ=; b=bOS7SBJOQHGw5JnWzYvkgt5HHwWG2um3E+0EVXxIRLMoqpQv9s4ZJlwPPXnFwvaZbb na7pqJgYNxSe9vQm/ZL8ShVWg0SoUKu62EuCel9oeXKo1fkhrBCB/zz1zOFXwkyUwqlz cH7NPDHqJ0696MVdxHZTV8PQ4vTcj4kBf9hmnq3HePIKTj7qtuKDUCsV9yH/2dYgQ7kQ siq+Npgfx26PCMjbFAFJHnMGQ2ITL9Bsmjujr/Ror4zYQCkXCIbgcDIlYRLa8nPWwy7m bEYboKSHZFMd3Ka/Ynk0+xprE6wHK0AW2pzlVB3sjNvi+ZEeJoJhJJOkmcP1Mp/nTMYx DfpA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jUMoGlY9; spf=pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139495; x=1721744295; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=xdIictTAYnYgwEr3bkpnkim8XzLLI9JCpIo5FXw+m+Y=; b=B10wck7gHwGK6QDZMQLElJMCPoeg4f+A3dAoxFlZ/rv6v/STUCzax9Q9zxUToHcS1l 8miP+zDk/mUOT1lXLDH9GBULH9V+kETWrH+taIi0CaKxxnKsHrJqPGACmg8TM4O3Y8MF WO9p7WN32SPszmjvHvWmMKK65pbCp0oa84Xa3dPWt83/K5ezrDN5I+0mZGTV5cMTD8VR LR8RnMrDBoXLCezi+VZNkghPVnWQzhG0ydblCFWlE0YkmwD/sFLeBVlxU9aqdp2M/rwz dv+yVLQc1NuGlUq5RVuRYVp+ExvkuiB2f2tm9+M1b/PzmGGuvKbOcGOcugPHrjxAjSfF JiXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139495; x=1721744295; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=xdIictTAYnYgwEr3bkpnkim8XzLLI9JCpIo5FXw+m+Y=; b=rkBB/IKYtnzcNfjBhSL5l2tC+V6jHklLHlC1tsR1m0utQOkDQPi913VaU3vBwD+Sdx uJKw+xnAubRb1WOCjwfuf1rwsACbrLJKTt5xGphZhWeT5j3ofJ4Cq1z2hEdGzAfOPHRE 08boDHhF7nrEzxirBWppGgU8zHpQHH64k95c1ZmEGM1Wf941IEg5PA59olpK+dhhrOB/ zIkFCNZv/f+mkZ+pl0GN/4cgfPgoJHPUclsjinqcLa3VMcJrB0n7AjSsvhh7gemYIwD6 /63taGYzVtdgcC2QKdlG539mT+dPbnNVMJmmMeqNcMMdDkEnOojKUMz7PKWyfSOcjXPo pR/w== X-Forwarded-Encrypted: i=2; AJvYcCXL5b8IL07Ns/+1w6gd9Rhi6clkJu4//UhV1XmZgFFkrfK/ze/lPBhjohjFH4Y1sUdj7lip2g/bT5lDbhxU4jkqUcKOqcE= X-Gm-Message-State: AOJu0YwiFCTOR1FKgKnvhqkZEga8bD6OgK7QCE2cpJdYXueNk3wAeIt0 oFJhICQmX4LCph54g6FHui2hDrNx7USJsRhz6PF6T6UsIP17SlJT X-Google-Smtp-Source: AGHT+IEkut5IBHADmdpr861edZmn+SVVXfW7QAtvZJo4vZakDZuRC5RMo8c0b7+Y2vA6li/x9c3QCw== X-Received: by 2002:a2e:2c1a:0:b0:2ee:4ab4:f752 with SMTP id 38308e7fff4ca-2eef41e333cmr15975261fa.49.1721139494269; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a481:0:b0:2ec:4d9a:63bc with SMTP id 38308e7fff4ca-2eec93a6e39ls20316771fa.1.-pod-prod-09-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a2e:9b86:0:b0:2ec:50ed:b59f with SMTP id 38308e7fff4ca-2eef41d739emr15106311fa.34.1721139491779; Tue, 16 Jul 2024 07:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=SO8CLW7mDHFvMWEOczJ52pyNKbXt8vDHHx5/DHe/2gSyk2kU4saodLYptAjOadnvMM ixzq/TYUd8YwSYd1bBz24hV/bZyepJaT7yKhyiXMCMzd6V8DuTA5NcW+RZ5Q70avZSVF sekQtVO6qY/paMfwbpS7iARCNxXdMZAqghdECrUUjVgodtISeuMY+hkrOSwjrEoHp/j+ cvHKYZz2xDw6V2HbcovnZrWPfzggQ6le84P4skHr9ucQYB/Q5mL5rgrv1TFmV5WLsHQj HBFGG5gV8XdZY2Ve3nIAWDpy4PKtxPsi5vLcZaAL1I4/BAyanYGcDGoln6DgB5sv3fOv e/4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=qT0qBD6zoZX/tPGQ0GTumBOagxBAE1nu2vnVszdRf/0=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=haMbP2oP3AdlVv5UIJ/OL9ViK/qqFxUha+2FbQdGu9Os+GpFGzsc6pquvSgHh05l3y lDw/Xjvb0araVsdRbuHDcw3MJw954V4cGLaXN2Js7LZM+YHNGLZKEs4hZ7z3ohZFVDsF DwWarwp/AwFpJrUBnHcHWgT2lIlcGaX6V7wmqF/1LIEwFLPEvawf1kjqcXwn0iD1PsSk bYK70QlJpveyPakZPwCW3nmFaMijCnkostATkqu6wmeStZ4bvHlfOOrr27pm+E40mPtz ywoYgWdlxgBdeduGKCOpjluZRaYmeYhMhCrbtGS4jvpa0C9XoJe6UEdBh13zDWHcAc8x X+yg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jUMoGlY9; spf=pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2eee19149ccsi1386831fa.4.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20240716141810bdb1dfc2f7c434762e for ; Tue, 16 Jul 2024 16:18:10 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 1/5] Introduce fetcher from container registries Date: Tue, 16 Jul 2024 16:18:05 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jUMoGlY9; spf=pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. The fetcher will try to pull all available variants of a multi-arch image. If this is not needed, you can also directly specify the image digest of a specific architecture. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..8513e246 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,90 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import os +import re +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd +from bb.progress import LineFilterProgressHandler + +class SkopeoProgressHandler(LineFilterProgressHandler): + def __init__(self, d): + super(SkopeoProgressHandler, self).__init__(d) + self._fire_progress(0) + + def writeline(self, line): + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line) + if match: + state = match[0].split('/') + progress = (int(state[0]) * 100) / int(state[1]) + self.update(progress) + return True + + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + + def download(self, ud, d): + progresshandler = SkopeoProgressHandler(d) + runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}", + d, log=progresshandler) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + arch = d.getVar('PACKAGE_ARCH') + variant_opt = "" + if arch == "armhf": + arch = "arm" + variant_opt = "--override-variant v7" + elif arch == "armel": + arch = "arm" + variant_opt = "--override-variant v6" + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \ + f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d) + + def clean(self, ud, d): + bb.utils.remove(ud.localpath, recurse=True) From patchwork Tue Jul 16 14:18:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3698 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:20 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f57.google.com (mail-ed1-f57.google.com [209.85.208.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIJEJ013324 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:20 +0200 Received: by mail-ed1-f57.google.com with SMTP id 4fb4d7f45d1cf-58c4f94b57csf4332391a12.0 for ; Tue, 16 Jul 2024 07:18:20 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139494; cv=pass; d=google.com; s=arc-20160816; b=ykzF9KwW/y1RESn9zjn2E64tIrjEDbh39FDjj0ud8Nr/844HTw8F8FWnyABT2cdZzX HWhVasUOVF6JKHsFFLCj1yVyWEiwiA16FnoRJrz2tlQH7a+XrdrSOrJD7hc52iPxKwfV 7s8aRBynkE0i78WV1cl7+N9bLk79NSpxuGxoZoSwLmTLDWvMz88VN64+IqqXHIPhEpaL V4AEuqE4VD/z5MKfHjT2x1O7/a/k2e2d/CoZh3j43tfMuX9zdVf/eS18mg5nt7fMNPpW wfvj++7PZdQwq1kDmUho89PoZtjsA61nOaGVYr4tPFKCl19eVkSMo2yvDyT8ZH/drhGs TwMA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; fh=1BDurDXYBFmfUnMK/x5QtycmSEnKbZv9ZRUufnZDkc4=; b=PRJnzz/xTWdHYMiV1rJ7knMN/5x9PmqSn57qnRhVOHZCLS3bBxts/LH2teKjrGwkjk Pww3Qf9GvP1RXiFP5h1PnkzcmiOZLaEvdpaDyzNDwn53tyJg3pI+p0ReuO+PZyZDxLGO MWMnNMG70jfOsdHjMRAOMEiPVQ5q0592lHDt5hf7i5VW6vXcsFlRsVNB8Qut57phweKi +UyUksDaAxfW+JSwr6PU65RWJISN0suZ1A47KXd7AD1eXvJWHYvl78CQ+92XyMUzyLdP h0vPdfdIIvxnpRdns2vOL0Xa447kvZU6xv8/AkI4VtDO75Bd6tq8tndWdgb7m8bEgiFV n1eA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139494; x=1721744294; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; b=o0MQQjPmBLtOu7FoXGheP53bMPGc80xV8/YkU5673wBRkKiZdODr70TVLo+SZkrLL5 tks74xDyqdpR6BWZ0Q0Z/Z0aQYNk+gE59dXszpjgrVufB+6gpDir0hxFvGjp/t4Hf0Jo l0gtOTQpIalycA4HKOb2xiebnLw9wjRsFkoixTRQqQ8UifcI2SfFj2GebHj72RoAodDx a6XFlSejnYJQ3hs4OuIrRDzA67xeXm0boqDq3l0/cfb0OfWAi5gbpYSV/wWMaGP5MRcz rB5nbZ1qw3Ry6qmg+5a2/by6+hLxk/n3YKjoI531gPpBjbSxVu0HYonNtF6CGfcj1x5S TN5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139494; x=1721744294; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; b=Ds/ztH07NsNA0t8ACkDv8lAT2ct9mI4cGBejQrt/AUXk1CQVIuOVSgiq1/tCyfDpQE /ItF58dEAbEkxjW9NUkNDQ1uQeFm2cOicG6bDxSIgwIybr0OkgkCbe+um3f0SArIbNnY dVfhxa9a/30V4pYhVqX4WU/mQOaofb/0rhZ0cGqdGY9+UH3086cj/+e1xQfZ2scJKVim yQ5TEqc1FKlvOrpk9Eft9GoEEg9Jn8up57xn988g2QbzObeQnzHNGsSNJZUF8FYAJMyZ GPq1xU7ZrVBEIiltcKcqqPnpCRV78EtUWBXGuhrM2uJFNwke7oS0APyS+9dziKTTHFMf OHbQ== X-Forwarded-Encrypted: i=2; AJvYcCUzqqrlcChiyJmINmmSk1DfKYB/BvThWEcWzZx1mfqO73Gak4CL28jHTg7RCO5+tCHAy2WxrRqJYEQqdpUtu5vcGDVkKZc= X-Gm-Message-State: AOJu0YyKpIjEaArPDduQ3UUCXd/rxYkrlliHMab5uZqT9f743FPvvK5e FDTO5BGlcaFMG8D3B54LiEhzQdUqwEvnsA70g5U/ZL8BS19ajm2n X-Google-Smtp-Source: AGHT+IGQjdjJUbhBeyxEgnBt2zTvIYjytX/SASyWiUCoTHSBsvGU0DTYBRHOxMuFj8HGtCKlYSqP0g== X-Received: by 2002:a50:aa93:0:b0:59f:9fc7:1e66 with SMTP id 4fb4d7f45d1cf-59f9fc721bcmr523379a12.9.1721139494100; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:4310:b0:599:9c73:c392 with SMTP id 4fb4d7f45d1cf-5999c73c484ls2421931a12.2.-pod-prod-04-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a05:6402:210b:b0:57c:672b:ca34 with SMTP id 4fb4d7f45d1cf-59ef01be830mr1701555a12.28.1721139491565; Tue, 16 Jul 2024 07:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=ooOFwoJhiLPMsL6IbIMRDku1pClUnZXOMGM9T8x6PcY969J4Ws7+Yzbszcckm3IpoU KA7cHJesZViRCtCkWKTk40TakIvrjAKA1yPd647+EfJtyWbsPQwQPZH3csTmk/NtyN02 QzFv+bFQbxDy51LbR8lOLrNEaXCyMrnuc5CdKhYB/XfC77aIMIxTJX9vouDcKYZPlqNH ACpa8QmnTe+NenQBf8P6m2Pfjme/GO4I2NSIxbZLs5Vo4K1og1l63BuA6b2/Filt5B5U yCMZPP/C/3enjqip4v+recPWjqR3EJD3bmkACXw/Wpc4ueD3Hj2aTAkmP9fQTteeBzxa +p1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=O6OIIUZOAeri4B2tkgLzjLGvlfizpgiO81DeH8YO/nY=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=ySOzk2+RuPxdXReAVqTjgewCNC6rXJRK0XvmFqzVv69EHLcPBn3AcpPRR1oiMqQGey QUdThfL++z/PzTRTbCZFZuiojUbgF+JO7zE5BXJeuo9Nd6DGx/sSt/31U3AltyvRBbEw GrkC9uOPz2LLWZMlZW5uWsi77m8aER5rY+HLwgEvVDGNkDGBw3oj09J5wseNXdgrtfA9 8aPLfM2IRe9a/jzTZ1CYjAoniKJZWjkAHPFpyAeXIwGA6fV8mbfC1ldG2T2pfTI3jDhi KUibKrgwetqY+u6bfaP5qjz4rGTgoYjt1nTOjd/4G7YPDCJWVcuMXD4IHOsfWx9vQFgZ Iydg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-59b27a0af16si219724a12.5.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 202407161418115ce2812f9612c0cc2e for ; Tue, 16 Jul 2024 16:18:11 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry Date: Tue, 16 Jul 2024 16:18:06 +0200 Message-ID: <3b7b8dbdde7fa3a4184daa3f8d567e72c8b50d2d.1721139489.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed, by default with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 101 ++++++++++++++++++ .../container-loader/docker-loader.inc | 10 ++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 ++ 5 files changed, 151 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..e97e829b --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,101 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_COMPRESSION ?= "zst" +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += " \ + ${CONTAINER_ENGINE_PACKAGES} \ + ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + ''}" + +CONTAINER_COMPRESSOR_CMD = "${@ \ + 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +CONTAINER_DECOMPRESSOR_CMD = "${@ \ + 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DECOMPRESSOR_CMD \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + import os + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + image_name = host + (path if path != "/" else "") + unpacked_image = workdir + "/" + image_name.replace('/', '.') + dest_dir = D + "/usr/share/" + BPN + "/images" + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar" + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest" + + bb.utils.remove(tar_image) + cmd = f"skopeo copy dir:{unpacked_image} " \ + f"docker-archive:{tar_image}:{image_name}{docker_ref}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \ + image_name + docker_ref + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service + + # Do not compress the package, most of its payload is already, and trying + # nevertheless will only cost time without any gain. + cat <> ${S}/debian/rules +override_dh_builddeb: + dh_builddeb -- -Znone +EOF +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..b6abec92 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman" From patchwork Tue Jul 16 14:18:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3699 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:21 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f62.google.com (mail-wr1-f62.google.com [209.85.221.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIJ9A013331 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:20 +0200 Received: by mail-wr1-f62.google.com with SMTP id ffacd0b85a97d-368255decf3sf745602f8f.3 for ; Tue, 16 Jul 2024 07:18:20 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139494; cv=pass; d=google.com; s=arc-20160816; b=K97XKT2KOWOLMOIQPxBO6ZjUQ9922U9eNXsm0Y1V6CU91GTt6ayC8i0EByUU/vqa0j dGmjznaKjTN/cZWV6UQMx3trWSBidE0mL1rMKfB+bZXJmd44t620uTgc8Ywc8Bta39IO uvVKm5IfpEHIGsrQeO0QS/MqW9+vyvSfUBajeaasVWMYVCFMXAwzaiR7z1//+fiuLdRm /JMG9C7UyAIlqNBl7v+KwKAIV0CXMNH8gt9m7jvfC4xg/bXhdpwyFXV5UKTel60JyLij Tt1YASIKGPxwpZBC+B7KIZUGPmtfSdYvbEOdHz7GNc8hZ/mvr/Ru9aNA+WkvNxowo+HW FUfQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2zd8gppMf7mdB8ZFU4DBi2XFirTckYPPlQv+KbxO1ZE=; fh=zI1UhxRaGrxasVKM9FsfW4bVVY+wmDQWU5MAMnki0Kk=; b=wk85Jfis2K2iaTKV2NFu5d3Xja/lM07f9ique163Or74h2ODCBlY4SO+7zAbxx8+6l tXwcOOjZMeqO5iqtzIvPbwbpS1aiE4MD+UAivJnCDvAv63QzKBNfU59atBraNCXdxo/L /99ZZRNUy2gVVIh1HeV9Ypf/TSsOez9qotX/fBlJXWzQYNywfJSnpxmQPkt/XCZlhnsJ fapvXoyY5jIo5Qrb+Z7onWULD08ng6Bu2A9UcbwBtMP+/C5vs48LPiCRqDuVt5C1DAdm 50gE3fnnYEetdUscQDpJpOJCdSJgL8BZ4WNcNb9dG7ppDOKi5xQSNafoMQywl7tPPI8y jDkA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=tfW4X4NW; spf=pass (google.com: domain of fm-294854-202407161418115715de5e2c90be43be-8rm7cm@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-294854-202407161418115715de5e2c90be43be-8RM7CM@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139494; x=1721744294; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=2zd8gppMf7mdB8ZFU4DBi2XFirTckYPPlQv+KbxO1ZE=; b=veBh9bDP65bEi5J0cbm56472BNnsw9sGLTPbmuywsmWBuDJg5z67p3KtAjhhOoFkPn XqzTAyBFpaQt06PlMOq16bExH6kjUmO9OzsXSr4V0h0q+ZprYvzwjELAcwh32PER/3tQ WP3Vz+zxolkAXiTOTFc3FovjnQ8oS5CJAJeO+mbxjsK48tXGtY1JMgVr4vEukbHr4Uuw UXhYp64Ggt6DxMd06r+FtCdk0sbw7hRKU+KWPfqaGGDnWQzMa9Zw1wBrryZrJDCJphKU VHeqzCapUB6iD2L8NG9z5FJRr36kc0scrjLZyOXK/gHJOMu4LHsQ/Pjp2Chi0XMnlZ/8 qaAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139494; x=1721744294; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2zd8gppMf7mdB8ZFU4DBi2XFirTckYPPlQv+KbxO1ZE=; b=kpY6IIzPmHaJlCGiKsUUBRPr2dYKm7FgVxecJuLH1jvgBhj3SOeokDPOmrzwY6Sksp Bxw3mI+25p38RtElj9/TXsqkDH8B17xz06jAKj+5fKSdD+lrHsIjHWRbTnIzMnwA/7mV sfLqx7Hls2nN1a2VpP5yNi82cAyCkSyk3M9naGZ8HsfM2XeNvN8q+Mosj4ODLzmnQJ+x j5hzij7jEpAwTwBGXW6WDlKJJm38Xxm84QQuZpsr/Pdt8Aziz1EuHV8Yl2uzGG6+anEC IOHm6R40J9ywyZtmS7mF8TCgEqVc5GioH3ENcmL8lrO4YkFV1M7jvh+RSx0v5i3zAJEc BAVA== X-Forwarded-Encrypted: i=2; AJvYcCX/KWSF0mbpfZ89W1noL41e7VxNsBtsbMpg8VWmHSAWOvZBbefGQUyxlsf/ZMokJXZcV3z74YPBl4DFHqjGG1UYWD70MEA= X-Gm-Message-State: AOJu0YzvuMRLZKgoQiAcHlszmS7pgamomFATUkogD7ETgyaWpsSEF1RB 2f0UvQMpExxtQX0vXhQp3TCwfW9WFX9sCecXEQHK+Qqe0T/o1fiD X-Google-Smtp-Source: AGHT+IHPlpkX6+Sr7EgkpyJAVHfTlMsNnwLJAXDHiwTbLADVgcewb10GlHCjeDDt4a+PJBIvDF2jVA== X-Received: by 2002:adf:f1c9:0:b0:367:92c7:3ac0 with SMTP id ffacd0b85a97d-36826099732mr1650882f8f.22.1721139494105; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6000:92a:b0:35f:2852:bb52 with SMTP id ffacd0b85a97d-367f113cf5cls2008440f8f.2.-pod-prod-05-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a5d:5f45:0:b0:362:4f55:6c43 with SMTP id ffacd0b85a97d-36825cec72amr2063619f8f.0.1721139491970; Tue, 16 Jul 2024 07:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=e+w8JVeySWa0qGHl6mk3RowZW91/bd4PyeeOYutsE6WlYU9bEj/BczVxCp9MIk1Epx QskIp6O7T9OM2qmjw85cUOjlDryfChwivAUvhBNjoCtzDb22+UC72THOu+GTo0czSJmq dqOpa9I9zAQIpYvIGRjHoEXCMSbOaDKwXfzn2LQY7nEtMNnGIdW95WjjvJAv7LDDe3Oz nvO8xXFuVO7Aw+zRfDF9AaibPOLDt07B9dBcB66+xU59rYB52Se6e6oiOYmFnba+zd9Y 8yE6RqDpfPgVjOLOdVpg3oH5AVxOc1TeR/cwPlYOlhROepSbLgJxNVcHdcOXLJ6ApPGg jPeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=PZNATY6IoSLm+0l8pnfs+jH3Y4sl0XJy+TjIS1m4FCs=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=gcOB3EUdGomgj300rKxk9XjYZ8LbAUOsIOkEY15dGKdJt/iEK37Hzhw4IX3NcC/bXP 0XCkOHXDB0rZYBnpuehYB8H4P9Wraccke7KCMWmtLqBVxlIrITPBync2vh7uIbLMGVsA jQD6sSWIiPQUBZsgoYeRI7D1Lj81OyZR7SQCddOSn8VUfWQf6yjG7em/8cl6XNP4xEu0 dSY5GcAt+6pwfRz0UtBf6jfLEeTUyN/0rMHov+lA6Znpf2sGVYiKspGlA4u3T3e5qeF2 7rV4l5sxWVo5YWhzelg50gGm7D6fgFocEd+O+judgSMRHucKcDOjBxg6V7Mchjk638u+ bT0w==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=tfW4X4NW; spf=pass (google.com: domain of fm-294854-202407161418115715de5e2c90be43be-8rm7cm@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-294854-202407161418115715de5e2c90be43be-8RM7CM@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net. [185.136.64.227]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-3680d92d07csi113969f8f.0.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407161418115715de5e2c90be43be-8rm7cm@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) client-ip=185.136.64.227; Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 202407161418115715de5e2c90be43be for ; Tue, 16 Jul 2024 16:18:11 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 3/5] meta-isar: Add demo packages for installing prebuilt containers Date: Tue, 16 Jul 2024 16:18:07 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=tfW4X4NW; spf=pass (google.com: domain of fm-294854-202407161418115715de5e2c90be43be-8rm7cm@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-294854-202407161418115715de5e2c90be43be-8RM7CM@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka One recipe for docker, one for podman. Both pull from a registry that, in contrast to infamous dockerhub, should not throttle CI jobs running these frequently for testing purposes. The podman variant of the recipe is intentionally leaving out the digest to trigger the related warning of the container fetcher. These demos also come with kas integration. Signed-off-by: Jan Kiszka --- kas/package/Kconfig | 19 +++++++++++++++++++ kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++ kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++ .../prebuilt-docker-img_0.1.bb | 12 ++++++++++++ .../prebuilt-podman-img_0.1.bb | 10 ++++++++++ 5 files changed, 59 insertions(+) create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb diff --git a/kas/package/Kconfig b/kas/package/Kconfig index 35ba7cf1..395c3a08 100644 --- a/kas/package/Kconfig +++ b/kas/package/Kconfig @@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP default "kas/package/pkg_sshd-regen-keys.yaml" depends on PACKAGE_ISAR_CI_SSH_SETUP + +config PACKAGE_PREBUILT_DOCKER_IMG + bool "prebuilt-docker-img" + default y + +config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG + string + default "kas/package/pkg_prebuilt-docker-img.yaml" + depends on PACKAGE_PREBUILT_DOCKER_IMG + +config PACKAGE_PREBUILT_PODMAN_IMG + bool "prebuilt-podman-img" + default y + +config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG + string + default "kas/package/pkg_prebuilt-podman-img.yaml" + depends on PACKAGE_PREBUILT_PODMAN_IMG + endmenu config KAS_IMAGE_PREINSTALL diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml new file mode 100644 index 00000000..df96a484 --- /dev/null +++ b/kas/package/pkg_prebuilt-docker-img.yaml @@ -0,0 +1,9 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 + +header: + version: 14 + +local_conf_header: + package-prebuilt-docker-img: | + IMAGE_INSTALL:append = " prebuilt-docker-img" diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml new file mode 100644 index 00000000..d0b8da1c --- /dev/null +++ b/kas/package/pkg_prebuilt-podman-img.yaml @@ -0,0 +1,9 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 + +header: + version: 14 + +local_conf_header: + package-prebuilt-podman-img: | + IMAGE_INSTALL:append = " prebuilt-podman-img" diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb new file mode 100644 index 00000000..0dfc9b8f --- /dev/null +++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb @@ -0,0 +1,12 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require recipes-support/container-loader/docker-loader.inc + +CONTAINER_DELETE_AFTER_LOAD = "1" + +SRC_URI += "\ + docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \ + " diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb new file mode 100644 index 00000000..e671a494 --- /dev/null +++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require recipes-support/container-loader/podman-loader.inc + +SRC_URI += "\ + docker://quay.io/libpod/alpine;tag=latest \ + " From patchwork Tue Jul 16 14:18:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3700 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:22 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f183.google.com (mail-lj1-f183.google.com [209.85.208.183]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEILjO013347 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:21 +0200 Received: by mail-lj1-f183.google.com with SMTP id 38308e7fff4ca-2ee8e0dab26sf51666551fa.0 for ; Tue, 16 Jul 2024 07:18:21 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139495; cv=pass; d=google.com; s=arc-20160816; b=qCEfM/7I9VOzLPBYxJBzVe5vDU0PXedeKeZ0mEE4mKD68YLmaO2I4I2y0vw2JFBDv+ eSOFr+os1d12nHu7Mx1kNx55jSla8ZYOQdTWFXc7rpKCItIqGR/dtDBb67n48AYixwdg VhvFA+frjhP3DK9DwnigM1N1pW7clXc+Omg+/8UFTKUULNCye0BwgLLw/O1Q5YG6A0d3 fPSd5++9V1ZK9/xohSfANJPhdqEUMywCLyL7QPZjKs7nCwLBKE7aaWtqP3wfCSOjcL97 FFo/nClqdxsEl5PuDY1AB7jS2F9f6nhonv+VVQWJeTlcYJa4vNOe68Tj3BzWVpxrDlCh /xJw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/2K9aUgWM0QpU/lzTXem730Z0/hiL3YRXNKSYWP2Fvw=; fh=rgmtAyv/Rp8PeiqqZPSraNrlYQh/QKnTWqOYoghKApA=; b=ESYwIcM56SJLizmyT01hXHWiwUX31IujdkYoXaP1MV7or8MuH/nuzxINHFRmn4MH+7 bZLHjwxbNRwzNbW4HS2oC9Ply/dnVvj16v6ugfp+wfdM56HLxIGBy73nfLWy5x4Jalwa XokeMyKj37k+KHQgd5vqt/XYyYA3cDERe9g8D7peU7yAomojS8eWCSe9TlW1sr2VaggC 0K2sfbbNG49739RccBh341dv39aG8sluI7xsEaFjK0KgEHyfIpVhSEh+HF5DKJFsqNRb sE1s9zAreufzdW4RWQnoSMzwlBRvN8lIwy95CbDL2BZTRNn6vRLErbpQ96jDtaWVW66U 296A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=YMBvnV5k; spf=pass (google.com: domain of fm-294854-2024071614181148dd4834af28e9bfa1-f0rcbb@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-2024071614181148dd4834af28e9bfa1-F0RcBB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139495; x=1721744295; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=/2K9aUgWM0QpU/lzTXem730Z0/hiL3YRXNKSYWP2Fvw=; b=S2hbcB1Iipq9NPwCTom2HiW+1NNV8YLI6mSgWwsslVW6CcykTv22PFvHGW5gEqjGUP Tq+7i9pCjnbXJr+3vMnSbm57pZQv3SNH8uPuOSgnOeL7Ut8DTQbPznNUSeBQvD0eXR48 +dIMzCL+f052qkNgp/nhF6FYnZSb0hgqedI6RvcUyG95JmzZdit+sHWVqhhaEM19V6Kn i14cpZ3dWCcv2kMfOLst7agP/uFzr9MAEvKswNNTfx6OMj16uDQUEX/ivtqSyGcH8r4r tow0FMs+/OFpK/HUVNpGwFX3W0+mv+hXkuvLcA2X7Dua0Teeq7ZN5PjEgN7KL0MZedIl WzCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139495; x=1721744295; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=/2K9aUgWM0QpU/lzTXem730Z0/hiL3YRXNKSYWP2Fvw=; b=L5x/NtIDZaJhTeVkyUKj780Rb8mpgfNAytT8VyVEJpNjxLFdQwBaSL4L0rt4pk9Nge y2NGZHVT/A+I9DriLxtFDof1Kt9n5iF30YbA1T2nIsZE5OeuSp7j1OTVLmHFBTtvCR26 pEDwGByqYZyqhGH6/0OjGZP6cZxvNJsM+8Boc60W3kIogN+i4TpT9NP9O4yu3nvVBuNw qED8s7nLXKxckdrGX+yE3Yb9FxcAblH0KeX3uMHg9grw10BJKuFXPiKvMzNFDYAHhWjf 1GBfEcbWRmjCc/ilMBXA+C6EmLB2DkjRiIPeilFTwNc8idrmcDPzxZpyD74FU9bCcKVn y7aQ== X-Forwarded-Encrypted: i=2; AJvYcCUZB+hHevSD93TNZlaV5una9YT0S7qj/y+fJBWlzncbSx4/JEp9GbCu30eQxL84BoO5eH7V1cmt8Lqd3TvzvyRA+PPJHaY= X-Gm-Message-State: AOJu0Yx5gYdL8BH3bmUxrBfiqMmqUySpTEul6l7DnEzMSVbC7MVI7Aq9 Gz229kJZ2vA9Vmboy1vgl17qRuxoHH82y8oWJSk3wy2AGXtw52HH X-Google-Smtp-Source: AGHT+IGuH464gL5nAU3jLfjzAzliPUtIz4VE2oVQ/HLQ+RKnIWCDCxHEA0rs2SJu93Zdr2UqDrBlvw== X-Received: by 2002:a05:6512:b94:b0:52c:c032:538d with SMTP id 2adb3069b0e04-52edf02300cmr1595569e87.27.1721139494351; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:410:b0:52e:9d2f:233 with SMTP id 2adb3069b0e04-52ec409092dls2910351e87.0.-pod-prod-02-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a05:651c:19a5:b0:2eb:68d0:88be with SMTP id 38308e7fff4ca-2eef416e54emr19155581fa.12.1721139492001; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=XVbTK4sJ6qJL6dM1D6YXy5JL7GgMpnYDJIQ1H8OmRj6c0P+rvtozY7h+9nFYI8MqzC TjzcgHWZo0BCwcSF0wHiBLAe+BlpKxDsHGVbmwZn+lTU3NTFd+QmMXvH0T8qYHNjritf UvWpLA3nOj7EncMspsjx+DL4i8sgL8RzoJbxB28Vs8d8H/wkLBF5bsBaOM/WLBYWBXwQ bqrNI4KDCkV8U0VYOGK9fe7A5W070aSCbgHhzuxLoFTjdEWfbYPUINTLpSwp/xi87rBN EAVSHA3ue/8zWTaOAuqF25N4fH8bemGL19RW5sKGms9onUCKpyd83RfMAhSWinsAxPrC mi+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=5Q8nkkkjzpXg64IKufplF2I9id/rT8Gipiy3rjOvdQU=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=z3DsT357qWNWrxsrhrBHkvi3QUEcgbqvtTtWk2x19Tg95hpUMynmP2WHOJQEnM9tta pMT4EamxJ2ISeU6WOwel8P6meSPxXk3zf7yix82ydYk+M78Sy4FgpgAd5HRLcGzri1k/ eOac9NzKdeP9o+ThI1ZSi5aagUPIyu4XImLiSEYPmO/4dy/MRGmmugt5ZcOKw5zIDg1z HKPE4ZOPVY9XVwtHmPCZ3ORjGyI0Gla2Q/HIs+l0MJBveBpCJ8ETWrQx7s1fgs4qiQWU /XqQYiRJ2MbZA8tKwcQgOJjKnOJt7uzetM/PNR0SvybDHUulPKEIb3TR3S2OOOBxXzp9 TDzg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=YMBvnV5k; spf=pass (google.com: domain of fm-294854-2024071614181148dd4834af28e9bfa1-f0rcbb@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-2024071614181148dd4834af28e9bfa1-F0RcBB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2eee19149ccsi1386831fa.4.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-2024071614181148dd4834af28e9bfa1-f0rcbb@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 2024071614181148dd4834af28e9bfa1 for ; Tue, 16 Jul 2024 16:18:11 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 4/5] ci: Add test cases for container fetching and loading Date: Tue, 16 Jul 2024 16:18:08 +0200 Message-ID: <2bf448610b18ca3ecc438f2a8342db7da0dacf02.1721139489.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=YMBvnV5k; spf=pass (google.com: domain of fm-294854-2024071614181148dd4834af28e9bfa1-f0rcbb@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-2024071614181148dd4834af28e9bfa1-F0RcBB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This plugs the two example recipes for loading container images into VM-based testing. The test consists of running 'true' in the installed alpine images. Rather than enabling the ci user to do password-less sudo, this uses su with the piped-in password. Another trick needed is to poll for the images because loading is performed asynchronously. Signed-off-by: Jan Kiszka --- .../recipes-core/images/isar-image-ci.bb | 2 ++ testsuite/citest.py | 21 +++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb index e5d51e6e..9133da74 100644 --- a/meta-test/recipes-core/images/isar-image-ci.bb +++ b/meta-test/recipes-core/images/isar-image-ci.bb @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys" # qemuamd64-bookworm WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks" +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img" # qemuamd64-bullseye IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz" @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL} # qemuarm64-bookworm IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz" IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}" +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img" diff --git a/testsuite/citest.py b/testsuite/citest.py index 8dd907d0..539c9440 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest): self.init() self.vm_start('mipsel','bookworm', image='isar-image-ci', script='test_kernel_module.sh example_module') + + + def test_amd64_bookworm_prebuilt_containers(self): + self.init() + self.vm_start('amd64', 'bookworm', image='isar-image-ci', + cmd='echo root | su -c \'' \ + 'PATH=\$PATH:/usr/sbin;' \ + 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \ + 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \ + 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \ + 'podman run --rm quay.io/libpod/alpine:latest true\'') + + def test_arm64_bookworm_prebuilt_containers(self): + self.init() + self.vm_start('arm64', 'bookworm', image='isar-image-ci', + cmd='echo root | su -c \'' \ + 'PATH=\$PATH:/usr/sbin;' \ + 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \ + 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \ + 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \ + 'podman run --rm quay.io/libpod/alpine:latest true\'') From patchwork Tue Jul 16 14:18:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3702 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:23 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f59.google.com (mail-lf1-f59.google.com [209.85.167.59]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIMdj013365 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:22 +0200 Received: by mail-lf1-f59.google.com with SMTP id 2adb3069b0e04-52ea883187csf5411991e87.3 for ; Tue, 16 Jul 2024 07:18:22 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139495; cv=pass; d=google.com; s=arc-20160816; b=YZgdaB8VDTrj+xz6XfBTJgA851ORrVcyY2IcSg1z7McIMSUC4QV3aCE4WGZuFzMXo7 wEDhgLXH6EyFaRsN3HoL0UoaJ4+XtAPKgKWfgkHeE6SyVwfF+p7LJrwA22S2koH2FZh3 uT8yD63iHFJbkl+AJ1VhGwaWTVSHhfBL/enNQyn4GiL7XKC3bi6oG5dFCPVZ5nG0Opoe OBeHC4zfgt2Jj7mlTnj8GNYd+xNubIRjCYTB6Cz12e0277+wt9F1/CraB7Tvo3b9ly0l fac2OpGoTkTA2VriGw/UVcfsTqMHQKu2L5sxAAL1USvBNCml83vbcTcdkO0AYgYNhuqu 4awA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NVtl5KSCs504vflhcJXHySR5kw0FxGPbKdCtbsRCdTk=; fh=V4zeEaFRScaizAoMW7X/RMWGyqA3qMGcelvOjSZhnC0=; b=mJpV9qPstVWESB7WxNBdflvS7UTW5Huh3i1A5x/V3sJBlrQAkXcS46qm1pIuA4jMEv h/ARiHCWGRMxoAEWjqDYbJgeZi+FNlOnuBGfDr6/ryy4/NQD5pSIjW/rFls6z5+pmBOT sfBLLz7L/TgW619CEhTqG8BKVTfdeTUkbK2e431EGMhR34R64gfOX1eTOVlngQjjxrcV TKJ2ZVFWGvKoFRhOyuJ57g/HSnWJxJLH/e4bgOR/t1aq64sGZihBan6L4oVNCPzOJmdn /g2W7E+LuNcWUK9Xr9yxprBWJx+lpil/T81XDbf2k3cAHSTWJhBkbJ2caz1o5L2Q9UZz m6Fg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=CnPCanMz; spf=pass (google.com: domain of fm-294854-20240716141811b22134322e75f578a7-mojsmh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-294854-20240716141811b22134322e75f578a7-Mojsmh@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139495; x=1721744295; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=NVtl5KSCs504vflhcJXHySR5kw0FxGPbKdCtbsRCdTk=; b=KruENsXhjONSTHYpOzjoe2m15DoOXXc0UiFbIc6mNu57kEQqpDBL8tK9lsoB/GrWy4 jJm0qRXFwSDQ5ysUmfSoW2qlpyXJ/VRB4aETmhZqEB9cbiZZGjt/FBOtKoTMjQqgXZvg ZDUeP+ywsmZCDvqrNSoCvpFSaAJNK11VUI/bgrYhj9XtMxyW0dDm857ELba1dkTHd4XM 5ZZvwqVkO3YMUb5rnEpI3FxWd2P6Nalkj5U/A4bc6rQxBxLXm7iva0vMdnGloP476ByH X/LQ4PeR1gzxFfHVQXRqCfdwWTkwGRX2RAxvG5E/KYiw8uV1nHjC8hN7+n7+3ePYdhv9 Czdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139495; x=1721744295; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=NVtl5KSCs504vflhcJXHySR5kw0FxGPbKdCtbsRCdTk=; b=OCCe6u2TSK2HYsIdO2bywgIaz9/f9Hubzufn84EjaBUZe0tkDDS5SXTuHSxxldqNBO 66yYiHqzwKKmNd3cOrDHFuBKgOiyQWyLwAQ6Ch+XLY3710RZcLJR/3rptpiUdq9biIRd /S2Ix/o7T5MQAGRKnkBrUXodsLOPkoaGTEeJg8QGHV4/sclu7yX679yULTzSgTtwJEuL wUeHWbHptCmP4XQQFEnnF2V0WPVefL61SlCNXgM0DRCRK+Xj53B21O6mgePz4GPfLDag 9V/yF82jNwu+MGU9BTM0pvN7Hu43LOqWHvqQnUs5pez8E0EsmT6GGj21nRuHWeV6MmM+ Rvnw== X-Forwarded-Encrypted: i=2; AJvYcCV1z3+fskjYgwxZWwEvL302aa8B8KX1sRaht/OhcsoVx6hbUBm/fN3wJzYcJGGYRphkhJVynSL5K5OV+rNOwPUv5E17oI0= X-Gm-Message-State: AOJu0YzWUpjyi2WDMtxqgxHpQJhg+w+14+27TtOWH2iicLafaPFopTtU 59KG1MiTXBBtQtrfWuB6c0SIW0qP6hRUOU08Kcssw5zu9dCn+Tmp X-Google-Smtp-Source: AGHT+IEnLbRlgQx53EHx5Gnd7c9qHV+XjxouOUvh47crxi5neIu9M+ZE2yEN++I42keuJyyfRHkFTw== X-Received: by 2002:a05:6512:280c:b0:52c:a88b:9997 with SMTP id 2adb3069b0e04-52edf030adcmr1883169e87.49.1721139494503; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:ac02:0:b0:52e:8072:8b91 with SMTP id 2adb3069b0e04-52ec4504d4als2693328e87.1.-pod-prod-08-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a05:6512:3f04:b0:52e:9481:eaa1 with SMTP id 2adb3069b0e04-52edef1eaedmr1775285e87.23.1721139492131; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139492; cv=none; d=google.com; s=arc-20160816; b=ZOBrroVR8nqooJ3wk0rjAPXfNPB91toxd5yv1r+oS3YQsTB4sEPX92aw0IeNNVbBXq 8akaYpkT6DvzdkpWeWnzYcearllkf2fZqur2hR67HJAT9qho6y2+PV9AFD9ZFBUU5Ptm Phy4wC1FR4l7F02k30vJ+1bDFIIXAcBJv3y3yRopOEMOLnt3Z3oeG54JUHDhjjPpWd3z qc5yKLyVH4PmuJW3x9faFWOupnb6ZHrPGxwdH2tsEQRwqnlWq49Zzh4aDJizvxIQ1nzk xS6qhDCAWf2pfiwrIr+CEctjMiNyhTBOnBSUIikmQA6n/hWibPW0jrOrRjS22xKQX9CF CJtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=4AcWpN0C7NKWvt2bQCCVT9D6e+6VhTXPCLccjjb62/o=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=o5bVb5EswtqdMV6WeY7jcDNxHSndWiSG/NlYmQqDuwupt2g/k5MwrN9ROPcIDFKNTL ujEfYi3zcR4vNsQPstw7fy7+EGRzziREAxu3rcOHGlQkBYTgTfU8/B4Uwu5PHUExw8iz gu5ZlyIuEFTfPq64rXoIErEw/4MkIDHVQfEELnsPAhA75uFCFQ5x/Kk0AZ4qLinKu3ay 0LP2Zw9imtTa+IkY6uzwVmFHlZOG2e9P6IXfjf+immUvHLzrQTnfJ2AQBmiIajWE4kiW RfC9HNtG6xmpNJy171yNALM8dpiCIeoF+VTOIFxmOyXin+mud6HaVWSXWazTWHG8M061 h2UA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=CnPCanMz; spf=pass (google.com: domain of fm-294854-20240716141811b22134322e75f578a7-mojsmh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-294854-20240716141811b22134322e75f578a7-Mojsmh@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52ed254a0f1si115976e87.7.2024.07.16.07.18.12 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:12 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240716141811b22134322e75f578a7-mojsmh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20240716141811b22134322e75f578a7 for ; Tue, 16 Jul 2024 16:18:12 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 5/5] doc: Describe how to use the container fetcher and loader Date: Tue, 16 Jul 2024 16:18:09 +0200 Message-ID: <80f1c915285e61b70623f0eb2ce5f930abaa6196.1721139489.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=CnPCanMz; spf=pass (google.com: domain of fm-294854-20240716141811b22134322e75f578a7-mojsmh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-294854-20240716141811b22134322e75f578a7-Mojsmh@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka Signed-off-by: Jan Kiszka --- doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/doc/user_manual.md b/doc/user_manual.md index 776ae52c..e97a2cd5 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "" Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = ""`. To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`. + +## Pre-install container images + +If an isar-generated image shall provide a container runtime, it may also be +desirable to pre-install container images to avoid having to download them on +first boot or because they may not be accessible outside of the build +environment. Isar supports this scenario via two services, a container fetcher +and a container loader. + +### Bitbake fetcher for containers + +The bitbake fetching protocol "docker://" allows to download pre-built images +from container registries. The URL consists of the image path, followed by +a recommended digest in the form `digest=sha256:` and an optional +tag in the form `tag=`. A digest is preferred over a tag to identify an +image when fetching because it also allows to validate its integrity. If a tag +is not specified, `latest` is used as tag name. + +When specifying a multi-arch image, the fetcher will download the images for +all available architectures. If this is not desired, directly specify the +digest of the desired architecture manifest instead of that the manifest list. + +The fetched container images are stored in a directory in the `WORKDIR` of the +requesting recipe. When a multi-arch image was specified, only the image +matching `PACKAGE_ARCH` will be stored. The name of the image directory is +derived from the container image name, replacing all `/` with `.`. + +### Container loader helpers + +To create a Debian package which can carry container images and load them into +local storage of docker or podman, there is a set of helpers available. To use +them in an own recipe, add +`require recipes-support/container-loader/docker-loader.inc` when using docker +and `require recipes-support/container-loader/podman-loader.inc` when using +podman. The loader will try to transfer the packaged image into the container +runtime storage on boot, but only if no container image of the same name and +tag is present already. + +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images +remain by default available and may be used again for loading the storage after +it may have been emptied later on (factory reset). + +Source container images may either be fetched as binaries from a registry, see +above, or built via isar as well. + +### Example + +This creates a debian package which will download, package and then load the +`debian:bookworm-20240701-slim` container image into the docker container +storage. The package will depend on `docker.io`, insuring that that basic +runtime services are installed on the target as well. The packaged image will +be deleted from the target device's rootfs after successful import. + +``` +require recipes-support/container-loader/docker-loader.inc + +CONTAINER_DELETE_AFTER_LOAD = "1" + +SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim" +```