From patchwork Mon Jul 15 10:08:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3689 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:08:44 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f186.google.com (mail-lj1-f186.google.com [209.85.208.186]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FA8hK8005843 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:08:43 +0200 Received: by mail-lj1-f186.google.com with SMTP id 38308e7fff4ca-2eebc944683sf42359391fa.0 for ; Mon, 15 Jul 2024 03:08:43 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038117; cv=pass; d=google.com; s=arc-20160816; b=Gpa8llAeNea2kSHC3CZTwu85pwqkfu2yPduszjeGb1s3ID9dyKSCMRaXH3GBiDrCYv aqVijjtbIIYJdFUa0Py3FAkcWkkaZRl4RKgoIDW5PgL8TSz4sJDNStHY0YwFnwJVQdzH TaBY38NzDTiVl+Md2+0A83GX3G5xNSvUBY4uRC5C6YGxJNwZcWUAhhTZurdPZQoekQjo bIwGle6FhHYbZZ4C+kNKRVjqHeYvCbQnI4E5dkcqopTsXuKUWRba2oZ4pEQB2C/B5aSE 7AkwAD4O0j4fWwvaoBzCh1rnj9ATLkkGDJlEoDOuIZooXlcJb+8T30a2j0nld4gEWRrh Nw0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; fh=pX/53+dQ2lkjFigwDlnf12FA2ODDehKXsA5Uygo2/LU=; b=K80klFpFiwhl21VPprtlfz3/uAHxcyGTsCTR4FmShKhvsz7Bv1LLfLmkZilVJAr2Fy T7yJKQWaHzR3i3dSfdqojYzjxnNjhplr+6nwvZssBPeES3h1R90QtL2l74NCmZiFAw4l Hg1XPzOXA4GpFT3qDuHIrjjnNX6aYKC/BGMYmt3ZUhNnpwUwFP8NEszWLGjcAmXG4zu7 c0bKLf2wzq/Bf1/M5R5pS0jOW+9LmwhK09jvk/YklJoyGILd3CPC9orGaoDynrliTV4e HZGXN7t0AWXtNUwb7haEz1tHr9cXjdUDopDMQrgFiOCnPDH5BA3Tv3Ov2T0TiH50P9YZ Gryg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038117; x=1721642917; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; b=PaisrenUlKjSQ2yuXaDQMj/HJErG/DQJghiWPy0z29ZUGqC1MEwvlqk1e9fJOcoksw OnV9m02Ffreft7n28amhGaJGGk0u1CyCatPGR3ncSaTCoC7LHLR4Q87ih4Z3TD6NyOO1 JnEd3G0U6I+nE7BJZQEk0ZGIWe7g8LvdvTPlFj/7Yy9UvEdUMtRJvnCl2fa2RGLTEXWQ WGaS6YIj4TZhm8CHCAw2QRYF5hkUQW3xmcX6/oY0jQvVJMYCx6V2U8NheJwrjocEMcEU 0/4riAo6mFv8Ex+meHSrD1pV3yjfNF868G8YQOtoUOoFWeicaf14prQRP2Em1M1BbtQi uiAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038117; x=1721642917; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; b=mBy7wFdw/Mtr8YdtFV8VZ2KGFBD6pXXhjLf5kt+Uy/hoaWoUohX/YjqATd5ap9VPYe IlNa1Wke6cgV9VKvs0MK8WlX4i1+aHkH6nCSbZk0ZJhOV3Y2cDpiU5jxeuKCpK4gprYs mXq0Wkb7ggm/PKODgj3sJQfNI07ZU5HbKj+rlFgsb1aM7zwm0QsbKhw2Ta6rIODN50lY PuwydgGROklhjlP5D9rESYl7ZTMUjZ7So8C3cgKCzhryig9v1sHXK4QpldvekZEClK+I TcTYW2kooCcan8opREBGzJ7Sc0Vn9hhSGK0hTNuPlTFtfvLv5O+cOv8YH3R4IjYDPDNe vW+g== X-Forwarded-Encrypted: i=2; AJvYcCXdOYGVlffSVHh7exw0T6ug92bt3OPTdg2TxOMYGwirC8qS+v7CJ2G7pWsBv2Y4PlkTZA3WgNhSv9WGluXJEDNrPEqjfYE= X-Gm-Message-State: AOJu0YzqaZ/auRbgV+RNo3NbFMpWpFmTF4SK0f0GxlWV05Q2OZvRAPR/ ZD973bW8pULtGNtWnUxCLaJDLAFHgeUR229xAMEHw/CEepcvWoWE X-Google-Smtp-Source: AGHT+IFZUg67Cmcta18vrBLhsdSqhwJpD8LW8jaUlnneW5JcfMO0tOKf6DhcZQ6SlvhIsnwkfF5tGQ== X-Received: by 2002:a2e:8081:0:b0:2ee:83f7:8b34 with SMTP id 38308e7fff4ca-2eeb30ba05cmr140867551fa.8.1721038116249; Mon, 15 Jul 2024 03:08:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:489:b0:2ee:7a6c:fcba with SMTP id 38308e7fff4ca-2eec938cc1fls18789091fa.1.-pod-prod-08-eu; Mon, 15 Jul 2024 03:08:34 -0700 (PDT) X-Received: by 2002:a2e:911a:0:b0:2eb:2e0b:72c with SMTP id 38308e7fff4ca-2eeb30dc843mr140984731fa.16.1721038113883; Mon, 15 Jul 2024 03:08:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038113; cv=none; d=google.com; s=arc-20160816; b=JU4wn0VRM5hy+yIn4zvGzsxkvRsgoHtT/q+G4apQH+GK66rwS9HzPanm5gjPUsdKQx 5UUWpLBwEVTSvC+yPVHUbTKfz/BFw1FacPWufxGOxPM1WPBO0g2PGNMbdkLNTFiT1jkT y/2qAajUQkpKVfz+MeBmJ71dLnvJnjjyfwtksW+5rblzq9hwBSQaKfAMp7MzcK/ey6jy D5EpNu9TTp9CEq88/xFhODnVomDfcEa0eae9IwKkOqvnESFqregVlh1rl5eMuHGva9tO C9HVBgk9ljSqDbVu9p/vyArtOS/1chLD3iefsxFQWwI0Yp98YEeEmB+OIsu7GAGwPw2v BUKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=qT0qBD6zoZX/tPGQ0GTumBOagxBAE1nu2vnVszdRf/0=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=AE6Fas+i8Fg5x6PpoF4G3P3lo6fGTAZg2OPujtmZa2vcT8q4VJLqHf2pUb46YW572F 3E0/OQsPoCUNoRyv1CtFG5yv0wF+JheNbL5xkTAPca+ZciRNPVj/roZDna0IC3L3jGSK 1N6+v/lEv4aDVRp+/0cyoSqaWolmlPkTYEbfl90FLuO8XgdLtnsjwJdF3BYNMnQE1k/A 9061UVWz9jEDZyU2WAbk0q9HY9/oMjlDAE7tLw+GjUSEad7YTTiKsKJ4OL5JijwI3PwF QGdW5q9jKqPLY8fiXn8UrVX7yMA2cLNRudX9rXYukR8CgNMNrcJ/ns59T8S7WwuijQIl kHCg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net. [185.136.65.228]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427a5ec6cb7si674365e9.1.2024.07.15.03.08.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:33 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) client-ip=185.136.65.228; Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 202407151008334113b2dafda22e9ba9 for ; Mon, 15 Jul 2024 12:08:33 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 1/5] Introduce fetcher from container registries Date: Mon, 15 Jul 2024 12:08:28 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. The fetcher will try to pull all available variants of a multi-arch image. If this is not needed, you can also directly specify the image digest of a specific architecture. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..8513e246 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,90 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import os +import re +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd +from bb.progress import LineFilterProgressHandler + +class SkopeoProgressHandler(LineFilterProgressHandler): + def __init__(self, d): + super(SkopeoProgressHandler, self).__init__(d) + self._fire_progress(0) + + def writeline(self, line): + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line) + if match: + state = match[0].split('/') + progress = (int(state[0]) * 100) / int(state[1]) + self.update(progress) + return True + + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + + def download(self, ud, d): + progresshandler = SkopeoProgressHandler(d) + runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}", + d, log=progresshandler) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + arch = d.getVar('PACKAGE_ARCH') + variant_opt = "" + if arch == "armhf": + arch = "arm" + variant_opt = "--override-variant v7" + elif arch == "armel": + arch = "arm" + variant_opt = "--override-variant v6" + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \ + f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d) + + def clean(self, ud, d): + bb.utils.remove(ud.localpath, recurse=True) From patchwork Mon Jul 15 10:08:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3693 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:20:54 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f62.google.com (mail-wm1-f62.google.com [209.85.128.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FAKr7I006227 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:20:53 +0200 Received: by mail-wm1-f62.google.com with SMTP id 5b1f17b1804b1-4266fbae4c6sf30823445e9.0 for ; Mon, 15 Jul 2024 03:20:53 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038848; cv=pass; d=google.com; s=arc-20160816; b=c4xi2j+9wU3gCr9eMLlLzNW4+3Jcw4GQ5TV2P1/Ix8HTcKhaApNIp7GTA8EuTwoPO/ IOKn3sELB2EX6YDumcyVn1IatHjDaD4SdZMLrxhxJ6lhqtubH9mHzqee40lfXJy7YAOD pZvRDVoM3gzGF521rwQ/7tGPB4BCiVFuaVxCPD7Uih69ttnJP36ZcB5a9a6lwuX0WLU5 x4HUHw4NFyaHQaVEJKsvEcU35qCKTPaXOwMRmjvxGqjytow7KJ9mcuNLXQk/NWjnXIE9 S/L+ZarjkpxeFEpr4duJJBB93EMPPNAmDaaKXRZ+1bB0q6838LvZJ6bB4oLgkGHA+0ps zU0Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EoDxlovoIdtwN9tjI7QFj8OPZS9oPqzi2cyN04XoWwI=; fh=rpyyCBOJCoDbgVEPkBXBUGe1wc13ws3DpS7A/fMmuDw=; b=T1mJP/vLrmgfeuB4cvHAwjfp8eu5NOfkUo0uzYSB2/6QAK0tq6obeZRDhcFFvAEo23 zmqRq4wkNqg3cLoJ4bE0ICbwv2xmgmIOaH3xL33bsJxtIi1M0GOq9b3V9U7FfKarrSJa ecZfJOWVpYzHOsyNsnYqsEUffIE/B51O6T2n41K0907ZWRoEdlsEekaskwMNvJExauFr vkTgnKDXd70w8Ba+HqzPzzTGiS76MV+42cKWWrUId8AQEfJ79Fe36W9+rrpOZIYKyCYC htOLq5gUYfgoDP7qCWaOsvj66fnKhxu5Nqxg2u4a6aoXkrTp4rBTFS8VrFdg737eRKhG FQgQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fxvQ3eLh; spf=pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-202407151008339faedc6c53b9c7b06d-XIRAWL@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038848; x=1721643648; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=EoDxlovoIdtwN9tjI7QFj8OPZS9oPqzi2cyN04XoWwI=; b=v4pJjg3c9adkSiTIcVDmCnIHuQ6b2flznYYLcn6+L6bZuUMQF+o21TuosNXwGf+Dlq 5Ey0MKCkx546BUESxrqbJiT8ew3WQCVmXkqXwzwZArcuEvFHPYY5BHKJDNMRPvpXzVnE EWOu3DWOtgThY0400ofdvqgfcLsW1BwLbJMeHTEPNtvDZga5N/lJ2v9K5iPrRGthHdEJ JBW8hP39Hqd2lsZ6dvFIE1nSIMKV/EwTJd+go1YUHnkMdfyB1PXiovaGCBjfP5jjpnFd w6smPJaUk6bzlGhAMSmyGsgdenpThGS3pBQp39jGMdrtw5cqXyBa5mBejGHXU5KKPjzU Tvsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038848; x=1721643648; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=EoDxlovoIdtwN9tjI7QFj8OPZS9oPqzi2cyN04XoWwI=; b=ZzIZXy1g+RTpbYGwYkrl8Z++FQ2plYjEMoO5HOOmDyG5JqclA4ximrHZq2GfB3bpSZ V5/BH/REzkupYukrgHK12GpUu4xFjELDuclWYRM7F+3N+GTnM62wVYr6oyJ60wLiLTlm YR6NxsLr7Tek91WtU6noHWCBQ4LgMn+16G707oEIw1ZkVCsESi59dLrurBm+7S6hHJru fsDI7RzyDmeERPL1BDC/kOMyZzXOzxxVloCa82mB1FLdoMX2KRgy9fEMgKq2pzYVOsap 8kPqdZP/ywA8PEteQ/Jgp9t3Mmqhloi7aStcxttQgE3csVQajV4HMeVWrPCqWQYGHLLl xr/g== X-Forwarded-Encrypted: i=2; AJvYcCX3fCQ4YjSLkXR8jvns/aIA5HU/6u/wr8TNHxE8ms79GYWF45pbhOfODbgEPZVbmU9nIqq3hb+En5thIqI7N7ST3/pQr0E= X-Gm-Message-State: AOJu0YxeFxEGmfkL+pLFAfGAkvryRDn4k+pH1cC2EQkgUhUbHl9pYvtj sstWXfxNAuz7jZ5opxrUQ34aaHDag4SzWUh34YMONtq/2DCRcW28 X-Google-Smtp-Source: AGHT+IFrDVuqNyvfv/BfGz2eEeD/eQh9ZAR0TSBqBobBbDOIxJ1n4JE7bX+OtQJvOXhl4Zd22isxrw== X-Received: by 2002:a05:6512:2350:b0:52c:987f:b355 with SMTP id 2adb3069b0e04-52eb99cc6abmr13515129e87.42.1721038116818; Mon, 15 Jul 2024 03:08:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3b02:b0:427:96b1:a684 with SMTP id 5b1f17b1804b1-42798478ec6ls21375715e9.0.-pod-prod-07-eu; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) X-Received: by 2002:a05:600c:4982:b0:426:6314:3336 with SMTP id 5b1f17b1804b1-426708fa8bbmr121731375e9.36.1721038114681; Mon, 15 Jul 2024 03:08:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038114; cv=none; d=google.com; s=arc-20160816; b=0RwfKGB8XAHuVu4ncfaXqXA8j+iyCUV5Kic2u2YtFG/lsWFj67yzmz7WNBTQlmVhMW sE663AdEBGapcQW8Knf7wfaeg48iBhOYjar9StUdUCKN2ht7/Rmjjk78G2I1jEeaOCLF e3KbjvAB3sKtzpV3zw8xAzZji/vlROXKcBE7+wXXYa+dlq62YNOy6dMrbMNR+OeJUzbh gWXDBXhjr4MgC6LM/0wsJsfrMPBvjGIHxSSPfjflpabAw6PtIx0f2HrtlK5HZyX0iOaq VreCIP0I8OYAauSKjh7ppvHR4P0TaEtMrF61UE9c3EG1FB9VckUB6Wk+UkjvWEx+2BOi PtqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=Z3WBY2NDqDJJkDez3e+RMxMNkjaN1jw0I79TYdvWnCo=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=B+/1jwLKtekR5U0/o/DOTYaLC3dCZBmha65FdzL3pbeks2nVleZj23tFwi5q8usgwR V2T+VGBNvFaLTrs3v4BEjwQRs2ojKvaNV1pYZnPB4XmxxOYkz7eZukkALlCQ3fyBczTd ZkffDuRkSC0tTaHANas03pYBz64ttFOcnCQrnLeJVdcmyGMm4+up/nHm6IYtfNxVwX7/ GQqFFa0EArVBJVYCRkqqLKuxCgvh6kvOe8env4iDOa2soVe+Udgi8T0ikdJ8+bWDAIGr WvgxE2gs4i5TdT8Lg/HzZmY/m2rrpIuqjOS83olfIoYb5Vxf3zhgkeqPnmEznOb06wTl Pc1Q==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fxvQ3eLh; spf=pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-202407151008339faedc6c53b9c7b06d-XIRAWL@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427a2db519esi841125e9.0.2024.07.15.03.08.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:34 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 202407151008339faedc6c53b9c7b06d for ; Mon, 15 Jul 2024 12:08:34 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 2/5] container-loader: Introduce helper to load container images into local registry Date: Mon, 15 Jul 2024 12:08:29 +0200 Message-ID: <27306a537db0da8d51a0ca709b9113248331d340.1721038111.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fxvQ3eLh; spf=pass (google.com: domain of fm-294854-202407151008339faedc6c53b9c7b06d-xirawl@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-202407151008339faedc6c53b9c7b06d-XIRAWL@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed, by default with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 94 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 ++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 ++ 5 files changed, 144 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..a0c2ddb3 --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,94 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_COMPRESSION ?= "zst" +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += " \ + ${CONTAINER_ENGINE_PACKAGES} \ + ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + ''}" + +CONTAINER_COMPRESSOR_CMD = "${@ \ + 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +CONTAINER_DECOMPRESSOR_CMD = "${@ \ + 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DECOMPRESSOR_CMD \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + import os + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + image_name = host + (path if path != "/" else "") + unpacked_image = workdir + "/" + image_name.replace('/', '.') + dest_dir = D + "/usr/share/" + BPN + "/images" + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar" + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest" + + bb.utils.remove(tar_image) + cmd = f"skopeo copy dir:{unpacked_image} " \ + f"docker-archive:{tar_image}:{image_name}{docker_ref}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \ + image_name + docker_ref + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..b6abec92 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman" From patchwork Mon Jul 15 10:08:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3691 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:08:45 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f57.google.com (mail-lf1-f57.google.com [209.85.167.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FA8iRQ005860 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:08:44 +0200 Received: by mail-lf1-f57.google.com with SMTP id 2adb3069b0e04-52e9b773505sf143654e87.1 for ; Mon, 15 Jul 2024 03:08:44 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038118; cv=pass; d=google.com; s=arc-20160816; b=wWRofWR39hPldD2k8Rg4uBWnXmdPiaBGY3Ld606jHFMLl/SsT5y9tO+DcBVr+eFyVE sIyN0/DsHYjuTEBPrBdl5aN4l8DLvYorQsUYMGSCvd+IavIR6Vr6cCapxVW9CSUf8N7w xOcDKkWu/m4VaNJdExfZBqCXQVmIKptnC3QObfP/q6B7aBo1YubpfiCGxs9/5T1IMWTA 4SDuZjjMaB8O2T1otXXiiccYireYZDo3bdNSTYZ+WHuYDz0CbQXznOUEgMNefta8eacL D0TZUjIuPZbwXq2Gjify27iww3vJq3GvZYKUsO0Yhd6EuzO88LbD41mG/Qr9TEnLjVCN TL6Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aaXlbdELNM5njS2AHhu/6g/g9G+Y7IDCpAeaQ1b2uPE=; fh=RKp1fUxwQ/wLMSLb3QfsYla85st0oUGClSFSZ5WiN0E=; b=QkbE2RgtbdILzt6q8JhwbSnV0w4Olga2I8fhuDWW+R31YcZyt5RLhCE5cFDQ5TxYNr UTI9M+jCXNTzwLyUScAiIw+FzYKreYf3sYtvspnhpoJzOCjfc9BJdX9LaFRk6Ehn+BQP 9E7MIXG1gzsQjX0rCYSwcQyGTqKgU3H6m4vl8vR6nDlwLcmBxvRQS2y8dRTn6OUXivy6 H2hCyvA6mdJ1bonja+ApOoWwJLmIaXwkuoP9WmzS51BKyc6Fvnk6AwNHKVcgW4GAx+se ymyiygupZGc/+p8keM69VJRGhG5yXbjHIAWfooMy8SaitDJWZ3uqkBwqWt1itxOPnhhp l1jw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=pyVC3gMj; spf=pass (google.com: domain of fm-294854-202407151008346382fe1be55c1fd90d-wkpl5u@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-294854-202407151008346382fe1be55c1fd90d-WKPL5U@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038118; x=1721642918; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=aaXlbdELNM5njS2AHhu/6g/g9G+Y7IDCpAeaQ1b2uPE=; b=ETzC4a3QvptjPqDxd2f7F3yOjX3tJo5k59HhaQDT9ORPYNFcfrN6is0l3Z+1unL2zk /uFTzQCcpOgJCuFVd7Ewysd545huRumw+2PpPQVjM+ftN2DZ4nX1aNDRTOeiAR4wgzY4 pHJIIP0lR8qtWGLFcpbPc3UVG6M47aq3PZ6/a/+jnOVNRycmYPMMtiBnS74e7lg0tSws 4gXc/LUrQuyW+AHkQXNVK0KFrekI+DC7WEpKRORcDkGDAZcOa66YSRjpmFJEI+AcDsKy ugsXh9YLn16a3Qj8zNGwaMpcrxANfANXIgRR4TiZDd60qaIyTUpSi4+SCKyd2fUo5W1p QcBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038118; x=1721642918; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=aaXlbdELNM5njS2AHhu/6g/g9G+Y7IDCpAeaQ1b2uPE=; b=fqIxOVY3u9ENNIJ1+ObJMPjRbNrCVQmA96HnGxRl7QCzCenxRSXyZ14lMUSwvuvXhL W3YvVt6n6lIDTTHdC9D16+LSQmymXxizfV9OxLy1Y2SXVmgq0NtZNugARg8/sVAaBuMG BhAVxse6uY8nXoaEx4B4nt7J62tN33zYZbl9cA55w8V2l6RNGanp4rbqz4Z1UUJVDDzM G7PgKEBWYkpR5cB3ZqJjeDwR2Pp6ZJWQKSzoIkGV5w8nj7/Q1nINCXym49Fm96GjV13Z ahe6oN0dUiAQ+O/581gkazXFgVRjW/KGJM4Ftoj4kLFnkbVsFFc9sRzJePhSYr64Jbu+ qzXg== X-Forwarded-Encrypted: i=2; AJvYcCXq2pffex5vANC7b1WLitZ/dh9BPEb2gXrcLNtKE3zvptzHLvi4n8E6VeqGgcwiFsuM2nZBW4jNIIKfZe9SDMhDlRdbU2E= X-Gm-Message-State: AOJu0YzCwaHv9HQc9wRwQh3z/HPR7QV7N/jFzqOeOoB4v0yTiFyRdzIm UY6bqUPameBBhMtN+GS8cGVZQxiszqC1bpk8Z5DFqGjZ0vp6gAd5 X-Google-Smtp-Source: AGHT+IHtBsAPIaRRfqnW4idePz40kpilocY9ID3aQqyJ7vnCWBVBSC4VLJwCCxlKb+FKekeNDquKpQ== X-Received: by 2002:a05:6512:3e02:b0:52b:8c88:2d6b with SMTP id 2adb3069b0e04-52eb9991281mr13298695e87.11.1721038117556; Mon, 15 Jul 2024 03:08:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:36d8:b0:52e:7693:6745 with SMTP id 2adb3069b0e04-52ec452193els1753401e87.2.-pod-prod-03-eu; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) X-Received: by 2002:a05:6512:b11:b0:52c:8591:1f7b with SMTP id 2adb3069b0e04-52eb9997315mr13748555e87.24.1721038115076; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038115; cv=none; d=google.com; s=arc-20160816; b=WAypA6drMz8+fLPtqGobkEDs2iM2fsKsx3EPA13FlVViFZwL7aVw89o6nVIryoHKN6 qM4OL3ORM9SwUkntO/X3jwFS7gFFQFGfDuHKF/aMojOzhrSHxYpJfz6732VmGFWrMA9L BmzVhmXfNMNX7gJGUDFLDw4FYDSiSsqOGqPhfP6La+KpLvQTZ88fHF3D/pNI6C56se0K Aek7fQWEf456+DRe8wvcxZUbVZ4mAFtv6ISMXySirYYNnvoYzi60wH/9PyhHY5947m6V TbzM5OYznJ5Xf9BNXcMI0PbYVHHcVZu8WbV8beuiHlwtNXPVf9cLNdQ0tZdknLxRq3da CfdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=PZNATY6IoSLm+0l8pnfs+jH3Y4sl0XJy+TjIS1m4FCs=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=TmJr4W0B6a8+quVMjGXFckNWofHlIobU/oTkdp4T7h9ku4RVh4KURYztkDbiy+u8+f jQWByn28BVBra/69EiLEEA8BpnAxv1xvzMLOJB5s9+pZb0a1y4+swioEImKAy9m7gx0Y o35VsrClj8WviZmJu95uC2o6KAhNu0txkE8ebgSEw7z67CHqq7DLVk/Y6NTTasT5rsfS jP867JJI5vNQ83Xc8Uyjk5PDS4xP6X+gPaUWzvT4VDSSKZW1mtBfpTUV1rhnbwe0IY8m usMjneviVUNn3O+dYtDKtVr0VJOb2SgWJ5tl3sFswxTk2LvIAykVaQMzSZeDhbWEv2bH XcgQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=pyVC3gMj; spf=pass (google.com: domain of fm-294854-202407151008346382fe1be55c1fd90d-wkpl5u@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-294854-202407151008346382fe1be55c1fd90d-WKPL5U@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-225.siemens.flowmailer.net (mta-65-225.siemens.flowmailer.net. [185.136.65.225]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52ed257c829si77019e87.9.2024.07.15.03.08.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:35 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407151008346382fe1be55c1fd90d-wkpl5u@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) client-ip=185.136.65.225; Received: by mta-65-225.siemens.flowmailer.net with ESMTPSA id 202407151008346382fe1be55c1fd90d for ; Mon, 15 Jul 2024 12:08:34 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 3/5] meta-isar: Add demo packages for installing prebuilt containers Date: Mon, 15 Jul 2024 12:08:30 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=pyVC3gMj; spf=pass (google.com: domain of fm-294854-202407151008346382fe1be55c1fd90d-wkpl5u@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-294854-202407151008346382fe1be55c1fd90d-WKPL5U@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka One recipe for docker, one for podman. Both pull from a registry that, in contrast to infamous dockerhub, should not throttle CI jobs running these frequently for testing purposes. The podman variant of the recipe is intentionally leaving out the digest to trigger the related warning of the container fetcher. These demos also come with kas integration. Signed-off-by: Jan Kiszka --- kas/package/Kconfig | 19 +++++++++++++++++++ kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++ kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++ .../prebuilt-docker-img_0.1.bb | 12 ++++++++++++ .../prebuilt-podman-img_0.1.bb | 10 ++++++++++ 5 files changed, 59 insertions(+) create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb diff --git a/kas/package/Kconfig b/kas/package/Kconfig index 35ba7cf1..395c3a08 100644 --- a/kas/package/Kconfig +++ b/kas/package/Kconfig @@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP default "kas/package/pkg_sshd-regen-keys.yaml" depends on PACKAGE_ISAR_CI_SSH_SETUP + +config PACKAGE_PREBUILT_DOCKER_IMG + bool "prebuilt-docker-img" + default y + +config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG + string + default "kas/package/pkg_prebuilt-docker-img.yaml" + depends on PACKAGE_PREBUILT_DOCKER_IMG + +config PACKAGE_PREBUILT_PODMAN_IMG + bool "prebuilt-podman-img" + default y + +config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG + string + default "kas/package/pkg_prebuilt-podman-img.yaml" + depends on PACKAGE_PREBUILT_PODMAN_IMG + endmenu config KAS_IMAGE_PREINSTALL diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml new file mode 100644 index 00000000..df96a484 --- /dev/null +++ b/kas/package/pkg_prebuilt-docker-img.yaml @@ -0,0 +1,9 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 + +header: + version: 14 + +local_conf_header: + package-prebuilt-docker-img: | + IMAGE_INSTALL:append = " prebuilt-docker-img" diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml new file mode 100644 index 00000000..d0b8da1c --- /dev/null +++ b/kas/package/pkg_prebuilt-podman-img.yaml @@ -0,0 +1,9 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 + +header: + version: 14 + +local_conf_header: + package-prebuilt-podman-img: | + IMAGE_INSTALL:append = " prebuilt-podman-img" diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb new file mode 100644 index 00000000..0dfc9b8f --- /dev/null +++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb @@ -0,0 +1,12 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require recipes-support/container-loader/docker-loader.inc + +CONTAINER_DELETE_AFTER_LOAD = "1" + +SRC_URI += "\ + docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \ + " diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb new file mode 100644 index 00000000..e671a494 --- /dev/null +++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require recipes-support/container-loader/podman-loader.inc + +SRC_URI += "\ + docker://quay.io/libpod/alpine;tag=latest \ + " From patchwork Mon Jul 15 10:08:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3690 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:08:44 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f187.google.com (mail-lj1-f187.google.com [209.85.208.187]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FA8hYi005845 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:08:43 +0200 Received: by mail-lj1-f187.google.com with SMTP id 38308e7fff4ca-2ee9c4475dbsf9948921fa.3 for ; Mon, 15 Jul 2024 03:08:43 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038117; cv=pass; d=google.com; s=arc-20160816; b=HNvMYRrUO4gpXPS4yR8qKQl4sybNYfGDC5zrl/FL6WOi5YwsOAykg65ntlnqPlaYax cTcZdgvOpZFm6zf8xvLE/ow6kWiznPs8lFICRj18Lt9zW4cnD8/tzml2YsBOuZRjAeHQ 1LRLpamCll3QMdDOiHYPrpG3q8gxsIsFvRc0f1R7AwhlKK5w7N/ycufU3VwRDjBM9guK hRChpjWvEVS0xVWPZFrNZUmcUFBc0RsC7ROzgvL8jrnOKRZyuUk5HWaPELvTHaEh2K62 EhfxsBvV6PblOBhMsudKcXY10Dyaf2cwpNK/gYlTb6EXatiaLpNpbskIBL4kC3YJZbni Na/w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bLOIRSkcypFG1xbTqmLEZqBbfOCDqjZQ2FmXPyVm8Zg=; fh=35VR6z/q64IK6GIPIPmG3XnEtKAK3XtUzDFv1KH1Dio=; b=i6Z7l4r3JPLRGsI9gn/OYo7/GZ9dOnIhM0X0T+3dPEU/Nu6hxvMxifwWjEyfLrbF/M PY//YGyFwr7I+CWud29lFFbB+mdRe8g/vQJXODkg4KGUGgMDpiTdXJuf5U5C0S08fWxX 6tzA/JlvAy34j/wakLyddqivgVZaFJY/puRBH5DyLgpO5fJ4+IkZDpy8fC/x/TA76lww 2azVBQo943slUAzCJ+ibcleezBJdE3tOPf4VcrkpPq7yAEYQSxsDp6Vk9mnMs/p1eoo/ 6OSYyNE0Olmrld3UANfuW0bLt6LrQuAz81Oo+T6eSscH1KRKgVyCDCfF/JsLW5gml9uO hSzw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="LW/jKcjT"; spf=pass (google.com: domain of fm-294854-2024071510083480682f41173ff8fbc5-f9qrg4@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-2024071510083480682f41173ff8fbc5-F9qrg4@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038117; x=1721642917; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=bLOIRSkcypFG1xbTqmLEZqBbfOCDqjZQ2FmXPyVm8Zg=; b=DyKmSg89yZSE0FR/ZQcpKiBccR3uxVPr7x8cyoeyKbbNXDaRSHnOZ64rohV0Z5ylLa RPQ5wPu5Jyw6JHgOltFompjA40+S3wEKTDj4AqPGzxSYgzaJQwpgMOsLE22iT31c9loG vWiACc2haoW+LzoUR79+9phSOwXh6u7CopLqw3yS6PCS45KYysjpvcIcNTnT1fYVlt1s 8YHjplqNGK17x4ZDsnZVw2xdWaSG+NLnEB/A9cmztCCBVRRTxV4CiW+w8Q38QNbnKg7d 3+avD4k1wiN9SPrzdEW16e+SqG0XHnxakztAvDRQlKkiiGEvlvfpqIEgXpsU01wp++M0 c+dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038117; x=1721642917; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=bLOIRSkcypFG1xbTqmLEZqBbfOCDqjZQ2FmXPyVm8Zg=; b=qv+njPNxSRDVFKNSZiqjriceKK5zHxVt2bHjd6XwCuew619ebIQt+YMtMZk8reQ4rZ 9bMU5w9iGdOznsNF5N6h4YtRSiydztz527nGb/4Nv1KXtR6NUS39GvhBtXcEZpu3HRli 8gQZJzBgrrYriebRun+mzUJwz2RtWTLyIBVuJDcGjhd88rFDn9OLAGjSBR/B3E8+lad8 ZQNg7Vb96JdoVi+flFnnH+uk98qLnOLUAhBbht3huwztosh8lSz50KYUiOKXwOZUBL1I zMWKRyPOvzFWCkq5/AWiu6Gq6pOWxtGPwXrFfKfXV8UI/O8Q+lgEaIAp9I8q7sThH2Hr Zo0Q== X-Forwarded-Encrypted: i=2; AJvYcCVx4lJc5qhhYFw99MPs4LbJAw5Aagv2US85b3NPr9EaQTGWVkmB+chwFIImiPfdaBlGyvdaVIn/2xC2qDyc03zF63rdo8M= X-Gm-Message-State: AOJu0YzheWYWyvfxRm8oUULIapSq+asE6mK5q0Hn8KUXegzVQLCMHpZw 2mTwU3UB/ybfKRfdWVcLibpl34Du60j6fWBLxFARPwZ+HcHdKrHX X-Google-Smtp-Source: AGHT+IHrE3FiFkefyNNsgG48ETyCa9AiuxvTmgAxvcehhwOcRI3oonMKYqPPia0wt4ub4ZOD8U4BiQ== X-Received: by 2002:ac2:5f54:0:b0:52e:9b18:9a89 with SMTP id 2adb3069b0e04-52ec3e53a53mr4554267e87.1.1721038117176; Mon, 15 Jul 2024 03:08:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d0a:b0:426:6243:ddf with SMTP id 5b1f17b1804b1-427989ed7c5ls16546505e9.2.-pod-prod-08-eu; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) X-Received: by 2002:a7b:cd98:0:b0:426:593c:9361 with SMTP id 5b1f17b1804b1-426708f1d94mr129390105e9.26.1721038115019; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038115; cv=none; d=google.com; s=arc-20160816; b=qRalZ4B74Nnbz+VWo6YobejDT9xQ3waH5hAhidAT6+qSaF39ubzz0EIYkICc7ObF18 HYP5gg5z/kqvdmNT4l5e7vk0abTctp4qu7YvkoUDmpLxoRcwvkp4leXXjfDKFezH+kFu yF+3qeAnwtzR/Dd6G8nNTX7SjH0A40kxb1OaVtI5Uxiu98rBckpvgmmr8B2fg2i6CAQL PFHDR4FdSFjvnNwRsastF+bNJP/mWi9b9lxszsuZhUfNg4+ikg9Wu/AUQszbi5P7wWQm Q8KLPEUSDaWvxJDLYOpcEsegQk9ntraxesJkqGkjS4Pq16ND2DWP7T2/kzpshaLCIYU8 9r2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=5Q8nkkkjzpXg64IKufplF2I9id/rT8Gipiy3rjOvdQU=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=qelkrNHA1VoM9fY+mfHZOsIhqZh+BAN0fU7Ss916O6fbiUwj77WgRzcZ9y3DiB6TCD hobMPZ9cZWYgLJWo+u5CqV3cX3wt0XJaKQ7mI4Xsdxt0vZG2NAJ8B8ZbPfTpXgcfimnH djeX73x+P5s5bNXkGwWl9b9oOymAf6DzRNiPb0byHDyCKupwAJResNflLLDO+Eid+g3V BFQQXOnEELJlQ3BUCZF65KDrS9RqsXT8OqkdMFMo99eL6xvoG7YSiBgR3Jsik0TtAkoH ndw6yzkU3rsqoSJNKqmD74wXbjjay1ra0J4umyIRdsF/BOUPZDzwaxrcnjMM9WAv9sxo zo+w==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="LW/jKcjT"; spf=pass (google.com: domain of fm-294854-2024071510083480682f41173ff8fbc5-f9qrg4@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-2024071510083480682f41173ff8fbc5-F9qrg4@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427a2db519esi841125e9.0.2024.07.15.03.08.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:35 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-2024071510083480682f41173ff8fbc5-f9qrg4@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 2024071510083480682f41173ff8fbc5 for ; Mon, 15 Jul 2024 12:08:34 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 4/5] ci: Add test cases for container fetching and loading Date: Mon, 15 Jul 2024 12:08:31 +0200 Message-ID: <67bcaf0f3ff3d5939f7df7061384758391e92443.1721038111.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="LW/jKcjT"; spf=pass (google.com: domain of fm-294854-2024071510083480682f41173ff8fbc5-f9qrg4@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-294854-2024071510083480682f41173ff8fbc5-F9qrg4@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This plugs the two example recipes for loading container images into VM-based testing. The test consists of running 'true' in the installed alpine images. Rather than enabling the ci user to do password-less sudo, this uses su with the piped-in password. Another trick needed is to poll for the images because loading is performed asynchronously. Signed-off-by: Jan Kiszka --- .../recipes-core/images/isar-image-ci.bb | 2 ++ testsuite/citest.py | 21 +++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb index e5d51e6e..9133da74 100644 --- a/meta-test/recipes-core/images/isar-image-ci.bb +++ b/meta-test/recipes-core/images/isar-image-ci.bb @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys" # qemuamd64-bookworm WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks" +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img" # qemuamd64-bullseye IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz" @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL} # qemuarm64-bookworm IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz" IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}" +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img" diff --git a/testsuite/citest.py b/testsuite/citest.py index 8dd907d0..539c9440 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest): self.init() self.vm_start('mipsel','bookworm', image='isar-image-ci', script='test_kernel_module.sh example_module') + + + def test_amd64_bookworm_prebuilt_containers(self): + self.init() + self.vm_start('amd64', 'bookworm', image='isar-image-ci', + cmd='echo root | su -c \'' \ + 'PATH=\$PATH:/usr/sbin;' \ + 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \ + 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \ + 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \ + 'podman run --rm quay.io/libpod/alpine:latest true\'') + + def test_arm64_bookworm_prebuilt_containers(self): + self.init() + self.vm_start('arm64', 'bookworm', image='isar-image-ci', + cmd='echo root | su -c \'' \ + 'PATH=\$PATH:/usr/sbin;' \ + 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \ + 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \ + 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \ + 'podman run --rm quay.io/libpod/alpine:latest true\'') From patchwork Mon Jul 15 10:08:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3692 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:08:45 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f55.google.com (mail-lf1-f55.google.com [209.85.167.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FA8ie1005869 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:08:44 +0200 Received: by mail-lf1-f55.google.com with SMTP id 2adb3069b0e04-52ebdbf8a7csf3099743e87.2 for ; Mon, 15 Jul 2024 03:08:44 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038118; cv=pass; d=google.com; s=arc-20160816; b=W44g5F+kTK2EygDGRPb60DVJb3nkcxVowZiNDasi8HDun/NtdDdEolaGg36OTPuMYi YvUKnEAbs710+A8sOY0Gbewqf/XPJaYrnqg5EWd0833Bihg3cb7xzcKwHX9q3nLWYqM/ Rto/KIcBwLp2Z27c/GxFfNrKbC0takTFkLBP6zJckLe8eYeX/OSrzTDyPH9uppzjQ2zT 90q91RO6FtlAMA8ulsOLmk/LDgJ85MgdpdvC94YO0Nq12zl/uAOt63Ovo5Bqsdx1rT93 I+7BrYyyZVpKYMgIT/RS1wBMqA9+T4VFsfbaK2lkvofnMW+8gBNrhp5CnLVSz3FizltI jc5Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IcEraZG8XQedbB0juPTShRBWi7IpMSDogj0nvMqPbow=; fh=n/a+DwmJADkrDYjJHmY97Apz5k9iSmzI8rPdg3c1V0I=; b=MJ4kepF3LFnt6lYa9oHpCQV3BDSDzrXVIbSZlEjnAwLIqmpHRugVkkx5ZAP0JAShUr CKtHPHVXtoJbIGJ/AFqr6MWtMyn/ZJoc5D3xEbq7xd3boc2+LuacwwPazsCtwbelH/rU C9daKZMnyyW8EADLkdo51wN/Qpt+82xZX1Ckhjy4fbOUjp0QVm4DI33Frmoofw9+RQ/e HkxPDCcznJEazgOc6wVweUCpINp3q/xDNvCa4tsowug4PnxaV9leb89Q4ZN99VgFqTjd LC5FTX6OHkc++Gy2mC1W9Vpl7vbiKsO1qD0D1hJNL8uHe5vM9mcrfHZebxgy5TLEiu94 8W4Q==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=EqVGyv6A; spf=pass (google.com: domain of fm-294854-202407151008342a7fd46636f45a806a-ubffof@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-294854-202407151008342a7fd46636f45a806a-UbFFOF@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038118; x=1721642918; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=IcEraZG8XQedbB0juPTShRBWi7IpMSDogj0nvMqPbow=; b=d9FWn3+Ngf1nBbWK0mN8MgGGYWB2A/Q1IV40gVjR5I4dGU3x6ebg2F8tH/NUcmGThc xPTh3sMD/zjf91IY3eqxn+D75rLGrYUKd5VXuDOfMpBq69nt/jnbD2SWhG9sb2Sxc6Pk z58J8yAGJtawaRn/n8koH88w1BSzvFX3GH8+edOwGXVGCL1DoxM6gGNcjtaKJYubNLmm XIxuPyjeSns++atY4fInF+eEXBT2cYb12M510QkjWN11ijaaFrz8LigPpnVK56Rrdff9 t2Rmu3SaaJA6AzamyDVUCueE79IgGDhnkKCBppHey1f8s4ll+cl3fUbm4AiztTedaM/8 CBsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038118; x=1721642918; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=IcEraZG8XQedbB0juPTShRBWi7IpMSDogj0nvMqPbow=; b=XUvAYeMZkP/TLgyWiwYzFQATEcV4BXwANbypnPSu0hhTUnplXE3Tv4GpFmFdrEXs9a NwUTPq/cR/krJB6fl+fwNEIvQPVQ0euOkjSeqyUDHegDKwKCOvxKrttbszDSX3e9qDFH uhOY3RyLVileQeBQC5eehFtjupgs3dxMMVrAsAbeEIGEMBUsN6dgbg4ntIsEV9HF1rBq SbyeT8yfO5cOJzQW7wvTt6rMMeHtFXNm4dqMREF6URErmNcsoYB2IakSnv59zuwJQq2h rckNu90toUpxuRm04SMt1oQCTUf3lkpre7E/dZ7ZpKGKZoo+eLrpf/VxNTcnmRK4s53G 4HqA== X-Forwarded-Encrypted: i=2; AJvYcCW3t1x86ofhFqYYfNEr5fsjT4TbZdyIDZNZ5NEMN8rEMw8TfK3mZnnY5Yt1wQ/wXbT3QdnjbQC4T6Qxf+PaIKsqgtSZ2Bs= X-Gm-Message-State: AOJu0YydWRa4AdvpOA0F+x2Sv4XlKIsjrBmDK0/KcnMIKthieJk3j3Ee b0Fti5n5r4+I7546QWNCGoy0bRJcHxPoePRrmAbyMgSIzLnRUbWI X-Google-Smtp-Source: AGHT+IEA/lUSUWxDA4D5Xy6dEFpb54qji4cJWNu0Wwx260eLIcuTZNML4RdSSK0ceImv3itqnxiAig== X-Received: by 2002:a05:6512:158f:b0:52e:9d60:7b4c with SMTP id 2adb3069b0e04-52eb99d4f52mr12265822e87.61.1721038117591; Mon, 15 Jul 2024 03:08:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:5a51:0:b0:52e:9923:a1ba with SMTP id 2adb3069b0e04-52ec4521d36ls1832997e87.2.-pod-prod-06-eu; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) X-Received: by 2002:a05:6512:2310:b0:52e:a7a6:ed7f with SMTP id 2adb3069b0e04-52eb99d4f79mr11963570e87.60.1721038115366; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038115; cv=none; d=google.com; s=arc-20160816; b=p7IuTXrMHT/hIvkJIh+kc/ZPFciRBgdJIgKCssC1zLEuh4ZrX5Q2DiWSoHLObpF8er 5yqSbC1nnljN/qIHn6MCSksUr46QXlp59oWsA3bbQxPcDioI+qS86buZ+AzIdkzEOQG6 IiTdMShqtpieeAdk+B20A/CNjBt2jVbcKBBrw30NMywMTFk6k8AgZfyto5i6snOZS3bt swPH93mDnZPMvk2f9XrwxfRykHR8CzHQli2/Yq74zauDOoRPsL6hARyx2rJtrNtv6KfW TI/tpfYMv+Z0VbEaRDQ1bBoo5GeMuyB9+I8av99Wbk/7Sr1/3cFiT3fyvWYwkFJs90g0 x5tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=hcbrhizW+zLGQ2MZFaZRcEJ3AvAEjfR5l/8EpKGdW60=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=SkH1Wy8KA4EKwPP8mtb4leTSeVT4VTTgx5hwEpJY0bZ6oJ2nNLwNflCB8z/N3zTR14 JFUquInO2qoEfeVk5gjd5FRYPm2YSzk/P8UzRSS5GqyrIJqXlRV9P9XLG33nce8iUuoS f0qpPGRc0rP4jtaKdvbw8NqE7zJIRA82Uy7481daDGgVEXobbRZK9WA93KoZ9HW3NHUf //7XOAZliRgj/xhjRUmvHQnSd5WyWFfpKNHkjr3gQmCbJx7x6jCP6nh69AO2jmCM1KBG C0jID5Z2UmCG784PBv1fmtRM2eu9nsrMGyn6UWvGwuZtEeV+2cYVTQjD0qwnE2yrKOWA +dVg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=EqVGyv6A; spf=pass (google.com: domain of fm-294854-202407151008342a7fd46636f45a806a-ubffof@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-294854-202407151008342a7fd46636f45a806a-UbFFOF@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-225.siemens.flowmailer.net (mta-65-225.siemens.flowmailer.net. [185.136.65.225]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52ed257c829si77019e87.9.2024.07.15.03.08.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:35 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407151008342a7fd46636f45a806a-ubffof@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) client-ip=185.136.65.225; Received: by mta-65-225.siemens.flowmailer.net with ESMTPSA id 202407151008342a7fd46636f45a806a for ; Mon, 15 Jul 2024 12:08:35 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 5/5] doc: Describe how to use the container fetcher and loader Date: Mon, 15 Jul 2024 12:08:32 +0200 Message-ID: <2e1537004e7ad48f33cd3d3daf2b8f7c3294d31a.1721038111.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=EqVGyv6A; spf=pass (google.com: domain of fm-294854-202407151008342a7fd46636f45a806a-ubffof@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-294854-202407151008342a7fd46636f45a806a-UbFFOF@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka Signed-off-by: Jan Kiszka --- doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/doc/user_manual.md b/doc/user_manual.md index 776ae52c..733b2b30 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "" Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = ""`. To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`. + +## Pre-install container images + +If an isar-generated image shall provides a container runtime, it may also be +desirable to pre-install container images to avoid having to download them on +first boot or because they may not be accessible outside of the build +environment. Isar supports this scenario via two services, a container fetcher +and a container loader. + +### Bitbake fetcher for containers + +The bitbake fetching protocol "docker://" allows to download pre-built images +from container registries. The URL consists of the image path, followed by +a recommened digests in the form `digest=sha256:` and an optional +tag in the form `tag=`. A digest is preferred over a tag to identify the +an when fetching as it also allows to validate its integrity. If tag is not +specified, `latest` is used as tag name. + +When specifying a multi-arch image, the fetcher will download the images for +all available architectures. If this is not desired, directly specify the +digest of the desired architecture manifest instead of that the manifest list. + +The fetched container image are stored in a directory in the `WORKDIR` of the +requesting recipe. When a multi-arch image was specified, only the image +matching `PACKAGE_ARCH` will be stored. The name of the image directory is +derived from the container image name, replacing all `/` with `.`. + +### Container loader helpers + +To create a Debian package which can carry container images and load them into +local storage of docker or podman, there is a set of helpers available. To use +them into an own recipe, add +`require recipes-support/container-loader/docker-loader.inc` when using docker +and `require recipes-support/container-loader/podman-loader.inc` when using +podman. The loader will try to transfer the packaged image into the container +runtime storage on boot, but only if no container image of the same name and +tag is present already. + +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images +remain by default available and may be used again for loading the storage after +it may have been emptied later on (factory reset). + +Source container images may either be fetched as binaries from a registry, see +above, or built via isar as well. + +### Example + +This creates debian package with will download, package and then load the +`debian:bookworm-20240701-slim` container image into the docker container +storage. The package will depend on `docker.io`, thus ensure that that basic +runtime services are installed on the target as well. The packaged image will +be deleted from the target device's rootfs after successful import. + +``` +require recipes-support/container-loader/docker-loader.inc + +CONTAINER_DELETE_AFTER_LOAD = "1" + +SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim" +```