From patchwork Mon Jul 15 10:08:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3689 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:08:44 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f186.google.com (mail-lj1-f186.google.com [209.85.208.186]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FA8hK8005843 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:08:43 +0200 Received: by mail-lj1-f186.google.com with SMTP id 38308e7fff4ca-2eebc944683sf42359391fa.0 for ; Mon, 15 Jul 2024 03:08:43 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038117; cv=pass; d=google.com; s=arc-20160816; b=Gpa8llAeNea2kSHC3CZTwu85pwqkfu2yPduszjeGb1s3ID9dyKSCMRaXH3GBiDrCYv aqVijjtbIIYJdFUa0Py3FAkcWkkaZRl4RKgoIDW5PgL8TSz4sJDNStHY0YwFnwJVQdzH TaBY38NzDTiVl+Md2+0A83GX3G5xNSvUBY4uRC5C6YGxJNwZcWUAhhTZurdPZQoekQjo bIwGle6FhHYbZZ4C+kNKRVjqHeYvCbQnI4E5dkcqopTsXuKUWRba2oZ4pEQB2C/B5aSE 7AkwAD4O0j4fWwvaoBzCh1rnj9ATLkkGDJlEoDOuIZooXlcJb+8T30a2j0nld4gEWRrh Nw0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; fh=pX/53+dQ2lkjFigwDlnf12FA2ODDehKXsA5Uygo2/LU=; b=K80klFpFiwhl21VPprtlfz3/uAHxcyGTsCTR4FmShKhvsz7Bv1LLfLmkZilVJAr2Fy T7yJKQWaHzR3i3dSfdqojYzjxnNjhplr+6nwvZssBPeES3h1R90QtL2l74NCmZiFAw4l Hg1XPzOXA4GpFT3qDuHIrjjnNX6aYKC/BGMYmt3ZUhNnpwUwFP8NEszWLGjcAmXG4zu7 c0bKLf2wzq/Bf1/M5R5pS0jOW+9LmwhK09jvk/YklJoyGILd3CPC9orGaoDynrliTV4e HZGXN7t0AWXtNUwb7haEz1tHr9cXjdUDopDMQrgFiOCnPDH5BA3Tv3Ov2T0TiH50P9YZ Gryg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038117; x=1721642917; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; b=PaisrenUlKjSQ2yuXaDQMj/HJErG/DQJghiWPy0z29ZUGqC1MEwvlqk1e9fJOcoksw OnV9m02Ffreft7n28amhGaJGGk0u1CyCatPGR3ncSaTCoC7LHLR4Q87ih4Z3TD6NyOO1 JnEd3G0U6I+nE7BJZQEk0ZGIWe7g8LvdvTPlFj/7Yy9UvEdUMtRJvnCl2fa2RGLTEXWQ WGaS6YIj4TZhm8CHCAw2QRYF5hkUQW3xmcX6/oY0jQvVJMYCx6V2U8NheJwrjocEMcEU 0/4riAo6mFv8Ex+meHSrD1pV3yjfNF868G8YQOtoUOoFWeicaf14prQRP2Em1M1BbtQi uiAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038117; x=1721642917; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; b=mBy7wFdw/Mtr8YdtFV8VZ2KGFBD6pXXhjLf5kt+Uy/hoaWoUohX/YjqATd5ap9VPYe IlNa1Wke6cgV9VKvs0MK8WlX4i1+aHkH6nCSbZk0ZJhOV3Y2cDpiU5jxeuKCpK4gprYs mXq0Wkb7ggm/PKODgj3sJQfNI07ZU5HbKj+rlFgsb1aM7zwm0QsbKhw2Ta6rIODN50lY PuwydgGROklhjlP5D9rESYl7ZTMUjZ7So8C3cgKCzhryig9v1sHXK4QpldvekZEClK+I TcTYW2kooCcan8opREBGzJ7Sc0Vn9hhSGK0hTNuPlTFtfvLv5O+cOv8YH3R4IjYDPDNe vW+g== X-Forwarded-Encrypted: i=2; AJvYcCXdOYGVlffSVHh7exw0T6ug92bt3OPTdg2TxOMYGwirC8qS+v7CJ2G7pWsBv2Y4PlkTZA3WgNhSv9WGluXJEDNrPEqjfYE= X-Gm-Message-State: AOJu0YzqaZ/auRbgV+RNo3NbFMpWpFmTF4SK0f0GxlWV05Q2OZvRAPR/ ZD973bW8pULtGNtWnUxCLaJDLAFHgeUR229xAMEHw/CEepcvWoWE X-Google-Smtp-Source: AGHT+IFZUg67Cmcta18vrBLhsdSqhwJpD8LW8jaUlnneW5JcfMO0tOKf6DhcZQ6SlvhIsnwkfF5tGQ== X-Received: by 2002:a2e:8081:0:b0:2ee:83f7:8b34 with SMTP id 38308e7fff4ca-2eeb30ba05cmr140867551fa.8.1721038116249; Mon, 15 Jul 2024 03:08:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:489:b0:2ee:7a6c:fcba with SMTP id 38308e7fff4ca-2eec938cc1fls18789091fa.1.-pod-prod-08-eu; Mon, 15 Jul 2024 03:08:34 -0700 (PDT) X-Received: by 2002:a2e:911a:0:b0:2eb:2e0b:72c with SMTP id 38308e7fff4ca-2eeb30dc843mr140984731fa.16.1721038113883; Mon, 15 Jul 2024 03:08:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038113; cv=none; d=google.com; s=arc-20160816; b=JU4wn0VRM5hy+yIn4zvGzsxkvRsgoHtT/q+G4apQH+GK66rwS9HzPanm5gjPUsdKQx 5UUWpLBwEVTSvC+yPVHUbTKfz/BFw1FacPWufxGOxPM1WPBO0g2PGNMbdkLNTFiT1jkT y/2qAajUQkpKVfz+MeBmJ71dLnvJnjjyfwtksW+5rblzq9hwBSQaKfAMp7MzcK/ey6jy D5EpNu9TTp9CEq88/xFhODnVomDfcEa0eae9IwKkOqvnESFqregVlh1rl5eMuHGva9tO C9HVBgk9ljSqDbVu9p/vyArtOS/1chLD3iefsxFQWwI0Yp98YEeEmB+OIsu7GAGwPw2v BUKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=qT0qBD6zoZX/tPGQ0GTumBOagxBAE1nu2vnVszdRf/0=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=AE6Fas+i8Fg5x6PpoF4G3P3lo6fGTAZg2OPujtmZa2vcT8q4VJLqHf2pUb46YW572F 3E0/OQsPoCUNoRyv1CtFG5yv0wF+JheNbL5xkTAPca+ZciRNPVj/roZDna0IC3L3jGSK 1N6+v/lEv4aDVRp+/0cyoSqaWolmlPkTYEbfl90FLuO8XgdLtnsjwJdF3BYNMnQE1k/A 9061UVWz9jEDZyU2WAbk0q9HY9/oMjlDAE7tLw+GjUSEad7YTTiKsKJ4OL5JijwI3PwF QGdW5q9jKqPLY8fiXn8UrVX7yMA2cLNRudX9rXYukR8CgNMNrcJ/ns59T8S7WwuijQIl kHCg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net. [185.136.65.228]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427a5ec6cb7si674365e9.1.2024.07.15.03.08.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:33 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) client-ip=185.136.65.228; Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 202407151008334113b2dafda22e9ba9 for ; Mon, 15 Jul 2024 12:08:33 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 1/5] Introduce fetcher from container registries Date: Mon, 15 Jul 2024 12:08:28 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. The fetcher will try to pull all available variants of a multi-arch image. If this is not needed, you can also directly specify the image digest of a specific architecture. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..8513e246 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,90 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import os +import re +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd +from bb.progress import LineFilterProgressHandler + +class SkopeoProgressHandler(LineFilterProgressHandler): + def __init__(self, d): + super(SkopeoProgressHandler, self).__init__(d) + self._fire_progress(0) + + def writeline(self, line): + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line) + if match: + state = match[0].split('/') + progress = (int(state[0]) * 100) / int(state[1]) + self.update(progress) + return True + + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + + def download(self, ud, d): + progresshandler = SkopeoProgressHandler(d) + runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}", + d, log=progresshandler) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + arch = d.getVar('PACKAGE_ARCH') + variant_opt = "" + if arch == "armhf": + arch = "arm" + variant_opt = "--override-variant v7" + elif arch == "armel": + arch = "arm" + variant_opt = "--override-variant v6" + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \ + f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d) + + def clean(self, ud, d): + bb.utils.remove(ud.localpath, recurse=True)