From patchwork Tue Jul 9 17:31:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3673 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 09 Jul 2024 19:31:50 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f62.google.com (mail-lf1-f62.google.com [209.85.167.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 469HVn1x006532 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 9 Jul 2024 19:31:50 +0200 Received: by mail-lf1-f62.google.com with SMTP id 2adb3069b0e04-52ea96517a6sf4654957e87.0 for ; Tue, 09 Jul 2024 10:31:50 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1720546304; cv=pass; d=google.com; s=arc-20160816; b=KtzLsS2zdi6yG3iikI28zpsGKURTpM4AeC6lQuZdFkoc+HusnoBX8G1c0LMdPu4ubW 1+4CLM8R5W+qDtI0DQzdkSy4lvQVMIexMlaazOEEGIzZrY+zNbUfBzKhta3vqfdEB2Xj BimxH0QEwZrDCxIplbZJqqEBQP0BV5Kch6Xs1WH4dbh4q5v1b9GKweQi62JgLUg3WpFd sXDC5/F8TCw5cRFzs7cnQOcrt0kYGngmbG98wYf2j+QCe1Etr1u40M/H+pZvfp9q2Xb3 FSoqY1QZekQd/owMLOfcF37dH6x09SbwSkqgffgUZMvV+dwImGVHmZhCnaKUpIdkD/D1 cfnQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=wUR46tngSfB5cZc8nNKXaUBTvDRWVT0hOZzOppbQG3o=; fh=zAgQillEVyp+qXIT1i2gYHOKKVEx56lyLs3BNR2VE2I=; b=DLkFQnEhiSbS/0xtiYKoU+tQFquGjwBQTTxNkPtEPu7pJOmq/PQRNkrZ6pF7EQzgZ6 /wYEgjXg9iJQlU55rbYDKI0EGaPaDA6fjob5QwWCUcDKsoHCxH/66e4PcKH/mjW+drnM nVEqweHy+RPSN1LO6glUsMgs7E4SiXZbMmQAosL1DlbiCnSbM3E4Bdlv8lFCyA37vvEg culokzfF1HamS5Zsb8AP047ME7gOS2aK81UxSUJMwdYlAK7fz630W9jUMxXm6gGrJXEQ KTTt2U4c4NVaeDgmR6ywPq3F74FKuxEB6uj0WK2orVfj8EynYLK6rxnKJJKlfhF5fF0g DTYg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=L3VrYHtm; spf=pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024070917314078e9b583f9bda058c6-FIUSpa@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1720546304; x=1721151104; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :from:to:cc:subject:date:message-id:reply-to; bh=wUR46tngSfB5cZc8nNKXaUBTvDRWVT0hOZzOppbQG3o=; b=lzobixiV5rrc/A2TVPp+KeGEwpGtSgoK061g3O4dja6iaA0xNG2Dt4HkBOM6Ndiv9G MHJKUyoe/Y9m+ITlHttYR6qRINT8LtfBJIfyWj2EsPbuGzcNWXa4wmKZUvnMRrvD46/+ QgcUGS4bKo2AXSM/3cCImAPgYFqv2qUSSe0ggNGsMl3O1Ca/Lx+VRgSFsk9A6VYbhuvO vb81gq/DMgz8s+m0366bZvgN/ffiISM8CldMO9eucIf1rDTngyATS+vXj3Uf4yjnw+B9 ai3K9MFHbkdBsp4CzWJChBNcYQ6/eEeOR1l5cw2TSFnp3SO1KLkzGCpXlN+jyVfjNiRd 3HBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720546304; x=1721151104; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :x-beenthere:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wUR46tngSfB5cZc8nNKXaUBTvDRWVT0hOZzOppbQG3o=; b=khdPX+ATShHQETROFjpmTgaCIYbqTuv34xwK4b8IeXGGlYo+Y58db60jUD5PGwcaEN vpIQSYqQTJzk0GeutDC64PBExsVcR0rV4lfmClxldZrYXG6bj6jvruMxtPPW1uVX0GpH PRRQoRTRTuCS5B4uxPE1YIh4VFxX4A+9bP3hFu6hS/THHw8/ZAu8XmXY6rHM4UmcHNL0 d/ud03hQvU5S0JQ9zxGrJl/4y5pMURhUws3Hyc7MNyoO/hOb+9BeLRkafq+5lzl5e7Vt 7m1Q+ZiABR+AQ3wPr5oiigFM1NXczZ/ESVTynNvFdZ8WJ/9Kz06Sar6I5rU0Jeqd4DQY IuVg== X-Forwarded-Encrypted: i=2; AJvYcCWF451OccETgoEsl9ly/Z8qkA44MAOgf9em4bzS7KjwRuAhoVXzxT8dgMzzLrXX0EaQB09aqjNa7l4o9sK+x+MKhwloH6Q= X-Gm-Message-State: AOJu0YzcIdq2gPupBPjQ/hZguZb5MTzZxCly+79Ccx12YfGssJKkSqUO wPzkHpUc3TyYGjH6CoHdwc+JUROd21X7luavkq6h+o+Xn0z+UvNV X-Google-Smtp-Source: AGHT+IFbzF9vTf5QEz/NXr6d4bHeiD8Y5qbN/UinPZKyVhF9jEHU8xhSb+v26eFdi1D/nf2tP/6MNA== X-Received: by 2002:a05:6512:3b87:b0:52e:9d2c:1c81 with SMTP id 2adb3069b0e04-52eb9990ccamr2336007e87.13.1720546303719; Tue, 09 Jul 2024 10:31:43 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:31d3:b0:52c:a105:90ee with SMTP id 2adb3069b0e04-52e9f301e67ls1488533e87.1.-pod-prod-01-eu; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) X-Received: by 2002:a05:6512:3d8f:b0:52e:7688:2817 with SMTP id 2adb3069b0e04-52eb9994ab3mr1863067e87.17.1720546301347; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1720546301; cv=none; d=google.com; s=arc-20160816; b=LNuVddjc93aBv7Ulep3tn4Z7qY/62ThyFWDJ05Z1Kh/rqH1CVe/LRU9fLrrG63LTVo pw0F40l8J3Uxy8lk0x9MTQfJNM6mzW28zVG7Stn2Vp0ep+qUE0XazlL1hjCDoWSU/VRr FuYGUzwJZPcFbVRicFGq6M/xnpMUK6mdPV2DO5nP7Tz9mPuV3W+MTekN12BucEgyu9gM acJe/2iDo+4pslaNuFRB0sv+XCd7kCKfQ1TEZc/7REw7xhc9s5z+lwngrwqDkZN5d4z5 LC7W+CryOWkzyWTGSAqoG0+C0gxlYeK4KxT87wWwA2F4vAsNYQqP64ZYDsdFyBN2hCIb 6hiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:to:from:dkim-signature; bh=fanyJ2CdPImoQzmlj8y6E58TDb+QAfcBjuwhNoStB48=; fh=hOBXq6nisExkIRIwN8IDOmOCLKwX8PM0ANlxQbNMEqA=; b=hGom/G19dkdpbmfRCVWh3ft5rZjChIdc6XNke4ElhMcrr23MkYGtipEb196XjXle17 gfqMqv9gjyF93zFq32Nuvw2oRmAvUWj8t5qpgP1uORzdJKoOHM1TIq/baOIA28MPrkxu Rfy0qZkSuwNh7NiWKzpjKHQkaQluX5i3vl8tfXuv7ZaDiGOevg4oYfXYsNtCjRlSpS+g 4wtIiM88+TOi1g+wL4lXQPr2gCaNIAsOws2uhpP9SulP6XBP0PIx/nJfiN5U6kMqN/Rj KYnpENROYRqJO0xo6aUD9O/1HIDMSmzkp5YT0JPN8XWRhmO5d2M8Jz2CGK7NQOlQ72cc A6hg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=L3VrYHtm; spf=pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024070917314078e9b583f9bda058c6-FIUSpa@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52eb8e5237bsi89199e87.7.2024.07.09.10.31.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Jul 2024 10:31:41 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 2024070917314078e9b583f9bda058c6 for ; Tue, 09 Jul 2024 19:31:40 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Subject: [RFC][PATCH 1/3] Introduce fetcher from container registries Date: Tue, 9 Jul 2024 19:31:37 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=L3VrYHtm; spf=pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024070917314078e9b583f9bda058c6-FIUSpa@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. The fetcher will try to pull all available variants of a multi-arch image. If this is not needed, you can also directly specify the image digest of a specific architecture. Future versions may also introduce full unpacking of the fetched container layers in workdir if use case come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..8c713795 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,90 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import os +import re +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd +from bb.progress import LineFilterProgressHandler + +class SkopeoProgressHandler(LineFilterProgressHandler): + def __init__(self, d): + super(SkopeoProgressHandler, self).__init__(d) + self._fire_progress(0) + + def writeline(self, line): + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line) + if match: + state = match[0].split('/') + progress = (int(state[0]) * 100) / int(state[1]) + self.update(progress) + return True + + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + + def download(self, ud, d): + progresshandler = SkopeoProgressHandler(d) + runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}", + d, log=progresshandler) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + arch = d.getVar('DISTRO_ARCH') + variant_opt = "" + if arch == "armhf": + arch = "arm" + variant_opt = "--override-variant v7" + elif arch == "armel": + arch = "arm" + variant_opt = "--override-variant v6" + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \ + f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d) + + def clean(self, ud, d): + bb.utils.remove(ud.localpath, recurse=True)