From patchwork Tue Jul 16 14:18:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3701 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:22 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIL5S013354 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:21 +0200 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-2ee8ceb0852sf52097191fa.0 for ; Tue, 16 Jul 2024 07:18:21 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139495; cv=pass; d=google.com; s=arc-20160816; b=BL6oGkUNJgT7BwhJSCdHgNdHI1EfdRM7OWUIgAcz4ddcO/8KFWX/MuA2F1Q6/JRfPg HQ1RN44zxsypvnMerD5F07Fn+ZfwMEF3tXfDfmHmX3G7D9jLNyMTeEJ8GSwkTWNthqJe fUitBq+mKpOn1okyoGe+RBw61iwLGr/nwQHesH1t51fdTVK6A/IuwzhoCUwizMBN/xKj XLShuE22OYd7u17NeZ6sRrSgTOkjWjeJDc5teZT3d88Izrhv7cYxKNssyRK3cdHq2OU9 X0G8nX9CPqJddT5FJLJ+pwLo/DzjPbw1k7oNZDzFgtGsS9Sa/3tRNuF4xMby0ZDASsMJ 2q3g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xdIictTAYnYgwEr3bkpnkim8XzLLI9JCpIo5FXw+m+Y=; fh=TcHzqIaBlqsn6gYEnG2DFOCPZkIFC7b96dPD7HupqJQ=; b=bOS7SBJOQHGw5JnWzYvkgt5HHwWG2um3E+0EVXxIRLMoqpQv9s4ZJlwPPXnFwvaZbb na7pqJgYNxSe9vQm/ZL8ShVWg0SoUKu62EuCel9oeXKo1fkhrBCB/zz1zOFXwkyUwqlz cH7NPDHqJ0696MVdxHZTV8PQ4vTcj4kBf9hmnq3HePIKTj7qtuKDUCsV9yH/2dYgQ7kQ siq+Npgfx26PCMjbFAFJHnMGQ2ITL9Bsmjujr/Ror4zYQCkXCIbgcDIlYRLa8nPWwy7m bEYboKSHZFMd3Ka/Ynk0+xprE6wHK0AW2pzlVB3sjNvi+ZEeJoJhJJOkmcP1Mp/nTMYx DfpA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jUMoGlY9; spf=pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139495; x=1721744295; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=xdIictTAYnYgwEr3bkpnkim8XzLLI9JCpIo5FXw+m+Y=; b=B10wck7gHwGK6QDZMQLElJMCPoeg4f+A3dAoxFlZ/rv6v/STUCzax9Q9zxUToHcS1l 8miP+zDk/mUOT1lXLDH9GBULH9V+kETWrH+taIi0CaKxxnKsHrJqPGACmg8TM4O3Y8MF WO9p7WN32SPszmjvHvWmMKK65pbCp0oa84Xa3dPWt83/K5ezrDN5I+0mZGTV5cMTD8VR LR8RnMrDBoXLCezi+VZNkghPVnWQzhG0ydblCFWlE0YkmwD/sFLeBVlxU9aqdp2M/rwz dv+yVLQc1NuGlUq5RVuRYVp+ExvkuiB2f2tm9+M1b/PzmGGuvKbOcGOcugPHrjxAjSfF JiXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139495; x=1721744295; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=xdIictTAYnYgwEr3bkpnkim8XzLLI9JCpIo5FXw+m+Y=; b=rkBB/IKYtnzcNfjBhSL5l2tC+V6jHklLHlC1tsR1m0utQOkDQPi913VaU3vBwD+Sdx uJKw+xnAubRb1WOCjwfuf1rwsACbrLJKTt5xGphZhWeT5j3ofJ4Cq1z2hEdGzAfOPHRE 08boDHhF7nrEzxirBWppGgU8zHpQHH64k95c1ZmEGM1Wf941IEg5PA59olpK+dhhrOB/ zIkFCNZv/f+mkZ+pl0GN/4cgfPgoJHPUclsjinqcLa3VMcJrB0n7AjSsvhh7gemYIwD6 /63taGYzVtdgcC2QKdlG539mT+dPbnNVMJmmMeqNcMMdDkEnOojKUMz7PKWyfSOcjXPo pR/w== X-Forwarded-Encrypted: i=2; AJvYcCXL5b8IL07Ns/+1w6gd9Rhi6clkJu4//UhV1XmZgFFkrfK/ze/lPBhjohjFH4Y1sUdj7lip2g/bT5lDbhxU4jkqUcKOqcE= X-Gm-Message-State: AOJu0YwiFCTOR1FKgKnvhqkZEga8bD6OgK7QCE2cpJdYXueNk3wAeIt0 oFJhICQmX4LCph54g6FHui2hDrNx7USJsRhz6PF6T6UsIP17SlJT X-Google-Smtp-Source: AGHT+IEkut5IBHADmdpr861edZmn+SVVXfW7QAtvZJo4vZakDZuRC5RMo8c0b7+Y2vA6li/x9c3QCw== X-Received: by 2002:a2e:2c1a:0:b0:2ee:4ab4:f752 with SMTP id 38308e7fff4ca-2eef41e333cmr15975261fa.49.1721139494269; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a481:0:b0:2ec:4d9a:63bc with SMTP id 38308e7fff4ca-2eec93a6e39ls20316771fa.1.-pod-prod-09-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a2e:9b86:0:b0:2ec:50ed:b59f with SMTP id 38308e7fff4ca-2eef41d739emr15106311fa.34.1721139491779; Tue, 16 Jul 2024 07:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=SO8CLW7mDHFvMWEOczJ52pyNKbXt8vDHHx5/DHe/2gSyk2kU4saodLYptAjOadnvMM ixzq/TYUd8YwSYd1bBz24hV/bZyepJaT7yKhyiXMCMzd6V8DuTA5NcW+RZ5Q70avZSVF sekQtVO6qY/paMfwbpS7iARCNxXdMZAqghdECrUUjVgodtISeuMY+hkrOSwjrEoHp/j+ cvHKYZz2xDw6V2HbcovnZrWPfzggQ6le84P4skHr9ucQYB/Q5mL5rgrv1TFmV5WLsHQj HBFGG5gV8XdZY2Ve3nIAWDpy4PKtxPsi5vLcZaAL1I4/BAyanYGcDGoln6DgB5sv3fOv e/4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=qT0qBD6zoZX/tPGQ0GTumBOagxBAE1nu2vnVszdRf/0=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=haMbP2oP3AdlVv5UIJ/OL9ViK/qqFxUha+2FbQdGu9Os+GpFGzsc6pquvSgHh05l3y lDw/Xjvb0araVsdRbuHDcw3MJw954V4cGLaXN2Js7LZM+YHNGLZKEs4hZ7z3ohZFVDsF DwWarwp/AwFpJrUBnHcHWgT2lIlcGaX6V7wmqF/1LIEwFLPEvawf1kjqcXwn0iD1PsSk bYK70QlJpveyPakZPwCW3nmFaMijCnkostATkqu6wmeStZ4bvHlfOOrr27pm+E40mPtz ywoYgWdlxgBdeduGKCOpjluZRaYmeYhMhCrbtGS4jvpa0C9XoJe6UEdBh13zDWHcAc8x X+yg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jUMoGlY9; spf=pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2eee19149ccsi1386831fa.4.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20240716141810bdb1dfc2f7c434762e for ; Tue, 16 Jul 2024 16:18:10 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 1/5] Introduce fetcher from container registries Date: Tue, 16 Jul 2024 16:18:05 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jUMoGlY9; spf=pass (google.com: domain of fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240716141810bdb1dfc2f7c434762e-rj_d6t@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. The fetcher will try to pull all available variants of a multi-arch image. If this is not needed, you can also directly specify the image digest of a specific architecture. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..8513e246 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,90 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import os +import re +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd +from bb.progress import LineFilterProgressHandler + +class SkopeoProgressHandler(LineFilterProgressHandler): + def __init__(self, d): + super(SkopeoProgressHandler, self).__init__(d) + self._fire_progress(0) + + def writeline(self, line): + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line) + if match: + state = match[0].split('/') + progress = (int(state[0]) * 100) / int(state[1]) + self.update(progress) + return True + + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + + def download(self, ud, d): + progresshandler = SkopeoProgressHandler(d) + runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}", + d, log=progresshandler) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + arch = d.getVar('PACKAGE_ARCH') + variant_opt = "" + if arch == "armhf": + arch = "arm" + variant_opt = "--override-variant v7" + elif arch == "armel": + arch = "arm" + variant_opt = "--override-variant v6" + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \ + f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d) + + def clean(self, ud, d): + bb.utils.remove(ud.localpath, recurse=True)