From patchwork Fri Jul 19 16:38:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 3708 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:56 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f57.google.com (mail-lf1-f57.google.com [209.85.167.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcrBR002645 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:54 +0200 Received: by mail-lf1-f57.google.com with SMTP id 2adb3069b0e04-52e993e31a9sf1705029e87.1 for ; Fri, 19 Jul 2024 09:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407128; cv=pass; d=google.com; s=arc-20160816; b=flwalhknr47IFNsMfs9ns+jSOlHZh0eD5erDnH2PCll/YR/KMuxySlMSPLcdIs7/I0 NQ5yWH6mUOJ4j98/5UmAZmqnjgnhoDZKUyzx6qkhXvjZbLX8x31yflfOQU3OdxeS9Yxx AG0V8hWeIDcpiYbt59rRINc0up83J2F47fY75wXGaolGN/70y/GPvVALN9GgBXUCeuCY AlkBBD7gVXZ05pdZlFLZa6sd1DyJGDFR+J4hVi/xTwTYv/8kA8OfQJQ2XJCjv5HyeCdK h4R1Lxxh9ISBwMUM/Bt7zw/hIr48kPZ9VXfz6U3uX66dBBkbozXKRB2reXGFBY8w4de0 ub/A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; fh=IHp74hTUh0ItTXC1vRjKc78GIVciWXPlmP5xzMrXiKE=; b=zdIM03IvLsV4cPWBjnTUSlY/dEupLR6+diaVFgceebw3Taj0XcFxc/ZKdhvBm8aslC 214BxH9moKtw4Pdq/zHqwSMV1GmCKw7TkJeZYng1oq0Els+yF9FgK02XTGV2yYgBswRh 3+gpwN5YSCYl0z5gkmFApr+XMfgMItG9EizIWdeTo1CkpeLk33hBajT0qlXhtyws3qnT 5zIuPoJlvmWs2XbZZIInP67KZOtCa05ndUg9W/YuGOHxyrRGWl1bp/jMBMuAQHBvGHut N7fnCOOTiTlgJYQEQnAYVz8USVyeye7NKDyR8BGalNjZEYyFCG47UaqvLhpr3uollzcK BAnQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407127; x=1722011927; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; b=NT1V/skf7BCP8JE+H531rKubo+j99CP/ir61W0gGI+KYsu6vIfqjf417Sm0KLYz1an VxJFv+q6w0cxAn2jwk86DK0AqVaEy0GEAQqTlmOgbImwuIu8eIOe4y1FhrDFNancKiwp dPGwjeTm7xzIZK9W74Q0avBKnfQZs2KGcO1InPMJyn/qe9RX9EjJyqSOxiDQswDaiEfk JNL0b+K93Ix2qx2cu+u7xXuSMVVjvSbmnp4vK5GWW0fFHFKsM4t38fOm+eiCm78Mryw4 WCVWWS86hoafie7kz7Qp5FMHEh+uUt410k16GXMBD/ipBARZpjxXzohYx2l2eix+eWZ0 Q3Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407127; x=1722011927; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; b=u+I3YYPt5Kad6ZKkcljboxpHoUXmEn75qOdmup4TLPjCiM24NCMwTDqWQNYGFQN/Y3 J9ROcfw9kUxMMwRWAYRZJ03MSGB6oP34aMkh/LGhoEqwTMohjAGhNDc+sZSFQ/C7VQJO S4zNl9h6Wucj+DUcd/202PQBpaFMkbtGoByWrA2GtiNeuFAG2FNAUwoXY0YweZLWXPZZ OoYLww9do3aXVdWf8e217Du+lPMIhHqWJtyyydwuo42AlqDPdfUTn5eaeDgc0NmOBCdN HdFnoEZB7cyOIVRz98QGznDgjT3/aG38xg09/0+Zgp9Fep4JRyOnEbUe4kszeIqDIojY XrOg== X-Forwarded-Encrypted: i=2; AJvYcCVHxYUXQo0fgeIBhtEigrMwxUU3DyHafOkLoZyY6qdyUCZ6uz3nR+mWuFl9gOnrL+29GlUjEgnhFXQy3WBkaCQ7ZT2EI9s= X-Gm-Message-State: AOJu0Yz36cHFoR0UssThoYD90PkkqcfO+xO5we7m9PbDpG0pi+0HIoO0 AvaoilTDyZF9x3po/hLxmnMx2zm9Txsx6i9Wlv/GGTemKkH+PVXh X-Google-Smtp-Source: AGHT+IHaC2gceeeJQruDY5QEsjbXZig7LE2ETEMxod6+WZ1HCBdh1KwzWImRgSvASowScyo9GBPaFQ== X-Received: by 2002:a05:6512:3f13:b0:52c:dc57:868b with SMTP id 2adb3069b0e04-52eeaeb5d8amr4001575e87.13.1721407127305; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:b0b:b0:52d:259b:377c with SMTP id 2adb3069b0e04-52eeacb4a2fls874116e87.1.-pod-prod-07-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a2e:9893:0:b0:2ee:bd1b:84c with SMTP id 38308e7fff4ca-2ef166214f5mr2378601fa.0.1721407124979; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=pamy7Fi4ghv/6AVkTTFSEYS1ALfuN/z09ikMhW7U3SRiTNB2O2S/w+zjsKSj2CCZ7k SGmWQWcGopxelCvI5rH69L+5z1+xc7TgWem1mV/4Acd5WOE+QG3AwJTAm6Fe/ZfgwBSo cbTtpf49xZSb+08on7Mk3gV1kzsWh/+LH5Uz2ysdcI3E6hiY9KPh4Vf+TVDm2qPhhFp1 vYNH8jn4Dv1FUnPHcGEtRKweNWgRgI73iy5S7nZwLnoGSOZg0GdaHIqT3Hpa6uUEFZal PG1c1Uvu2b3Fg2COFES1pxFe0WVY910iXSUa88mchSWzsiT1epXYOXr/27ZA6b8DmrMw E/8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=tur0DYltz4BuUK9fghA1yjJMijPlyfHlWAkmvVWKtn4=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=YR1f/i+xrRysNZxKzcftpZT6n1rsjBJJV9uuOTcUqL65nf4msVH82P4ODZKeoyuYzO l0Ya/99XleAQ8GGzNSOmQPqonIUTSoujk4A8s8GpJ2ypPZvBmXJ8cH3y0946nCrdi0G8 1R/DcQegGDsosB4FdUDUtxdLZLouduLCdfIfWZYj0jjx2vXSbra2goP/AOY9gLxDgdJB ZoffPBbiFwHNLSuxCHo7Ylu0TSOtTMKmM+ZeqwHLH6NUiHNGSQe3GNm21GLIDdhCsz4x bTA0ZcHUIuInbs+7SkqQxXazaBSXqW19/d7weGgqWrc3BGANKhekwzexC9u0kvwl5wG8 bZyg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2ef0fd30618si385501fa.8.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 20240719163844e28a1b2b4382c7a18d for ; Fri, 19 Jul 2024 18:38:44 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 1/5] Introduce fetcher from container registries Date: Fri, 19 Jul 2024 18:38:39 +0200 Message-ID: <82da88bf02bf928d8807bc93bfb5fcdeece1f558.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. In case a multi-arch image is specified, the fetcher will only pull for the package architecture of the requesting recipe. The image is stored compressed in docker-archive format and, wherever possible, hard-linked from DL_DIR to WORKDIR. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..0d659154 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,86 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import oe.path +import os +import tempfile +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + ud.arch = d.getVar('PACKAGE_ARCH') + ud.variant = None + if ud.arch == "armhf": + ud.arch = "arm" + ud.variant = "v7" + elif ud.arch == "armel": + ud.arch = "arm" + ud.variant = "v6" + + ud.container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = ud.container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = ud.container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.arch + "/" + \ + (ud.variant + "/" if ud.variant else "") + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \ + ".zst" + + def download(self, ud, d): + tarball = ud.localfile[:-len('.zst')] + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: + # Take a two steps for downloading into a docker archive because + # not all source may have the required Docker schema 2 manifest. + runfetchcmd("skopeo copy --preserve-digests " + \ + f"--override-arch {ud.arch} " + \ + (f"--override-variant {ud.variant} " if ud.variant else "") + \ + f"docker://{ud.container_src} dir:{tmpdir}", d) + runfetchcmd(f"skopeo copy dir:{tmpdir} " + \ + f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d) + zstd_defaults = d.getVar('ZSTD_DEFAULTS') + runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + image_file = ud.localname + ":" + ud.tag + ".zst" + oe.path.remove(rootdir + "/" + image_file) + oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file)