From patchwork Tue Jul 16 14:18:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 199 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:21 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f64.google.com (mail-lf1-f64.google.com [209.85.167.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIKmW013341 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:21 +0200 Received: by mail-lf1-f64.google.com with SMTP id 2adb3069b0e04-52eaef92c91sf5983950e87.1 for ; Tue, 16 Jul 2024 07:18:21 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139494; cv=pass; d=google.com; s=arc-20160816; b=Cir3YsGWou8/3mXEg8gU164kMS3r7pQfLT69vuL5N+BhF/T9B5NX57Hq74wJ8d5upT Wso1d+s+0DoRuOcYSQpOeGa+LIrLfR7S34ZOeZPIkKUxue0KQzKBRg8rmdgNYiQEvCoS yc4FOOu1/DVd3ij/ZsAx9s3/TGeL73kVqhBFtV4+G/Hd0KZnaAw36V9fuCJPHx9ZUzd/ efgY+5bIysPtfgwqOmlNLE+lZUevWafT3bMYfkuppGXkEwIMPx9FGWvB/kzC6/9ZBcse VZn2LjB6KtAX1qH4tIuitLMepbutCmK1nXTyhAXn7WWYGT763IueTHuUrRD3koQjs/9C bgsw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=6EbuFDe5jmr3q0BTGBKz0HhiCejOeYZIkgCNoZdUuRc=; fh=unb0sGwqomYETyvc78PCDSLJa7Zrge4NjNHqQ9mfOL4=; b=Fthb7w//hQdWg5dQMmXwEygwuBLlg8rBLfhx+qC0+qadFzARktxG8VVZr+lcalsifY NXzMKv5MEU13K95bz9yahmXavASmCTpDCO4UCmj4atTEuN/JTdUgqANCKtGeGDIxyaPE X2Zm1KQ14jzJkpD+R3IpCZKuq+enbi9mmJLfZSyDzYbVCcRTwPNNGCjQnLVMAONxNR+N xwt1upH1cCXDHYH6cOeZobAAOo+sufIt+ga4hxTtxDXB58+OB8zlBzLTz6eiMA+3fl8+ IehnZitqsQdPytsYUFgHVhuRM7+Acqjiv1Uq56qrd4SkOu2ThQAEvTq3dL8q8WhZwKwB FWBg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="JbuhA4Q/"; spf=pass (google.com: domain of fm-294854-202407161418109cd3c870492fb78a85-30xggh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-294854-202407161418109cd3c870492fb78a85-30xGGH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139494; x=1721744294; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=6EbuFDe5jmr3q0BTGBKz0HhiCejOeYZIkgCNoZdUuRc=; b=Zl8mTxE8rWOF/rKgaaOR04r0cW8nBats8ZGFj+dFoxqRpOKYeOSgonp6qTJYDwCyGO jv0IUkyn/uhCfnHb9tZ5vxBqksjgdM8FKJ+FtmISIXL20eHocLQxpDouU1CPwdaXQ+4Y axVNgmlJpoyJokwlr7VkhhF70PH6xZkCPs4Ebr1GkkYrc1E1YHNopgBf15JBIi0/k3OW qqPT4I676i9TL7gdBDWCzaRdV1rCPD89KGHtim6OQMrLZAHCDlabMt3LCDDrd9qT8Iyi CxmOKtTxowUl2RqDfJWwLpTkujDUqsl0m3xPemW0kG8AkANGB5eHqa0RCixM+45QbL9c ReTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139494; x=1721744294; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6EbuFDe5jmr3q0BTGBKz0HhiCejOeYZIkgCNoZdUuRc=; b=VWtfGsd4xHhhSfOn92kfNqeRr/RMMKYtFb0whKA7kDVLpFn8frjtlhnFZkypRmDMZC cOpTOTIpTIiHC7DQweVwFQPYPGUyrUhZwkInDkT+XfhvhD3Td8/8AQYrVsyvIxp/Rztt vofYOz/o9avQ/zm90glkO+auBA0+g8oBuVE74sC4Wy9uyy0UGP/C9+hdRuGnJnawmiSW RJ5XnRYCjWgGNV8yd4g3vwFnBZ2Wzb7SiMNRFFbbkTVxEIGersCRCa534Om4WABSjcA5 ibXJ2YPoNnOqmFcY48OkJPlNM29TXpuWFvV3DOxL/oHdqPiTJ1r/bqLZHsfl+QXARrr8 2ROw== X-Forwarded-Encrypted: i=2; AJvYcCX73OhwPn/VwfKERhfrOOh1SbkLInzR3JddCa6RD0Hq1VtA5OK0idNdeEnejKxU7PwTtELusi8UjGNqPk/8JQuJ2qwwUDA= X-Gm-Message-State: AOJu0Yz5xlxBme7GC9hSdtw05lsX7jwaPXNm4Or9gm/0MJ183FXxCUUP TcMvrEWj3GJUVOlvHsO1TbsuAyUqmdrrscMzMywPeWq/uNIFA0MC X-Google-Smtp-Source: AGHT+IGiBvTgsKrf781OyV3Z7ENC6mkos0+54jIGURb0TbZfZpHs6cAVYSBVLsQg3K+kl7IYFfuSZg== X-Received: by 2002:a05:6512:2349:b0:52e:9b92:4999 with SMTP id 2adb3069b0e04-52edef10785mr1754332e87.2.1721139494125; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:ac02:0:b0:52e:9707:ab58 with SMTP id 2adb3069b0e04-52ec45047fels2429393e87.1.-pod-prod-04-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a05:6512:3992:b0:516:d692:5e0b with SMTP id 2adb3069b0e04-52edf0345admr1511285e87.54.1721139491725; Tue, 16 Jul 2024 07:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=Ak1LsPp1/e2S2padxeCklj8Lm9JkcVLpCAAEMK3ZeJ/W8c4EXqQlORMYyZzVTakIiI 1M60mzJz1xBe17sLpYWPeQZOIifps+WVyhMuYMWN62zbjigP6d+coARCH4cUKHqMoGLA KXkvgG2tm9GvL8AcbR2Q7Dpb0iqQjHixArGF86EmvYJoeiN2Pp6YzkvPbwfYD2sB4hS+ 0IGngae3bkF9V9vxnetyKjSspmQFnG7uMtGin7WGXoHXuiQx5TgLNEq0jITS2jmt8ibD tI3Ebi2mIDLYQc7Q3bnPh2aI93M32mqBAnX1tAvUDsVV+UR46GLkoKL7Z7ubosoXgsc4 5zjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=x6wU5bafMD9ODBxDxbrSL60NtRFyCN4lsXx4NYYArCw=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=k8IzAOM02woldsZ8dDgmDLAu5AwOFADdrJx5fseg0WR71R2IsEeK1Eu1rTy3X2GV2v EdHNRUzv0YjuXcEDtihvW3MudAf2INnRJnbeL+oVuhGpWmxOPEXKecRK8N+F7rfzb9yK 168aH1MLAPdJzjoasfZ5Ya7VHe1l+U1Ypdxz0CwIF4CQzhfWaHyCNlkAvkK8es1adAMK kXyPvcf/1CNBLhago9gtGydcGeIVMJ8rxnz/Ogdc+gk2SUApc4R8z7WZ/hANJItWA55i bR69iKK9gJRSGH7ZhFqvDcejW9F1kXRvOvFQwZxA8NMKb0yJQcHPXU+tde+yGZQ5zFdO h5qA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="JbuhA4Q/"; spf=pass (google.com: domain of fm-294854-202407161418109cd3c870492fb78a85-30xggh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-294854-202407161418109cd3c870492fb78a85-30xGGH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52ed254a0f1si115976e87.7.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407161418109cd3c870492fb78a85-30xggh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 202407161418109cd3c870492fb78a85 for ; Tue, 16 Jul 2024 16:18:10 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 0/5] Introduce container fetcher and pre-loader Date: Tue, 16 Jul 2024 16:18:04 +0200 Message-ID: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="JbuhA4Q/"; spf=pass (google.com: domain of fm-294854-202407161418109cd3c870492fb78a85-30xggh@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-294854-202407161418109cd3c870492fb78a85-30xGGH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= We noticed internally that there are at least 3 implementations for solving the task of pre-loading container images into isar-built base systems, may it be that the images are not publicly available or the device is only poorly connected, if at all. This series tries to solve the task generically - hopefully. Changes in v3: - do not compress container deb packages [Benedikt] - fix various spelling issues in the doc [Adriaan, Benedikt] Changes in v2: - switch to open-coded unpack before loading - switch to zstd as default compression - use default compression settings from bitbake.conf - add support for delete-after-load (opt-in) - factor out fetched container installation into separate task (should make self-built container packaging easier) - use BPN instead of PN - add "Requires" to systemd service - use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH) - add CI test - add documentation Jan Jan Kiszka (5): Introduce fetcher from container registries container-loader: Introduce helper to load container images into local registry meta-isar: Add demo packages for installing prebuilt containers ci: Add test cases for container fetching and loading doc: Describe how to use the container fetcher and loader doc/user_manual.md | 60 +++++++++++ kas/package/Kconfig | 19 ++++ kas/package/pkg_prebuilt-docker-img.yaml | 9 ++ kas/package/pkg_prebuilt-podman-img.yaml | 9 ++ .../prebuilt-docker-img_0.1.bb | 12 +++ .../prebuilt-podman-img_0.1.bb | 10 ++ .../recipes-core/images/isar-image-ci.bb | 2 + meta/classes/dpkg-base.bbclass | 6 ++ meta/lib/container_fetcher.py | 90 ++++++++++++++++ .../container-loader/container-loader.inc | 101 ++++++++++++++++++ .../container-loader/docker-loader.inc | 10 ++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 ++ testsuite/citest.py | 21 ++++ 15 files changed, 389 insertions(+) create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb create mode 100644 meta/lib/container_fetcher.py create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc