From patchwork Mon Jul 15 10:08:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 197 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:08:42 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f56.google.com (mail-lf1-f56.google.com [209.85.167.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FA8gES005833 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:08:42 +0200 Received: by mail-lf1-f56.google.com with SMTP id 2adb3069b0e04-52ea1a947aesf4155676e87.1 for ; Mon, 15 Jul 2024 03:08:42 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038116; cv=pass; d=google.com; s=arc-20160816; b=X2NG+mb749KQ0tQqiBJ33mlCxUidLadF/ZqcZiBO97RmB27acv2p4QVrFIk72Y60yE devRrmB09RxO5D06ySc3jzB8Od2bg+wcCq/NRRDr9U9bkuf+6y6MgjWJgazcuv9xOxC1 HStn9JM9RrPcRTVZMxKNjHIVAu8FUfedHmuWFlbvRk7MAsrp7SChb4eCqBzdOuuxpHtf T2xMqGy5r8UgqDRpB0u2/DRXgqQeolpYnRdv2mtxvZW3Lijrt3x1JngHeJhM2TQo94sb 0CoFAgo4XdBeLdc7TV+a466QneyjHoVT04V0wkYcbCy4npx2tn+M7xw4F9XYx96vqJE4 tBvg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=r7/ZvZaP9wpNLVQZp8Sc3Au2XVqRkqcyMNCmIPCHqAk=; fh=uys/LUUVi3ADwVqmvPR1vuT0CTXjZhKHBkbQGCd5uBs=; b=ZZHKXw7VU/KGsnGbIahyNZvgGb1IVdfFCEQwNZB/slhtvdhxY8aUS4BqSzBClxznvk 3m1BmVKuClQVTH1PhDvj7h0DVc9ucX2G8vlYkM+amabpb7/5Vm2+fY52bjuMPzvP9rxE 5np/tgFTYE4zl9zs5kUoQ6KzTAAW65zvpP/EvM7J5pSwwQunLZ6IB/tNhJm8WlZDYUhZ qA7rmFgegJfkyg9Yu0XNyk8+lJNvVR23ZtPp8wG/tPEPP6a5FThlnM6uRdc+ha5V1uDv zIUfiWA3jomGHOYJCH8tSI5eMS7MZg6nNxvk4HFIyb9MjZBkA21iuILQ3pXyafQCKSZ6 e9/A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=vhuz4Pgz; spf=pass (google.com: domain of fm-294854-2024071510083328912e780f7ddbf5fd-gjdgqw@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-2024071510083328912e780f7ddbf5fd-GjDGqW@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038116; x=1721642916; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=r7/ZvZaP9wpNLVQZp8Sc3Au2XVqRkqcyMNCmIPCHqAk=; b=BS+Mnfoc/Mr128Kbnddcyvp8Jtc362AAHnzhxeZHO3d4UwhH2PlBE3QF3C5qOYZkz9 XaSWAAUdbRKTFoKsWqWkuG9gQ2+cdyNxWe5hdsJmB0jA1wHw+hmVIofbOgvtF8mh2fmm iduuQmvZmKIlpz72SthJOPIMuwF8kTJhI1im5R7+UygZUSm7fyFu8tpMsgr0U9UTf+mK N2usqQRKqAh+/5g/5n1+iTX4i4Eivfp47yq/1LGyiiCkGcOovHv3/WWaFzxtQsHXEbfs ewsUyXJt1tug+5aOrxRuIqBAMtI5bN3C8voiRsVGV5imePrcgQO7bp63dZYjBM1wqu3Z zatw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038116; x=1721642916; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r7/ZvZaP9wpNLVQZp8Sc3Au2XVqRkqcyMNCmIPCHqAk=; b=DT8/+VtBxxtc1kdhIk8StR0gShwqETE225cXI8ws1rGj99XAnbK6GefxWr8Y1LFXsm WtJHydCcNnQVSJ0LG3CcwpznNGEdO66p4jzEf3tqRzr3tyQjLnncpu6TNPDYcFzd/cKI 1QviMuVYLD5SsxTZ2Run8U3KUTfpNucKtrG0mxZe/G+OLxvUdeuwMBApigvjwTGsp6B4 uXOByFFOQGyIqA0sTK5yxDCCmZxwuqmXcoMOdixBfXTCwXsHjuZ7uCy0oxdiixsp8gbt xz6T0u52jrMePQfKYaIF3L+A10+xLZL6pLFTyjEQ8BNn7wt3ax5SeKZIruAIhZH5VAF6 QAXQ== X-Forwarded-Encrypted: i=2; AJvYcCXvt3iwH1OkXbObC5Y2RwBC93LcY4ZnF87OZSmcevr4sEVa6km1D/PU4IL2j2kxQpEbgBdW4yMrnS76hPLcudr8mYudmXM= X-Gm-Message-State: AOJu0Yxc5FYuGRlS9QRq6dcaROp3ByNbuPbdzBgkMzkmr916xVSAMV5j oFNnTkz5jUayMA9yMUAf/NiweRWAPQk76Dj3fRDxYHtBX0ACrCdV X-Google-Smtp-Source: AGHT+IHMUBB7kY9Nr4cHqpEoXL/HZ1kWX6fZQ3sscEjZF3+T0PjbalGOH4/Mv4ZTqs85uY9tv8mQOw== X-Received: by 2002:a05:6512:1051:b0:52b:c0b1:ab9e with SMTP id 2adb3069b0e04-52eb999124fmr13030541e87.5.1721038115951; Mon, 15 Jul 2024 03:08:35 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3e2a:b0:52e:d12c:cf91 with SMTP id 2adb3069b0e04-52ed12cd053ls1058425e87.2.-pod-prod-02-eu; Mon, 15 Jul 2024 03:08:34 -0700 (PDT) X-Received: by 2002:ac2:42d8:0:b0:52b:b327:9c1c with SMTP id 2adb3069b0e04-52eb99d59e8mr9901603e87.62.1721038113667; Mon, 15 Jul 2024 03:08:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038113; cv=none; d=google.com; s=arc-20160816; b=1Is2BrycmWw386WRX0+TM6ebl6gzFxVhcRXfgDVoc/NZjuXYJunXP9yvYqe4uCJHz+ 82yYUE+wLtBFtJg3eAEarmM8sgCkkEB74aoFtJvcqHYGKBYWDGJJjbaNIgiHcw/cSmxL TennAuNdlGx3YUpBleFBISl2Y8LYXlNNc/ZvmrmKlIxpypA0XUQScTa4FpFzymor4g7t jMuH0SHVLiaznz5tYMLk/vo6wad0SL65300h6ISEabzp8Cif40WYIpfpVr37wz+1Um7q pi1CMtqVYScwmtUAePbvk1Ox5nLdsTK0lUVPo1z7RdRDEM/ZtkKBqyGpxHMbdryDbPDv kaAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=EY+d4pEq6Zhz4VSIEvw+n/1HEjxAoWPNrCNGjB/lzUo=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=h1dnUwzyr+o7Jg1wcW38UXN4G5DscQ0bfWyuor/VPfiH7g1OmuYoIJD9vk27c3Vgt2 WFG79voU9L6H/hdbN4mD9+ASDlkxVrEJKwTeov/NXgWQdKFYzEryM1qKt8FpKW+/b0Qq YyI9x2uC0IOKmg5ENTjkmCyoDVFpAr838LxvWhL+IKG/hmxZjvEpIpaX7zKciyIy3tbv xDlzBZECDE4+AdGHdQ56iufM+PNzbIRF8PzGnyyBZYGBQVl5RFa7TocwiaNix81aJLW4 WBPciBEj15ry+vt12eJUFqiWyKHnyNuqIFa0198cgGHSF9jawYCo7oA0AO3+0rvIxWfL dBYw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=vhuz4Pgz; spf=pass (google.com: domain of fm-294854-2024071510083328912e780f7ddbf5fd-gjdgqw@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-2024071510083328912e780f7ddbf5fd-GjDGqW@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net. [185.136.65.228]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427a5ec6cb7si674365e9.1.2024.07.15.03.08.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:33 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-2024071510083328912e780f7ddbf5fd-gjdgqw@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) client-ip=185.136.65.228; Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 2024071510083328912e780f7ddbf5fd for ; Mon, 15 Jul 2024 12:08:33 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 0/5] Introduce container fetcher and pre-loader Date: Mon, 15 Jul 2024 12:08:27 +0200 Message-ID: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=vhuz4Pgz; spf=pass (google.com: domain of fm-294854-2024071510083328912e780f7ddbf5fd-gjdgqw@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-2024071510083328912e780f7ddbf5fd-GjDGqW@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= We noticed internally that there are at least 3 implementations for solving the task of pre-loading container images into isar-built base systems, may it be that the images are not publicly available or the device is only poorly connected, if at all. This series tries to solve the task generically - hopefully. Changes in v2: - switch to open-coded unpack before loading - switch to zstd as default compression - use default compression settings from bitbake.conf - add support for delete-after-load (opt-in) - factor out fetched container installation into separate task (should make self-built container packaging easier) - use BPN instead of PN - add "Requires" to systemd service - use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH) - add CI test - add documentation Jan Jan Kiszka (5): Introduce fetcher from container registries container-loader: Introduce helper to load container images into local registry meta-isar: Add demo packages for installing prebuilt containers ci: Add test cases for container fetching and loading doc: Describe how to use the container fetcher and loader doc/user_manual.md | 60 ++++++++++++ kas/package/Kconfig | 19 ++++ kas/package/pkg_prebuilt-docker-img.yaml | 9 ++ kas/package/pkg_prebuilt-podman-img.yaml | 9 ++ .../prebuilt-docker-img_0.1.bb | 12 +++ .../prebuilt-podman-img_0.1.bb | 10 ++ .../recipes-core/images/isar-image-ci.bb | 2 + meta/classes/dpkg-base.bbclass | 6 ++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++ .../container-loader/container-loader.inc | 94 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 ++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 ++ testsuite/citest.py | 21 +++++ 15 files changed, 382 insertions(+) create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb create mode 100644 meta/lib/container_fetcher.py create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc