From patchwork Fri Jul 19 16:38:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kiszka X-Patchwork-Id: 201 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:54 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f57.google.com (mail-wr1-f57.google.com [209.85.221.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcqtN002629 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:52 +0200 Received: by mail-wr1-f57.google.com with SMTP id ffacd0b85a97d-36879bb9129sf148174f8f.2 for ; Fri, 19 Jul 2024 09:38:52 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407127; cv=pass; d=google.com; s=arc-20160816; b=FfMC/S5FA9YYOMR/QegCqMQo/BHVzsZ9GgQCFGsBs0FtoC+0o+/nribnWC7hLIr3LO 3xTlfEOiilMi332DLkb5IahW96YhlbG8EL0RSOiD0kjzuRiAD9is6j8+02564r0bWdqA f9FJu7jwaihvO/SNK/xX/5VssNbeu0eVs8yPd9V6AnloXcjp1AnB9MXq2qWZQea2xrwf +7bgdch1P4er8D0S3/ZNPaTrBthSnV5BykmBOnvC0G6gwUTNbjHMpobqTE2tbKJ1Le5X kY83KWwFiuhpe9uamm5uqSE5ZeM0HKq8zKbcqJ2u+GqKHFFxg472Td7WiXmzhI+iPiuz RlNg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=wVGdmxEWXjjK36JufY9jKaOTGHBl3LdsJRx/k6m5zIg=; fh=KiVcPixTy+z5t1H/GXqgNrPbQ2bUnxQkDMeD7jwxe0k=; b=0pUVff6DW68VrNGyi5zmYd2MWPi9NSyRhjcuzhlUx90gJw4zPh28ZfAZwGBtW0tMKU bCrOOMaNoVO5i023X+PlrZMDfl8SGQORV7KvSyQL5tNhhTFNnBKcAHY20JjmKRO7JA7M x0/MGEeqM3SnZCTH67+4FuQc5gaDPxKlNowOiHGF85r/wyWx5lQa6LR99qQUc0KyiG+b Ypcnx6WfIhTcbklQGIFSyBq/6XMaPUTp/BG2ASaj5JJX5yzAUzdSOcJjeH79umaaU1mL FKSN3dtTtVm+FIYmIquEhqKKUZq5F/NnIufDKe4MgtfD+SuucBlX543m1OHiY6Mu8XpX 7uKg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=hVVBMGX6; spf=pass (google.com: domain of fm-294854-20240719163843ba90b9cba9133cc524-0oqczr@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163843ba90b9cba9133cc524-0OQCZR@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407127; x=1722011927; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=wVGdmxEWXjjK36JufY9jKaOTGHBl3LdsJRx/k6m5zIg=; b=M15yvmFLyXqaCv+6wMuRi2CcU3qBHnkVaVFD98DpdmaCXYLwP7yT0dMgZmm9bBYIQY tlp+3/H4VfaA/pfErcWiUDXy3L5ZZ3lhyuYGrK+f4ehSGs/dW1DqagRAnHUuk5pYe7iS c3bEFhlYiOwvpvIsjLZgd8Lo8BEiqb96hwIue+DsE8rEHH/XltUXNUn/HTcgF7mM+w97 O+3pm7nuHhTKmcz6ZpvmUoTjz1quVWShqG/AVhWq1PNLqYx5FHZ+7fpCEY418r7qwrqd 3qhFfxj4u0SH2sp/whjeimLDq09RTE0DkgySUk5EFJC17WfpJROO2QFvqhJcmcoURD8J 1xNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407127; x=1722011927; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wVGdmxEWXjjK36JufY9jKaOTGHBl3LdsJRx/k6m5zIg=; b=mbTGTeOImGUOCU/qNv7QB3ILuUYmXEyqU/YDYPkToZMfQyvyPzGUHuLvCZF6VW5f9n wo3Ctg1ICBcNM9TXXhEKDqZqtVxJ2ac48QCpHtT7M3/vMNhPNE+KNVrcCpW6d1zCd9ax 2Q6RCuBdIcdc7ZXL2XR7i4dWlXztOLIpyo7zv4p7boa2d+w7r9JxsdP4k5JO2KvoNIQq WOvr9LNgEhNvpMhuQVOelqhEoJwSKq9SvMPAwXtcO0PC0TibVsWxK+o7vSr/cMn2mlz7 cZYuqCtlk23CMrg46JBNMLCcnmhBuAVE5Ow+Kkt8qS1L+XcvD7NvSM8bl4TUxcN9KvrY McOA== X-Forwarded-Encrypted: i=2; AJvYcCUzxl4SekfBZcbJM26H1Al0alGG71CoAIarWQTs1GDDgYKneAKrBOC2bHN0nn0uSJoN1FXwD1+xte0hKfFTek9dNT0X83M= X-Gm-Message-State: AOJu0YyAnt9mj6sVqRG15Jv6A4OnUBW31pGrHZPfWRQk4YjwK8n51h7n B1rXH7aNOdQ0ZrqtZn9klzkp3uNfP8jd1YmGFCx1tUACUstSvatJ X-Google-Smtp-Source: AGHT+IG3hIjM5Knv6ww8Ne3RUIcB209uCwCQST6DIfckLP/qB7lzPFrq8xRZyt53vuXalQpDaZf4Zw== X-Received: by 2002:a05:600c:1d0e:b0:426:6fc0:5910 with SMTP id 5b1f17b1804b1-427daa29d80mr297805e9.1.1721407126518; Fri, 19 Jul 2024 09:38:46 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3b8b:b0:426:64c1:8974 with SMTP id 5b1f17b1804b1-427c815639als9954445e9.0.-pod-prod-01-eu; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) X-Received: by 2002:a05:600c:cc6:b0:426:6f48:2dad with SMTP id 5b1f17b1804b1-427daa67d53mr323425e9.35.1721407124421; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=Vj76Cx+bRiFCEcpIAJz5ZU8uAOwJcWTCcbmF/vSj2AB6eivmIYW1cAAL+sPMhkJntZ ZNMLP6mmy1SLRrsFzcb+UgsVIYgKhS7e93vk/v4UkzMXvECHPAtm/nDMFGZDvALhdGFX ZhoDQsJ08AYEtm4JEEAk9qH5Zyba6+zS8lXNvPwjCQtmXXSyGsZzLUJjfW5bP9zEXbpx jedascG4QF/rI2qUrt+cpZl+GqtYHGsH6nSg8sbQtYc8bRzGg2Aq3wmm2iGJEwpnJgJz CCWCoaujgpKB1Pevn5RQf1gZ9/qTFFivMWqr9SlwXNFWBZpPpiE4t7E63od3u+YzHk87 rBlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=ZwcYfVTuBZdjYZAYZHKneiJFNmDCMmetpeNI7YKwhQ8=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=YzOUtx7rVTZN/KiWfJaH7yb9lR+uARNcdhIHt3LVG5z8NPa7ZKvmWKxet8KHb9bFau wRlYnswyKtkU6MaoB1w+OfA0hO8NRtl2M8n7Ha84aryO6P0tBK6VkFBKQEQ7TsGwb1ZA r9q9gi83S8iNINwHC0i6dCGIli3DPZ87D6p7azqnJ8giZ20YEhBqj7G/SYJmhqgM6hw+ HYFFBW5kJb3olWpZdEHpZH9sYmMP1uANgqnYyUdrQOugtNyQw5xXc9DOeRIgbsfSa1uS vTr380XjeCL97iYPcd6J7PX1Hzwveu3Sjx2me8flTPYN32f6ztwXYIAlvO0Za6djRikA sgNQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=hVVBMGX6; spf=pass (google.com: domain of fm-294854-20240719163843ba90b9cba9133cc524-0oqczr@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163843ba90b9cba9133cc524-0OQCZR@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427d2911f57si1635545e9.0.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240719163843ba90b9cba9133cc524-0oqczr@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20240719163843ba90b9cba9133cc524 for ; Fri, 19 Jul 2024 18:38:44 +0200 X-Patchwork-Original-From: "'Jan Kiszka' via isar-users" From: Jan Kiszka To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 0/5] Introduce container fetcher and pre-loader Date: Fri, 19 Jul 2024 18:38:38 +0200 Message-ID: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=hVVBMGX6; spf=pass (google.com: domain of fm-294854-20240719163843ba90b9cba9133cc524-0oqczr@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163843ba90b9cba9133cc524-0OQCZR@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= We noticed internally that there are at least 3 implementations for solving the task of pre-loading container images into isar-built base systems, may it be that the images are not publicly available or the device is only poorly connected, if at all. This series tries to solve the task generically - hopefully. Changes in v4 - fetch per architecture, no longer full multi-arch - directly archive and compress in download task - use hard links where possible to reduce disk usage - always use zstd, remove configurability Changes in v3: - do not compress container deb packages [Benedikt] - fix various spelling issues in the doc [Adriaan, Benedikt] Changes in v2: - switch to open-coded unpack before loading - switch to zstd as default compression - use default compression settings from bitbake.conf - add support for delete-after-load (opt-in) - factor out fetched container installation into separate task (should make self-built container packaging easier) - use BPN instead of PN - add "Requires" to systemd service - use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH) - add CI test - add documentation Jan Jan Kiszka (5): Introduce fetcher from container registries container-loader: Introduce helper to load container images into local registry meta-isar: Add demo packages for installing prebuilt containers ci: Add test cases for container fetching and loading doc: Describe how to use the container fetcher and loader doc/user_manual.md | 58 +++++++++++++ kas/package/Kconfig | 19 ++++ kas/package/pkg_prebuilt-docker-img.yaml | 9 ++ kas/package/pkg_prebuilt-podman-img.yaml | 9 ++ .../prebuilt-docker-img_0.1.bb | 12 +++ .../prebuilt-podman-img_0.1.bb | 10 +++ .../recipes-core/images/isar-image-ci.bb | 2 + meta/classes/dpkg-base.bbclass | 6 ++ meta/lib/container_fetcher.py | 86 +++++++++++++++++++ .../container-loader/container-loader.inc | 73 ++++++++++++++++ .../container-loader/docker-loader.inc | 10 +++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 +++ testsuite/citest.py | 21 +++++ 15 files changed, 355 insertions(+) create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb create mode 100644 meta/lib/container_fetcher.py create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc