From patchwork Thu Feb 11 22:51:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Silvano Cirujano Cuesta X-Patchwork-Id: 312 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:c583:0:0:0:0:0 with SMTP id x3csp2985641oop; Fri, 12 Feb 2021 01:06:16 -0800 (PST) X-Received: by 2002:a2e:b603:: with SMTP id r3mr788264ljn.410.1613120776738; Fri, 12 Feb 2021 01:06:16 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1613120776; cv=pass; d=google.com; s=arc-20160816; b=0w3IrNzDTszOWIta6MqpmnUaqGdKWYZTgx+DydmxvteT76uRBeF11evcC5LyyKUGwv pMZ9MI9KBv3Jy2k/oEu43ET4p/swM8b5QqDNwsvM72PV/KHucXa+BW4WOdTWI2UJnW/Y P14KxNwgh7it0Nsiy+ohRkzXath1lXKMtE/8FQeYWD6hvpXMXQGGDJVbJY0/2jhIqN2t wg8afqDT8fkhD/wv8+m4vlCsFyuErLVNKYkfBdPsXT9hEE6j+pOZZoawW/f8simEgVi6 NO5GOQ36V6K1tOngpBnREoTVH4dQBLn+c+DDk+Yit0JfHHsFVtMhC2m3ZMu8IjHuVBoj +APg== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=Ryy+MMTf8JgM6EPaK6/4pT+enfDuqTAtQDX8tJEHD/k=; b=MSYSajeC8z3NbwjSv6R+1XEymYoRiiK5Jt+8uRE9cvHSYBnGcxWLP5TkUa/7bzK7rj Q+B70Y0PtXwmE5XZpO3wjkyot4qNyqA/DZ6ZUkyqgLzu/o4C8INQNd46/2HtOXqwBuIV x8w1J7F782qxQHBe/kuOlSj9TgoEVlNUNUck+sj7iLlCfuwVo/AMkztx8oGk3J822yrx 7KV581KETeiTi8e0uXvg4V83r4qX7bGuWBMQH5q9cqNPSzQBetp7TCST/UcYMka+H00V DhMf2nWVw74IQ2WIZ2eg1qYiB2nYwqjaYyqXsAPT+F/gz6iTJomFE0H9+QqqmZcWlNL9 26BA== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=Eau4rDTX; arc=pass (i=2 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of isar-users+bncbcd6jj6cqqfrbcektgaqmgqevm4nkra@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCD6JJ6CQQFRBCEKTGAQMGQEVM4NKRA@googlegroups.com; dmarc=fail (p=NONE sp=NONE dis=NONE arc=pass) header.from=siemens.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id a3sor3950852ljq.73.2021.02.12.01.06.16 (Google Transport Security); Fri, 12 Feb 2021 01:06:16 -0800 (PST) Received-SPF: pass (google.com: domain of isar-users+bncbcd6jj6cqqfrbcektgaqmgqevm4nkra@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=Eau4rDTX; arc=pass (i=2 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of isar-users+bncbcd6jj6cqqfrbcektgaqmgqevm4nkra@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCD6JJ6CQQFRBCEKTGAQMGQEVM4NKRA@googlegroups.com; dmarc=fail (p=NONE sp=NONE dis=NONE arc=pass) header.from=siemens.com ARC-Seal: i=2; a=rsa-sha256; t=1613120776; cv=pass; d=google.com; s=arc-20160816; b=ePUQA8Cr0WMaaoygomNQgAKrIFRKIwofLTvGwgsH8inzvqPaV1fcaBHaZ0uz9Iqbmr ae0SiAahoCuULZiYvh+xtJbYbPJybjcNSOz78G7h10mmEGl7leW1sPXEKnTwoh5K+U4v +/01sUAcWwZlV0aWKHZ5yA3Ly6lCySf+CWj+9X4pCLz4nvaIbV7rmDMe5Ku/fE/2lcss lIt/beel5m+LCYhjo4e+ZtnXvk1tNMAcPpM1rTeyeXr12dGQ9rURAeAAXIfzOsRifbf/ WQiZC3Z/iL2ZL9TI1Xv6ww3nMhFLjwZE+T/h4fauK/X0J1iDx53nu4cPUqePBxDiy01F Wsyg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=Ryy+MMTf8JgM6EPaK6/4pT+enfDuqTAtQDX8tJEHD/k=; b=SC7p25A3jtqjQ+rLspt+0NVmIrlTHpnyHcIYrsnKPoS/FvEr6NuHKz3x9o5XlHLrv9 FBTvFa+JDb/cjeNwatbAY9oSs/ji59P+Fv+p8zj7rsYRPxKaMfnXDWBtIlUK2+tAIcKn G/VxtixGTrxxGCwnyCE4ymeiexSe9E06x0xpO13QXEqCE0roO2sJzewkymdaPYIFRE4k zOf4nbSSBHrsbO5IVR63jC61YWtziGoIUggTiTK7y1UZLDtpTqMsC8az9SvhVlUklgxE 7NuTUmYL6TU3P1oqLWhiglwtBFNolv7I4q3hYDG/VEcvk0gXYjIkj+v8k0zO3tZu9zwc JRJw== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Ryy+MMTf8JgM6EPaK6/4pT+enfDuqTAtQDX8tJEHD/k=; b=Eau4rDTXSWxIVOxZD4jHLMXh552wrsmU6hyusDHOZUBiK/E1r0iO2sAwPKs5ssjvS9 2WGWl2WUk0du4Kk3hu+35bG7DxoH5OZB6gzsoFi7FhTUu6iL88zbRscVbrppCqKmjGDA 0CSNBeMHZ3C1EFZZItDN5Ebj8F76ONR8YohiJK1g1eREB6AY2Ns0t7PswsVRJgZtNd+t TJlEqxSWKRhwjz865CsRyYUuAYjdzBoZE/BvPt7kSYL0whv2n3SS8XZtKjIGfXmVK4XG Mr++bJGqMwFp2oCHWN+zgWGHpGe4IwwlNPfil4GsSiXVpcTKZH7G1zj/9FjB8WgQWdLb qojQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:to:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Ryy+MMTf8JgM6EPaK6/4pT+enfDuqTAtQDX8tJEHD/k=; b=uSeyRej8ynAwfUOjrt1q9BsAKgzsn6k8mhEl5IFrmBuHqd6iNYcp3Y7Sdc9R90P6BS JA3Lq7X+1fJMdUyYXUHDCyKy1yM3dwNxj479DUwVkf2K44YYcwT7Ru1PzOwIXidZDMoT fzZqUkKF6LnnB4KoVbi2GEVlkemE9t6BEia4gn6r0XlqckbpmToIk8RoPL9WOexpzX46 fwu7XHYczPPwSUclHYQs3c5NjWZtI6tPrvhPWcSP9z8NFRi3RgsAVOs5gxw0MYaIQnHx R36XivmmZKOU6asWE7koUFWIh39zMHnZYKw4PfUpShJr+ZhGH9bPnTg02imILARSbMrd GoEg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM530b6l0BrDNOMhCiKK1AOygWDKVwjFhvzqq6vbBDLBM6limujzQ/ i0eSQ5VOka+CTIJAJWWuIVM= X-Google-Smtp-Source: ABdhPJxVYANqbDFqOPRv8dBNNIClWOrHZyytv821oX0Eu0HzDOt9Svy7SkLVbE3RcajZyneQOu8Klg== X-Received: by 2002:a05:651c:15:: with SMTP id n21mr1138824lja.347.1613120776294; Fri, 12 Feb 2021 01:06:16 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:9696:: with SMTP id q22ls749101lji.9.gmail; Fri, 12 Feb 2021 01:06:15 -0800 (PST) X-Received: by 2002:a2e:93c7:: with SMTP id p7mr1136515ljh.75.1613120775369; Fri, 12 Feb 2021 01:06:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613120775; cv=none; d=google.com; s=arc-20160816; b=s/O1s9caAj5p8FWesrNA4kY2MUvxqU3UkPGrSyiPeQXGKtuBjplqC18WXrnxijAPs1 d61n42JyTCpJfgBnK35OI7+YiwGjttBRC6V5so4pze88YrqdZMmgoOU7S/YJjuQoVVhc A3kxcaE4xQy4FRe/ZDqWD/DUXccDJGYkbWvFv8a8S3VyLr9xCXcnApB8z4yDFmg0kr+M I6ozz3QKm4p27S08GQgi59dW96EyA1TGA6dNya9lCopuIIW2NZ72BV8XfrLAu+cd0hZB XRsEGwiG/kjDxmRq3iETKbM4QvzeCoGubmauOtBrXxMEcseEXqurm7/qvWt6RRFGxGc0 hyyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=ec0f5w+LaAA2X2q7zzFtsJSYyJPrZpWY9PekOIYKCYM=; b=rZ98eaE7f9QLXb4qsogaa8loSWKZktdlhLEIj5usMTUgRBdWNDiibRK2DAN9ze2Vfq pGBPWpIZY+Mc1pb5SsVX4HoAOh3lfv9zxPQwp5580OKsNXNs5DwhFUQXLhU0dyX9Ld8W FnEmG6jpIdM1O6h5BjOfjb8/X/wUSxf/KStypzO0N6UYEU4PU9Vmy+p2ALFEmUu7XM92 2gDCBpCrcBDwi78qkj25QNDRxT+i5dPffpzdpdw4xrK/Jwkbthi+UZMAEa+y89x58LBf JsktxsjQDsS8w2IYjsXAuYZTKGsoSxwH87koqGDHsA8vBbAeavZ6XNyrNAyknm2WAiNN Ktrg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id y8si8966lfy.2.2021.02.12.01.06.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Feb 2021 01:06:15 -0800 (PST) Received-SPF: pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 11C96EQA006224 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 12 Feb 2021 10:06:14 +0100 Received: from md1sf36c.ad001.siemens.net ([167.87.23.75]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 11C8pDgL017325 for ; Fri, 12 Feb 2021 09:51:14 +0100 From: Silvano Cirujano Cuesta To: isar-users@googlegroups.com Subject: [PATCH v3 1/2] images: add support for container images Date: Fri, 12 Feb 2021 09:51:12 +0100 Message-Id: <20210212085113.11013-2-silvano.cirujano-cuesta@siemens.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210212085113.11013-1-silvano.cirujano-cuesta@siemens.com> References: <20210212085113.11013-1-silvano.cirujano-cuesta@siemens.com> MIME-Version: 1.0 X-Original-Sender: silvano.cirujano-cuesta@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1691479731579776791?= X-GMAIL-MSGID: =?utf-8?q?1691479731579776791?= Add support for creation of container images with the build root filesystems. Extend also task "populate_sdk" to support the creation of a container image containing the SDK. Signed-off-by: Silvano Cirujano Cuesta --- meta/classes/container-img.bbclass | 88 ++++++++++++++++++++++++ meta/classes/image-sdk-extension.bbclass | 51 ++++++++++++-- meta/classes/image.bbclass | 1 + 3 files changed, 133 insertions(+), 7 deletions(-) create mode 100644 meta/classes/container-img.bbclass diff --git a/meta/classes/container-img.bbclass b/meta/classes/container-img.bbclass new file mode 100644 index 0000000..35c7bbc --- /dev/null +++ b/meta/classes/container-img.bbclass @@ -0,0 +1,88 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2021 +# +# SPDX-License-Identifier: MIT +# +# This class provides the tasks 'containerize_rootfs' and 'containerize_sdk' +# to create container images containing the target rootfs and the SDK +# respectively. + +CONTAINER_FORMATS ?= "docker-archive" + +containerize_rootfs() { + local cmd="/bin/dash" + local empty_tag="empty" + local full_tag="latest" + local oci_img_dir="${WORKDIR}/oci-image" + local rootfs="$1" + local rootfs_id="$2" + local container_formats="$3" + + # prepare OCI container image skeleton + bbdebug 1 "prepare OCI container image skeleton" + rm -rf "${oci_img_dir}" + sudo umoci init --layout "${oci_img_dir}" + sudo umoci new --image "${oci_img_dir}:${empty_tag}" + sudo umoci config --image "${oci_img_dir}:${empty_tag}" \ + --config.cmd="${cmd}" + sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \ + "${oci_img_dir}_unpacked" + + # add root filesystem as the flesh of the skeleton + sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/" + + # pack container image + bbdebug 1 "pack container image" + sudo umoci repack --image "${oci_img_dir}:${full_tag}" \ + "${oci_img_dir}_unpacked" + sudo umoci remove --image "${oci_img_dir}:${empty_tag}" + sudo rm -rf "${oci_img_dir}_unpacked" + + # no root needed anymore + sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}" + + # convert the OCI container image to the desired format + image_name="isar-${rootfs_id}" + for image_type in ${CONTAINER_FORMATS} ; do + image_archive="${DEPLOY_DIR_IMAGE}/${rootfs_id}-${image_type}.tar" + bbdebug 1 "Creating container image type: ${image_type}" + case "${image_type}" in + "docker-archive" | "oci-archive") + if [ "${image_type}" = "oci-archive" ] ; then + target="${image_type}:${image_archive}:latest" + else + target="${image_type}:${image_archive}:${image_name}:latest" + fi + rm -f "${image_archive}" "${image_archive}.xz" + bbdebug 2 "Converting OCI image to ${image_type}" + skopeo --insecure-policy copy \ + "oci:${oci_img_dir}:${full_tag}" "${target}" + bbdebug 2 "Compressing image" + xz -T0 "${image_archive}" + ;; + "oci") + tar --create --xz --directory "${oci_img_dir}" \ + --file "${image_archive}.xz" . + ;; + "docker-daemon" | "containers-storage") + skopeo --insecure-policy copy \ + "oci:${oci_img_dir}:${full_tag}" \ + "${image_type}:${image_name}:latest" + ;; + *) + die "Unsupported format for containerize_rootfs: ${image_type}" + ;; + esac + done +} + +do_container_image[stamp-extra-info] = "${DISTRO}-${MACHINE}" +do_container_image[vardeps] += "CONTAINER_FORMATS" +do_container_image(){ + rootfs_id="${DISTRO}-${DISTRO_ARCH}" + + bbnote "Generate container image in these formats: ${CONTAINER_FORMATS}" + containerize_rootfs "${IMAGE_ROOTFS}" "${rootfs_id}" "${CONTAINER_FORMATS}" +} + +addtask container_image before do_image after do_image_tools diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass index a8c708a..63138da 100644 --- a/meta/classes/image-sdk-extension.bbclass +++ b/meta/classes/image-sdk-extension.bbclass @@ -6,11 +6,25 @@ # This class extends the image.bbclass to supply the creation of a sdk SDK_INCLUDE_ISAR_APT ?= "0" +SDK_FORMATS ?= "tar-xz" + +sdk_tar_xz() { + # Copy mount_chroot.sh for convenience + sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR} + + # Create SDK archive + cd -P ${SDKCHROOT_DIR}/.. + sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \ + -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz + bbnote "SDK rootfs available in ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz" +} do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}" do_populate_sdk[depends] = "sdkchroot:do_build" -do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT" +do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT SDK_FORMATS" do_populate_sdk() { + local sdk_container_formats="" + if [ "${SDK_INCLUDE_ISAR_APT}" = "1" ]; then # Copy isar-apt with deployed Isar packages sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/isar-apt @@ -48,12 +62,35 @@ do_populate_sdk() { done done - # Copy mount_chroot.sh for convenience - sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR} + # separate SDK formats: TAR and container formats + for sdk_format in ${SDK_FORMATS} ; do + case ${sdk_format} in + "tar-xz") + sdk_tar_xz + ;; + "docker-archive" | "oci" | "oci-archive") + if [ -z "${sdk_container_formats}" ] ; then + sdk_container_formats="${sdk_format}" + else + sdk_container_formats="${sdk_container_formats} ${sdk_format}" + fi + ;; + "docker-daemon" | "containers-storage") + if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then + die "Adding the SDK container image to a container runtime (${sdk_format}) not supported if running from a container (e.g. 'kas-container')" + fi + ;; + *) + die "unsupported SDK format specified: ${sdk_format}" + ;; + esac + done - # Create SDK archive - cd -P ${SDKCHROOT_DIR}/.. - sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \ - -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz + # generate the SDK in all the desired container formats + if [ -n "${sdk_container_formats}" ] ; then + bbnote "Generating SDK container in ${sdk_container_formats} format" + containerize_rootfs "${SDKCHROOT_DIR}" "sdk-${DISTRO}-${DISTRO_ARCH}" "${sdk_container_formats}" + fi } + addtask populate_sdk after do_rootfs diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index eddc444..7fb7b7e 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -76,6 +76,7 @@ inherit image-tools-extension inherit image-postproc-extension inherit image-locales-extension inherit image-account-extension +inherit container-img # Extra space for rootfs in MB ROOTFS_EXTRA ?= "64" From patchwork Thu Feb 11 22:51:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Silvano Cirujano Cuesta X-Patchwork-Id: 311 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:c583:0:0:0:0:0 with SMTP id x3csp2985637oop; Fri, 12 Feb 2021 01:06:16 -0800 (PST) X-Received: by 2002:aa7:dc4e:: with SMTP id g14mr1666869edu.7.1613120776346; Fri, 12 Feb 2021 01:06:16 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1613120776; cv=pass; d=google.com; s=arc-20160816; b=d45bUhKqjQZjUka94Sw6G/3CTfvrnInHZSdkx6al70qmi1sQ0IGYu6syQrmA6b8EDl CyveQeUiZ09uDjfiGE6fRE5roDgpRHyUAh/k/8OjU4HP+mWnAib27MhbYX92rZXaxi3i OzoLpqigW2NCMOwJCzDFjaPV6/j/l8ZcAXdylbTtbRdjtXHLnDOKuH//zWaQH213rqqW SY++z2xyEiyyFstD0Qz8bvMz/jdJyjBTj2sfz+HpQg+cAhOtzsnVSkp9J08kLuW042eZ 87Bzt5EDS5Q3VRvX/uMMQwFesqpYuKzmkUKlrc7f4/o+0xCz3sRKj8srj+F3xmfZXZfb vUmw== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :sender:dkim-signature; bh=l8ws9zIMRt4Il48eXLi4VRq41kl58TA9zGh6gB+JdN8=; b=CAmi8X7pXZ+sZZUzh5Hu/kvDlxc7nsz4GYhK2dsu1mmCHar9fZm0X+VrCGE+NqSAMp OuqNwftSB3IjgQXsNozk7wfuNMqy2Zd6dHkmeV7YlWPsk8yjBL3sbys0y/K2ndggywoY VJtiHxrxaRZaFaFCex5V43BRYkXksJdB+plmJqwQa9Io4zYyL/ThzLaiBQaSCr1h8Z3/ /9ynoy7jN37wK2j8oYHfYAY1Jrr6ed0ablJhk2gT2iW06m6eYRxXrAyLYP0yFAC5Uigr WXV8MKc6KV/uPyvXCYxxYloc+LQfuEpF/3I4viA5yTxOiBmXnJWUuzIGO/Hng4p8cUrc Hvmw== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=X7WuEIif; arc=pass (i=2 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of isar-users+bncbcd6jj6cqqfrbb4ktgaqmgqe2hojx5q@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCD6JJ6CQQFRBB4KTGAQMGQE2HOJX5Q@googlegroups.com; dmarc=fail (p=NONE sp=NONE dis=NONE arc=pass) header.from=siemens.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id i23sor7695411edg.15.2021.02.12.01.06.16 (Google Transport Security); Fri, 12 Feb 2021 01:06:16 -0800 (PST) Received-SPF: pass (google.com: domain of isar-users+bncbcd6jj6cqqfrbb4ktgaqmgqe2hojx5q@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=X7WuEIif; arc=pass (i=2 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of isar-users+bncbcd6jj6cqqfrbb4ktgaqmgqe2hojx5q@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCD6JJ6CQQFRBB4KTGAQMGQE2HOJX5Q@googlegroups.com; dmarc=fail (p=NONE sp=NONE dis=NONE arc=pass) header.from=siemens.com ARC-Seal: i=2; a=rsa-sha256; t=1613120776; cv=pass; d=google.com; s=arc-20160816; b=UWUG87ssMzGajLP5f7oRasXPlYLL0b4itz6wB8eTlSjnqOik0HG/zPlj4YszkPgDUt OhxtrdmokA1DHuIsnsSBCEYdqiV1tvsEdAwt6lMTk2frTksIdq4dWUJCUSPCJ2TdCVmH NKpKAp7fL1hmJliUaRNkDPuov1pkltT4YuYCjAf2y8cO+TaqadTORvRPr9MQLxziQ6JO v9du3j4vLrmnMPWIGm+RXleCX/fqV6VmftRe969MZZr37BVAeZj3SXUH7g1jLbczpfVt d4kOOhGic0DLRvkZ74g3GQ+KBvGV8hD5SY+waPTYa96u3aSiwgQIPnvCIhPU+59UtWTQ nPPw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :sender:dkim-signature; bh=l8ws9zIMRt4Il48eXLi4VRq41kl58TA9zGh6gB+JdN8=; b=NR5kBiOD++cwwcftsMJDHspjcTIy+YG3uBwwowwACHh6bj9oj9DwfrWvGxO95YmOpD eW/eL/VVIS8qd5954bwBGK6M+Ct+quzFGTyHRyAXRteamQfoMxxG/SFlqOeOTJiJ6xYk H39lNNukeiorDMDQIPbLhr42yOzyjJW557SNZO5lJQkIXFsIhHfN/duD98gdCl14tD/J +hjVy1KVmsNoVjV5auw7JroeYRTGQ4B4m6FjQDE8n8XuSB3xw1bg8PhK1gEliKZRievr BjHOsy35QOYgvd2LoZjOedcl9FS0nF0SGsojz5yhLbQjw0BBPFs1qulYDvoaiT5snHa6 zraA== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=l8ws9zIMRt4Il48eXLi4VRq41kl58TA9zGh6gB+JdN8=; b=X7WuEIifGX5mUbZs7t0zMAYXwFdRlcnRXPeGEz052M7WVGq4b2/yydlhngFXuE3jwY L8cLzQfKFqyN0W2shY0+iF3Dr7irpaQ3cUAjaqQPVMP4VtsOQ72NEUCWe8EUPOMlqTQR UO9gs8Jla5YDld2hdgLJx15jSzb4IsK+Xj5G2Nspn2pHduS/ME4Bp7OHoycrevjTwhGe T+Xt1EAYmtMJ6xbfRU8YKoCEkazv7Mb3+rnwt1BSE1DueaaaorrLJqmOHi8F8Kc7ABhd y7VENimcozLPflTfwASmlGwfzf7n9i4Qsv03mCDESkCPWygU64xZZW1R6EtBwShvCXyz 4VsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:x-spam-checked-in-group:list-post:list-help :list-archive:list-subscribe:list-unsubscribe; bh=l8ws9zIMRt4Il48eXLi4VRq41kl58TA9zGh6gB+JdN8=; b=PIcTwuswsF2v7bGTVp/+e9BgCTShMtsinlXq5mzJnegLBYACuc3z7k7fviQTR440cm pVa9qClxYpMwVvZw5DdOFWPanmhrx0WYcFNRunEpKDrzsFay8Nug+BwFOJsc7giyCpq+ fRHdP+Rd3oEcHyabWbvWDcuTs/o95ugxtVjWrD/Vlwv/t1wWz3QtY9GS6zgr2GJYlmiK NsGOTF+88eTSXRUt2xp5rhR1FOVkz5enOkHPJLmcc3oYoLn8D0RAz7yKPK/MTMqz/Kof SnEBkoITqZDBFJPC21S+mnBWjSwSNU40h4Xw+29//CAi+M17ZsX0thq829BZDgM+iBHU NprQ== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM5309yNbUla5FEJYWroYAmnr5CeSHg4HfHcULWgdKhCTG0N3iHz0V 9fiOlXGrIRsI7ArqOKXkl54= X-Google-Smtp-Source: ABdhPJzP86vV6kJ4WjxhHx71bCYEtYt+PE9HzaJ5ohAn+k9SeJoYkJ6FKjO0kRg3CZnVAgBMwN1UMg== X-Received: by 2002:aa7:ce18:: with SMTP id d24mr2393197edv.376.1613120776060; Fri, 12 Feb 2021 01:06:16 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:fd14:: with SMTP id i20ls3462166eds.2.gmail; Fri, 12 Feb 2021 01:06:15 -0800 (PST) X-Received: by 2002:aa7:c94c:: with SMTP id h12mr2402704edt.40.1613120775237; Fri, 12 Feb 2021 01:06:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613120775; cv=none; d=google.com; s=arc-20160816; b=We09MwPrzZY4BBHZdsrC1gSQOQqqfJRWxW3N1F/T39ljKKp7ftYAP1zEauz70WXF/5 9Jf5SUuJ1iciQwK+DqfdE+BWWS5qqeKme+mcxO/GOz1SAsukApK4bn8ndm6HE+A7l9V+ NsMat2THjl/rbvksbSixSt2w9C74XdzMLw5ltMyIkIOvqpAJyn5YFfJYDzDPW+0KJkxZ N0Q9IjFTDvOqLxXLmy0TNNXmlECiP2six0u0M22Va0KBFRLoAnsiIHqVTPEA8dVNLaxk c942tY8WfZcYHn1K/mqtLvSCKDI/247c9OyHW5149i187KhBgbDbWeRhP+mawXA9/q92 XYPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=SxYQRnfWtdcCt2TK6gpVSxJ0DaKTtc6xbX4unwjm3vY=; b=E5tXCDg3KGakP20XvKR0rmRvILmjk9OsVcNCgEN9/5wfT5S8K2pi7SVnelrpezSQ/L t/ttKCbV+GlU12aUayXFeoPFdfHlFZl/smdOR6Ij+SYMJCxsMRnDrHC+paPLnJOKGP51 V1krFtnzL04ps/HJOAZjoGasdfoGBQEDydAMpPecoOoO8rEIi/d7rUDLmMiSTugKsFPj /M3TNcfih7c7Lvi9YPIlWq0PR5FrG5TXVvAMwqdiF2B9jiVYjB63ebdn/u79m/M5mDZh Z0yrK89eCNgMfmbtIQ4DlPlBF9rnQkIPT0UWZVbPNSli91wMY9uwiSsW9f5wuJ3vKKnA lIrQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id c14si440505edr.4.2021.02.12.01.06.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Feb 2021 01:06:15 -0800 (PST) Received-SPF: pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 11C96E3g004382 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 12 Feb 2021 10:06:14 +0100 Received: from md1sf36c.ad001.siemens.net ([167.87.23.75]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 11C8pDgM017325 for ; Fri, 12 Feb 2021 09:51:14 +0100 From: Silvano Cirujano Cuesta To: isar-users@googlegroups.com Subject: [PATCH v3 2/2] docs: document creation of container images Date: Fri, 12 Feb 2021 09:51:13 +0100 Message-Id: <20210212085113.11013-3-silvano.cirujano-cuesta@siemens.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210212085113.11013-1-silvano.cirujano-cuesta@siemens.com> References: <20210212085113.11013-1-silvano.cirujano-cuesta@siemens.com> MIME-Version: 1.0 X-Original-Sender: silvano.cirujano-cuesta@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1691479731217968199?= X-GMAIL-MSGID: =?utf-8?q?1691479731217968199?= Signed-off-by: Silvano Cirujano Cuesta --- doc/user_manual.md | 127 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) diff --git a/doc/user_manual.md b/doc/user_manual.md index a4f3d1d..f6f49bc 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -19,6 +19,7 @@ Copyright (C) 2016-2019, ilbers GmbH - [Add a Custom Application](#add-a-custom-application) - [Enabling Cross-compilation](#isar-cross-compilation) - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem) + - [Create a containerized ISAR SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem) - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages) @@ -84,6 +85,9 @@ If your host is >= buster, also install the following package. apt install python3-distutils ``` +If you want to generate containerized SDKs, also install the following packages: `umoci` and `skopeo`. +Umoci is provided by Debian Buster and can be installed with `apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to be installed either manually downloading the DEB and installing it (no other packages required) or with `apt install -t bullseye skopeo` (if unstable/bullseye included in `/etc/apt/sources.list[.d]`). + Notes: * BitBake requires Python 3.4+. @@ -223,6 +227,54 @@ qemu-system-x86_64 -m 256M -nographic -bios edk2/Build/OvmfX64/RELEASE_*/FV/OVMF qemu-system-i386 -m 256M -nographic -hda tmp/deploy/images/qemui386/isar-image-base-debian-buster-qemui386.wic.img ``` +### Generate container image with root-filesystem + +A runnable container image is generated if you set IMAGE_TYPE to 'container-img'. +Getting a container image can be the main purpose of an ISAR configuration, but not only. +A container image created from an ISAR configuration meant for bare-metal or virtual machines can be helpfull to test certain applications which requirements (e.g. libraries) can be easily resolved in a containerized environment. + +Container images can be generated in different formats, selected with the variable `CONTAINER_FORMAT`. One or more (whitespace separated) of following options can be given: + - `docker-archive`: (default) an archive containing a Docker image that can be imported with [`docker import`](https://docs.docker.com/engine/reference/commandline/import/) + - `docker-daemon`: resulting container image is made available on the local Docker Daemon + - `containers-storage`: resulting container image is made available to tools using containers/storage back-end (e.g. Podman, CRIO, buildah,...) + - `oci-archive`: an archive containing an OCI image, mostly for archiving as seed for any of the above formats + +Following formats don't work if running `bitbake ...` (to build the image) from inside of a container (e.g. using `kas-container`): `docker-daemon` and `containers-storage`. +It's technically possible, but requires making host resources (e.g. the Docker Daemon socket) accessible in the container. +What can endanger the stability and security of the host. + +The resulting container image archives (only for `docker-archive` and `oci-archive`) are made available as `tmp/deploy/images/${MACHINE}/${DISTRO}-${DISTRO_ARCH}-${container_format}.tar.xz` (being `container_format` each one of the formats specified in `CONTAINER_FORMAT`). + +### Example + + - Make the relevant environment variables available to the task + +For one-shot builds (use `local.conf` otherwise): + +``` +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE IMAGE_TYPE CONTAINER_FORMAT" +export IMAGE_TYPE="container-img" +export CONTAINER_FORMAT="docker-archive" +``` + + - Trigger creation of container image from root filesystem + +``` +bitbake mc:qemuarm-buster:isar-image-base +``` + + - Load the container image into the Docker Daemon + +``` +xzcat build/tmp/deploy/images/qemuarm/debian-buster-armhf-docker-archive.tar.xz | docker load +``` + + - Run a container using the container image (following commands starting with `#~:` are to be run in the container) + +``` +docker run --rm -ti --volume "$(pwd):/build" isar-buster-armhf:latest +``` + --- ## Terms and Definitions @@ -834,6 +886,81 @@ ii crossbuild-essential-armhf 12.3 all Inf ~# ``` +## Create a containerized ISAR SDK root filesystem + +### Motivation + +Distributing and using the SDK root filesystem created following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" becomes easier using container images (at least for those using containers anyway) +A "containerized" SDK adds to those advantages of a normal SDK root filesystem the comfort of container images. + +### Approach + +Create container image with SDK root filesystem with installed cross-toolchain for target architecture and ability to install already prebuilt target binary artifacts. +Developer: + - runs a container based on the resulting container image mounting the source code to be built, + - develops applications for target platform on the container and + - leaves the container getting the results on the mounted directory. + +### Solution + +User specifies the variable `SDK_FORMAT` providing a space-separated list of SDK formats to generate. + +Supported formats are: + - `tar-xz`: (default) is the non-containerized format that results from following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" + - `docker-archive`: an archive containing a Docker image that can be imported with [`docker import`](https://docs.docker.com/engine/reference/commandline/import/) + - `docker-daemon`: resulting container image is made available on the local Docker Daemon + - `containers-storage`: resulting container image is made available to tools using containers/storage back-end (e.g. Podman, CRIO, buildah,...) + - `oci-archive`: an archive containing an OCI image, mostly for archiving as seed for any of the above formats + +User manually triggers creation of SDK formats for his target platform by launching the task `do_populate_sdk` for target image, f.e. +`bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. +Packages that should be additionally installed into the SDK can be appended to `SDK_PREINSTALL` (external repositories) and `SDK_INSTALL` (self-built). + +Following formats don't work if running `bitbake -c do_populate_sdk ...` (to generate the containerized SDK) from inside of a container (e.g. using `kas-container`): `docker-daemon` and `containers-storage`. +It's technically possible, but requires making host resources (e.g. the Docker Daemon socket) accessible in the container. +What can endanger the stability and security of the host. + +The resulting SDK formats are archived into `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}-${sdk_format}.tar.xz` (being `sdk_format` each one of the formats specified in `SDK_FORMATS`). +The SDK container directory `/isar-apt` contains a copy of isar-apt repo with locally prebuilt target debian packages (for ). +One may get into an SDK container and install required target packages with the help of `apt-get install :` command. +The directory with the source code to develop on should be mounted on the container (with `--volume :`) to be able to edit files in the host with an IDE and build in the container. + +### Example + + - Make the SDK formats to generate available to the task + +For one-shot builds (use `local.conf` otherwise): + +``` +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SDK_FORMATS" +export SDK_FORMATS="docker-archive" +``` + + - Trigger creation of SDK root filesystem + +``` +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base +``` + + - Load the SDK container image into the Docker Daemon + +``` +xzcat build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf-docker-archive.tar.xz | docker load +``` + + - Run a container using the SDK container image (following commands starting with `#~:` are to be run in the container) + +``` +docker run --rm -ti --volume "$(pwd):/build" isar-sdk-buster-armhf:latest +``` + + - Check that cross toolchains are installed + +``` +:~# dpkg -l | grep crossbuild-essential-armhf +ii crossbuild-essential-armhf 12.3 all Informational list of cross-build-essential packages +``` + ## Creation of local apt repo caching upstream Debian packages ### Motivation