From patchwork Mon Mar 29 06:56:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Silvano Cirujano Cuesta X-Patchwork-Id: 392 Return-Path: Delivered-To: ilbers.mnt@gmail.com Received: by 2002:a4a:378f:0:0:0:0:0 with SMTP id r137csp3546332oor; Mon, 29 Mar 2021 08:56:43 -0700 (PDT) X-Received: by 2002:adf:b30f:: with SMTP id j15mr29467385wrd.132.1617033403396; Mon, 29 Mar 2021 08:56:43 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1617033403; cv=pass; d=google.com; s=arc-20160816; b=S2a1HYc8I+KdmlAydew9zCA+Nh29Tj0VA+tTcWHHRgmo0BsnLZGA1RBFep0qidHw5/ u+Yn4eOOPki7vs6wzIslJ2+VcLAyEWS/7XKvYgA/tRVdgPKckM7eVzliByuPEeHNSPbD 0C0ktRDXjPRn186aoFwsSd/CI1ZNMW2ChrFzxG4wzYzkyG3Zcvx9Tq0gQJ3sBvRvXvqg zauHFXQ++kRBB2cKNbFdvij2kRbGWN4XAskgXG6t4ToK5W8qcFo9XJocReIceILHgCAF Dowy0sjGf5LRUAQ5T8j5cNS/XtnZm9DWQiMorm1muKBEhsqmp7AcEu/KRrR9hbHV/2Zz Sl/g== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=y7dyZoVFlQw/soy8qErJT5GiOZmji8Sy+dUtwRI2VnA=; b=VOslR0gLJCPpan+whqKgkDtZNeqYKLVarqV3C1zldkLF5ePfcMCS03+61zSlLC3Q1a Jf6w46sLHVgA4MgnJDp1uRhoojtWyPcYIBZSU6DJ79cL+9jOI0dkNcQKkwGyBjb7sZ/t Ue94qfRzfUPh9AiJsiz/Aov8zWO0KmQArI0i+ZXzASc8WgV1SZak1yuwFrdfQwHb6BV+ lD51wyUVer8Y2bBVEsbBDoli92mgXX2lWXrZUnQKPzaXBf5BT89R8OBGqnLv3Wcl6+xz 4MU5+Ozq8ChaVdMR4KPYcj8Pm0cFOg8xGkz23+SZ7V33HIFXYa9l6H9FWRnN05xbK8N2 qUUg== ARC-Authentication-Results: i=3; mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=iK3wmlCL; arc=pass (i=2 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of isar-users+bncbcd6jj6cqqfrboxrq6bqmgqe7gl7qja@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCD6JJ6CQQFRBOXRQ6BQMGQE7GL7QJA@googlegroups.com; dmarc=fail (p=NONE sp=NONE dis=NONE arc=pass) header.from=siemens.com Received: from mail-sor-f55.google.com (mail-sor-f55.google.com. [209.85.220.55]) by mx.google.com with SMTPS id l2sor8033801wml.24.2021.03.29.08.56.43 (Google Transport Security); Mon, 29 Mar 2021 08:56:43 -0700 (PDT) Received-SPF: pass (google.com: domain of isar-users+bncbcd6jj6cqqfrboxrq6bqmgqe7gl7qja@googlegroups.com designates 209.85.220.55 as permitted sender) client-ip=209.85.220.55; Authentication-Results: mx.google.com; dkim=pass header.i=@googlegroups.com header.s=20161025 header.b=iK3wmlCL; arc=pass (i=2 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of isar-users+bncbcd6jj6cqqfrboxrq6bqmgqe7gl7qja@googlegroups.com designates 209.85.220.55 as permitted sender) smtp.mailfrom=isar-users+bncBCD6JJ6CQQFRBOXRQ6BQMGQE7GL7QJA@googlegroups.com; dmarc=fail (p=NONE sp=NONE dis=NONE arc=pass) header.from=siemens.com ARC-Seal: i=2; a=rsa-sha256; t=1617033403; cv=pass; d=google.com; s=arc-20160816; b=i1BXhB+0x0AHpjLHkEtP4MY8GYax3QqJm0Tr0WnKoYxon4Q1JxlhC9+SZaB/aN5L5s BwVb54KYAs9eyFhMJEAsSbhaWltEu5o0+YLddAy2d9L/lDU3lZ0ai6XVPqjkg33ERM/0 BPqtxoy8EYF+yN8a++F/4hZ8d5FT3fRwsYyndVSdeRRGO41Q5tjzV+kS4pMakA5BVR5y dXMjAfRm/MmRqsMgSUhf9h/PdgsHXnG25dRI6eRnFwfz1zk9aQYmSevbCQlpvISE8Ft/ 4xRCgjpDI5io3Jz+yfrcrh144ZqqiCSRFFhs/mvDR5jyG7W+C243K86Vd9f3vtAvvrGF t0sw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=y7dyZoVFlQw/soy8qErJT5GiOZmji8Sy+dUtwRI2VnA=; b=ffDZExihK+8ibr3FLmM18xZ1J0mYpOUD0KBjlR6aWJz0XbGB+VIoFbeSOvR2FSQAKw D7eF1aT1TvxZ1JhwTAsdzw7dAUxMXleweDV3lKu7IX0qe2qylKYWsrwmGEONCMB5+3Ef COpSrAjiTpvClm9Xeyq1gGWXJMNsFEcZXQiCFLUhpqzTgAWhV8JY54Zl5cTjGM4OukI7 di5WzkPDFZwrwN214ZevAmwjfCqULQxovh6QS1A6f1DJiLWiNElTOD6SzH9mtgoQZhjO K3jsxmAn9iNfUXF3V5IDnmDzuWHoF5OzSM+tZILiH+5poEkanXI9+fKcggrm0hGs+vCX oLGQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=y7dyZoVFlQw/soy8qErJT5GiOZmji8Sy+dUtwRI2VnA=; b=iK3wmlCLJ62ly7ruuk8acawIo1DrF9hgGsbMOXGk2lp7O19bZPfJjnET1sZK91kioZ hjBLXwUMEz4VFTaPZ30PI2pptfnRo9ef3Xo9jwcQfNLsQkMsXcg6Rnkz9ATieAEN2fcD t3B30/kw7DgtQHa1Yb5E+6iYVgfF+Kuh8Kdh3rck/UuOG6vZYcyAKcBPX998778R6dN7 4y/GksWUyfc9rv56siBu5zrbWueh2YOyZj23pWXIg/edm9fCapLTbCnAB8ni/PHjYD3N J6beP2L1TV64Ru4c143FDfTQsZjGpJpAJrSSIdVAEkG5JeCfyliQIh5yayEbkhsFfs/Z KHAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:to:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=y7dyZoVFlQw/soy8qErJT5GiOZmji8Sy+dUtwRI2VnA=; b=BViXRUS/mcv99z2bW6L86MWGYs9/cvOlsPd64mmx20s54Qr1h+7Z7ASXry6MyGLoDH Zsd7EodpJDfxyT9UKiLuN3+MLdbfW0gS/NEchJ/gNFTb8X20Tta0WErN9K6Exh0lQA+f vIUjbICjOLofV867Ytt11unzwAdNQ4t2R8ccKRPEjFBIyxY/EdMYkIeFAnDKHBuoLzZx aMBd46XbWLdkrDrGlx3/SxyA8Z5IUY9fgqAAimRaz2TnWqa6n630JdXzn+u5fyGYQbgg 9Sx+H0RHi0RZI8kVl6wRRsG2Umam0H0OY/UFtCG0LlIEOuky0FDucSXxk9wQs7881HPs F1ow== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM533cm5ljLPM3cB8uKaA3r/2HC+Y+fCYy8741wvwzbRg9OPe+HOM/ DvnIXDx2l1U6v7LgtbV+UkY= X-Google-Smtp-Source: ABdhPJxFYQAr2lLYo/6AoowUFWmwaK2IRpunimkFgiXqi061hoMfZAfHofon48UYgGG+6NrEKoQcaQ== X-Received: by 2002:a1c:7715:: with SMTP id t21mr25107630wmi.132.1617033403030; Mon, 29 Mar 2021 08:56:43 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:9ac6:: with SMTP id c189ls5088303wme.3.canary-gmail; Mon, 29 Mar 2021 08:56:42 -0700 (PDT) X-Received: by 2002:a1c:9d0e:: with SMTP id g14mr25570227wme.30.1617033402257; Mon, 29 Mar 2021 08:56:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1617033402; cv=none; d=google.com; s=arc-20160816; b=muD9aaPXO/lT3E191bHpQZecz0r1YmfvwcatVrlUZbSmdoTQhd5BJMu9gmEUoWxZ1g Q1cu+Z06TQMD2nPcOtZUHDDXkymyFDnb0u2LPNKbDotcjlXdGDvV8XTQG3m+gafyhfud iYsD1aVcZBklR9dnhzsXQ0vEfpu2dDzfT8KkwAAVxWZibwkykdvgz7DIIoA4WwGQS2GS XkYUQgy2KLWVehQ+SFmWgLwKqq9W6id8Zq6QlfV1LB7RJuYLnKKjtO06w2IZ1RQDvyj2 dE9HniAsgL663Ozckt/v9D6TjJxG0qDlJ6Gnysb9wiIiJ9HueWYa+JOPXAX3HH7lag2c 17Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=la/esXpFR0osGZuUC0Au3TjQAgrmai4QHShMqZ8JKJo=; b=dZgydf2Hw+II7KPQK2nGE511VMDXlzBTk1uYgjvhZkoR+Bz/UOfDW1rvv3RberbKCE qqwoYRuMG4n21qrobEA1nevDk7+szoTiQB6M1gvTc+RznSM18PG3Oajdl5BI+GQiFI/0 Sqas0mRDY9EWKlJpK7v0pDWvrGs1H93fnM0dB6H/U+5fMah8/vj/hMbQhCNHs7ovXUw4 VQQa2BJ8e5ynNvdu1PfD9LZzOVFdO9fdgnrU8wWfgEgxYvo1pjyl9bp/qHxqL2QOIYfH mu5ny2i1gaJM+fBToieH0cCUFVdvhxhlG2YH88XyqxY3mIAtflIVrq2nxs84XKpLjbFO KMIg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id b6si15483wmc.2.2021.03.29.08.56.42 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Mar 2021 08:56:42 -0700 (PDT) Received-SPF: pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 12TFufcB000549 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 29 Mar 2021 17:56:42 +0200 Received: from md1sf36c.ad001.siemens.net ([167.87.46.205]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 12TFufsn031984 for ; Mon, 29 Mar 2021 17:56:41 +0200 From: Silvano Cirujano Cuesta To: isar-users@googlegroups.com Subject: [PATCH v8 4/5] docs: document creation of container images Date: Mon, 29 Mar 2021 17:56:39 +0200 Message-Id: <20210329155640.62445-5-silvano.cirujano-cuesta@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210329155640.62445-1-silvano.cirujano-cuesta@siemens.com> References: <20210329155640.62445-1-silvano.cirujano-cuesta@siemens.com> MIME-Version: 1.0 X-Original-Sender: silvano.cirujano-cuesta@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of silvano.cirujano-cuesta@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=silvano.cirujano-cuesta@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1695582417495904461?= X-GMAIL-MSGID: =?utf-8?q?1695582417495904461?= Signed-off-by: Silvano Cirujano Cuesta --- doc/user_manual.md | 179 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) diff --git a/doc/user_manual.md b/doc/user_manual.md index a4f3d1d..e57551b 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -19,6 +19,7 @@ Copyright (C) 2016-2019, ilbers GmbH - [Add a Custom Application](#add-a-custom-application) - [Enabling Cross-compilation](#isar-cross-compilation) - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem) + - [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem) - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages) @@ -84,6 +85,14 @@ If your host is >= buster, also install the following package. apt install python3-distutils ``` +If you want to generate containerized SDKs, also install the following +packages: `umoci` and `skopeo`. +Umoci is provided by Debian Buster and can be installed with +`apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to +be installed either manually downloading the DEB and installing it (no other +packages required) or with `apt install -t bullseye skopeo` (if +unstable/bullseye included in `/etc/apt/sources.list[.d]`). + Notes: * BitBake requires Python 3.4+. @@ -223,6 +232,73 @@ qemu-system-x86_64 -m 256M -nographic -bios edk2/Build/OvmfX64/RELEASE_*/FV/OVMF qemu-system-i386 -m 256M -nographic -hda tmp/deploy/images/qemui386/isar-image-base-debian-buster-qemui386.wic.img ``` +### Generate container image with root filesystem + +A runnable container image is generated if you set IMAGE_TYPE to +'container-img'. +Getting a container image can be the main purpose of an Isar configuration, +but not only. +A container image created from an Isar configuration meant for bare-metal or +virtual machines can be helpfull to test certain applications which +requirements (e.g. libraries) can be easily resolved in a containerized +environment. + +Container images can be generated in different formats, selected with the +variable `CONTAINER_FORMAT`. One or more (whitespace separated) of following +options can be given: + - `docker-archive`: (default) an archive containing a Docker image that can + be imported with [`docker import`](https://docs.docker.com/engine/reference/commandline/import/) + - `docker-daemon`: resulting container image is made available on the local + Docker Daemon + - `containers-storage`: resulting container image is made available to tools + using containers/storage back-end (e.g. Podman, CRIO, buildah,...) + - `oci-archive`: an archive containing an OCI image, mostly for archiving as + seed for any of the above formats + +Following formats don't work if running `bitbake ...` (to build the image) +from inside of a container (e.g. using `kas-container`): `docker-daemon` and +`containers-storage`. +It's technically possible, but requires making host resources (e.g. the +Docker Daemon socket) accessible in the container, which can endanger the +stability and security of the host. + +The resulting container image archives (only for `docker-archive` and +`oci-archive`) are made available as +`tmp/deploy/images/${MACHINE}/${DISTRO}-${DISTRO_ARCH}-${container_format}.tar.xz` +(being `container_format` each one of the formats specified in +`CONTAINER_FORMAT`). + +### Example + + - Make the relevant environment variables available to the task + +For one-shot builds (use `local.conf` otherwise): + +``` +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE IMAGE_TYPE CONTAINER_FORMAT" +export IMAGE_TYPE="container-img" +export CONTAINER_FORMAT="docker-archive" +``` + + - Trigger creation of container image from root filesystem + +``` +bitbake mc:qemuarm-buster:isar-image-base +``` + + - Load the container image into the Docker Daemon + +``` +xzcat build/tmp/deploy/images/qemuarm/debian-buster-armhf-docker-archive.tar.xz | docker load +``` + + - Run a container using the container image (following commands starting with + `#~:` are to be run in the container) + +``` +docker run --rm -ti --volume "$(pwd):/build" isar-buster-armhf:latest +``` + --- ## Terms and Definitions @@ -834,6 +910,109 @@ ii crossbuild-essential-armhf 12.3 all Inf ~# ``` +## Create a containerized Isar SDK root filesystem + +### Motivation + +Distributing and using the SDK root filesystem created following the +instructions in +"[Create an Isar SDK root filesystem](#create-an-isar-sdk-root-filesystem)" +becomes easier using container images (at least for those using containers +anyway). +A "containerized" SDK adds to those advantages of a normal SDK root filesystem +the comfort of container images. + +### Approach + +Create container image with SDK root filesystem with installed cross-toolchain +for target architecture and ability to install already prebuilt target binary +artifacts. +Developer: + - runs a container based on the resulting container image mounting the source + code to be built, + - develops applications for target platform on the container and + - leaves the container getting the results on the mounted directory. + +### Solution + +User specifies the variable `SDK_FORMATS` providing a space-separated list of +SDK formats to generate. + +Supported formats are: + - `tar-xz`: (default) is the non-containerized format that results from + following the instructions in + "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" + - `docker-archive`: an archive containing a Docker image that can be imported + with + [`docker import`](https://docs.docker.com/engine/reference/commandline/import/) + - `docker-daemon`: resulting container image is made available on the local + Docker Daemon + - `containers-storage`: resulting container image is made available to tools + using containers/storage back-end (e.g. Podman, CRIO, buildah,...) + - `oci-archive`: an archive containing an OCI image, mostly for archiving as + seed for any of the above formats + +User manually triggers creation of SDK formats for his target platform by +launching the task `do_populate_sdk` for target image, f.e. +`bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. +Packages that should be additionally installed into the SDK can be appended to +`SDK_PREINSTALL` (external repositories) and `SDK_INSTALL` (self-built). + +Following formats don't work if running `bitbake -c do_populate_sdk ...` (to +generate the containerized SDK) from inside of a container (e.g. using +`kas-container`): `docker-daemon` and `containers-storage`. +It's technically possible, but requires making host resources (e.g. the Docker +Daemon socket) accessible in the container. +What can endanger the stability and security of the host. + +The resulting SDK formats are archived into +`tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}-${sdk_format}.tar.xz` +(being `sdk_format` each one of the formats specified in `SDK_FORMATS`). +The SDK container directory `/isar-apt` contains a copy of isar-apt repo with +locally prebuilt target debian packages (for ). +One may get into an SDK container and install required target packages with +the help of `apt-get install :` command. +The directory with the source code to develop on should be mounted on the +container (with `--volume :`) to be able +to edit files in the host with an IDE and build in the container. + +### Example + + - Make the SDK formats to generate available to the task + +For one-shot builds (use `local.conf` otherwise): + +``` +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SDK_FORMATS" +export SDK_FORMATS="docker-archive" +``` + + - Trigger creation of SDK root filesystem + +``` +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base +``` + + - Load the SDK container image into the Docker Daemon + +``` +xzcat build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf-docker-archive.tar.xz | docker load +``` + + - Run a container using the SDK container image (following commands starting + with `#~:` are to be run in the container) + +``` +docker run --rm -ti --volume "$(pwd):/build" isar-sdk-buster-armhf:latest +``` + + - Check that cross toolchains are installed + +``` +:~# dpkg -l | grep crossbuild-essential-armhf +ii crossbuild-essential-armhf 12.3 all Informational list of cross-build-essential packages +``` + ## Creation of local apt repo caching upstream Debian packages ### Motivation