| Message ID | 20210205090827.17788-3-silvano.cirujano-cuesta@siemens.com |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | Add support for containerized SDKs | expand |
On 05.02.21 10:08, [ext] Silvano Cirujano Cuesta wrote: > Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> > --- > doc/user_manual.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 79 insertions(+) > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index a4f3d1d..7863241 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -19,6 +19,7 @@ Copyright (C) 2016-2019, ilbers GmbH > - [Add a Custom Application](#add-a-custom-application) > - [Enabling Cross-compilation](#isar-cross-compilation) > - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem) > + - [Create a containerized ISAR SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem) > - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages) > > > @@ -84,6 +85,9 @@ If your host is >= buster, also install the following package. > apt install python3-distutils > ``` > > +If you want to generate containerized SDKs, also install the following packages: `umoci` and `skopeo`. This packages should probably also be listed under https://github.com/ilbers/isar/blob/master/doc/user_manual.md#install-host-tools, as optional and with a pointer to here for all the details. > +Umoci is provided by Debian Buster and can be installed with `apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to be installed either manually downloading the DEB and installing it (no other packages required) or with `apt install -t bullseye skopeo` (if unstable/bullseye included in `/etc/apt/sources.list[.d]`). > + > Notes: > > * BitBake requires Python 3.4+. > @@ -834,6 +838,81 @@ ii crossbuild-essential-armhf 12.3 all Inf > ~# > ``` > > +## Create a containerized ISAR SDK root filesystem > + > +### Motivation > + > +Distributing and using the SDK root filesystem created following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" becomes easier using container images (at least for those using containers anyway) > +A "containerized" SDK adds to those advantages of a normal SDK root filesystem the comfort of container images. > + > +### Approach > + > +Create container image with SDK root filesystem with installed cross-toolchain for target architecture and ability to install already prebuilt target binary artifacts. > +Developer: > + - runs a container based on the resulting container image mounting the source code to be built, > + - develops applications for target platform on the container and > + - leaves the container getting the results on the mounted directory. > + > +### Solution > + > +User specifies the variable `SDK_FORMAT` providing a space-separated list of SDK formats to generate. > + > +Supported formats are: > + - `tar-xz`: (default) is the non-containerized format that results from following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" > + - `docker-archive`: an archive containing a Docker image that can be imported with [`docker import`](https://docs.docker.com/engine/reference/commandline/import/) > + - `docker-daemon`: resulting container image is made available on the local Docker Daemon > + - `containers-storage`: resulting container image is made available to tools using containers/storage back-end (e.g. Podman, CRIO, buildah,...) > + - `oci-archive`: an archive containing an OCI image, mostly for archiving as seed for any of the above formats > + > +User manually triggers creation of SDK formats for his target platform by launching the task `do_populate_sdk` for target image, f.e. > +`bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. > +Packages that should be additionally installed into the SDK can be appended to `SDK_PREINSTALL` (external repositories) and `SDK_INSTALL` (self-built). > + > +Following formats don't work if running `bitbake -c do_populate_sdk ...` (to generate the containerized SDK) from inside of a container (e.g. using `kas-container`): `docker-daemon` and `containers-storage`. > +It's technically possible, but requires making host resources (e.g. the Docker Daemon socket) accessible in the container. > +What can endanger the stability and security of the host. > + > +The resulting SDK formats are archived into `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}-${sdk_format}.tar.xz` (being `sdk_format` each one of the formats specified in `SDK_FORMATS`). > +The SDK container directory `/isar-apt` contains a copy of isar-apt repo with locally prebuilt target debian packages (for <HOST_DISTRO>). > +One may get into an SDK container and install required target packages with the help of `apt-get install <package_name>:<DISTRO_ARCH>` command. > +The directory with the source code to develop on should be mounted on the container (with `--volume <host-directory>:<container-directory>`) to be able to edit files in the host with an IDE and build in the container. > + > +### Example > + > + - Make the SDK formats to generate available to the task > + > +For one-shot builds (use `local.conf` otherwise): > + > +``` > +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SDK_FORMATS" > +export SDK_FORMATS="docker-archive" > +``` > + > + - Trigger creation of SDK root filesystem > + > +``` > +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base > +``` > + > + - Load the SDK container image into the Docker Daemon > + > +``` > +xzcat build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf-docker-archive.tar.xz | docker load > +``` > + > + - Run a container using the SDK container image (following commands starting with `#~:` are to be run in the container) > + > +``` > +docker run --rm -ti --volume "$(pwd):/build" isar-sdk-buster-armhf:latest > +``` > + > + - Check that cross toolchains are installed > + > +``` > +:~# dpkg -l | grep crossbuild-essential-armhf > +ii crossbuild-essential-armhf 12.3 all Informational list of cross-build-essential packages > +``` > + > ## Creation of local apt repo caching upstream Debian packages > > ### Motivation > Jan
On 05/02/2021 12:07, Jan Kiszka wrote: > On 05.02.21 10:08, [ext] Silvano Cirujano Cuesta wrote: >> Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> >> --- >> doc/user_manual.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 79 insertions(+) >> >> diff --git a/doc/user_manual.md b/doc/user_manual.md >> index a4f3d1d..7863241 100644 >> --- a/doc/user_manual.md >> +++ b/doc/user_manual.md >> @@ -19,6 +19,7 @@ Copyright (C) 2016-2019, ilbers GmbH >> - [Add a Custom Application](#add-a-custom-application) >> - [Enabling Cross-compilation](#isar-cross-compilation) >> - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem) >> + - [Create a containerized ISAR SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem) >> - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages) >> >> >> @@ -84,6 +85,9 @@ If your host is >= buster, also install the following package. >> apt install python3-distutils >> ``` >> >> +If you want to generate containerized SDKs, also install the following packages: `umoci` and `skopeo`. > This packages should probably also be listed under > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fdoc%2Fuser_manual.md%23install-host-tools&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C6b8f60d2fae94496197d08d8c9c6403a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481200603194744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VF1U9PGhurVgbH%2BoILbmSfAnBYWUL%2BwylXZ0N2xoG6g%3D&reserved=0, > as optional and with a pointer to here for all the details. I'm puzzled now. That's the document and section where this addition goes to... Am I missing something? Silvano >> +Umoci is provided by Debian Buster and can be installed with `apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to be installed either manually downloading the DEB and installing it (no other packages required) or with `apt install -t bullseye skopeo` (if unstable/bullseye included in `/etc/apt/sources.list[.d]`). >> + >> Notes: >> >> * BitBake requires Python 3.4+. >> @@ -834,6 +838,81 @@ ii crossbuild-essential-armhf 12.3 all Inf >> ~# >> ``` >> >> +## Create a containerized ISAR SDK root filesystem >> + >> +### Motivation >> + >> +Distributing and using the SDK root filesystem created following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" becomes easier using container images (at least for those using containers anyway) >> +A "containerized" SDK adds to those advantages of a normal SDK root filesystem the comfort of container images. >> + >> +### Approach >> + >> +Create container image with SDK root filesystem with installed cross-toolchain for target architecture and ability to install already prebuilt target binary artifacts. >> +Developer: >> + - runs a container based on the resulting container image mounting the source code to be built, >> + - develops applications for target platform on the container and >> + - leaves the container getting the results on the mounted directory. >> + >> +### Solution >> + >> +User specifies the variable `SDK_FORMAT` providing a space-separated list of SDK formats to generate. >> + >> +Supported formats are: >> + - `tar-xz`: (default) is the non-containerized format that results from following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" >> + - `docker-archive`: an archive containing a Docker image that can be imported with [`docker import`](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.docker.com%2Fengine%2Freference%2Fcommandline%2Fimport%2F&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C6b8f60d2fae94496197d08d8c9c6403a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481200603194744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fMI3heAmHRkFvQj3euZcbFYS0wkhIjM3hyw5FzOjyes%3D&reserved=0) >> + - `docker-daemon`: resulting container image is made available on the local Docker Daemon >> + - `containers-storage`: resulting container image is made available to tools using containers/storage back-end (e.g. Podman, CRIO, buildah,...) >> + - `oci-archive`: an archive containing an OCI image, mostly for archiving as seed for any of the above formats >> + >> +User manually triggers creation of SDK formats for his target platform by launching the task `do_populate_sdk` for target image, f.e. >> +`bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. >> +Packages that should be additionally installed into the SDK can be appended to `SDK_PREINSTALL` (external repositories) and `SDK_INSTALL` (self-built). >> + >> +Following formats don't work if running `bitbake -c do_populate_sdk ...` (to generate the containerized SDK) from inside of a container (e.g. using `kas-container`): `docker-daemon` and `containers-storage`. >> +It's technically possible, but requires making host resources (e.g. the Docker Daemon socket) accessible in the container. >> +What can endanger the stability and security of the host. >> + >> +The resulting SDK formats are archived into `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}-${sdk_format}.tar.xz` (being `sdk_format` each one of the formats specified in `SDK_FORMATS`). >> +The SDK container directory `/isar-apt` contains a copy of isar-apt repo with locally prebuilt target debian packages (for <HOST_DISTRO>). >> +One may get into an SDK container and install required target packages with the help of `apt-get install <package_name>:<DISTRO_ARCH>` command. >> +The directory with the source code to develop on should be mounted on the container (with `--volume <host-directory>:<container-directory>`) to be able to edit files in the host with an IDE and build in the container. >> + >> +### Example >> + >> + - Make the SDK formats to generate available to the task >> + >> +For one-shot builds (use `local.conf` otherwise): >> + >> +``` >> +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SDK_FORMATS" >> +export SDK_FORMATS="docker-archive" >> +``` >> + >> + - Trigger creation of SDK root filesystem >> + >> +``` >> +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base >> +``` >> + >> + - Load the SDK container image into the Docker Daemon >> + >> +``` >> +xzcat build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf-docker-archive.tar.xz | docker load >> +``` >> + >> + - Run a container using the SDK container image (following commands starting with `#~:` are to be run in the container) >> + >> +``` >> +docker run --rm -ti --volume "$(pwd):/build" isar-sdk-buster-armhf:latest >> +``` >> + >> + - Check that cross toolchains are installed >> + >> +``` >> +:~# dpkg -l | grep crossbuild-essential-armhf >> +ii crossbuild-essential-armhf 12.3 all Informational list of cross-build-essential packages >> +``` >> + >> ## Creation of local apt repo caching upstream Debian packages >> >> ### Motivation >> > Jan >
On 05.02.21 15:58, [ext] Silvano Cirujano Cuesta wrote: > > On 05/02/2021 12:07, Jan Kiszka wrote: >> On 05.02.21 10:08, [ext] Silvano Cirujano Cuesta wrote: >>> Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> >>> --- >>> doc/user_manual.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++ >>> 1 file changed, 79 insertions(+) >>> >>> diff --git a/doc/user_manual.md b/doc/user_manual.md >>> index a4f3d1d..7863241 100644 >>> --- a/doc/user_manual.md >>> +++ b/doc/user_manual.md >>> @@ -19,6 +19,7 @@ Copyright (C) 2016-2019, ilbers GmbH >>> - [Add a Custom Application](#add-a-custom-application) >>> - [Enabling Cross-compilation](#isar-cross-compilation) >>> - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem) >>> + - [Create a containerized ISAR SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem) >>> - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages) >>> >>> >>> @@ -84,6 +85,9 @@ If your host is >= buster, also install the following package. >>> apt install python3-distutils >>> ``` >>> >>> +If you want to generate containerized SDKs, also install the following packages: `umoci` and `skopeo`. >> This packages should probably also be listed under >> https://github.com/ilbers/isar/blob/master/doc/user_manual.md#install-host-tools >> as optional and with a pointer to here for all the details. > > I'm puzzled now. That's the document and section where this addition goes to... Am I missing something? > Forget it, missed that it is already in the right section. Jan
diff --git a/doc/user_manual.md b/doc/user_manual.md index a4f3d1d..7863241 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -19,6 +19,7 @@ Copyright (C) 2016-2019, ilbers GmbH - [Add a Custom Application](#add-a-custom-application) - [Enabling Cross-compilation](#isar-cross-compilation) - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem) + - [Create a containerized ISAR SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem) - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages) @@ -84,6 +85,9 @@ If your host is >= buster, also install the following package. apt install python3-distutils ``` +If you want to generate containerized SDKs, also install the following packages: `umoci` and `skopeo`. +Umoci is provided by Debian Buster and can be installed with `apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to be installed either manually downloading the DEB and installing it (no other packages required) or with `apt install -t bullseye skopeo` (if unstable/bullseye included in `/etc/apt/sources.list[.d]`). + Notes: * BitBake requires Python 3.4+. @@ -834,6 +838,81 @@ ii crossbuild-essential-armhf 12.3 all Inf ~# ``` +## Create a containerized ISAR SDK root filesystem + +### Motivation + +Distributing and using the SDK root filesystem created following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" becomes easier using container images (at least for those using containers anyway) +A "containerized" SDK adds to those advantages of a normal SDK root filesystem the comfort of container images. + +### Approach + +Create container image with SDK root filesystem with installed cross-toolchain for target architecture and ability to install already prebuilt target binary artifacts. +Developer: + - runs a container based on the resulting container image mounting the source code to be built, + - develops applications for target platform on the container and + - leaves the container getting the results on the mounted directory. + +### Solution + +User specifies the variable `SDK_FORMAT` providing a space-separated list of SDK formats to generate. + +Supported formats are: + - `tar-xz`: (default) is the non-containerized format that results from following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" + - `docker-archive`: an archive containing a Docker image that can be imported with [`docker import`](https://docs.docker.com/engine/reference/commandline/import/) + - `docker-daemon`: resulting container image is made available on the local Docker Daemon + - `containers-storage`: resulting container image is made available to tools using containers/storage back-end (e.g. Podman, CRIO, buildah,...) + - `oci-archive`: an archive containing an OCI image, mostly for archiving as seed for any of the above formats + +User manually triggers creation of SDK formats for his target platform by launching the task `do_populate_sdk` for target image, f.e. +`bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. +Packages that should be additionally installed into the SDK can be appended to `SDK_PREINSTALL` (external repositories) and `SDK_INSTALL` (self-built). + +Following formats don't work if running `bitbake -c do_populate_sdk ...` (to generate the containerized SDK) from inside of a container (e.g. using `kas-container`): `docker-daemon` and `containers-storage`. +It's technically possible, but requires making host resources (e.g. the Docker Daemon socket) accessible in the container. +What can endanger the stability and security of the host. + +The resulting SDK formats are archived into `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}-${sdk_format}.tar.xz` (being `sdk_format` each one of the formats specified in `SDK_FORMATS`). +The SDK container directory `/isar-apt` contains a copy of isar-apt repo with locally prebuilt target debian packages (for <HOST_DISTRO>). +One may get into an SDK container and install required target packages with the help of `apt-get install <package_name>:<DISTRO_ARCH>` command. +The directory with the source code to develop on should be mounted on the container (with `--volume <host-directory>:<container-directory>`) to be able to edit files in the host with an IDE and build in the container. + +### Example + + - Make the SDK formats to generate available to the task + +For one-shot builds (use `local.conf` otherwise): + +``` +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SDK_FORMATS" +export SDK_FORMATS="docker-archive" +``` + + - Trigger creation of SDK root filesystem + +``` +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base +``` + + - Load the SDK container image into the Docker Daemon + +``` +xzcat build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf-docker-archive.tar.xz | docker load +``` + + - Run a container using the SDK container image (following commands starting with `#~:` are to be run in the container) + +``` +docker run --rm -ti --volume "$(pwd):/build" isar-sdk-buster-armhf:latest +``` + + - Check that cross toolchains are installed + +``` +:~# dpkg -l | grep crossbuild-essential-armhf +ii crossbuild-essential-armhf 12.3 all Informational list of cross-build-essential packages +``` + ## Creation of local apt repo caching upstream Debian packages ### Motivation
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> --- doc/user_manual.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+)