Message ID | 20220405132431.2895832-4-adriaan.schmidt@siemens.com |
---|---|
State | Superseded, archived |
Headers | show |
Series | imagetypes | expand |
Hello, Adriaan 05.04.2022 16:24, Adriaan Schmidt wrote: > This makes a number of changes: > - Replace all do_*_image tasks with IMAGE_CMD_* definitions > - Remove all implicit conversions (gz, xz) > - Add IMAGE_CONVERSION_* commands > - Create tasks (do_image_*) on demand in image.bbclass for > all requested IMAGE_FSTYPES > - When creating the tasks: > - image_do_mounts is inserted automatically > - a final chown is inserted automatically > - variables IMAGE_FILE_HOST and IMAGE_FILE_CHROOT are > set to reference the image > - variable SUDO_CHROOT contains the chroot command > - Create conversions on demand based on IMAGE_FSTYPES. > A conversion is defined by CONVERSION_CMD_type, and its > dependencies given as CONVERSION_DEPS_type > - In conversion commands > - the input file is named ${IMAGE_FULLNAME}.${type} > - the conversions appends its own type, e.g. the output file > would be ${IMAGE_FULLNAME}.${type}.xz > - a final chown is appended automatically > - Image types now longer have a -img suffix, e.g., ext4 instead > of ext4-img, and conversions are appended like tar.gz instead > of targz-img > - Imager dependencies are set as IMAGER_INSTALL_type > - Dependencies between image types are modelled like > IMAGE_TYPEDEP_ova = "wic" > - Required arguments/variables are modelled by > IMAGE_CMD_REQUIRED_ARGUMENTS = "A B C" > - Container types (previously CONTAINER_IMAGE_FORMATS) are now > first class image types (oci, oci-archive, docker-archive, > docker-daemon, containers.storage) > - The logic of image-container-extension has moved to > container-img > - The VM image now has type ova (instead of vm-img) > > Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com> > --- > meta/classes/container-img.bbclass | 94 ++++++++- > meta/classes/cpiogz-img.bbclass | 20 +- > meta/classes/ext4-img.bbclass | 21 +- > meta/classes/fit-img.bbclass | 17 +- > .../classes/image-container-extension.bbclass | 83 -------- > meta/classes/image.bbclass | 184 +++++++++++++++++- > meta/classes/rootfs.bbclass | 1 + > meta/classes/sdk.bbclass | 2 +- > meta/classes/targz-img.bbclass | 11 +- > meta/classes/ubi-img.bbclass | 24 +-- > meta/classes/ubifs-img.bbclass | 29 +-- > meta/classes/vm-img.bbclass | 24 +-- > meta/classes/wic-img.bbclass | 45 ++--- > 13 files changed, 331 insertions(+), 224 deletions(-) > delete mode 100644 meta/classes/image-container-extension.bbclass > > diff --git a/meta/classes/container-img.bbclass b/meta/classes/container-img.bbclass > index 322889a9..436a0051 100644 > --- a/meta/classes/container-img.bbclass > +++ b/meta/classes/container-img.bbclass > @@ -3,15 +3,93 @@ > # > # SPDX-License-Identifier: MIT > # > -# This class provides the task 'container_image' > +# This class provides the task 'containerize' > # to create container images containing the target rootfs. > > -do_container_image[dirs] = "${DEPLOY_DIR_IMAGE}" > -do_container_image[stamp-extra-info] = "${DISTRO}-${MACHINE}" > -do_container_image[vardeps] += "CONTAINER_IMAGE_FORMATS" > -do_container_image(){ > - bbdebug 1 "Generate container image in these formats: ${CONTAINER_IMAGE_FORMATS}" > - containerize_rootfs "${IMAGE_ROOTFS}" "${CONTAINER_IMAGE_FORMATS}" > +CONTAINER_TYPES = "oci oci-archive docker-archive docker-daemon containers-storage" > +USING_CONTAINER = "${@bb.utils.contains_any('IMAGE_BASETYPES', d.getVar('CONTAINER_TYPES').split(), '1', '0', d)}" > + > +CONTAINER_IMAGE_NAME ?= "${PN}-${DISTRO}-${DISTRO_ARCH}" > +CONTAINER_IMAGE_TAG ?= "${PV}-${PR}" > + > +python() { > + if not d.getVar('USING_CONTAINER') == '1': > + return > + for t in d.getVar('CONTAINER_TYPES').split(): > + t_clean = t.replace('-', '_').replace('.', '_') > + d.setVar('IMAGE_CMD_' + t_clean, 'convert_container %s "${CONTAINER_IMAGE_NAME}" "${IMAGE_FILE_HOST}"' % t) > + d.setVar('IMAGE_FULLNAME_' + t_clean, '${PN}-${DISTRO}-${DISTRO_ARCH}') > + bb.build.addtask('containerize', 'do_image_' + t_clean, 'do_image_tools', d) > +} > + > +do_containerize() { > + local cmd="/bin/dash" > + local empty_tag="empty" > + local tag="${CONTAINER_IMAGE_TAG}" > + local oci_img_dir="${WORKDIR}/oci-image" > + local rootfs="${IMAGE_ROOTFS}" > + > + # prepare OCI container image skeleton > + bbdebug 1 "prepare OCI container image skeleton" > + sudo rm -rf "${oci_img_dir}" "${oci_img_dir}_unpacked" > + sudo umoci init --layout "${oci_img_dir}" > + sudo umoci new --image "${oci_img_dir}:${empty_tag}" > + sudo umoci config --image "${oci_img_dir}:${empty_tag}" \ > + --config.cmd="${cmd}" > + sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \ > + "${oci_img_dir}_unpacked" > + > + # add root filesystem as the flesh of the skeleton > + sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/" > + # clean-up temporary files > + sudo find "${oci_img_dir}_unpacked/rootfs/tmp" -mindepth 1 -delete > + > + # pack container image > + bbdebug 1 "pack container image" > + sudo umoci repack --image "${oci_img_dir}:${tag}" \ > + "${oci_img_dir}_unpacked" > + sudo umoci remove --image "${oci_img_dir}:${empty_tag}" > + sudo rm -rf "${oci_img_dir}_unpacked" > + > + # no root needed anymore > + sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}" > } > > -addtask container_image before do_image after do_image_tools > +convert_container() { > + local tag="${CONTAINER_IMAGE_TAG}" > + local oci_img_dir="${WORKDIR}/oci-image" > + local container_type="$1" > + local image_name="$2" > + local image_archive="$3" > + > + # convert the OCI container image to the desired format > + bbdebug 1 "Creating container image type: ${container_type}" > + case "${container_type}" in > + "docker-archive" | "oci-archive") > + if [ "${container_type}" = "oci-archive" ] ; then > + target="${container_type}:${image_archive}:${tag}" > + else > + target="${container_type}:${image_archive}:${image_name}:${tag}" > + fi > + rm -f "${image_archive}" > + bbdebug 2 "Converting OCI image to ${container_type}" > + skopeo --insecure-policy copy \ > + "oci:${oci_img_dir}:${tag}" "${target}" > + ;; > + "oci") > + tar --create --directory "${oci_img_dir}" \ > + --file "${image_archive}" . > + ;; > + "docker-daemon" | "containers-storage") > + if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then > + die "Adding the container image to a container runtime (${container_type}) not supported if running from a container (e.g. 'kas-container')" > + fi > + skopeo --insecure-policy copy \ > + "oci:${oci_img_dir}:${tag}" \ > + "${container_type}:${image_name}:${tag}" > + ;; > + *) > + die "Unsupported format for convert_container: ${container_type}" > + ;; > + esac > +} > diff --git a/meta/classes/cpiogz-img.bbclass b/meta/classes/cpiogz-img.bbclass > index 2a49456b..f4c33bd9 100644 > --- a/meta/classes/cpiogz-img.bbclass > +++ b/meta/classes/cpiogz-img.bbclass > @@ -3,20 +3,12 @@ > # > # SPDX-License-Identifier: MIT > > -CPIOGZ_FNAME ?= "${IMAGE_FULLNAME}.cpio.gz" > -CPIOGZ_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${CPIOGZ_FNAME}" > -IMAGER_INSTALL += "cpio" > +IMAGER_INSTALL_cpio += "cpio" > CPIO_IMAGE_FORMAT ?= "newc" > > -do_cpiogz_image() { > - sudo rm -f ${CPIOGZ_IMAGE_FILE} > - image_do_mounts > - sudo chroot ${BUILDCHROOT_DIR} \ > - sh -c "cd ${PP_ROOTFS}; /usr/bin/find . | \ > - /usr/bin/cpio -H ${CPIO_IMAGE_FORMAT} -o | /usr/bin/gzip -9 > \ > - ${PP_DEPLOY}/${CPIOGZ_FNAME}" > - sudo chown $(id -u):$(id -g) ${CPIOGZ_IMAGE_FILE} > +IMAGE_CMD_cpio() { > + ${SUDO_CHROOT} \ > + sh -c "cd ${PP_ROOTFS}; /usr/bin/find . | \ > + /usr/bin/cpio -H ${CPIO_IMAGE_FORMAT} -o > \ > + ${IMAGE_FILE_CHROOT}" > } > - > -addtask cpiogz_image before do_image after do_image_tools > -do_cpiogz_image[dirs] = "${DEPLOY_DIR_IMAGE}" > diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass > index 5085afcc..73d1bb57 100644 > --- a/meta/classes/ext4-img.bbclass > +++ b/meta/classes/ext4-img.bbclass > @@ -1,24 +1,15 @@ > # This software is a part of ISAR. > # Copyright (C) 2015-2017 ilbers GmbH > > -EXT4_IMAGE_FILE = "${IMAGE_FULLNAME}.ext4.img" > - > -IMAGER_INSTALL += "e2fsprogs" > +IMAGER_INSTALL_ext4 += "e2fsprogs" > > MKE2FS_ARGS ?= "-t ext4" > > # Generate ext4 filesystem image > -do_ext4_image() { > - rm -f '${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}' > - > - truncate -s ${ROOTFS_SIZE}K '${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}' > +IMAGE_CMD_ext4() { > + truncate -s ${ROOTFS_SIZE}K '${IMAGE_FILE_HOST}' > > - image_do_mounts > - > - sudo chroot ${BUILDCHROOT_DIR} /sbin/mke2fs ${MKE2FS_ARGS} \ > - -F -d '${PP_ROOTFS}' '${PP_DEPLOY}/${EXT4_IMAGE_FILE}' > + ${SUDO_CHROOT} /sbin/mke2fs ${MKE2FS_ARGS} \ > + -F -d '${PP_ROOTFS}' '${IMAGE_FILE_CHROOT}' > } > - > -addtask ext4_image before do_image after do_image_tools > -do_ext4_image[prefuncs] = 'set_image_size' > -do_ext4_image[dirs] = "${DEPLOY_DIR_IMAGE}" > +#IMAGE_CMD_ext4[vardepsexclude] = "ROOTFS_SIZE ROOTFS_EXTRA" > diff --git a/meta/classes/fit-img.bbclass b/meta/classes/fit-img.bbclass > index 1ad0c5b8..ef65af88 100644 > --- a/meta/classes/fit-img.bbclass > +++ b/meta/classes/fit-img.bbclass > @@ -6,24 +6,17 @@ > MKIMAGE_ARGS ??= "" > > FIT_IMAGE_SOURCE ??= "fitimage.its" > -FIT_IMAGE_FILE ?= "${IMAGE_FULLNAME}.fit.img" > > -IMAGER_INSTALL += "u-boot-tools device-tree-compiler" > +IMAGER_INSTALL_fit += "u-boot-tools device-tree-compiler" > > # Generate fit image > -do_fit_image() { > +IMAGE_CMD_fit() { > if [ ! -e "${WORKDIR}/${FIT_IMAGE_SOURCE}" ]; then > die "FIT_IMAGE_SOURCE does not contain fitimage source file" > fi > > - rm -f '${DEPLOY_DIR_IMAGE}/${FIT_IMAGE_FILE}' > - > - image_do_mounts > - > # Create fit image using buildchroot tools > - sudo chroot ${BUILDCHROOT_DIR} /usr/bin/mkimage ${MKIMAGE_ARGS} \ > - -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${PP_DEPLOY}/${FIT_IMAGE_FILE}' > - sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${FIT_IMAGE_FILE}' > + ${SUDO_CHROOT} /usr/bin/mkimage ${MKIMAGE_ARGS} \ > + -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${IMAGE_FILE_CHROOT}' > } > -addtask fit_image before do_image after do_image_tools do_transform_template > -do_fit_image[dirs] = "${DEPLOY_DIR_IMAGE}" > +IMAGE_CMD_fit[depends] = "${PN}:do_transform_template" > diff --git a/meta/classes/image-container-extension.bbclass b/meta/classes/image-container-extension.bbclass > deleted file mode 100644 > index cdec4633..00000000 > --- a/meta/classes/image-container-extension.bbclass > +++ /dev/null > @@ -1,83 +0,0 @@ > -# This software is a part of ISAR. > -# Copyright (C) Siemens AG, 2021 > -# > -# SPDX-License-Identifier: MIT > -# > -# This class extends the image.bbclass for containerizing the root filesystem. > - > -CONTAINER_IMAGE_FORMATS ?= "docker-archive" > -CONTAINER_IMAGE_NAME ?= "${PN}-${DISTRO}-${DISTRO_ARCH}" > -CONTAINER_IMAGE_TAG ?= "${PV}-${PR}" > - > -containerize_rootfs() { > - local cmd="/bin/dash" > - local empty_tag="empty" > - local tag="${CONTAINER_IMAGE_TAG}" > - local oci_img_dir="${WORKDIR}/oci-image" > - local rootfs="$1" > - local container_formats="$2" > - local container_name_prefix="$3" > - > - # prepare OCI container image skeleton > - bbdebug 1 "prepare OCI container image skeleton" > - sudo rm -rf "${oci_img_dir}" "${oci_img_dir}_unpacked" > - sudo umoci init --layout "${oci_img_dir}" > - sudo umoci new --image "${oci_img_dir}:${empty_tag}" > - sudo umoci config --image "${oci_img_dir}:${empty_tag}" \ > - --config.cmd="${cmd}" > - sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \ > - "${oci_img_dir}_unpacked" > - > - # add root filesystem as the flesh of the skeleton > - sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/" > - # clean-up temporary files > - sudo find "${oci_img_dir}_unpacked/rootfs/tmp" -mindepth 1 -delete > - > - # pack container image > - bbdebug 1 "pack container image" > - sudo umoci repack --image "${oci_img_dir}:${tag}" \ > - "${oci_img_dir}_unpacked" > - sudo umoci remove --image "${oci_img_dir}:${empty_tag}" > - sudo rm -rf "${oci_img_dir}_unpacked" > - > - # no root needed anymore > - sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}" > - > - # convert the OCI container image to the desired format > - image_name="${container_name_prefix}${CONTAINER_IMAGE_NAME}" > - for image_type in ${CONTAINER_IMAGE_FORMATS} ; do > - image_archive="${DEPLOY_DIR_IMAGE}/${image_name}-${tag}-${image_type}.tar" > - bbdebug 1 "Creating container image type: ${image_type}" > - case "${image_type}" in > - "docker-archive" | "oci-archive") > - if [ "${image_type}" = "oci-archive" ] ; then > - target="${image_type}:${image_archive}:${tag}" > - else > - target="${image_type}:${image_archive}:${image_name}:${tag}" > - fi > - rm -f "${image_archive}" "${image_archive}.xz" > - bbdebug 2 "Converting OCI image to ${image_type}" > - skopeo --insecure-policy copy \ > - "oci:${oci_img_dir}:${tag}" "${target}" > - bbdebug 2 "Compressing image" > - xz -T0 "${image_archive}" > - ;; > - "oci") > - tar --create --xz --directory "${oci_img_dir}" \ > - --file "${image_archive}.xz" . > - ;; > - "docker-daemon" | "containers-storage") > - if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then > - die "Adding the container image to a container runtime (${image_type}) not supported if running from a container (e.g. 'kas-container')" > - fi > - skopeo --insecure-policy copy \ > - "oci:${oci_img_dir}:${tag}" \ > - "${image_type}:${image_name}:${tag}" > - ;; > - *) > - die "Unsupported format for containerize_rootfs: ${image_type}" > - ;; > - esac > - done > -} > - > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index f87b76e7..4413a7d6 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -10,7 +10,8 @@ STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*" > SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${MACHINE}-${DISTRO}-${DISTRO_ARCH}" > > IMAGE_INSTALL ?= "" > -IMAGE_FSTYPES ?= "${@ d.getVar("IMAGE_TYPE", True) if d.getVar("IMAGE_TYPE", True) else "ext4-img"}" > +IMAGE_FSTYPES ?= "${@ d.getVar("IMAGE_TYPE", True) if d.getVar("IMAGE_TYPE", True) else "ext4"}" > +IMAGE_CONVERSIONS = "gz xz" > IMAGE_ROOTFS ?= "${WORKDIR}/rootfs" > > KERNEL_IMAGE_PKG ??= "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}" > @@ -83,7 +84,183 @@ inherit image-tools-extension > inherit image-postproc-extension > inherit image-locales-extension > inherit image-account-extension > -inherit image-container-extension > + > +def get_base_type(t, d): > + bt = t > + for c in d.getVar('IMAGE_CONVERSIONS').split(): > + if t.endswith('.' + c): > + bt = t[:-len('.' + c)] > + break > + return bt if bt == t else get_base_type(bt, d) > + > +# determine image basetypes, just so we can use it in imagetypes* classes > +python() { > + basetypes = set() > + for t in (d.getVar('IMAGE_FSTYPES') or '').split(): > + bt = get_base_type(t, d) > + if bt.endswith('-img'): > + # be backwards-compatible > + bt = bt[:-len('-img')] > + bb.warn("IMAGE_TYPE '{0}-img' is deprecated. Please use '{0}' instead.".format(bt)) > + basetypes.add(bt) > + deps = (d.getVar('IMAGE_TYPEDEP_' + bt.replace('-', '_').replace('.', '_')) or '').split() > + basetypes |= set([get_base_type(t, d) for t in deps]) > + d.setVar('IMAGE_BASETYPES', ' '.join(basetypes)) > +} > + > +# image types > +IMAGE_CLASSES ??= "" > +IMGCLASSES = "container-img cpiogz-img ext4-img fit-img targz-img ubi-img ubifs-img vm-img wic-img" > +IMGCLASSES += "${IMAGE_CLASSES}" > +inherit ${IMGCLASSES} > + > +# image conversions > +CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" > +CONVERSION_DEPS_gz = "gzip" > + > +XZ_OPTIONS ?= "" > +CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'cat ${IMAGE_FILE_CHROOT} | xz ${XZ_OPTIONS} > ${IMAGE_FILE_CHROOT}.xz'" > +CONVERSION_DEPS_xz = "xz-utils" > + > +# hook up IMAGE_CMD_* > +python() { > + image_types = (d.getVar('IMAGE_FSTYPES') or '').split() > + conversions = set(d.getVar('IMAGE_CONVERSIONS').split()) > + > + basetypes = {} > + typedeps = {} > + vardeps = set() > + > + def collect_image_type(t): > + bt = get_base_type(t, d) > + if bt.endswith('-img'): > + # be backwards-compatible > + bt = bt[:-len('-img')] > + bb.warn("IMAGE_TYPE '{0}-img' is deprecated. Please use '{0}' instead.".format(bt)) > + > + if bt not in basetypes: > + basetypes[bt] = [] > + if t not in basetypes[bt]: > + basetypes[bt].append(t) > + t_clean = t.replace('-', '_').replace('.', '_') > + deps = (d.getVar('IMAGE_TYPEDEP_' + t_clean) or '').split() > + vardeps.add('IMAGE_TYPEDEP_' + t_clean) > + if bt not in typedeps: > + typedeps[bt] = set() > + for dep in deps: > + if dep not in image_types: > + image_types.append(dep) > + collect_image_type(dep) > + typedeps[bt].add(get_base_type(dep, d)) > + if bt != t: > + collect_image_type(bt) > + > + for t in image_types[:]: > + collect_image_type(t) > + > + # TODO: OE uses do_image, but Isar is different... > + d.appendVarFlag('do_image_tools', 'vardeps', ' '.join(vardeps)) > + > + imager_install = set() > + imager_build_deps = set() > + conversion_install = set() > + for bt in basetypes: > + vardeps = set() > + cmds = [] > + bt_clean = bt.replace('-', '_').replace('.', '_') > + > + # prepare local environment > + localdata = bb.data.createCopy(d) > + localdata.setVar('OVERRIDES', bt_clean + ':' + d.getVar('OVERRIDES', False)) > + localdata.setVar('PV', d.getVar('PV')) > + localdata.delVar('DATETIME') > + localdata.delVar('DATE') > + localdata.delVar('TMPDIR') > + vardepsexclude = (d.getVarFlag('IMAGE_CMD_' + bt_clean, 'vardepsexclude', True) or '').split() > + for dep in vardepsexclude: > + localdata.delVar(dep) > + > + # check if required args are set > + required_args = (localdata.getVar('IMAGE_CMD_REQUIRED_ARGS') or '').split() > + if any([d.getVar(arg) is None for arg in required_args]): > + bb.fatal("IMAGE_TYPE '%s' requires these arguments: %s" % (image_type, ', '.join(required_args))) > + > + # convenience variables to be used by CMDs > + localdata.setVar('IMAGE_FILE_HOST', '${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.${type}') > + #bb.warn("FULLNAME is %s -> %s" % (localdata.getVar('IMAGE_FULLNAME', False), localdata.getVar('IMAGE_FULLNAME', True))) > + localdata.setVar('IMAGE_FILE_CHROOT', '${PP_DEPLOY}/${IMAGE_FULLNAME}.${type}') > + localdata.setVar('SUDO_CHROOT', localdata.expand('sudo chroot ${BUILDCHROOT_DIR}')) > + > + # imager install > + for dep in (d.getVar('IMAGER_INSTALL_' + bt_clean) or '').split(): > + imager_install.add(dep) > + for dep in (d.getVar('IMAGER_BUILD_DEPS_' + bt_clean) or '').split(): > + imager_build_deps.add(dep) > + > + # construct image command > + cmds.append('\timage_do_mounts') > + image_cmd = localdata.getVar('IMAGE_CMD_' + bt_clean) > + if image_cmd: > + localdata.setVar('type', bt) > + cmds.append(localdata.expand(image_cmd)) > + #bb.warn("IMAGE_CMD\n*** %s\n*** %s" % (image_cmd, localdata.expand(image_cmd))) > + cmds.append(localdata.expand('\tsudo chown $(id -u):$(id -g) ${IMAGE_FILE_HOST}')) > + else: > + bb.fatal("No IMAGE_CMD for %s" % bt) > + vardeps.add('IMAGE_CMD_' + bt_clean) > + d.delVarFlag('IMAGE_CMD_' + bt_clean, 'func') > + task_deps = d.getVarFlag('IMAGE_CMD_' + bt_clean, 'depends') > + > + # add conversions > + conversion_depends = set() > + rm_images = set() > + def create_conversions(t): > + for c in sorted(conversions): > + if t.endswith('.' + c): > + t = t[:-len(c) - 1] > + create_conversions(t) > + localdata.setVar('type', t) > + cmd = '\t' + localdata.getVar('CONVERSION_CMD_' + c) > + if cmd not in cmds: > + cmds.append(cmd) > + cmds.append(localdata.expand('\tsudo chown $(id -u):$(id -g) ${IMAGE_FILE_HOST}.%s' % c)) > + vardeps.add('CONVERSION_CMD_' + c) > + for dep in (localdata.getVar('CONVERSION_DEPS_' + c) or '').split(): > + conversion_install.add(dep) > + # remove temporary image files > + if t not in image_types: > + rm_images.add(localdata.expand('${IMAGE_FILE_HOST}')) > + > + for t in basetypes[bt]: > + create_conversions(t) > + > + if bt not in image_types: > + localdata.setVar('type', t) > + rm_images.add(localdata.expand('${IMAGE_FILE_HOST}')) > + > + for image in rm_images: > + cmds.append('\trm ' + image) > + > + # image type dependencies > + after = 'do_image_tools' > + for dep in typedeps[bt]: > + after += ' do_image_%s' % dep.replace('-', '_').replace('.', '_') > + > + # create the task > + task = 'do_image_%s' % bt_clean > + d.setVar(task, '\n'.join(cmds)) > + d.setVarFlag(task, 'func', '1') > + d.appendVarFlag(task, 'prefuncs', ' set_image_size') > + d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) > + d.appendVarFlag(task, 'vardepsexclude', ' ' + ' '.join(vardepsexclude)) > + d.appendVarFlag(task, 'dirs', localdata.expand(' ${DEPLOY_DIR_IMAGE}')) > + if task_deps: > + d.appendVarFlag(task, 'depends', task_deps) > + bb.build.addtask(task, 'do_image', after, d) > + > + d.appendVar('IMAGER_INSTALL', ' ' + ' '.join(sorted(imager_install | conversion_install))) > + d.appendVar('IMAGER_BUILD_DEPS', ' ' + ' '.join(sorted(imager_build_deps))) > +} > > # Extra space for rootfs in MB > ROOTFS_EXTRA ?= "64" > @@ -256,6 +433,3 @@ do_rootfs_quality_check() { > } > > addtask rootfs_quality_check after do_rootfs_finalize before do_rootfs > - > -# Last so that the image type can overwrite tasks if needed > -inherit ${IMAGE_FSTYPES} > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > index b021e728..7e9fb0f3 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -172,6 +172,7 @@ rootfs_install_pkgs_install() { > > do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" > do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}" > +do_rootfs_install[vardepsexclude] += "IMAGE_ROOTFS" > do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build" > do_rootfs_install[recrdeptask] = "do_deploy_deb" > python do_rootfs_install() { > diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass > index adf9a1fe..477dff70 100644 > --- a/meta/classes/sdk.bbclass > +++ b/meta/classes/sdk.bbclass > @@ -31,7 +31,7 @@ SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${BPN}-${MACHINE}" > > # SDK settings > SDK_INCLUDE_ISAR_APT ?= "0" > -SDK_FORMATS ?= "targz-img" > +SDK_FORMATS ?= "tar.xz" > SDK_INSTALL ?= "" > SDK_PREINSTALL += " \ > debhelper \ > diff --git a/meta/classes/targz-img.bbclass b/meta/classes/targz-img.bbclass > index bf94af02..74d34e29 100644 > --- a/meta/classes/targz-img.bbclass > +++ b/meta/classes/targz-img.bbclass > @@ -3,13 +3,6 @@ > # > # SPDX-License-Identifier: MIT > > -TARGZ_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz" > - > -do_targz_image() { > - rm -f ${TARGZ_IMAGE_FILE} > - sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} . > - sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE} > +IMAGE_CMD_tar() { > + sudo tar -cvzf ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz --one-file-system -C ${IMAGE_ROOTFS} . > } > - > -addtask targz_image before do_image after do_image_tools > -do_targz_image[dirs] = "${DEPLOY_DIR_IMAGE}" > diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass > index efaf058e..92acb6f8 100644 > --- a/meta/classes/ubi-img.bbclass > +++ b/meta/classes/ubi-img.bbclass > @@ -3,30 +3,18 @@ > # > # SPDX-License-Identifier: MIT > > -python() { > - if not d.getVar("UBINIZE_ARGS"): > - raise bb.parse.SkipRecipe("UBINIZE_ARGS must be set") > -} > - > UBINIZE_CFG ??= "ubinize.cfg" > -UBI_IMAGE_FILE ?= "${IMAGE_FULLNAME}.ubi.img" > > -IMAGER_INSTALL += "mtd-utils" > +IMAGER_INSTALL_ubi += "mtd-utils" > > # Generate ubi filesystem image > -do_ubi_image() { > +IMAGE_CMD_ubi() { > if [ ! -e "${WORKDIR}/${UBINIZE_CFG}" ]; then > die "UBINIZE_CFG does not contain ubinize config file." > fi > > - rm -f '${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}' > - > - image_do_mounts > - > - # Create ubi image using buildchroot tools > - sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \ > - -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' '${PP_WORK}/${UBINIZE_CFG}' > - sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}' > + ${SUDO_CHROOT} /usr/sbin/ubinize ${UBINIZE_ARGS} \ > + -o '${IMAGE_FILE_CHROOT}' '${PP_WORK}/${UBINIZE_CFG}' > } > -addtask ubi_image before do_image after do_image_tools do_transform_template > -do_ubi_image[dirs] = "${DEPLOY_DIR_IMAGE}" > +IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template" > +IMAGE_CMD_REQUIRED_ARGS_ubi = "UBINIZE_ARGS" > diff --git a/meta/classes/ubifs-img.bbclass b/meta/classes/ubifs-img.bbclass > index 229eb3ef..60c3bf14 100644 > --- a/meta/classes/ubifs-img.bbclass > +++ b/meta/classes/ubifs-img.bbclass > @@ -3,30 +3,19 @@ > # > # SPDX-License-Identifier: MIT > > -python() { > - if not d.getVar("MKUBIFS_ARGS"): > - raise bb.parse.SkipRecipe("mkubifs_args must be set") > -} > - > -UBIFS_IMAGE_FILE ?= "${IMAGE_FULLNAME}.ubifs.img" > - > -IMAGER_INSTALL += "mtd-utils" > +IMAGER_INSTALL_ubifs += "mtd-utils" > > # glibc bug 23960 https://sourceware.org/bugzilla/show_bug.cgi?id=23960 > # should not use QEMU on armhf target with mkfs.ubifs < v2.1.3 > -ISAR_CROSS_COMPILE_armhf = "1" > +python() { > + if 'ubifs' in (d.getVar('IMAGE_BASETYPES') or '').split(): > + d.setVar('ISAR_CROSS_COMPILE_armhf', '1') > +} > This will also not work and actually fails on imx6-sabrelite-bullseye target. Reasons: 1) IMAGE_BASETYPES = "ubi ubi-ubifs" for that target and condition is False 2) ISAR_CROSS_COMPILE should be set before anonymous python code because buildchroot selection is done also in python() {} Also it better to document (or at least comment) IMAGE_BASETYPES as a new variable. > # Generate ubifs filesystem image > -do_ubifs_image() { > - rm -f '${DEPLOY_DIR_IMAGE}/${UBIFS_IMAGE_FILE}' > - > - image_do_mounts > - > +IMAGE_CMD_ubifs() { > # Create ubifs image using buildchroot tools > - sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} \ > - -r '${PP_ROOTFS}' '${PP_DEPLOY}/${UBIFS_IMAGE_FILE}' > - sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${UBIFS_IMAGE_FILE}' > + ${SUDO_CHROOT} /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} \ > + -r '${PP_ROOTFS}' '${IMAGE_FILE_CHROOT}' > } > - > -addtask ubifs_image before do_image after do_image_tools > -do_ubifs_image[dirs] = "${DEPLOY_DIR_IMAGE}" > +IMAGE_CMD_REQUIRED_ARGS_ubifs = "MKUBIFS_ARGS" > diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass > index 4bc977b9..8a676aca 100644 > --- a/meta/classes/vm-img.bbclass > +++ b/meta/classes/vm-img.bbclass > @@ -5,16 +5,18 @@ > # > > inherit buildchroot > -inherit wic-img > + > +USING_OVA = "${@bb.utils.contains('IMAGE_BASETYPES', 'ova', '1', '0', d)}" > > FILESEXTRAPATHS_prepend := "${LAYERDIR_core}/classes/vm-img:" > OVF_TEMPLATE_FILE ?= "vm-img-virtualbox.ovf.tmpl" > -SRC_URI += "file://${OVF_TEMPLATE_FILE}" > +SRC_URI += "${@'file://${OVF_TEMPLATE_FILE}' if d.getVar('USING_OVA') == '1' else ''}" > > -IMAGER_INSTALL += "qemu-utils gawk uuid-runtime" > +IMAGE_TYPEDEP_ova = "wic" > +IMAGER_INSTALL_ova += "qemu-utils gawk uuid-runtime" > > # virtual machine disk settings > -SOURCE_IMAGE_FILE ?= "${IMAGE_FULLNAME}.wic.img" > +SOURCE_IMAGE_FILE ?= "${IMAGE_FULLNAME}.wic" > > # For VirtualBox, this needs to be "monolithicSparse" (default to it). > # VMware needs this to be "streamOptimized". > @@ -34,7 +36,7 @@ def set_convert_options(d): > > CONVERSION_OPTIONS = "${@set_convert_options(d)}" > > -do_convert_wic() { > +convert_wic() { > rm -f '${DEPLOY_DIR_IMAGE}/${VIRTUAL_MACHINE_IMAGE_FILE}' > image_do_mounts > bbnote "Creating ${VIRTUAL_MACHINE_IMAGE_FILE} from ${SOURCE_IMAGE_FILE}" > @@ -43,8 +45,6 @@ do_convert_wic() { > '${PP_DEPLOY}/${SOURCE_IMAGE_FILE}' '${VIRTUAL_MACHINE_DISK}' > } > > -addtask convert_wic before do_build after do_wic_image do_copy_boot_files do_install_imager_deps do_transform_template > - > # User settings for OVA > OVA_NAME ?= "${IMAGE_FULLNAME}" > OVA_MEMORY ?= "8192" > @@ -67,10 +67,11 @@ OVA_VARS = "OVA_NAME OVA_MEMORY OVA_NUMBER_OF_CPU OVA_VRAM \ > OVA_FIRMWARE OVA_ACPI OVA_3D_ACCEL \ > OVA_SHA_ALG VIRTUAL_MACHINE_IMAGE_FILE" > > -TEMPLATE_FILES += "${OVF_TEMPLATE_FILE}" > +TEMPLATE_FILES += "${@'${OVF_TEMPLATE_FILE}' if d.getVar('USING_OVA') == '1' else ''}" > TEMPLATE_VARS += "${OVA_VARS}" > > -do_create_ova() { > +do_image_ova[prefuncs] += "convert_wic" > +IMAGE_CMD_ova() { > if [ ! ${VIRTUAL_MACHINE_IMAGE_TYPE} = "vmdk" ]; then > exit 0 > fi > @@ -81,10 +82,7 @@ do_create_ova() { > export PRIMARY_MAC=$(macgen) > export LAST_CHANGE=$(date -u "+%Y-%m-%dT%H:%M:%SZ") > export OVA_FIRMWARE_UPPERCASE=$(echo ${OVA_FIRMWARE} | tr '[a-z]' '[A-Z]') > - > export OVF_TEMPLATE_STAGE2=$(echo ${OVF_TEMPLATE_FILE} | sed 's/.tmpl$//' ) > - image_do_mounts > - > sudo -Es chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} <<'EOSUDO' > set -e > export DISK_SIZE_BYTES=$(qemu-img info -f vmdk "${VIRTUAL_MACHINE_DISK}" \ > @@ -104,5 +102,3 @@ do_create_ova() { > tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${VIRTUAL_MACHINE_IMAGE_FILE} > EOSUDO > } > - > -addtask do_create_ova after do_convert_wic before do_deploy > diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass > index 7537a27b..cfcc94c7 100644 > --- a/meta/classes/wic-img.bbclass > +++ b/meta/classes/wic-img.bbclass > @@ -4,7 +4,8 @@ > # this class is heavily inspired by OEs ./meta/classes/image_types_wic.bbclass > # > > -WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}')}" > +USING_WIC = "${@bb.utils.contains('IMAGE_BASETYPES', 'wic', '1', '0', d)}" > +WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if d.getVar('USING_WIC') == '1' else ''}" > > WKS_FILE ??= "sdimage-efi" > > @@ -14,6 +15,9 @@ do_copy_wks_template () { > } > > python () { > + if not d.getVar('USING_WIC') == '1': > + return > + > import itertools > import re > > @@ -74,13 +78,13 @@ python () { > except (IOError, OSError) as exc: > pass > else: > - bb.build.addtask('do_copy_wks_template', 'do_transform_template do_wic_image', None, d) > - bb.build.addtask('do_transform_template', 'do_wic_image', None, d) > + bb.build.addtask('do_copy_wks_template', 'do_transform_template do_image_wic', None, d) > + bb.build.addtask('do_transform_template', 'do_image_wic', None, d) > } > > inherit buildchroot > > -IMAGER_INSTALL += "${WIC_IMAGER_INSTALL}" > +IMAGER_INSTALL_wic += "${WIC_IMAGER_INSTALL}" > # wic comes with reasonable defaults, and the proper interface is the wks file > ROOTFS_EXTRA ?= "0" > > @@ -125,32 +129,23 @@ python do_rootfs_wicenv () { > > } > > -addtask do_rootfs_wicenv after do_rootfs before do_wic_image > +addtask do_rootfs_wicenv after do_rootfs before do_image_wic > do_rootfs_wicenv[vardeps] += "${WICVARS}" > do_rootfs_wicenv[prefuncs] = 'set_image_size' > > -WIC_IMAGE_FILE ="${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic.img" > - > -python check_for_wic_warnings() { > - with open("{}/log.do_wic_image".format(d.getVar("T"))) as f: > - for line in f.readlines(): > - if line.startswith("WARNING"): > - bb.warn(line.strip()) > +check_for_wic_warnings() { > + WARN="$(grep -e '^WARNING' ${T}/log.do_image_wic || true)" > + if [ -n "$WARN" ]; then > + bbwarn "$WARN" > + fi > } > > -do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" > -do_wic_image[dirs] = "${DEPLOY_DIR_IMAGE}" > -python do_wic_image() { > - cmds = ['wic_do_mounts', 'generate_wic_image', 'check_for_wic_warnings'] > - weights = [5, 90, 5] > - progress_reporter = bb.progress.MultiStageProgressReporter(d, weights) > - > - for cmd in cmds: > - progress_reporter.next_stage() > - bb.build.exec_func(cmd, d) > - progress_reporter.finish() > +do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}" > +IMAGE_CMD_wic() { > + wic_do_mounts > + generate_wic_image > + check_for_wic_warnings > } > -addtask wic_image before do_image after do_image_tools > > wic_do_mounts() { > buildchroot_do_mounts > @@ -209,7 +204,7 @@ generate_wic_image() { > sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} > find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" | while read f; do > suffix=$(basename $f | sed 's/\(.*\)\(\.direct\)\(.*\)/\3/') > - mv -f ${f} ${WIC_IMAGE_FILE}${suffix} > + mv -f ${f} "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}" > done > rm -rf ${BUILDCHROOT_DIR}/${WICTMP} > rm -rf ${IMAGE_ROOTFS}/../pseudo
Hi Anton, Anton Mikanovich, Donnerstag, 14. April 2022 19:47: > > > > # glibc bug 23960 https://sourceware.org/bugzilla/show_bug.cgi?id=23960 > > # should not use QEMU on armhf target with mkfs.ubifs < v2.1.3 > > -ISAR_CROSS_COMPILE_armhf = "1" > > +python() { > > + if 'ubifs' in (d.getVar('IMAGE_BASETYPES') or '').split(): > > + d.setVar('ISAR_CROSS_COMPILE_armhf', '1') > > +} > > > This will also not work and actually fails on imx6-sabrelite-bullseye > target. > Reasons: > 1) IMAGE_BASETYPES = "ubi ubi-ubifs" for that target and condition is False Yes, this is actually a mistake I made. When caclucating IMAGE_BASETYPES, we need to add IMAGE_TYPEDEPS_xxx recursively. Fixed in v3. > 2) ISAR_CROSS_COMPILE should be set before anonymous python code because > buildchroot selection is done also in python() {} Fixed in v3: THIS_ISAR_CROSS_COMPILE := "${ISAR_CROSS_COMPILE}" ISAR_CROSS_COMPILE_armhf = "${@bb.utils.contains('IMAGE_BASETYPES', 'ubifs', '1', '${THIS_ISAR_CROSS_COMPILE}', d)}" It looks a little ugly because I'm trying to preserve any previous value of ISAR_CROSS_COMPILE without the variable referencing itself... With these changes I can now build the imx6-sabrelite-bullseye target, and I confirmed that ISAR_CROSS_COMPILE is not set when using armhf with image types != ubifs. > Also it better to document (or at least comment) IMAGE_BASETYPES as a > new variable. For the user manual, I think adding this would be much more detail than we generally have throughout the document, but I added a comment in v3. Adriaan
diff --git a/meta/classes/container-img.bbclass b/meta/classes/container-img.bbclass index 322889a9..436a0051 100644 --- a/meta/classes/container-img.bbclass +++ b/meta/classes/container-img.bbclass @@ -3,15 +3,93 @@ # # SPDX-License-Identifier: MIT # -# This class provides the task 'container_image' +# This class provides the task 'containerize' # to create container images containing the target rootfs. -do_container_image[dirs] = "${DEPLOY_DIR_IMAGE}" -do_container_image[stamp-extra-info] = "${DISTRO}-${MACHINE}" -do_container_image[vardeps] += "CONTAINER_IMAGE_FORMATS" -do_container_image(){ - bbdebug 1 "Generate container image in these formats: ${CONTAINER_IMAGE_FORMATS}" - containerize_rootfs "${IMAGE_ROOTFS}" "${CONTAINER_IMAGE_FORMATS}" +CONTAINER_TYPES = "oci oci-archive docker-archive docker-daemon containers-storage" +USING_CONTAINER = "${@bb.utils.contains_any('IMAGE_BASETYPES', d.getVar('CONTAINER_TYPES').split(), '1', '0', d)}" + +CONTAINER_IMAGE_NAME ?= "${PN}-${DISTRO}-${DISTRO_ARCH}" +CONTAINER_IMAGE_TAG ?= "${PV}-${PR}" + +python() { + if not d.getVar('USING_CONTAINER') == '1': + return + for t in d.getVar('CONTAINER_TYPES').split(): + t_clean = t.replace('-', '_').replace('.', '_') + d.setVar('IMAGE_CMD_' + t_clean, 'convert_container %s "${CONTAINER_IMAGE_NAME}" "${IMAGE_FILE_HOST}"' % t) + d.setVar('IMAGE_FULLNAME_' + t_clean, '${PN}-${DISTRO}-${DISTRO_ARCH}') + bb.build.addtask('containerize', 'do_image_' + t_clean, 'do_image_tools', d) +} + +do_containerize() { + local cmd="/bin/dash" + local empty_tag="empty" + local tag="${CONTAINER_IMAGE_TAG}" + local oci_img_dir="${WORKDIR}/oci-image" + local rootfs="${IMAGE_ROOTFS}" + + # prepare OCI container image skeleton + bbdebug 1 "prepare OCI container image skeleton" + sudo rm -rf "${oci_img_dir}" "${oci_img_dir}_unpacked" + sudo umoci init --layout "${oci_img_dir}" + sudo umoci new --image "${oci_img_dir}:${empty_tag}" + sudo umoci config --image "${oci_img_dir}:${empty_tag}" \ + --config.cmd="${cmd}" + sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \ + "${oci_img_dir}_unpacked" + + # add root filesystem as the flesh of the skeleton + sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/" + # clean-up temporary files + sudo find "${oci_img_dir}_unpacked/rootfs/tmp" -mindepth 1 -delete + + # pack container image + bbdebug 1 "pack container image" + sudo umoci repack --image "${oci_img_dir}:${tag}" \ + "${oci_img_dir}_unpacked" + sudo umoci remove --image "${oci_img_dir}:${empty_tag}" + sudo rm -rf "${oci_img_dir}_unpacked" + + # no root needed anymore + sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}" } -addtask container_image before do_image after do_image_tools +convert_container() { + local tag="${CONTAINER_IMAGE_TAG}" + local oci_img_dir="${WORKDIR}/oci-image" + local container_type="$1" + local image_name="$2" + local image_archive="$3" + + # convert the OCI container image to the desired format + bbdebug 1 "Creating container image type: ${container_type}" + case "${container_type}" in + "docker-archive" | "oci-archive") + if [ "${container_type}" = "oci-archive" ] ; then + target="${container_type}:${image_archive}:${tag}" + else + target="${container_type}:${image_archive}:${image_name}:${tag}" + fi + rm -f "${image_archive}" + bbdebug 2 "Converting OCI image to ${container_type}" + skopeo --insecure-policy copy \ + "oci:${oci_img_dir}:${tag}" "${target}" + ;; + "oci") + tar --create --directory "${oci_img_dir}" \ + --file "${image_archive}" . + ;; + "docker-daemon" | "containers-storage") + if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then + die "Adding the container image to a container runtime (${container_type}) not supported if running from a container (e.g. 'kas-container')" + fi + skopeo --insecure-policy copy \ + "oci:${oci_img_dir}:${tag}" \ + "${container_type}:${image_name}:${tag}" + ;; + *) + die "Unsupported format for convert_container: ${container_type}" + ;; + esac +} diff --git a/meta/classes/cpiogz-img.bbclass b/meta/classes/cpiogz-img.bbclass index 2a49456b..f4c33bd9 100644 --- a/meta/classes/cpiogz-img.bbclass +++ b/meta/classes/cpiogz-img.bbclass @@ -3,20 +3,12 @@ # # SPDX-License-Identifier: MIT -CPIOGZ_FNAME ?= "${IMAGE_FULLNAME}.cpio.gz" -CPIOGZ_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${CPIOGZ_FNAME}" -IMAGER_INSTALL += "cpio" +IMAGER_INSTALL_cpio += "cpio" CPIO_IMAGE_FORMAT ?= "newc" -do_cpiogz_image() { - sudo rm -f ${CPIOGZ_IMAGE_FILE} - image_do_mounts - sudo chroot ${BUILDCHROOT_DIR} \ - sh -c "cd ${PP_ROOTFS}; /usr/bin/find . | \ - /usr/bin/cpio -H ${CPIO_IMAGE_FORMAT} -o | /usr/bin/gzip -9 > \ - ${PP_DEPLOY}/${CPIOGZ_FNAME}" - sudo chown $(id -u):$(id -g) ${CPIOGZ_IMAGE_FILE} +IMAGE_CMD_cpio() { + ${SUDO_CHROOT} \ + sh -c "cd ${PP_ROOTFS}; /usr/bin/find . | \ + /usr/bin/cpio -H ${CPIO_IMAGE_FORMAT} -o > \ + ${IMAGE_FILE_CHROOT}" } - -addtask cpiogz_image before do_image after do_image_tools -do_cpiogz_image[dirs] = "${DEPLOY_DIR_IMAGE}" diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass index 5085afcc..73d1bb57 100644 --- a/meta/classes/ext4-img.bbclass +++ b/meta/classes/ext4-img.bbclass @@ -1,24 +1,15 @@ # This software is a part of ISAR. # Copyright (C) 2015-2017 ilbers GmbH -EXT4_IMAGE_FILE = "${IMAGE_FULLNAME}.ext4.img" - -IMAGER_INSTALL += "e2fsprogs" +IMAGER_INSTALL_ext4 += "e2fsprogs" MKE2FS_ARGS ?= "-t ext4" # Generate ext4 filesystem image -do_ext4_image() { - rm -f '${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}' - - truncate -s ${ROOTFS_SIZE}K '${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}' +IMAGE_CMD_ext4() { + truncate -s ${ROOTFS_SIZE}K '${IMAGE_FILE_HOST}' - image_do_mounts - - sudo chroot ${BUILDCHROOT_DIR} /sbin/mke2fs ${MKE2FS_ARGS} \ - -F -d '${PP_ROOTFS}' '${PP_DEPLOY}/${EXT4_IMAGE_FILE}' + ${SUDO_CHROOT} /sbin/mke2fs ${MKE2FS_ARGS} \ + -F -d '${PP_ROOTFS}' '${IMAGE_FILE_CHROOT}' } - -addtask ext4_image before do_image after do_image_tools -do_ext4_image[prefuncs] = 'set_image_size' -do_ext4_image[dirs] = "${DEPLOY_DIR_IMAGE}" +#IMAGE_CMD_ext4[vardepsexclude] = "ROOTFS_SIZE ROOTFS_EXTRA" diff --git a/meta/classes/fit-img.bbclass b/meta/classes/fit-img.bbclass index 1ad0c5b8..ef65af88 100644 --- a/meta/classes/fit-img.bbclass +++ b/meta/classes/fit-img.bbclass @@ -6,24 +6,17 @@ MKIMAGE_ARGS ??= "" FIT_IMAGE_SOURCE ??= "fitimage.its" -FIT_IMAGE_FILE ?= "${IMAGE_FULLNAME}.fit.img" -IMAGER_INSTALL += "u-boot-tools device-tree-compiler" +IMAGER_INSTALL_fit += "u-boot-tools device-tree-compiler" # Generate fit image -do_fit_image() { +IMAGE_CMD_fit() { if [ ! -e "${WORKDIR}/${FIT_IMAGE_SOURCE}" ]; then die "FIT_IMAGE_SOURCE does not contain fitimage source file" fi - rm -f '${DEPLOY_DIR_IMAGE}/${FIT_IMAGE_FILE}' - - image_do_mounts - # Create fit image using buildchroot tools - sudo chroot ${BUILDCHROOT_DIR} /usr/bin/mkimage ${MKIMAGE_ARGS} \ - -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${PP_DEPLOY}/${FIT_IMAGE_FILE}' - sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${FIT_IMAGE_FILE}' + ${SUDO_CHROOT} /usr/bin/mkimage ${MKIMAGE_ARGS} \ + -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${IMAGE_FILE_CHROOT}' } -addtask fit_image before do_image after do_image_tools do_transform_template -do_fit_image[dirs] = "${DEPLOY_DIR_IMAGE}" +IMAGE_CMD_fit[depends] = "${PN}:do_transform_template" diff --git a/meta/classes/image-container-extension.bbclass b/meta/classes/image-container-extension.bbclass deleted file mode 100644 index cdec4633..00000000 --- a/meta/classes/image-container-extension.bbclass +++ /dev/null @@ -1,83 +0,0 @@ -# This software is a part of ISAR. -# Copyright (C) Siemens AG, 2021 -# -# SPDX-License-Identifier: MIT -# -# This class extends the image.bbclass for containerizing the root filesystem. - -CONTAINER_IMAGE_FORMATS ?= "docker-archive" -CONTAINER_IMAGE_NAME ?= "${PN}-${DISTRO}-${DISTRO_ARCH}" -CONTAINER_IMAGE_TAG ?= "${PV}-${PR}" - -containerize_rootfs() { - local cmd="/bin/dash" - local empty_tag="empty" - local tag="${CONTAINER_IMAGE_TAG}" - local oci_img_dir="${WORKDIR}/oci-image" - local rootfs="$1" - local container_formats="$2" - local container_name_prefix="$3" - - # prepare OCI container image skeleton - bbdebug 1 "prepare OCI container image skeleton" - sudo rm -rf "${oci_img_dir}" "${oci_img_dir}_unpacked" - sudo umoci init --layout "${oci_img_dir}" - sudo umoci new --image "${oci_img_dir}:${empty_tag}" - sudo umoci config --image "${oci_img_dir}:${empty_tag}" \ - --config.cmd="${cmd}" - sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \ - "${oci_img_dir}_unpacked" - - # add root filesystem as the flesh of the skeleton - sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/" - # clean-up temporary files - sudo find "${oci_img_dir}_unpacked/rootfs/tmp" -mindepth 1 -delete - - # pack container image - bbdebug 1 "pack container image" - sudo umoci repack --image "${oci_img_dir}:${tag}" \ - "${oci_img_dir}_unpacked" - sudo umoci remove --image "${oci_img_dir}:${empty_tag}" - sudo rm -rf "${oci_img_dir}_unpacked" - - # no root needed anymore - sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}" - - # convert the OCI container image to the desired format - image_name="${container_name_prefix}${CONTAINER_IMAGE_NAME}" - for image_type in ${CONTAINER_IMAGE_FORMATS} ; do - image_archive="${DEPLOY_DIR_IMAGE}/${image_name}-${tag}-${image_type}.tar" - bbdebug 1 "Creating container image type: ${image_type}" - case "${image_type}" in - "docker-archive" | "oci-archive") - if [ "${image_type}" = "oci-archive" ] ; then - target="${image_type}:${image_archive}:${tag}" - else - target="${image_type}:${image_archive}:${image_name}:${tag}" - fi - rm -f "${image_archive}" "${image_archive}.xz" - bbdebug 2 "Converting OCI image to ${image_type}" - skopeo --insecure-policy copy \ - "oci:${oci_img_dir}:${tag}" "${target}" - bbdebug 2 "Compressing image" - xz -T0 "${image_archive}" - ;; - "oci") - tar --create --xz --directory "${oci_img_dir}" \ - --file "${image_archive}.xz" . - ;; - "docker-daemon" | "containers-storage") - if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then - die "Adding the container image to a container runtime (${image_type}) not supported if running from a container (e.g. 'kas-container')" - fi - skopeo --insecure-policy copy \ - "oci:${oci_img_dir}:${tag}" \ - "${image_type}:${image_name}:${tag}" - ;; - *) - die "Unsupported format for containerize_rootfs: ${image_type}" - ;; - esac - done -} - diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index f87b76e7..4413a7d6 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -10,7 +10,8 @@ STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*" SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${MACHINE}-${DISTRO}-${DISTRO_ARCH}" IMAGE_INSTALL ?= "" -IMAGE_FSTYPES ?= "${@ d.getVar("IMAGE_TYPE", True) if d.getVar("IMAGE_TYPE", True) else "ext4-img"}" +IMAGE_FSTYPES ?= "${@ d.getVar("IMAGE_TYPE", True) if d.getVar("IMAGE_TYPE", True) else "ext4"}" +IMAGE_CONVERSIONS = "gz xz" IMAGE_ROOTFS ?= "${WORKDIR}/rootfs" KERNEL_IMAGE_PKG ??= "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}" @@ -83,7 +84,183 @@ inherit image-tools-extension inherit image-postproc-extension inherit image-locales-extension inherit image-account-extension -inherit image-container-extension + +def get_base_type(t, d): + bt = t + for c in d.getVar('IMAGE_CONVERSIONS').split(): + if t.endswith('.' + c): + bt = t[:-len('.' + c)] + break + return bt if bt == t else get_base_type(bt, d) + +# determine image basetypes, just so we can use it in imagetypes* classes +python() { + basetypes = set() + for t in (d.getVar('IMAGE_FSTYPES') or '').split(): + bt = get_base_type(t, d) + if bt.endswith('-img'): + # be backwards-compatible + bt = bt[:-len('-img')] + bb.warn("IMAGE_TYPE '{0}-img' is deprecated. Please use '{0}' instead.".format(bt)) + basetypes.add(bt) + deps = (d.getVar('IMAGE_TYPEDEP_' + bt.replace('-', '_').replace('.', '_')) or '').split() + basetypes |= set([get_base_type(t, d) for t in deps]) + d.setVar('IMAGE_BASETYPES', ' '.join(basetypes)) +} + +# image types +IMAGE_CLASSES ??= "" +IMGCLASSES = "container-img cpiogz-img ext4-img fit-img targz-img ubi-img ubifs-img vm-img wic-img" +IMGCLASSES += "${IMAGE_CLASSES}" +inherit ${IMGCLASSES} + +# image conversions +CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" +CONVERSION_DEPS_gz = "gzip" + +XZ_OPTIONS ?= "" +CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'cat ${IMAGE_FILE_CHROOT} | xz ${XZ_OPTIONS} > ${IMAGE_FILE_CHROOT}.xz'" +CONVERSION_DEPS_xz = "xz-utils" + +# hook up IMAGE_CMD_* +python() { + image_types = (d.getVar('IMAGE_FSTYPES') or '').split() + conversions = set(d.getVar('IMAGE_CONVERSIONS').split()) + + basetypes = {} + typedeps = {} + vardeps = set() + + def collect_image_type(t): + bt = get_base_type(t, d) + if bt.endswith('-img'): + # be backwards-compatible + bt = bt[:-len('-img')] + bb.warn("IMAGE_TYPE '{0}-img' is deprecated. Please use '{0}' instead.".format(bt)) + + if bt not in basetypes: + basetypes[bt] = [] + if t not in basetypes[bt]: + basetypes[bt].append(t) + t_clean = t.replace('-', '_').replace('.', '_') + deps = (d.getVar('IMAGE_TYPEDEP_' + t_clean) or '').split() + vardeps.add('IMAGE_TYPEDEP_' + t_clean) + if bt not in typedeps: + typedeps[bt] = set() + for dep in deps: + if dep not in image_types: + image_types.append(dep) + collect_image_type(dep) + typedeps[bt].add(get_base_type(dep, d)) + if bt != t: + collect_image_type(bt) + + for t in image_types[:]: + collect_image_type(t) + + # TODO: OE uses do_image, but Isar is different... + d.appendVarFlag('do_image_tools', 'vardeps', ' '.join(vardeps)) + + imager_install = set() + imager_build_deps = set() + conversion_install = set() + for bt in basetypes: + vardeps = set() + cmds = [] + bt_clean = bt.replace('-', '_').replace('.', '_') + + # prepare local environment + localdata = bb.data.createCopy(d) + localdata.setVar('OVERRIDES', bt_clean + ':' + d.getVar('OVERRIDES', False)) + localdata.setVar('PV', d.getVar('PV')) + localdata.delVar('DATETIME') + localdata.delVar('DATE') + localdata.delVar('TMPDIR') + vardepsexclude = (d.getVarFlag('IMAGE_CMD_' + bt_clean, 'vardepsexclude', True) or '').split() + for dep in vardepsexclude: + localdata.delVar(dep) + + # check if required args are set + required_args = (localdata.getVar('IMAGE_CMD_REQUIRED_ARGS') or '').split() + if any([d.getVar(arg) is None for arg in required_args]): + bb.fatal("IMAGE_TYPE '%s' requires these arguments: %s" % (image_type, ', '.join(required_args))) + + # convenience variables to be used by CMDs + localdata.setVar('IMAGE_FILE_HOST', '${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.${type}') + #bb.warn("FULLNAME is %s -> %s" % (localdata.getVar('IMAGE_FULLNAME', False), localdata.getVar('IMAGE_FULLNAME', True))) + localdata.setVar('IMAGE_FILE_CHROOT', '${PP_DEPLOY}/${IMAGE_FULLNAME}.${type}') + localdata.setVar('SUDO_CHROOT', localdata.expand('sudo chroot ${BUILDCHROOT_DIR}')) + + # imager install + for dep in (d.getVar('IMAGER_INSTALL_' + bt_clean) or '').split(): + imager_install.add(dep) + for dep in (d.getVar('IMAGER_BUILD_DEPS_' + bt_clean) or '').split(): + imager_build_deps.add(dep) + + # construct image command + cmds.append('\timage_do_mounts') + image_cmd = localdata.getVar('IMAGE_CMD_' + bt_clean) + if image_cmd: + localdata.setVar('type', bt) + cmds.append(localdata.expand(image_cmd)) + #bb.warn("IMAGE_CMD\n*** %s\n*** %s" % (image_cmd, localdata.expand(image_cmd))) + cmds.append(localdata.expand('\tsudo chown $(id -u):$(id -g) ${IMAGE_FILE_HOST}')) + else: + bb.fatal("No IMAGE_CMD for %s" % bt) + vardeps.add('IMAGE_CMD_' + bt_clean) + d.delVarFlag('IMAGE_CMD_' + bt_clean, 'func') + task_deps = d.getVarFlag('IMAGE_CMD_' + bt_clean, 'depends') + + # add conversions + conversion_depends = set() + rm_images = set() + def create_conversions(t): + for c in sorted(conversions): + if t.endswith('.' + c): + t = t[:-len(c) - 1] + create_conversions(t) + localdata.setVar('type', t) + cmd = '\t' + localdata.getVar('CONVERSION_CMD_' + c) + if cmd not in cmds: + cmds.append(cmd) + cmds.append(localdata.expand('\tsudo chown $(id -u):$(id -g) ${IMAGE_FILE_HOST}.%s' % c)) + vardeps.add('CONVERSION_CMD_' + c) + for dep in (localdata.getVar('CONVERSION_DEPS_' + c) or '').split(): + conversion_install.add(dep) + # remove temporary image files + if t not in image_types: + rm_images.add(localdata.expand('${IMAGE_FILE_HOST}')) + + for t in basetypes[bt]: + create_conversions(t) + + if bt not in image_types: + localdata.setVar('type', t) + rm_images.add(localdata.expand('${IMAGE_FILE_HOST}')) + + for image in rm_images: + cmds.append('\trm ' + image) + + # image type dependencies + after = 'do_image_tools' + for dep in typedeps[bt]: + after += ' do_image_%s' % dep.replace('-', '_').replace('.', '_') + + # create the task + task = 'do_image_%s' % bt_clean + d.setVar(task, '\n'.join(cmds)) + d.setVarFlag(task, 'func', '1') + d.appendVarFlag(task, 'prefuncs', ' set_image_size') + d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) + d.appendVarFlag(task, 'vardepsexclude', ' ' + ' '.join(vardepsexclude)) + d.appendVarFlag(task, 'dirs', localdata.expand(' ${DEPLOY_DIR_IMAGE}')) + if task_deps: + d.appendVarFlag(task, 'depends', task_deps) + bb.build.addtask(task, 'do_image', after, d) + + d.appendVar('IMAGER_INSTALL', ' ' + ' '.join(sorted(imager_install | conversion_install))) + d.appendVar('IMAGER_BUILD_DEPS', ' ' + ' '.join(sorted(imager_build_deps))) +} # Extra space for rootfs in MB ROOTFS_EXTRA ?= "64" @@ -256,6 +433,3 @@ do_rootfs_quality_check() { } addtask rootfs_quality_check after do_rootfs_finalize before do_rootfs - -# Last so that the image type can overwrite tasks if needed -inherit ${IMAGE_FSTYPES} diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index b021e728..7e9fb0f3 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -172,6 +172,7 @@ rootfs_install_pkgs_install() { do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}" +do_rootfs_install[vardepsexclude] += "IMAGE_ROOTFS" do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build" do_rootfs_install[recrdeptask] = "do_deploy_deb" python do_rootfs_install() { diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass index adf9a1fe..477dff70 100644 --- a/meta/classes/sdk.bbclass +++ b/meta/classes/sdk.bbclass @@ -31,7 +31,7 @@ SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${BPN}-${MACHINE}" # SDK settings SDK_INCLUDE_ISAR_APT ?= "0" -SDK_FORMATS ?= "targz-img" +SDK_FORMATS ?= "tar.xz" SDK_INSTALL ?= "" SDK_PREINSTALL += " \ debhelper \ diff --git a/meta/classes/targz-img.bbclass b/meta/classes/targz-img.bbclass index bf94af02..74d34e29 100644 --- a/meta/classes/targz-img.bbclass +++ b/meta/classes/targz-img.bbclass @@ -3,13 +3,6 @@ # # SPDX-License-Identifier: MIT -TARGZ_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz" - -do_targz_image() { - rm -f ${TARGZ_IMAGE_FILE} - sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} . - sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE} +IMAGE_CMD_tar() { + sudo tar -cvzf ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz --one-file-system -C ${IMAGE_ROOTFS} . } - -addtask targz_image before do_image after do_image_tools -do_targz_image[dirs] = "${DEPLOY_DIR_IMAGE}" diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass index efaf058e..92acb6f8 100644 --- a/meta/classes/ubi-img.bbclass +++ b/meta/classes/ubi-img.bbclass @@ -3,30 +3,18 @@ # # SPDX-License-Identifier: MIT -python() { - if not d.getVar("UBINIZE_ARGS"): - raise bb.parse.SkipRecipe("UBINIZE_ARGS must be set") -} - UBINIZE_CFG ??= "ubinize.cfg" -UBI_IMAGE_FILE ?= "${IMAGE_FULLNAME}.ubi.img" -IMAGER_INSTALL += "mtd-utils" +IMAGER_INSTALL_ubi += "mtd-utils" # Generate ubi filesystem image -do_ubi_image() { +IMAGE_CMD_ubi() { if [ ! -e "${WORKDIR}/${UBINIZE_CFG}" ]; then die "UBINIZE_CFG does not contain ubinize config file." fi - rm -f '${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}' - - image_do_mounts - - # Create ubi image using buildchroot tools - sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \ - -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' '${PP_WORK}/${UBINIZE_CFG}' - sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}' + ${SUDO_CHROOT} /usr/sbin/ubinize ${UBINIZE_ARGS} \ + -o '${IMAGE_FILE_CHROOT}' '${PP_WORK}/${UBINIZE_CFG}' } -addtask ubi_image before do_image after do_image_tools do_transform_template -do_ubi_image[dirs] = "${DEPLOY_DIR_IMAGE}" +IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template" +IMAGE_CMD_REQUIRED_ARGS_ubi = "UBINIZE_ARGS" diff --git a/meta/classes/ubifs-img.bbclass b/meta/classes/ubifs-img.bbclass index 229eb3ef..60c3bf14 100644 --- a/meta/classes/ubifs-img.bbclass +++ b/meta/classes/ubifs-img.bbclass @@ -3,30 +3,19 @@ # # SPDX-License-Identifier: MIT -python() { - if not d.getVar("MKUBIFS_ARGS"): - raise bb.parse.SkipRecipe("mkubifs_args must be set") -} - -UBIFS_IMAGE_FILE ?= "${IMAGE_FULLNAME}.ubifs.img" - -IMAGER_INSTALL += "mtd-utils" +IMAGER_INSTALL_ubifs += "mtd-utils" # glibc bug 23960 https://sourceware.org/bugzilla/show_bug.cgi?id=23960 # should not use QEMU on armhf target with mkfs.ubifs < v2.1.3 -ISAR_CROSS_COMPILE_armhf = "1" +python() { + if 'ubifs' in (d.getVar('IMAGE_BASETYPES') or '').split(): + d.setVar('ISAR_CROSS_COMPILE_armhf', '1') +} # Generate ubifs filesystem image -do_ubifs_image() { - rm -f '${DEPLOY_DIR_IMAGE}/${UBIFS_IMAGE_FILE}' - - image_do_mounts - +IMAGE_CMD_ubifs() { # Create ubifs image using buildchroot tools - sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} \ - -r '${PP_ROOTFS}' '${PP_DEPLOY}/${UBIFS_IMAGE_FILE}' - sudo chown $(id -u):$(id -g) '${DEPLOY_DIR_IMAGE}/${UBIFS_IMAGE_FILE}' + ${SUDO_CHROOT} /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} \ + -r '${PP_ROOTFS}' '${IMAGE_FILE_CHROOT}' } - -addtask ubifs_image before do_image after do_image_tools -do_ubifs_image[dirs] = "${DEPLOY_DIR_IMAGE}" +IMAGE_CMD_REQUIRED_ARGS_ubifs = "MKUBIFS_ARGS" diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass index 4bc977b9..8a676aca 100644 --- a/meta/classes/vm-img.bbclass +++ b/meta/classes/vm-img.bbclass @@ -5,16 +5,18 @@ # inherit buildchroot -inherit wic-img + +USING_OVA = "${@bb.utils.contains('IMAGE_BASETYPES', 'ova', '1', '0', d)}" FILESEXTRAPATHS_prepend := "${LAYERDIR_core}/classes/vm-img:" OVF_TEMPLATE_FILE ?= "vm-img-virtualbox.ovf.tmpl" -SRC_URI += "file://${OVF_TEMPLATE_FILE}" +SRC_URI += "${@'file://${OVF_TEMPLATE_FILE}' if d.getVar('USING_OVA') == '1' else ''}" -IMAGER_INSTALL += "qemu-utils gawk uuid-runtime" +IMAGE_TYPEDEP_ova = "wic" +IMAGER_INSTALL_ova += "qemu-utils gawk uuid-runtime" # virtual machine disk settings -SOURCE_IMAGE_FILE ?= "${IMAGE_FULLNAME}.wic.img" +SOURCE_IMAGE_FILE ?= "${IMAGE_FULLNAME}.wic" # For VirtualBox, this needs to be "monolithicSparse" (default to it). # VMware needs this to be "streamOptimized". @@ -34,7 +36,7 @@ def set_convert_options(d): CONVERSION_OPTIONS = "${@set_convert_options(d)}" -do_convert_wic() { +convert_wic() { rm -f '${DEPLOY_DIR_IMAGE}/${VIRTUAL_MACHINE_IMAGE_FILE}' image_do_mounts bbnote "Creating ${VIRTUAL_MACHINE_IMAGE_FILE} from ${SOURCE_IMAGE_FILE}" @@ -43,8 +45,6 @@ do_convert_wic() { '${PP_DEPLOY}/${SOURCE_IMAGE_FILE}' '${VIRTUAL_MACHINE_DISK}' } -addtask convert_wic before do_build after do_wic_image do_copy_boot_files do_install_imager_deps do_transform_template - # User settings for OVA OVA_NAME ?= "${IMAGE_FULLNAME}" OVA_MEMORY ?= "8192" @@ -67,10 +67,11 @@ OVA_VARS = "OVA_NAME OVA_MEMORY OVA_NUMBER_OF_CPU OVA_VRAM \ OVA_FIRMWARE OVA_ACPI OVA_3D_ACCEL \ OVA_SHA_ALG VIRTUAL_MACHINE_IMAGE_FILE" -TEMPLATE_FILES += "${OVF_TEMPLATE_FILE}" +TEMPLATE_FILES += "${@'${OVF_TEMPLATE_FILE}' if d.getVar('USING_OVA') == '1' else ''}" TEMPLATE_VARS += "${OVA_VARS}" -do_create_ova() { +do_image_ova[prefuncs] += "convert_wic" +IMAGE_CMD_ova() { if [ ! ${VIRTUAL_MACHINE_IMAGE_TYPE} = "vmdk" ]; then exit 0 fi @@ -81,10 +82,7 @@ do_create_ova() { export PRIMARY_MAC=$(macgen) export LAST_CHANGE=$(date -u "+%Y-%m-%dT%H:%M:%SZ") export OVA_FIRMWARE_UPPERCASE=$(echo ${OVA_FIRMWARE} | tr '[a-z]' '[A-Z]') - export OVF_TEMPLATE_STAGE2=$(echo ${OVF_TEMPLATE_FILE} | sed 's/.tmpl$//' ) - image_do_mounts - sudo -Es chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} <<'EOSUDO' set -e export DISK_SIZE_BYTES=$(qemu-img info -f vmdk "${VIRTUAL_MACHINE_DISK}" \ @@ -104,5 +102,3 @@ do_create_ova() { tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO } - -addtask do_create_ova after do_convert_wic before do_deploy diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index 7537a27b..cfcc94c7 100644 --- a/meta/classes/wic-img.bbclass +++ b/meta/classes/wic-img.bbclass @@ -4,7 +4,8 @@ # this class is heavily inspired by OEs ./meta/classes/image_types_wic.bbclass # -WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}')}" +USING_WIC = "${@bb.utils.contains('IMAGE_BASETYPES', 'wic', '1', '0', d)}" +WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if d.getVar('USING_WIC') == '1' else ''}" WKS_FILE ??= "sdimage-efi" @@ -14,6 +15,9 @@ do_copy_wks_template () { } python () { + if not d.getVar('USING_WIC') == '1': + return + import itertools import re @@ -74,13 +78,13 @@ python () { except (IOError, OSError) as exc: pass else: - bb.build.addtask('do_copy_wks_template', 'do_transform_template do_wic_image', None, d) - bb.build.addtask('do_transform_template', 'do_wic_image', None, d) + bb.build.addtask('do_copy_wks_template', 'do_transform_template do_image_wic', None, d) + bb.build.addtask('do_transform_template', 'do_image_wic', None, d) } inherit buildchroot -IMAGER_INSTALL += "${WIC_IMAGER_INSTALL}" +IMAGER_INSTALL_wic += "${WIC_IMAGER_INSTALL}" # wic comes with reasonable defaults, and the proper interface is the wks file ROOTFS_EXTRA ?= "0" @@ -125,32 +129,23 @@ python do_rootfs_wicenv () { } -addtask do_rootfs_wicenv after do_rootfs before do_wic_image +addtask do_rootfs_wicenv after do_rootfs before do_image_wic do_rootfs_wicenv[vardeps] += "${WICVARS}" do_rootfs_wicenv[prefuncs] = 'set_image_size' -WIC_IMAGE_FILE ="${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic.img" - -python check_for_wic_warnings() { - with open("{}/log.do_wic_image".format(d.getVar("T"))) as f: - for line in f.readlines(): - if line.startswith("WARNING"): - bb.warn(line.strip()) +check_for_wic_warnings() { + WARN="$(grep -e '^WARNING' ${T}/log.do_image_wic || true)" + if [ -n "$WARN" ]; then + bbwarn "$WARN" + fi } -do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" -do_wic_image[dirs] = "${DEPLOY_DIR_IMAGE}" -python do_wic_image() { - cmds = ['wic_do_mounts', 'generate_wic_image', 'check_for_wic_warnings'] - weights = [5, 90, 5] - progress_reporter = bb.progress.MultiStageProgressReporter(d, weights) - - for cmd in cmds: - progress_reporter.next_stage() - bb.build.exec_func(cmd, d) - progress_reporter.finish() +do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}" +IMAGE_CMD_wic() { + wic_do_mounts + generate_wic_image + check_for_wic_warnings } -addtask wic_image before do_image after do_image_tools wic_do_mounts() { buildchroot_do_mounts @@ -209,7 +204,7 @@ generate_wic_image() { sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" | while read f; do suffix=$(basename $f | sed 's/\(.*\)\(\.direct\)\(.*\)/\3/') - mv -f ${f} ${WIC_IMAGE_FILE}${suffix} + mv -f ${f} "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}" done rm -rf ${BUILDCHROOT_DIR}/${WICTMP} rm -rf ${IMAGE_ROOTFS}/../pseudo
This makes a number of changes: - Replace all do_*_image tasks with IMAGE_CMD_* definitions - Remove all implicit conversions (gz, xz) - Add IMAGE_CONVERSION_* commands - Create tasks (do_image_*) on demand in image.bbclass for all requested IMAGE_FSTYPES - When creating the tasks: - image_do_mounts is inserted automatically - a final chown is inserted automatically - variables IMAGE_FILE_HOST and IMAGE_FILE_CHROOT are set to reference the image - variable SUDO_CHROOT contains the chroot command - Create conversions on demand based on IMAGE_FSTYPES. A conversion is defined by CONVERSION_CMD_type, and its dependencies given as CONVERSION_DEPS_type - In conversion commands - the input file is named ${IMAGE_FULLNAME}.${type} - the conversions appends its own type, e.g. the output file would be ${IMAGE_FULLNAME}.${type}.xz - a final chown is appended automatically - Image types now longer have a -img suffix, e.g., ext4 instead of ext4-img, and conversions are appended like tar.gz instead of targz-img - Imager dependencies are set as IMAGER_INSTALL_type - Dependencies between image types are modelled like IMAGE_TYPEDEP_ova = "wic" - Required arguments/variables are modelled by IMAGE_CMD_REQUIRED_ARGUMENTS = "A B C" - Container types (previously CONTAINER_IMAGE_FORMATS) are now first class image types (oci, oci-archive, docker-archive, docker-daemon, containers.storage) - The logic of image-container-extension has moved to container-img - The VM image now has type ova (instead of vm-img) Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com> --- meta/classes/container-img.bbclass | 94 ++++++++- meta/classes/cpiogz-img.bbclass | 20 +- meta/classes/ext4-img.bbclass | 21 +- meta/classes/fit-img.bbclass | 17 +- .../classes/image-container-extension.bbclass | 83 -------- meta/classes/image.bbclass | 184 +++++++++++++++++- meta/classes/rootfs.bbclass | 1 + meta/classes/sdk.bbclass | 2 +- meta/classes/targz-img.bbclass | 11 +- meta/classes/ubi-img.bbclass | 24 +-- meta/classes/ubifs-img.bbclass | 29 +-- meta/classes/vm-img.bbclass | 24 +-- meta/classes/wic-img.bbclass | 45 ++--- 13 files changed, 331 insertions(+), 224 deletions(-) delete mode 100644 meta/classes/image-container-extension.bbclass