From patchwork Thu Apr 10 05:27:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4148 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 10 Apr 2025 07:28:04 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f58.google.com (mail-wm1-f58.google.com [209.85.128.58]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 53A5S3Kj009420 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Apr 2025 07:28:03 +0200 Received: by mail-wm1-f58.google.com with SMTP id 5b1f17b1804b1-43e9a3d2977sf3572615e9.1 for ; Wed, 09 Apr 2025 22:28:03 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1744262878; cv=pass; d=google.com; s=arc-20240605; b=kT2/5+SiWuEuIci6jIniJrZGhzpqQ9gIaHdWsyzLxrEG+YZUg/wQqA9+i/+l/FL7En lNselA6SbzMcpcXU9k3KT8gewVrVumMfNuqhN/KrbQfKuASbXU6q/YPdWY7anwmaXFTl oUuFowVI9MHEElTtPI17Mq4CI7kzhVoKZ7STqTQstBFQEW6KJgbD1YtJQruZzee60fJi ZUUlH5HCj5nofBkulL4VvmpFGgT3BBWi11+R+fqqfkRNrLfEOPSwagKxgyPRQAx3pnm/ IhrHfwkt1QNqanKrxVP6gG34Z6xNOQx8QBiTbsC0BxLPgZ/+hpj310XUyK9bdOpVOtZA kzxg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Y+QEZMOYJEkMkZ1bRgZaRA9wWSinrPGYNSBj6V0ysog=; fh=OiXIdi3RQM5W08Pxd3GqvfsQlv3asYtlsEAfuD1AILE=; b=I425wrK5WX2VAPQPkAcyW+5I0qYs2sVU7aDuT88pUb2O1N1GfYyy356IzASfJKcS1f r6Lff7Pbe6FkVSuWbrdrqK/+iODLA1Us0jtAPj4zSPvVbAhG58/fZsn7YjI+Fxvlq+zn X0OUZr0hUst1JtCZx+xj9Pui4vOBzDWJJwbI2lQwAUHXcK2o7pEV0Bcs1eQL0k+m2lUl SbDdtsk8blIFuM961SUIsE+Xbm7jvO2LPXcYPf9sbDxnJkKmycZE/hXjqICD6869DZPz eKD6yWCJ9EfZO5hWnI+bdMORfMEEIUbPfPStlzlY1ZPRz2vtyKRU9ffeW/h0Wwh9F2Ol /mzg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fAZdyhnl; spf=pass (google.com: domain of fm-1321639-20250410052754348f8329b973459864-i5rhmq@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-1321639-20250410052754348f8329b973459864-I5rhmq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1744262878; x=1744867678; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=Y+QEZMOYJEkMkZ1bRgZaRA9wWSinrPGYNSBj6V0ysog=; b=lXUf9noS7hBH+gpgBv+vWmJ41EJl8XWVTxOO1ypSRHZqessXV9YlZsQp0HxYwqONPe HbJwtni7T4UiRAumtetnstTXVV7nYTdD0KgLXq9lqYdoy5Bylq3QkO9nAlGy5vjFHUjd 95WnuCWLQhxI8l3SSXbTvSt9DODNKrSWx2Lg6F0weIUr/E72VSWCIRXdH3OD5vq8iwpd prMruIWn+CwRkON9/akwK07bUd7hXZzU1ofEwPWBEPTv27X+Sa3CTrfcFO9ocKuhbFnA TP3vtnJVKbgZgDooOMMDQfROBp2i3Q1ITy9EvxwbszowJY2ZNK2iTz0n52hDSFyRIU+f hoag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744262878; x=1744867678; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Y+QEZMOYJEkMkZ1bRgZaRA9wWSinrPGYNSBj6V0ysog=; b=shkIkqf7+Cl+08fODe5/0gHddDugTtt75VlBAamUS0jVct4+V8yF3ZYuYYjxSAK/Te caGTb2juK3eY+RvGAFzD0S5a0uihOM4mPzgXVHMLpychRPdtPPWyhqMHkp6XU3xcuKql tSp4HoCX6yczo6BfeCdphNYzDaVIGCiYnM8YFd2rHd8su/pESU3W3dhDaWHiG0I+SroN Cb40z5QufrXVyS8MHTIlu+99/UhtOmSotjx24gBMtVk9Jzs+stHS2jppOAiV95QvoCrD SacgB35dzfzlmXqYzsQT1197/bXy4NVAvdNwMXi3R6TK5JgUhhHt1tjJmjbopgVdQsGZ 50tg== X-Forwarded-Encrypted: i=2; AJvYcCXOyLd6lSKiZDCTw5HA4NKYCFtELMtTQek533mR5xKaIgei6Ywe6z0MLJL/aAaDa99nS0ev9hE=@isar-build.org X-Gm-Message-State: AOJu0YxkxW97MA0g5tqw6Ug8ySyqfJqGFa7RhgQi1XIwaCWLgHN8dP13 YtpDdfXirfzgrHewrEE1xPb6cYRvpmngBBIgA5vZI28fytogAZxJ X-Google-Smtp-Source: AGHT+IE085N5edLD+BrEIyGjhDhRJ2I4ov35t31rUTU7tKVl9/6Mh0iWEdjKIaZq9RsuhIyvfldKaA== X-Received: by 2002:a05:600c:348a:b0:43c:fdbe:4398 with SMTP id 5b1f17b1804b1-43f2fdceccfmr8243045e9.6.1744262877744; Wed, 09 Apr 2025 22:27:57 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAKkmGqc/43bj4iCUBNGek0ujvHNVHsqBowO0+vthk1vRQ== Received: by 2002:a05:600c:1e19:b0:43c:eb7b:1403 with SMTP id 5b1f17b1804b1-43f2c571b23ls3446125e9.1.-pod-prod-09-eu; Wed, 09 Apr 2025 22:27:55 -0700 (PDT) X-Received: by 2002:a05:6000:18a5:b0:39a:c9ed:8657 with SMTP id ffacd0b85a97d-39d8fd72838mr631544f8f.17.1744262874920; Wed, 09 Apr 2025 22:27:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1744262874; cv=none; d=google.com; s=arc-20240605; b=Gmaverof7gb4CdAMzukM4HMfnPmDhSF69tlL0PYVju9r4ICj8KKFI0IEOkBsRzaEVj SsHlMdf3oIi7wR5xfqDYaBjLYc/oP5yXhsgSPBuTpoglK2lhmhituuvxYAHbtqv22z1z mThERwTZu+L4+9budBoBQFMPJCh5BE52ZPbe5aNt5zqm5Rt6OZChlt0UCubEriMYzhNT U+SEwvguQDlntNuoKl0VNb4oSAZTrgLLG39+7gbqQp56V2ADUGlJNFxVsK5qda1HPkW0 /3lEEe6vMqj7nUje+Yhf71dBz3wgMwJYX5QGsjIOMJP7jX75ofMO34jEuCYOzAJu5S+O i/5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=LyBw2t3c8aMFo5JKLihLjBSJvkOxmyvoBLO+SEVIfvs=; fh=ihiGjvo7NUvpz5Iu2nNH4iKRnWbkT1eQ96FbYrmsmqY=; b=Gu7Q9AtaXIt0W5dwuQRnQ++Io6nPbxDPa4aIAlYRbXchJWLKrNicisTrELegvKDbw1 wKZxTLXG6ENVHMuVpDh6wShInjNRWBTuyQFu6it+tuBTY34i/CQBHhlbbUSzGbYG7ssb h1TvVencmUgeyTW6cgVXRVgjY9CrSInfMTexIsv06Up5Jc8ve+xWCMFqC0eEHMrP0/OG XbQWEeZKny/PCy7WklmePHIOweO7VF9mTVgM/gXrHzsQpG+hGxxgDTYKEhk3rPNsTh87 +8JVTuWsUfLGWDqWsdRMA2Z++4znJSE/rarxLFpSb+LMy8nHjkveoZD0L48ir5S43ru5 kAeQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fAZdyhnl; spf=pass (google.com: domain of fm-1321639-20250410052754348f8329b973459864-i5rhmq@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-1321639-20250410052754348f8329b973459864-I5rhmq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-225.siemens.flowmailer.net (mta-65-225.siemens.flowmailer.net. [185.136.65.225]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-43f231320e9si503725e9.0.2025.04.09.22.27.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Apr 2025 22:27:54 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20250410052754348f8329b973459864-i5rhmq@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) client-ip=185.136.65.225; Received: by mta-65-225.siemens.flowmailer.net with ESMTPSA id 20250410052754348f8329b973459864 for ; Thu, 10 Apr 2025 07:27:54 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, cedric.hombourger@siemens.com, Felix Moessbauer Subject: [PATCH 1/2] delay creation of initrd until end of rootfs install Date: Thu, 10 Apr 2025 07:27:50 +0200 Message-Id: <20250410052751.3835154-2-felix.moessbauer@siemens.com> In-Reply-To: <20250410052751.3835154-1-felix.moessbauer@siemens.com> References: <20250410052751.3835154-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fAZdyhnl; spf=pass (google.com: domain of fm-1321639-20250410052754348f8329b973459864-i5rhmq@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-1321639-20250410052754348f8329b973459864-I5rhmq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch solves major performance issues around the initramfs creation by ensuring that the initrd is only created once. This is implemented by stubbing the update-initramfs call during the package installing. After all apt operations are completed, we manually trigger the initrd creation. In case a custom initramfs is used, the creation is completely skipped in the image rootfs, as this would anyways not be used. We further unify the initrd generation in the rootfs and in the initramfs class. Before that, each package install that made a initrd relevant change triggered the update of the initrd. As we have multiple apt calls during the build, this step was sometimes executed multiple times. In addition, the apt install step is emulated, further slowing down the initrd generation. On a test build of the RPi4b target with a detached initramfs and a distro kernel, this patch set reduced the build time form ~50min to ~15min. Signed-off-by: Felix Moessbauer --- meta/classes/image.bbclass | 17 ++--- meta/classes/initramfs.bbclass | 39 +---------- meta/classes/rootfs.bbclass | 66 ++++++++++++++++++- .../sbuild-chroot/sbuild-chroot.inc | 1 + 4 files changed, 72 insertions(+), 51 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index ff3cd737..8523a662 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -67,6 +67,8 @@ inherit essential ROOTFSDIR = "${IMAGE_ROOTFS}" ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache" +# when using a custom initrd, do not generate one as part of the image rootfs +ROOTFS_FEATURES += "${@ '' if d.getVar('INITRD_IMAGE') == '' else 'no-generate-initrd'}" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}" ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" @@ -341,17 +343,6 @@ do_copy_boot_files() { sudo cat "$kernel" > "${DEPLOYDIR}/${KERNEL_IMAGE}" fi - if [ -z "${INITRD_IMAGE}" ]; then - # deploy default initrd if no custom one is build - initrd="$(realpath -q '${IMAGE_ROOTFS}/initrd.img')" - if [ ! -f "$initrd" ]; then - initrd="$(realpath -q '${IMAGE_ROOTFS}/boot/initrd.img')" - fi - if [ -f "$initrd" ]; then - cp -f "$initrd" '${DEPLOYDIR}/${INITRD_DEPLOY_FILE}' - fi - fi - for file in ${DTB_FILES}; do dtb="$(find '${IMAGE_ROOTFS}/usr/lib' -type f \ -iwholename '*linux-image-*/'${file} | head -1)" @@ -450,7 +441,7 @@ EOSUDO -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' } do_rootfs_finalize[network] = "${TASK_USE_SUDO}" -addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess +addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess do_generate_initramfs ROOTFS_QA_FIND_ARGS ?= "" @@ -461,6 +452,8 @@ do_rootfs_quality_check() { args="${ROOTFS_QA_FIND_ARGS}" # rootfs_finalize chroot-setup.sh args="${args} ! -path ${ROOTFSDIR}/var/lib/dpkg/diversions" + # initramfs is generated outside of the image rootfs + args="${args} ! -path ${ROOTFSDIR}/boot/initrd.img*" for cmd in ${ROOTFS_POSTPROCESS_COMMAND}; do case "${cmd}" in image_postprocess_mark) diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass index 0197a60b..3a996f78 100644 --- a/meta/classes/initramfs.bbclass +++ b/meta/classes/initramfs.bbclass @@ -6,14 +6,11 @@ DEPLOYDIR = "${WORKDIR}/deploy" STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*" -# Sstate also needs to be machine-specific -SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${MACHINE}-${DISTRO}-${DISTRO_ARCH}" -SSTATETASKS += "do_generate_initramfs" - INITRAMFS_INSTALL ?= "" INITRAMFS_PREINSTALL ?= "" INITRAMFS_ROOTFS ?= "${WORKDIR}/rootfs" INITRAMFS_IMAGE_NAME = "${INITRAMFS_FULLNAME}.initrd.img" +INITRD_DEPLOY_FILE = "${INITRAMFS_IMAGE_NAME}" # Install proper kernel INITRAMFS_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME")) if d.getVar("KERNEL_NAME") else ""}" @@ -28,37 +25,3 @@ ROOTFS_FEATURES = "" ROOTFS_PACKAGES = "initramfs-tools ${INITRAMFS_PREINSTALL} ${INITRAMFS_INSTALL}" inherit rootfs - -do_generate_initramfs[network] = "${TASK_USE_SUDO}" -do_generate_initramfs[cleandirs] += "${DEPLOYDIR}" -do_generate_initramfs[sstate-inputdirs] = "${DEPLOYDIR}" -do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" -do_generate_initramfs() { - rootfs_do_mounts - - trap 'exit 1' INT HUP QUIT TERM ALRM USR1 - trap 'rootfs_do_umounts' EXIT - - rootfs_do_qemu - - sudo -E chroot "${INITRAMFS_ROOTFS}" sh -c '\ - export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ - if [ -n "$kernel_version" ]; then \ - update-initramfs -u -v -k "$kernel_version"; \ - else \ - update-initramfs -u -v ; \ - fi' - - rootfs_do_umounts - - if [ ! -e "${INITRAMFS_ROOTFS}/initrd.img" ]; then - bberror "No initramfs was found after generation!" - fi - cp ${INITRAMFS_ROOTFS}/initrd.img ${DEPLOYDIR}/${INITRAMFS_IMAGE_NAME} -} -addtask generate_initramfs after do_rootfs before do_build - -python do_generate_initramfs_setscene () { - sstate_setscene(d) -} -addtask do_generate_initramfs_setscene diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 446d92d6..d49cd1ef 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}" # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR} # 'export-dpkg-status' - exports /var/lib/dpkg/status file to ${ROOTFS_DPKGSTATUS_DEPLOY_DIR} # 'clean-log-files' - delete log files that are not owned by packages +# 'no-generate-initrd' - do not generate debian default initrd ROOTFS_FEATURES ?= "" ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" @@ -22,6 +23,9 @@ ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf" ROOTFS_PACKAGE_SUFFIX ?= "${PN}-${DISTRO}-${DISTRO_ARCH}" +# path to deploy stubbed versions of initrd update scripts during do_rootfs_install +ROOTFS_STUBS_DIR = "/usr/local/isar-sbin" + # Useful environment variables: export E = "${@ isar_export_proxies(d)}" export DEBIAN_FRONTEND = "noninteractive" @@ -165,6 +169,13 @@ rootfs_configure_apt() { EOSUDO } +ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation" +rootfs_disable_initrd_generation[weight] = "1" +rootfs_disable_initrd_generation() { + # fully disable initrd generation + sudo mkdir -p "${ROOTFSDIR}${ROOTFS_STUBS_DIR}" + sudo cp -a ${ROOTFSDIR}/usr/bin/true ${ROOTFSDIR}${ROOTFS_STUBS_DIR}/update-initramfs +} ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" rootfs_install_pkgs_update[weight] = "5" @@ -227,7 +238,21 @@ rootfs_install_pkgs_install[progress] = "custom:rootfs_progress.PkgsInstallProgr rootfs_install_pkgs_install[network] = "${TASK_USE_SUDO}" rootfs_install_pkgs_install() { sudo -E chroot "${ROOTFSDIR}" \ - /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES} + /usr/bin/apt-get ${ROOTFS_APT_ARGS} \ + -o DPkg::Path='${ROOTFS_STUBS_DIR}:/usr/sbin:/usr/bin:/sbin:/bin' \ + ${ROOTFS_PACKAGES} +} + +ROOTFS_INSTALL_COMMAND += "rootfs_restore_initrd_tooling" +rootfs_restore_initrd_tooling[weight] = "1" +rootfs_restore_initrd_tooling() { + sudo rm -rf "${ROOTFSDIR}${ROOTFS_STUBS_DIR}" +} + +ROOTFS_INSTALL_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', 'rootfs_clear_initrd_symlinks', '', d)}" +rootfs_clear_initrd_symlinks() { + sudo rm -f ${ROOTFSDIR}/initrd.img + sudo rm -f ${ROOTFSDIR}/initrd.img.old } do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" @@ -421,6 +446,45 @@ python do_rootfs_postprocess() { } addtask rootfs_postprocess before do_rootfs after do_unpack +SSTATETASKS += "do_generate_initramfs" +do_generate_initramfs[network] = "${TASK_USE_SUDO}" +do_generate_initramfs[cleandirs] += "${DEPLOYDIR}" +do_generate_initramfs[sstate-inputdirs] = "${DEPLOYDIR}" +do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +python do_generate_initramfs() { + bb.build.exec_func('rootfs_do_mounts', d) + bb.build.exec_func('rootfs_do_qemu', d) + try: + bb.build.exec_func('rootfs_generate_initramfs', d) + finally: + bb.build.exec_func('rootfs_do_umounts', d) +} + +python do_generate_initramfs_setscene () { + sstate_setscene(d) +} + +rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHandler" +rootfs_generate_initramfs() { + if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then + sudo -E chroot "${ROOTFSDIR}" sh -c '\ + export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ + echo "Generating initrd"; \ + update-initramfs -u -v -k "$kernel_version";' + if [ -n "${INITRD_DEPLOY_FILE}" ]; then + cp ${ROOTFSDIR}/initrd.img ${DEPLOYDIR}/${INITRD_DEPLOY_FILE} + fi + else + echo "no kernel in this rootfs, do not generate initrd" + fi +} + +python() { + if 'no-generate-initrd' not in d.getVar('ROOTFS_FEATURES', True).split(): + bb.build.addtask('do_generate_initramfs', 'do_rootfs', 'do_rootfs_postprocess', d) + bb.build.addtask('do_generate_initramfs_setscene', None, None, d) +} + python do_rootfs() { """Virtual task""" pass diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc index 6478d3a6..98e427e5 100644 --- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc +++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc @@ -55,6 +55,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \ SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs" ROOTFSDIR = "${SBUILD_CHROOT_DIR}" ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}" +ROOTFS_FEATURES += "no-generate-initrd" ROOTFS_POSTPROCESS_COMMAND:remove = "rootfs_cleanup_isar_apt" From patchwork Thu Apr 10 05:27:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4149 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 10 Apr 2025 07:28:07 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 53A5S6TH009440 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Apr 2025 07:28:06 +0200 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-30c4168ffd8sf1510081fa.3 for ; Wed, 09 Apr 2025 22:28:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1744262880; cv=pass; d=google.com; s=arc-20240605; b=HuZgJ0sRqkkYzdT6Lkbgy1Y9IrVgGkvplLoWK5PGmvWdYVLm8n+eMxYHzvHbGus4Kn wzv3x6cjtLXMuCPHtAchpESCBJPaF/UB+b9o8KXTcr7dThgIQLwIq4wYOLaxzCYnjJO8 U+AqVe3FOrIpX2Qm73TjAT89aUcgUuX67IOGbFHtD/ulRQulN3Cs7b/mdJxU8plSkvxA 7QBEX5Zjq3P7L49Yr+zFnAB4kB84U/y05uQnePkAFphgHS/Wb5gLxKVNkDmPgEIlCqir rb9zelRne5gz5RPCDIlrxkzYLqmNzl/qYJ1FqWO6pmtwaf39CMZx56AOqf/KWm2O8kMX iIqg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Yjze8Ey4+iuacgtl1qMzGaY89CgGrwtbQ+cN5nC57KE=; fh=TQIADco/gAcnBKc5TXHZnBPLTLlHlOkBkS/jYZ/O990=; b=KA89NG5itl6sIHxaXcbBQm837zpfiQbgbsL2gEr1CFAdLj9FEWqcpPTNE1Uv+Gfy0b 5m/xiCd68iyH9MjmmZkd/fp9RlAmtPEZ5+juB1o68hZvy7DppXsHWrLCNZNTbCNEvvCq zlQf0Y43JI0Um1hK2Sgl/cHiHJvS0aGAXRsLSPpWZJBEBSuu+soR5Xz/F1dOPXy5vJ29 01K+6OPX5LWOY5UqUZhYoavVaJMMiZg8Bz+KMYhTb8LCzM1cN+Wb/kG5v0pQ2zOKb8vT HLRLpzX/i6dEphUFqcLkzTiHvf/i33vG0HaDV01epIwS4pP4ekNGhkPlXWokzgeBOxkD WUbQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="PvwXQJp/"; spf=pass (google.com: domain of fm-1321639-20250410052754604757b2cd5f10d5f3-oa_ab3@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1321639-20250410052754604757b2cd5f10d5f3-oA_Ab3@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1744262880; x=1744867680; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=Yjze8Ey4+iuacgtl1qMzGaY89CgGrwtbQ+cN5nC57KE=; b=oIjg/hoU93xGbfnduAc/Zs4P7iRLv5cVlLeUVeODITwZRIfYBhJgtxdMpClYEKGCRB U0wEQllFFZQCLOgDOtdnPV4OEwWbp7gGJVZJUW7ntbA4F+r1xOxqblk5cL3LTDqYXhsi syFPeX2asG7LISACgR7YgD3PYt0g0EZ+qZ9Cdi6ggvivKR5p0wHsuCwGbiwvPqrTrsHW Vi0JC1MbzL97IAKNlxDkx20y8s07cihlWe2LiwHrWL//OFIjpKMQU74TgJI6fxac7f8n P6STAUHLQO6EckeegV9oLS7t+J7KP3ykZ7wIqPU9GlyOtL8RYDa36glk5mDN0LlgOOoT O66A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744262880; x=1744867680; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Yjze8Ey4+iuacgtl1qMzGaY89CgGrwtbQ+cN5nC57KE=; b=ZHbj9GeqtIRw06Kq/ay/BxrY6uK8onomB1KG4CNLE1Aqdh26lC/o4BIFhTOWcoJSTl jWX9bpw0zrYrQTo6ppltWmHzz9FtBY7vogCJzYv5FyOnKL/1VCdJ/rW4DhrEOrvaj7jW 3jTary9cNh4qHwFRAx0jThn0j70JZHLxttgW2CNfieY/LXOtfehdhh4eP4wl9Net2L/k 97mjmg9T7NYbqnl+MHJ0wBofMtINkBxn05CsrVp73neKSrKLaeie5XWv9Vfa8lXwMLxN qF9dWF66ATY3PRFHgO5J1JA0rRiN07o+o1YSybG6Cas7RAJj7KK2pTSelZaKBmHlnBos Lhaw== X-Forwarded-Encrypted: i=2; AJvYcCWOWmgaATbN8JPwWRrn7AvBDsfVF93ue8fZYbzNKfqhWuQJc38NN6ap0BxKEtoxziERJO+c2A0=@isar-build.org X-Gm-Message-State: AOJu0YycORJHNlEfR/70J5BjMnSpUkiNYhAuQk7I2xxdxuURXmWedmc9 mNsm8mra/uBdcVQY3gE0EKUExFWrXrXrn7r8z2hzmpXwRIjwaRt5 X-Google-Smtp-Source: AGHT+IHEZl82F6/RTr322WlCl7B6RzPYHkt3Twxhs80NRnb38xXL4feDefQ3YU/Alw3fnN++Sm10Gg== X-Received: by 2002:a05:651c:3225:b0:30c:cb7:560b with SMTP id 38308e7fff4ca-3103eacd376mr2302681fa.12.1744262879101; Wed, 09 Apr 2025 22:27:59 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPALsmKNfi2A+Z8ikcWhc1qDSjvt1csVazeuAwl87MWERQA== Received: by 2002:a2e:ab0b:0:b0:30b:f139:c898 with SMTP id 38308e7fff4ca-30f4c933a96ls1496911fa.1.-pod-prod-09-eu; Wed, 09 Apr 2025 22:27:56 -0700 (PDT) X-Received: by 2002:a05:651c:88f:b0:30b:efa5:69a8 with SMTP id 38308e7fff4ca-3103ed5bf1fmr3431291fa.36.1744262875699; Wed, 09 Apr 2025 22:27:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1744262875; cv=none; d=google.com; s=arc-20240605; b=W1w5e7bK7s5B46F5Nqk9K9RR9CpKUFVehV/knv5SjH+7ZALRja8letwc5XRc4U8aCM /nM636/qQZZimp21qPQqG0F37W88NpNQr0t7nhPzunVtvqUCifEjC/CJHSXVYBZeapbw mxggKmDLdrrE0/y0KkosbtaFQfUVAcGCMG/Smm9yipk4P+dxaES/lFnvoOZ9gtoHnAwd FC7WGVti7MBtqssVqCsD2BORMg+kQNJsX1sVwI4o2JEIKfJcxvMJ/iPm9bKjxWlO9D3D DSC73Mun8ih4+VAloGCWS1dQyhQ9d7DutENYj5uhLrFN6DgJYd58lDRwZr3Dpvkiua1N YgQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=Y6NiNo5UkALfqncbE+8v4VXL6SbBRP9VTaPjaXZwJkE=; fh=ihiGjvo7NUvpz5Iu2nNH4iKRnWbkT1eQ96FbYrmsmqY=; b=OnHUf/cwJwcAkfdqIUCop0MxerS+IcmITl+d4UwtOS1CaCP8y+6K6cAlpFiyymm/zc ltIMe4c75I2tOhgVi+oqP1jSB5st7LmQs7vGoCL0UJ0BXugZ03Og9FhrM4jNIVgHvskh qlgh4vYUsFPKCL6rf/8283ADb64aI6O9i//ScP1u3tDHK5PeGVLUywpFVKdakHy+26wi RJmrHQOdbn2ZuM8vQmqs1wK3c0FeeFD0f9Nc6yG01CmnQWtc22dYEv3E0EAlQcueBtig qyCtu9z0qIVZJcyZ/CLD5wjO1CxRZpbtJ5qUoWHbkm0o1glwnCBBNo5b53Ns7BVd9Z2f H0Qw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="PvwXQJp/"; spf=pass (google.com: domain of fm-1321639-20250410052754604757b2cd5f10d5f3-oa_ab3@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1321639-20250410052754604757b2cd5f10d5f3-oA_Ab3@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net. [185.136.64.228]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-30f465d6583si471661fa.6.2025.04.09.22.27.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Apr 2025 22:27:55 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20250410052754604757b2cd5f10d5f3-oa_ab3@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) client-ip=185.136.64.228; Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20250410052754604757b2cd5f10d5f3 for ; Thu, 10 Apr 2025 07:27:54 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, cedric.hombourger@siemens.com, Felix Moessbauer Subject: [PATCH 2/2] report approximate progress during initrd generation Date: Thu, 10 Apr 2025 07:27:51 +0200 Message-Id: <20250410052751.3835154-3-felix.moessbauer@siemens.com> In-Reply-To: <20250410052751.3835154-1-felix.moessbauer@siemens.com> References: <20250410052751.3835154-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b="PvwXQJp/"; spf=pass (google.com: domain of fm-1321639-20250410052754604757b2cd5f10d5f3-oa_ab3@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1321639-20250410052754604757b2cd5f10d5f3-oA_Ab3@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= On non native architectures, the initrd generation has to be emulated. In combination with kernels that provide many modules (like the distro kernels), this leads to long initrd build times. To give a rough estimation of the duration, we add progress reporting to that step. As we always build the initrd with MODULES=most, we know from experiments, that ~50% of the kernel modules of the rootfs are included in the initrd. We use that number as a guesstimate, as we don't have precise numbers when starting the task. Signed-off-by: Felix Moessbauer --- meta/classes/rootfs.bbclass | 6 ++++++ meta/lib/rootfs_progress.py | 33 ++++++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index d49cd1ef..e0aa7377 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -454,6 +454,10 @@ do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" python do_generate_initramfs() { bb.build.exec_func('rootfs_do_mounts', d) bb.build.exec_func('rootfs_do_qemu', d) + + progress_reporter = bb.progress.ProgressHandler(d) + d.rootfs_progress = progress_reporter + try: bb.build.exec_func('rootfs_generate_initramfs', d) finally: @@ -468,7 +472,9 @@ rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHand rootfs_generate_initramfs() { if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then sudo -E chroot "${ROOTFSDIR}" sh -c '\ + mods_total="$(find /usr/lib/modules -type f -name '*.ko*' | wc -l)"; \ export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ + echo "Total number of modules: $mods_total"; \ echo "Generating initrd"; \ update-initramfs -u -v -k "$kernel_version";' if [ -n "${INITRD_DEPLOY_FILE}" ]; then diff --git a/meta/lib/rootfs_progress.py b/meta/lib/rootfs_progress.py index f808852f..1cc70c87 100644 --- a/meta/lib/rootfs_progress.py +++ b/meta/lib/rootfs_progress.py @@ -28,14 +28,16 @@ class PkgsProgressHandler(bb.progress.ProgressHandler): self._linebuffer = self._linebuffer[breakpos:] if self._stage == 'prepare': - match = re.search( - r'^([0-9]+) upgraded, ([0-9]+) newly installed', line) - if match: - self._num_pkgs = int(match.group(1)) + int(match.group(2)) - self._stage = 'post-prepare' + self.process_total(line) else: self.process_line(line) + def process_total(self, line): + m = re.search(r'^([0-9]+) upgraded, ([0-9]+) newly installed', line) + if m: + self._num_pkgs = int(m.group(1)) + int(m.group(2)) + self._stage = 'post-prepare' + def process_line(self, line): return @@ -65,3 +67,24 @@ class PkgsInstallProgressHandler(PkgsProgressHandler): progress = 99 * (self._pkg + self._pkg_set_up) / (self._num_pkgs * 2) self._progress.update(progress) + + +class InitrdProgressHandler(PkgsProgressHandler): + def __init__(self, d, outfile, otherargs=None): + super().__init__(d, outfile) + + def process_total(self, line): + m = re.search(r'^Total number of modules: ([0-9]+)', line) + if m: + # in MODULES=most mode, we install ~half of all modules + self._num_pkgs = int(m.group(1)) // 2 + self._stage = 'post-prepare' + + def process_line(self, line): + if line.startswith('Adding module'): + self._pkg += 1 + elif line.startswith('(excluding'): + self._pkg += len(line.split(' ')) - 1 + else: + return + self._progress.update(99 * self._pkg / self._num_pkgs)