From patchwork Wed Apr 23 12:43:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4194 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 23 Apr 2025 14:44:16 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f57.google.com (mail-wm1-f57.google.com [209.85.128.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 53NCiF5j011673 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 23 Apr 2025 14:44:15 +0200 Received: by mail-wm1-f57.google.com with SMTP id 5b1f17b1804b1-43cf5196c25sf35967195e9.0 for ; Wed, 23 Apr 2025 05:44:15 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1745412250; cv=pass; d=google.com; s=arc-20240605; b=YqFVDALsMH15OJu50PLxqS0/4KJ0fo7uLF1JH5ha9I2LrTIIs7kxYyfRvNwCMHoxub aSQU2C9YyU+ODN6scPOBzYpwSfag1Vt+PBPOa1upmeBEMbiLgAcIsviIp4mMcCH2Vnas wsGdleezJLarbzJuGrlA7e0V0j+tYTaEdniiQHiPHbyDUqBJe+YmNhxKE6verfHJs8vO G19r+kd4jwgeNT0Qjx8I5DeHt+Xk4QN4EqKCqAZuDjQ2CbMQWi3Z7lPP3ReBUcqc2UsG DmSI5RNBp7rMtli/yRaqvA6YZJD5f9pA7zzPXydip+00jl+enIvxVNXMYeeo9W242zmE 98/w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OIkpVeKBFlSqHRYrDTOtPZIdJSG4c5lNWmejhM5jTP0=; fh=3hhs+t0Y5hoB8S6gE4G4jO3wPUVm7tBKbgQkU8pMuA8=; b=T+WtERnfOLezGStQUjhNOWtLDJT+aojRzOJI4rdPiuuoAYeWUjIMH9z3l+BA8DQczC TcgVFwiccjjaI1LYLvqbaVde3pxYFFZ/Oy1zqtHP81Y/mZkCbVN84iqRomeyOa6/Ldg6 CQ3YET9cPpRF5Q/3SmmViVjlte7iRIfyuh4idHeUKrbcJqPqam1t0pqXu50XPfU4de9N mgZbCUG/g3KYSUruYG3p4c57u0TmAMlSwoy5D+LzTrXjPADhmEGzzBIvnBWnHFYW2Mk7 4m+lpj91yxjsx7NR9uyCgzZGr8Y3ukQKxKziwUNfhH0fs1i/liF5TpL4d6EKJo71Awf5 oPdA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fJTCBByj; spf=pass (google.com: domain of fm-1321639-20250423124405a5a6117b58d01431cb-gxrsp2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-1321639-20250423124405a5a6117b58d01431cb-gXrsp2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1745412250; x=1746017050; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=OIkpVeKBFlSqHRYrDTOtPZIdJSG4c5lNWmejhM5jTP0=; b=lhBLELKlTqw07gSrKC7Tk1sv7O7jMGQTCfBrr0VF9jpoXlDZHMS9+IBEpgD3D3ZfWU 0CZ8qgpXN6RmDn9sj+1h7NYLRaJK3v8LXQa8/bLswzSpC/jRr/TSDFZUaKq8V1medHrF KJOie74TOyoHvZlobUe5pwwietns8EGRw6qO7VVQHxh+s5vucRwK/Xzi33UtbW01RvFD JSGyGj9zP7NJlJ9Q2Kdz6XDCSYPC9fnuHT9rDkyiSMZAByV4vYjUz5wnKk+KEUu9y6aD tMy6MiAy25WN3qO0UgZ0jShHWHdKcqu9LNA1YkdV64W8NOF6zdtLAnasZaqlOrfW3UeZ jIeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745412250; x=1746017050; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OIkpVeKBFlSqHRYrDTOtPZIdJSG4c5lNWmejhM5jTP0=; b=KO+W9TM6tVEYkq72UDaoaaRUOwWU3whPvIrJsvXH25C7BJayESQbqFAyAsLDQT94og dlLNYyZRmsx6hpd84X1nNbhL1AXIPua39453ChxsjFpRCdR0Hg0osFQ7JV8KifxrNIVB U1Nq+s7mkNiY9WV1PQNvaDCG2eP6/xLHl3NrbOtG7/y2L99swg3Wdkdwup0CcrIHQ73l 12VTcWwijj9G4Cq07NGk3uTo0Qtt6rmuOUUpQCYhIBCfeoCSIMnI7eb4j2i/VPJungy/ /AAEDVSgkdKbhSS825rmIq91MZrRkqtvHsBHr/yJmR4KQ+FbOsEsf/7iZigztHRPcStY Er0Q== X-Forwarded-Encrypted: i=2; AJvYcCXxDcKyuMRZPZ76/bpJcaAxmZi1nTt3fkImv2ZP3Qfdjv9NVe/BdukVqIzzSASn7WQqG15cuPw=@isar-build.org X-Gm-Message-State: AOJu0YxlLWSNKsRulSes0GIGsvYT84YbkkOdlZZvactdy1/cnr42Jfpx hX5jNRlJkrg1WY9xxmczDG/3mvP8iOn2gvToyk0zmwsiOzrTXohX X-Google-Smtp-Source: AGHT+IHreMKgG7rgg7/nyDtwwUxls8Enpe24cyn/ceauEpl7bGX8aHEMwQVDEfS4TTjanPokzSmOdw== X-Received: by 2002:a05:600c:cc8:b0:43d:54a:221c with SMTP id 5b1f17b1804b1-44074f18848mr158367975e9.18.1745412249102; Wed, 23 Apr 2025 05:44:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAJGP95z74XbKsy3mLdQh1qX9KiYfEh1JWXauxxe3z1LRw== Received: by 2002:a05:600c:1d05:b0:43d:1776:2ebe with SMTP id 5b1f17b1804b1-4406234ba4als762935e9.2.-pod-prod-08-eu; Wed, 23 Apr 2025 05:44:06 -0700 (PDT) X-Received: by 2002:a05:600c:3d96:b0:43d:97ea:2f4 with SMTP id 5b1f17b1804b1-4406ab97d5emr177143825e9.12.1745412246274; Wed, 23 Apr 2025 05:44:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1745412246; cv=none; d=google.com; s=arc-20240605; b=iB4tpn9KXP7zIdwHyWBogFCNfJJi2ZNFlg2uz7GIB5dw4FviYCjifg5u5Y0e+zHRu5 1ZNoDChEmcHdF4/MLQTqFT9ogMdfJzDkBjyAl117pPziYMh5TfEvhhaIJBFnNZIAoh1X DuIrJnki2JmuLdQRmzULyN63Hq7J2Xd2YdwUx0lwe7BaMtiQ95viq+h+bsCscKQolr4G PUHXK8AIsEKuP1QHnosZr3b0vlJZ8HrMpl0o4TQeJ/v2qzX2Fmu1U3o9/Fn/mvFNvYB1 n6y9cZN7n92Z9D4EJFWHzHXVdDPCQJofjeLNuehkqRSRTjhk4ng8xh9QH0tPlpFP79E4 DkSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=RQW710NhC9dLQXZlv4hwrK8OMptc97otoUIf/Y1zoUk=; fh=ihiGjvo7NUvpz5Iu2nNH4iKRnWbkT1eQ96FbYrmsmqY=; b=QllBvnyhHjXGbHO3VK6g3/6aHtGZJvCGuBrBqoPU6ba0GLC1zj7ocXxnkmIVn/v6T0 /M/CFRPJwC8n9tO6q0UblBzdUrUGIywZjm+JwMikHI4dae8GMT40S0JUyM4gEBx1K9Tt 9KeD86GI1wCMh2QscB4IgxoVAe5TP78YS2wYbtrBa9j71+M4UMHpw3HeSQiAV9D3qg0B vYHoWjd0WtYtUp4EF5kcsCZ7qea2iisVP3DQHNTJk8CZXL5GWuWxEDDPq+MEvjpeUmY2 urQTOYeCcLIJw92uSYw9GLb3raNlS6GIYvarW4FngGNMvpdw0KJc/zIyCTuQEdlZH3s8 n3pw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fJTCBByj; spf=pass (google.com: domain of fm-1321639-20250423124405a5a6117b58d01431cb-gxrsp2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-1321639-20250423124405a5a6117b58d01431cb-gXrsp2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-4408d83a894si2742125e9.0.2025.04.23.05.44.06 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Apr 2025 05:44:06 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20250423124405a5a6117b58d01431cb-gxrsp2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 20250423124405a5a6117b58d01431cb for ; Wed, 23 Apr 2025 14:44:05 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, cedric.hombourger@siemens.com, Felix Moessbauer Subject: [PATCH v2 1/2] delay creation of initrd until end of rootfs install Date: Wed, 23 Apr 2025 14:43:58 +0200 Message-Id: <20250423124359.1081781-2-felix.moessbauer@siemens.com> In-Reply-To: <20250423124359.1081781-1-felix.moessbauer@siemens.com> References: <20250423124359.1081781-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=fJTCBByj; spf=pass (google.com: domain of fm-1321639-20250423124405a5a6117b58d01431cb-gxrsp2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-1321639-20250423124405a5a6117b58d01431cb-gXrsp2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch solves major performance issues around the initramfs creation by ensuring that the initrd is only created once. This is implemented by stubbing the update-initramfs call during the package installing. After all apt operations are completed, we manually trigger the initrd creation. In case a custom initramfs is used, the creation is completely skipped in the image rootfs, as this would anyways not be used. We further unify the initrd generation in the rootfs and in the initramfs class. Before that, each package install that made a initrd relevant change triggered the update of the initrd. As we have multiple apt calls during the build, this step was sometimes executed multiple times. In addition, the apt install step is emulated, further slowing down the initrd generation. On a test build of the RPi4b target with a detached initramfs and a distro kernel, this patch set reduced the build time form ~50min to ~15min. Signed-off-by: Felix Moessbauer --- meta/classes/image.bbclass | 15 +--- meta/classes/initramfs.bbclass | 39 +--------- meta/classes/rootfs.bbclass | 72 ++++++++++++++++++- .../sbuild-chroot/sbuild-chroot.inc | 1 + 4 files changed, 76 insertions(+), 51 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index ff3cd737..a766f7d5 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -67,6 +67,8 @@ inherit essential ROOTFSDIR = "${IMAGE_ROOTFS}" ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache" +# when using a custom initrd, do not generate one as part of the image rootfs +ROOTFS_FEATURES += "${@ '' if d.getVar('INITRD_IMAGE') == '' else 'no-generate-initrd'}" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}" ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" @@ -341,17 +343,6 @@ do_copy_boot_files() { sudo cat "$kernel" > "${DEPLOYDIR}/${KERNEL_IMAGE}" fi - if [ -z "${INITRD_IMAGE}" ]; then - # deploy default initrd if no custom one is build - initrd="$(realpath -q '${IMAGE_ROOTFS}/initrd.img')" - if [ ! -f "$initrd" ]; then - initrd="$(realpath -q '${IMAGE_ROOTFS}/boot/initrd.img')" - fi - if [ -f "$initrd" ]; then - cp -f "$initrd" '${DEPLOYDIR}/${INITRD_DEPLOY_FILE}' - fi - fi - for file in ${DTB_FILES}; do dtb="$(find '${IMAGE_ROOTFS}/usr/lib' -type f \ -iwholename '*linux-image-*/'${file} | head -1)" @@ -450,7 +441,7 @@ EOSUDO -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' } do_rootfs_finalize[network] = "${TASK_USE_SUDO}" -addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess +addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess do_generate_initramfs ROOTFS_QA_FIND_ARGS ?= "" diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass index 0197a60b..3a996f78 100644 --- a/meta/classes/initramfs.bbclass +++ b/meta/classes/initramfs.bbclass @@ -6,14 +6,11 @@ DEPLOYDIR = "${WORKDIR}/deploy" STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*" -# Sstate also needs to be machine-specific -SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${MACHINE}-${DISTRO}-${DISTRO_ARCH}" -SSTATETASKS += "do_generate_initramfs" - INITRAMFS_INSTALL ?= "" INITRAMFS_PREINSTALL ?= "" INITRAMFS_ROOTFS ?= "${WORKDIR}/rootfs" INITRAMFS_IMAGE_NAME = "${INITRAMFS_FULLNAME}.initrd.img" +INITRD_DEPLOY_FILE = "${INITRAMFS_IMAGE_NAME}" # Install proper kernel INITRAMFS_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME")) if d.getVar("KERNEL_NAME") else ""}" @@ -28,37 +25,3 @@ ROOTFS_FEATURES = "" ROOTFS_PACKAGES = "initramfs-tools ${INITRAMFS_PREINSTALL} ${INITRAMFS_INSTALL}" inherit rootfs - -do_generate_initramfs[network] = "${TASK_USE_SUDO}" -do_generate_initramfs[cleandirs] += "${DEPLOYDIR}" -do_generate_initramfs[sstate-inputdirs] = "${DEPLOYDIR}" -do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" -do_generate_initramfs() { - rootfs_do_mounts - - trap 'exit 1' INT HUP QUIT TERM ALRM USR1 - trap 'rootfs_do_umounts' EXIT - - rootfs_do_qemu - - sudo -E chroot "${INITRAMFS_ROOTFS}" sh -c '\ - export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ - if [ -n "$kernel_version" ]; then \ - update-initramfs -u -v -k "$kernel_version"; \ - else \ - update-initramfs -u -v ; \ - fi' - - rootfs_do_umounts - - if [ ! -e "${INITRAMFS_ROOTFS}/initrd.img" ]; then - bberror "No initramfs was found after generation!" - fi - cp ${INITRAMFS_ROOTFS}/initrd.img ${DEPLOYDIR}/${INITRAMFS_IMAGE_NAME} -} -addtask generate_initramfs after do_rootfs before do_build - -python do_generate_initramfs_setscene () { - sstate_setscene(d) -} -addtask do_generate_initramfs_setscene diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 446d92d6..fd6bb24a 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}" # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR} # 'export-dpkg-status' - exports /var/lib/dpkg/status file to ${ROOTFS_DPKGSTATUS_DEPLOY_DIR} # 'clean-log-files' - delete log files that are not owned by packages +# 'no-generate-initrd' - do not generate debian default initrd ROOTFS_FEATURES ?= "" ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" @@ -22,6 +23,9 @@ ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf" ROOTFS_PACKAGE_SUFFIX ?= "${PN}-${DISTRO}-${DISTRO_ARCH}" +# path to deploy stubbed versions of initrd update scripts during do_rootfs_install +ROOTFS_STUBS_DIR = "/usr/local/isar-sbin" + # Useful environment variables: export E = "${@ isar_export_proxies(d)}" export DEBIAN_FRONTEND = "noninteractive" @@ -165,6 +169,13 @@ rootfs_configure_apt() { EOSUDO } +ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation" +rootfs_disable_initrd_generation[weight] = "1" +rootfs_disable_initrd_generation() { + # fully disable initrd generation + sudo mkdir -p "${ROOTFSDIR}${ROOTFS_STUBS_DIR}" + sudo cp -a ${ROOTFSDIR}/usr/bin/true ${ROOTFSDIR}${ROOTFS_STUBS_DIR}/update-initramfs +} ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" rootfs_install_pkgs_update[weight] = "5" @@ -227,7 +238,21 @@ rootfs_install_pkgs_install[progress] = "custom:rootfs_progress.PkgsInstallProgr rootfs_install_pkgs_install[network] = "${TASK_USE_SUDO}" rootfs_install_pkgs_install() { sudo -E chroot "${ROOTFSDIR}" \ - /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES} + /usr/bin/apt-get ${ROOTFS_APT_ARGS} \ + -o DPkg::Path='${ROOTFS_STUBS_DIR}:/usr/sbin:/usr/bin:/sbin:/bin' \ + ${ROOTFS_PACKAGES} +} + +ROOTFS_INSTALL_COMMAND += "rootfs_restore_initrd_tooling" +rootfs_restore_initrd_tooling[weight] = "1" +rootfs_restore_initrd_tooling() { + sudo rm -rf "${ROOTFSDIR}${ROOTFS_STUBS_DIR}" +} + +ROOTFS_INSTALL_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', 'rootfs_clear_initrd_symlinks', '', d)}" +rootfs_clear_initrd_symlinks() { + sudo rm -f ${ROOTFSDIR}/initrd.img + sudo rm -f ${ROOTFSDIR}/initrd.img.old } do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" @@ -421,6 +446,51 @@ python do_rootfs_postprocess() { } addtask rootfs_postprocess before do_rootfs after do_unpack +SSTATETASKS += "do_generate_initramfs" +do_generate_initramfs[network] = "${TASK_USE_SUDO}" +do_generate_initramfs[cleandirs] += "${DEPLOYDIR}" +do_generate_initramfs[sstate-inputdirs] = "${DEPLOYDIR}" +do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +python do_generate_initramfs() { + bb.build.exec_func('rootfs_do_mounts', d) + bb.build.exec_func('rootfs_do_qemu', d) + try: + bb.build.exec_func('rootfs_generate_initramfs', d) + finally: + bb.build.exec_func('rootfs_do_umounts', d) +} + +python do_generate_initramfs_setscene () { + sstate_setscene(d) +} + +rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHandler" +rootfs_generate_initramfs() { + if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then + sudo -E chroot "${ROOTFSDIR}" sh -c '\ + export kernel_version=$(basename /boot/vmlinu[xz]-* | cut -d'-' -f2-); \ + echo "Generating initrd for kernel version: $kernel_version"; \ + update-initramfs -u -v -k "$kernel_version";' + if [ -n "${INITRD_DEPLOY_FILE}" ]; then + if [ -f "${ROOTFSDIR}/initrd.img" ]; then + # debian (mkinitramfs) + cp ${ROOTFSDIR}/initrd.img ${DEPLOYDIR}/${INITRD_DEPLOY_FILE} + else + # ubuntu (dracut) + cp ${ROOTFSDIR}/boot/initrd.img ${DEPLOYDIR}/${INITRD_DEPLOY_FILE} + fi + fi + else + echo "no kernel in this rootfs, do not generate initrd" + fi +} + +python() { + if 'no-generate-initrd' not in d.getVar('ROOTFS_FEATURES', True).split(): + bb.build.addtask('do_generate_initramfs', 'do_rootfs', 'do_rootfs_postprocess', d) + bb.build.addtask('do_generate_initramfs_setscene', None, None, d) +} + python do_rootfs() { """Virtual task""" pass diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc index 6478d3a6..98e427e5 100644 --- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc +++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc @@ -55,6 +55,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \ SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs" ROOTFSDIR = "${SBUILD_CHROOT_DIR}" ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}" +ROOTFS_FEATURES += "no-generate-initrd" ROOTFS_POSTPROCESS_COMMAND:remove = "rootfs_cleanup_isar_apt" From patchwork Wed Apr 23 12:43:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4195 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 23 Apr 2025 14:44:17 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f64.google.com (mail-ed1-f64.google.com [209.85.208.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 53NCiFwT011672 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 23 Apr 2025 14:44:15 +0200 Received: by mail-ed1-f64.google.com with SMTP id 4fb4d7f45d1cf-5f05d36993asf6561345a12.0 for ; Wed, 23 Apr 2025 05:44:15 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1745412250; cv=pass; d=google.com; s=arc-20240605; b=lUOhdvopigosDG4vjbQkFRSo2p22DWpKooHzLEHYtuUqf7AiMlMHHU2jS5tsH11m0J uDfmR/PMcNV7WdjEXlWHn/9vehumRHJwGvE9G/fLPmYY3SNiLZtnT0ONH2uJMj8yeCbg VwwOacZCcPhp/qOn+X03ro8xCdN+pEAAVQTgaCbbJYaqrW1vY9aMFJ7uh4WbvBe6U9nt s7bYxXP7loV3ZtL5PZZ+pJsJvDfyguO022ykLFsVP7RCEIU79bzZS1htp+t2lHJpkGU3 /IbzDx6rupDLOnowJOQl1NG9mEBAz+ugbUcKxJiXbx21nxbfd7dBNuFo0s2Xu+2bL98c qR2A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=t5edAy4yFNmFWmYFwaBG027bNLZyJJwgqu307084pZA=; fh=QJCGRiqUBMM3y8sb729fQRJwf02xsB9p2lNZWN4eJzI=; b=lCokagxUdM8KVW92k0MfBadj9g251+brSZlpm7Vdky/1TTmO8XIOPPerXlfTGhDBEc BexDzFOdPboDifVK2Lky6UW4M8IZF4Pue42JhCmxU4zImJxnoCOrrU61rq1pMfwgoAIV WB6bVaML1JpVB6H47C4WZKkqSWMhslXvBKODtoLiy+0Z9IkpU1WcSdhXupTUICO8ggSl mQyzLB27OSVOfQ6+dSkWdT0dd/d65Vta9luBOmTvIsIbsN8quS06JBpDGp1rOkHbASzt SvSc2R9MwvTxBJtCf+fTddBsX6Yo5rBm6x+WzuXEVZXGdc3eLm3SmWws7Ch0ANRcre+2 Mjkg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=eD0nlK9V; spf=pass (google.com: domain of fm-1321639-202504231244058ce314500f709e708a-gfsaf1@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-202504231244058ce314500f709e708a-gFSaF1@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1745412250; x=1746017050; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=t5edAy4yFNmFWmYFwaBG027bNLZyJJwgqu307084pZA=; b=sK5tn+oEXT5KzNG/S/JDf2QDZ0kH30w85Z3hq/TQAcBhB8YPwVhP0emIwFEssZO4pH sfcnXhNReTOsFxBQS9tPllAymA+gp6pNrG6KFpsEYWjW7+VMEmjfUKDGP3c1Bgppe0Kx 1wMb7gqhWok5F/R7U1Z8UzJiTBH4I9uLHLt+7UtGY/NhrAl7nDrwq6YJkOzqeTQZNxBe RRmqns/oNVPyHsmmbeJ16meoVp/mSABvc9RFxA199ONpSY3LNI5YUvr6U5ui5TwbO5Ue z1z5n9WPxl0PtukQhkc5BhuAkOlVL72KTNl6nCYl+8iY5uFwN6Pir4hnH+boE6ExGR4t MQxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745412250; x=1746017050; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=t5edAy4yFNmFWmYFwaBG027bNLZyJJwgqu307084pZA=; b=QEHzjMVmpQbSZXdOXwBNf/vkFduYxiBGFy+XVWqedDniVIXRXR1UDLavPNbP7Ar1um fPDDfpdA7pLgcMNH7AQyDsR2CpB1n09LuigM8bdflYqUr6FAnR88GMOQNz4NGGg/Ql+s dsMfecJzV4m1eQ5LqNCpnhsSlVIvxi7Zz1B/+Cc0oiu5B6LayKe+frlW4OF9eUPYo0gg TiDrBdanEXE2pWvi8ad2AyqMbsGW0rips/7YTlfGZPW7SfJB6XwOjKQ8emS3vDT67Ad1 DJTHfOAz10PaJhCIZu67GHdryq0j78OLkVMmwXzXj2jEgckLfH2XO22Qkxfn9jX4IkaS OTZQ== X-Forwarded-Encrypted: i=2; AJvYcCVlIb6SjtCRn4vI8IaAKIXB9cgUNFiIPFV4UDLW7t46EjF4HuHYRkgQNwAXSVPmQCsQHh8pRfY=@isar-build.org X-Gm-Message-State: AOJu0YzjNu1nZFBjCsEuaalISym06LI9Ki4raxMQCYoI+Q4Q6FlTR89I CTzNSKMYOtcjrB0wnN1p9ll0YV1m6ZGjLbTpgIPqps1nFoqQPOjF X-Google-Smtp-Source: AGHT+IHmeTwPZGFJ/XNzktWfeIRkacI2ximTZ5vrPj6c0RhnXxBmk9jG0eTVDFmjUrI1QTV/RGPz3g== X-Received: by 2002:a05:6402:40cf:b0:5f4:cad5:360f with SMTP id 4fb4d7f45d1cf-5f6285d430dmr17237115a12.28.1745412249182; Wed, 23 Apr 2025 05:44:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAJpA9k1rbqEsu4U4D0G70TdW9s6LQUg3giemLOna2LIrg== Received: by 2002:a05:6402:42c8:b0:5dc:cf28:2128 with SMTP id 4fb4d7f45d1cf-5f6b4f88bfals169167a12.0.-pod-prod-03-eu; Wed, 23 Apr 2025 05:44:06 -0700 (PDT) X-Received: by 2002:a05:6402:13ce:b0:5e6:de5:312f with SMTP id 4fb4d7f45d1cf-5f6285a89e4mr16260393a12.23.1745412246288; Wed, 23 Apr 2025 05:44:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1745412246; cv=none; d=google.com; s=arc-20240605; b=RzdkKRzckGuiKJOEQ2Zf490SpwD7ui3VkcB69bQp20tC4oJsvUg3aKwwRKowQCQViS xZP717rF1tg6+tRrYLKiN3Jo9JGeLO48s9orcqIOU+yr3rf7/nuVWK9BMmH3KSHqs08t YxbKwf3VAmCDxu2no1FFkKpfcCikuMW/r3WXd76VyW+NRH7JzfGLCCzuSwQzZFMQe8zH GSsZ3NDI7CDeKhql0rOoDlQPbZ5+tiEbC5/lSNUXl/3fdrCbT/zSf4EUdz3YHD7vfydx 0vQi6SUUioxvmadwPudUeLX5fEV+o+B9mufH1ai+z/jmXhtkzIIsNKOgIAJAov2BxYTH dOXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=vVVAObvm+npG1BgVLJffsHmP8s57zQVpO2J5vj9c9pw=; fh=ihiGjvo7NUvpz5Iu2nNH4iKRnWbkT1eQ96FbYrmsmqY=; b=KeH5ITjqIFgbDeo2i9qRNZ9VNURK4a3PScpnWL4eAdshAuGKsc61SDOrE9glGfJWSG PMen7bwFf7euAWJRoccZPzyfq+P6L+eQdLlOmFVHjFo8JwaQ7B8MYuVNgg7IDbeuGa+o Z/te29R90CowqBdGuchXgJbVuHgHN8+thHtFxKmPPh8UB1d2c/9cTc2R8S8CDuDHzKx1 GopVKR2MfoG2PZD6/sDBXbTTESPTCQuW6Z7qtYM+A2d3XdJuJ8HoMzIju+vWfkQKMMR5 qcoOKjU5TgcQzNrSnpYoCEtBiyY3x2QQCL9HLTQvCbv5dxaGUdKGnS6jvTev8A0zopFq 7/qQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=eD0nlK9V; spf=pass (google.com: domain of fm-1321639-202504231244058ce314500f709e708a-gfsaf1@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-202504231244058ce314500f709e708a-gFSaF1@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net. [185.136.64.227]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5f6258343edsi427225a12.4.2025.04.23.05.44.06 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Apr 2025 05:44:06 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-202504231244058ce314500f709e708a-gfsaf1@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) client-ip=185.136.64.227; Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 202504231244058ce314500f709e708a for ; Wed, 23 Apr 2025 14:44:06 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, cedric.hombourger@siemens.com, Felix Moessbauer Subject: [PATCH v2 2/2] report approximate progress during initrd generation Date: Wed, 23 Apr 2025 14:43:59 +0200 Message-Id: <20250423124359.1081781-3-felix.moessbauer@siemens.com> In-Reply-To: <20250423124359.1081781-1-felix.moessbauer@siemens.com> References: <20250423124359.1081781-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=eD0nlK9V; spf=pass (google.com: domain of fm-1321639-202504231244058ce314500f709e708a-gfsaf1@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-202504231244058ce314500f709e708a-gFSaF1@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= On non native architectures, the initrd generation has to be emulated. In combination with kernels that provide many modules (like the distro kernels), this leads to long initrd build times. To give a rough estimation of the duration, we add progress reporting to that step. As we always build the initrd with MODULES=most, we know from experiments, that ~50% of the kernel modules of the rootfs are included in the initrd. We use that number as a guesstimate, as we don't have precise numbers when starting the task. Signed-off-by: Felix Moessbauer --- meta/classes/rootfs.bbclass | 6 ++++++ meta/lib/rootfs_progress.py | 33 ++++++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index fd6bb24a..ab30ab91 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -454,6 +454,10 @@ do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" python do_generate_initramfs() { bb.build.exec_func('rootfs_do_mounts', d) bb.build.exec_func('rootfs_do_qemu', d) + + progress_reporter = bb.progress.ProgressHandler(d) + d.rootfs_progress = progress_reporter + try: bb.build.exec_func('rootfs_generate_initramfs', d) finally: @@ -468,7 +472,9 @@ rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHand rootfs_generate_initramfs() { if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then sudo -E chroot "${ROOTFSDIR}" sh -c '\ + mods_total="$(find /usr/lib/modules -type f -name '*.ko*' | wc -l)"; \ export kernel_version=$(basename /boot/vmlinu[xz]-* | cut -d'-' -f2-); \ + echo "Total number of modules: $mods_total"; \ echo "Generating initrd for kernel version: $kernel_version"; \ update-initramfs -u -v -k "$kernel_version";' if [ -n "${INITRD_DEPLOY_FILE}" ]; then diff --git a/meta/lib/rootfs_progress.py b/meta/lib/rootfs_progress.py index f808852f..1cc70c87 100644 --- a/meta/lib/rootfs_progress.py +++ b/meta/lib/rootfs_progress.py @@ -28,14 +28,16 @@ class PkgsProgressHandler(bb.progress.ProgressHandler): self._linebuffer = self._linebuffer[breakpos:] if self._stage == 'prepare': - match = re.search( - r'^([0-9]+) upgraded, ([0-9]+) newly installed', line) - if match: - self._num_pkgs = int(match.group(1)) + int(match.group(2)) - self._stage = 'post-prepare' + self.process_total(line) else: self.process_line(line) + def process_total(self, line): + m = re.search(r'^([0-9]+) upgraded, ([0-9]+) newly installed', line) + if m: + self._num_pkgs = int(m.group(1)) + int(m.group(2)) + self._stage = 'post-prepare' + def process_line(self, line): return @@ -65,3 +67,24 @@ class PkgsInstallProgressHandler(PkgsProgressHandler): progress = 99 * (self._pkg + self._pkg_set_up) / (self._num_pkgs * 2) self._progress.update(progress) + + +class InitrdProgressHandler(PkgsProgressHandler): + def __init__(self, d, outfile, otherargs=None): + super().__init__(d, outfile) + + def process_total(self, line): + m = re.search(r'^Total number of modules: ([0-9]+)', line) + if m: + # in MODULES=most mode, we install ~half of all modules + self._num_pkgs = int(m.group(1)) // 2 + self._stage = 'post-prepare' + + def process_line(self, line): + if line.startswith('Adding module'): + self._pkg += 1 + elif line.startswith('(excluding'): + self._pkg += len(line.split(' ')) - 1 + else: + return + self._progress.update(99 * self._pkg / self._num_pkgs)