From patchwork Wed Apr 9 11:28:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4146 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 09 Apr 2025 13:29:17 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f64.google.com (mail-wr1-f64.google.com [209.85.221.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 539BTG8T005641 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 9 Apr 2025 13:29:16 +0200 Received: by mail-wr1-f64.google.com with SMTP id ffacd0b85a97d-399744f742bsf1908457f8f.1 for ; Wed, 09 Apr 2025 04:29:16 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1744198150; cv=pass; d=google.com; s=arc-20240605; b=PZIvB6Cf5dL/vrUaO0f4HihwiMXV/Yzi3JISlKVlSwueyYTGNxjkAC1X+yHGjGnlTf bdD2fCxoErROg4SVOMaKuI8T65QBh6BfF0Pfij1mDE8hxVZ0gXX2E/q+3jE12ch4gc4u M4BmgUMTPrPtu1tbh0S0QwWpMKbQYD7suYq4jpK3szDNk/BHddv3eT6jh2YNBATx6wFB pwwdX2gs2xjODzJ/xr4QHHao+UZqUcuDpZ1g8rDnQPlFDTWMl/jPuJjz0BCFr3UpqGxf Qsju3r1NdP0nlk+pf6ZGuaLjh4Q+zHRy9C0nO/OTuNmPV+Ipui5EwVNS9KG8hhkdqpR+ +2Xg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vKwaZpRgbUf2TrjigVumW/oGCmhB/FqPrFWBQZmFZvM=; fh=y17g41OmpJbRkL8S/PREIfa+NDt0aNzui116N+C5yTE=; b=h8OSRjmOP6XXIcGljpSH+4hILUEITA2WMFsa+ZKVJ30QsKK5upIJaTa5VxkNJsB7pf p+5XQfpiKpGCAwIbn10tcJjjDVvwOINHzDuqWKdAMtQPIILp6Q34wgEWfNsfdtwoTro+ qnsbxq9rNSXys5w0fcnJC2fUWzWNauAGIK9TYRgrS3mN4KfMOLlr5eGWgjV++stok7XS yVDnIdmCTgX4wla+qE1Z7iBybJPywddOafMogeoTmwFkt17AbiLhLPYmwgL3gJVm6MhA fOhrBsXEoeViWbw4F1DbGNFpk7UCdq/nkcNIDghaC20u2IscVvZO8wLKxBAJ40iXqjIR X7+Q==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=Py4Trqdc; spf=pass (google.com: domain of fm-1321639-20250409112906d38995b5b14c759d49-yqa0hl@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250409112906d38995b5b14c759d49-YQA0Hl@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1744198150; x=1744802950; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=vKwaZpRgbUf2TrjigVumW/oGCmhB/FqPrFWBQZmFZvM=; b=kcHPm9xohV5Xhst8N0gvf3GLPlVgnnC1bqbIxPjV3Vsuqk+DYMSbtHzBOczvNcjo5g 9KKCab7aH4z+7yzdK6Vg3FbiYFoiNBVqzBIaOpyMA+XKMq8gZ/GN10EtmVPnWf4ze+yo JbAF1IrfTqoycObzdY18qelhJ/Ash2c4T+nRL9i7QI052Z0/jHyiUkHSUdMhDWtR7NJv 1IFhfHxzrSVUL4P+0uEhuigiYZB6eziSO8N04nPuJvdEkjZNORn1+PoVAI+7crHP5CXl 2R3IT49sz74Ar0rLkumZwZN/MtpNWTooP2ZNZgRRjcbgpFOO60/cCFF19ylbJXbfETn4 rnbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744198150; x=1744802950; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=vKwaZpRgbUf2TrjigVumW/oGCmhB/FqPrFWBQZmFZvM=; b=K9uKQVSN2iXNSkIVRR0PEEi/P4fy5RrwUGeGzicoBPtXSnIeBXt2hR3hADkdau+vs3 tY4vAriLnS+Ymz1surGEYahJr3jl6qZRIEk4R9OsxoDraWE7i5E56y4lsiMoQhXTqIlE Dbbj+P5XtRZ7vIJHY+vbxrox0IFxiWhMZa40HqUH8Dt1Zfhvlz1t9TO/LrPZxv1XA4yO sNrBATxZ+yzun79L54+50N2838WLeLk+LDwJ4dM208rqabTpPc8M3Mpz3C+i9nFg/G8q PwK2zef9A9jSzsjMCvg6/8wr9hAEm7M5hoASxj8x//KONqZKyO0TqqM+awXECzeSjTOj GC0Q== X-Forwarded-Encrypted: i=2; AJvYcCVwY9wcTEK4pqCgeMw3do39K93Al89G0jYjewzu+HwN/neeakhJItzmHHaIio8dbqKcTPlcsJQ=@isar-build.org X-Gm-Message-State: AOJu0YzTyxErki/7f9CDSOrYyUBoB/X3qBGRCyiWkqBxun8yYK1Opqnr AfcltCBt/bQeL2jE17KKtRZ5dELUNgBlQpJTf9OXOefH36pPt+jU X-Google-Smtp-Source: AGHT+IFUMRfgic8eBaA5f/ww5kNT55htLYW6KGusBmpU9ypvi5v4sgH1tM9wDDO9P5Hm5pgZMfWrQQ== X-Received: by 2002:a05:6000:1786:b0:39c:268b:83a7 with SMTP id ffacd0b85a97d-39d87aa8457mr2286536f8f.5.1744198149757; Wed, 09 Apr 2025 04:29:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAJolW0HiUt30KyI5adt47rspOCNakxcd4zCrLM4oOiH8g== Received: by 2002:a05:600c:b58:b0:43c:f009:43c2 with SMTP id 5b1f17b1804b1-43ebea4903fls29805385e9.0.-pod-prod-03-eu; Wed, 09 Apr 2025 04:29:07 -0700 (PDT) X-Received: by 2002:a05:600c:b90:b0:43c:e7ae:4bcf with SMTP id 5b1f17b1804b1-43f1eae345fmr28256365e9.0.1744198147023; Wed, 09 Apr 2025 04:29:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1744198147; cv=none; d=google.com; s=arc-20240605; b=RaV9iB3Qp0C6tAp7QUb3SsNjdqS3GTpqy8B6K/fDqnCheiiCdzOtypelTB8NkM+5CE b27BTDWgJlkKxntba4dNi0dY5hPfhCePxcTItBKW7DsXj4tyL9Nbnil2IHaZL536LmMO byJAXrvld/BTnT8fM8utqg04CEtDB58ERBtKdRx1Dgm+4D9sEi+AZICD15q4nG8/sm2E dWQ1pVAgeJmAQAaKas27GRLk8LHeOAmnaxFq6YueWRN4yys1mh40vSltHXY6f5Jx93ff XOsy/xo/TkHyDMFq0/5p9RhOWOFt6CTUbgP+5hG/5ITFGSXbEUYmVbgcGZXVFOLqcjBL uMew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=bxBtNTDXQmc7CseNuXS20be/uZ/OIfjLkO0U5FzyG9c=; fh=ihiGjvo7NUvpz5Iu2nNH4iKRnWbkT1eQ96FbYrmsmqY=; b=cQTcTM/otcKrTmGo973cZ3J+8g31fT5j3RHICbgT8xcvaM/XB0yz0seEa7u8d3Awky 9GcWS+MjCXRTMucCBYKapyuXWhc++hZGr+e0wKS4ogCQZ+PyIYaqGX/3z3aGEFlE3BgT Ty8AIWJfUHxYQlf4DNe3uFotWeN8tc57oKFX9CzeUIh/oW63VEA8XZ71aLh9xUHz8XbI 0W7W4TepD44q2MIv9flv/1h5mDTi4mAjKOk9NhiY2N9ssmDTgkD9Cfnk6S/kkonkf2Mi Di3CDjP9gYRxwmJyA3Nkj/Tak3FgUrMe3k3ZUgzFKUoaoPeUEVctrTNvPY+gPE7xxktT 4xqg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=Py4Trqdc; spf=pass (google.com: domain of fm-1321639-20250409112906d38995b5b14c759d49-yqa0hl@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250409112906d38995b5b14c759d49-YQA0Hl@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-43f20904af4si1010945e9.0.2025.04.09.04.29.06 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Apr 2025 04:29:06 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20250409112906d38995b5b14c759d49-yqa0hl@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20250409112906d38995b5b14c759d49 for ; Wed, 09 Apr 2025 13:29:06 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, cedric.hombourger@siemens.com, Felix Moessbauer Subject: [RFC v3 1/2] delay creation of initrd until end of rootfs install Date: Wed, 9 Apr 2025 13:28:46 +0200 Message-Id: <20250409112847.3395116-2-felix.moessbauer@siemens.com> In-Reply-To: <20250409112847.3395116-1-felix.moessbauer@siemens.com> References: <20250409112847.3395116-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=Py4Trqdc; spf=pass (google.com: domain of fm-1321639-20250409112906d38995b5b14c759d49-yqa0hl@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250409112906d38995b5b14c759d49-YQA0Hl@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch solves major performance issues around the initramfs creation by ensuring that the initrd is only created once. This is implemented by stubbing the update-initramfs call during the package installing. After all apt operations are completed, we manually trigger the initrd creation. In case a custom initramfs is used, the creation is completely skipped in the image rootfs, as this would anyways not be used. We further unify the initrd generation in the rootfs and in the initramfs class. Before that, each package install that made a initrd relevant change triggered the update of the initrd. As we have multiple apt calls during the build, this step was sometimes executed multiple times. In addition, the apt install step is emulated, further slowing down the initrd generation. On a test build of the RPi4b target with a detached initramfs and a distro kernel, this patch set reduced the build time form ~50min to ~15min. Signed-off-by: Felix Moessbauer --- meta/classes/image.bbclass | 17 ++--- meta/classes/initramfs.bbclass | 39 +---------- meta/classes/rootfs.bbclass | 64 +++++++++++++++++++ .../isar-bootstrap/isar-bootstrap.inc | 2 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 2 + .../sbuild-chroot/sbuild-chroot.inc | 1 + 6 files changed, 75 insertions(+), 50 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index ff3cd737..8523a662 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -67,6 +67,8 @@ inherit essential ROOTFSDIR = "${IMAGE_ROOTFS}" ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache" +# when using a custom initrd, do not generate one as part of the image rootfs +ROOTFS_FEATURES += "${@ '' if d.getVar('INITRD_IMAGE') == '' else 'no-generate-initrd'}" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}" ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" @@ -341,17 +343,6 @@ do_copy_boot_files() { sudo cat "$kernel" > "${DEPLOYDIR}/${KERNEL_IMAGE}" fi - if [ -z "${INITRD_IMAGE}" ]; then - # deploy default initrd if no custom one is build - initrd="$(realpath -q '${IMAGE_ROOTFS}/initrd.img')" - if [ ! -f "$initrd" ]; then - initrd="$(realpath -q '${IMAGE_ROOTFS}/boot/initrd.img')" - fi - if [ -f "$initrd" ]; then - cp -f "$initrd" '${DEPLOYDIR}/${INITRD_DEPLOY_FILE}' - fi - fi - for file in ${DTB_FILES}; do dtb="$(find '${IMAGE_ROOTFS}/usr/lib' -type f \ -iwholename '*linux-image-*/'${file} | head -1)" @@ -450,7 +441,7 @@ EOSUDO -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' } do_rootfs_finalize[network] = "${TASK_USE_SUDO}" -addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess +addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess do_generate_initramfs ROOTFS_QA_FIND_ARGS ?= "" @@ -461,6 +452,8 @@ do_rootfs_quality_check() { args="${ROOTFS_QA_FIND_ARGS}" # rootfs_finalize chroot-setup.sh args="${args} ! -path ${ROOTFSDIR}/var/lib/dpkg/diversions" + # initramfs is generated outside of the image rootfs + args="${args} ! -path ${ROOTFSDIR}/boot/initrd.img*" for cmd in ${ROOTFS_POSTPROCESS_COMMAND}; do case "${cmd}" in image_postprocess_mark) diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass index 0197a60b..3a996f78 100644 --- a/meta/classes/initramfs.bbclass +++ b/meta/classes/initramfs.bbclass @@ -6,14 +6,11 @@ DEPLOYDIR = "${WORKDIR}/deploy" STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*" -# Sstate also needs to be machine-specific -SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${MACHINE}-${DISTRO}-${DISTRO_ARCH}" -SSTATETASKS += "do_generate_initramfs" - INITRAMFS_INSTALL ?= "" INITRAMFS_PREINSTALL ?= "" INITRAMFS_ROOTFS ?= "${WORKDIR}/rootfs" INITRAMFS_IMAGE_NAME = "${INITRAMFS_FULLNAME}.initrd.img" +INITRD_DEPLOY_FILE = "${INITRAMFS_IMAGE_NAME}" # Install proper kernel INITRAMFS_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME")) if d.getVar("KERNEL_NAME") else ""}" @@ -28,37 +25,3 @@ ROOTFS_FEATURES = "" ROOTFS_PACKAGES = "initramfs-tools ${INITRAMFS_PREINSTALL} ${INITRAMFS_INSTALL}" inherit rootfs - -do_generate_initramfs[network] = "${TASK_USE_SUDO}" -do_generate_initramfs[cleandirs] += "${DEPLOYDIR}" -do_generate_initramfs[sstate-inputdirs] = "${DEPLOYDIR}" -do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" -do_generate_initramfs() { - rootfs_do_mounts - - trap 'exit 1' INT HUP QUIT TERM ALRM USR1 - trap 'rootfs_do_umounts' EXIT - - rootfs_do_qemu - - sudo -E chroot "${INITRAMFS_ROOTFS}" sh -c '\ - export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ - if [ -n "$kernel_version" ]; then \ - update-initramfs -u -v -k "$kernel_version"; \ - else \ - update-initramfs -u -v ; \ - fi' - - rootfs_do_umounts - - if [ ! -e "${INITRAMFS_ROOTFS}/initrd.img" ]; then - bberror "No initramfs was found after generation!" - fi - cp ${INITRAMFS_ROOTFS}/initrd.img ${DEPLOYDIR}/${INITRAMFS_IMAGE_NAME} -} -addtask generate_initramfs after do_rootfs before do_build - -python do_generate_initramfs_setscene () { - sstate_setscene(d) -} -addtask do_generate_initramfs_setscene diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 446d92d6..8ab44472 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}" # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR} # 'export-dpkg-status' - exports /var/lib/dpkg/status file to ${ROOTFS_DPKGSTATUS_DEPLOY_DIR} # 'clean-log-files' - delete log files that are not owned by packages +# 'no-generate-initrd' - do not generate debian default initrd ROOTFS_FEATURES ?= "" ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" @@ -165,6 +166,15 @@ rootfs_configure_apt() { EOSUDO } +ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation" +rootfs_disable_initrd_generation[weight] = "1" +rootfs_disable_initrd_generation() { + # fully disable initrd generation + echo "replace update-initramfs with stub" + sudo mv "${ROOTFSDIR}/usr/sbin/update-initramfs" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" + sudo chroot "${ROOTFSDIR}" ln -s "/usr/bin/true" "/usr/sbin/update-initramfs" +} ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" rootfs_install_pkgs_update[weight] = "5" @@ -395,6 +405,15 @@ rootfs_cleanup_base_apt() { EOSUDO } +ROOTFS_POSTPROCESS_COMMAND += "rootfs_restore_initrd_tooling" +rootfs_restore_initrd_tooling[weight] = "1" +rootfs_restore_initrd_tooling() { + if [ -e "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" ]; then + sudo mv -f "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs" + fi +} + do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" do_rootfs_postprocess[network] = "${TASK_USE_SUDO}" python do_rootfs_postprocess() { @@ -421,6 +440,51 @@ python do_rootfs_postprocess() { } addtask rootfs_postprocess before do_rootfs after do_unpack +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', 'rootfs_clear_initrd', '', d)}" +rootfs_clear_initrd() { + sudo rm -f ${ROOTFSDIR}/initrd.img + sudo rm -f ${ROOTFSDIR}/initrd.img.old +} + +SSTATETASKS += "do_generate_initramfs" +do_generate_initramfs[network] = "${TASK_USE_SUDO}" +do_generate_initramfs[cleandirs] += "${DEPLOYDIR}" +do_generate_initramfs[sstate-inputdirs] = "${DEPLOYDIR}" +do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +python do_generate_initramfs() { + bb.build.exec_func('rootfs_do_mounts', d) + bb.build.exec_func('rootfs_do_qemu', d) + try: + bb.build.exec_func('rootfs_generate_initramfs', d) + finally: + bb.build.exec_func('rootfs_do_umounts', d) +} + +python do_generate_initramfs_setscene () { + sstate_setscene(d) +} + +rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHandler" +rootfs_generate_initramfs() { + if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then + sudo -E chroot "${ROOTFSDIR}" sh -c '\ + export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ + echo "Generating initrd"; \ + update-initramfs -u -v -k "$kernel_version";' + if [ -n "${INITRD_DEPLOY_FILE}" ]; then + cp ${ROOTFSDIR}/initrd.img ${DEPLOYDIR}/${INITRD_DEPLOY_FILE} + fi + else + echo "no kernel in this rootfs, do not generate initrd" + fi +} + +python() { + if 'no-generate-initrd' not in d.getVar('ROOTFS_FEATURES', True).split(): + bb.build.addtask('do_generate_initramfs', 'do_rootfs', 'do_rootfs_postprocess', d) + bb.build.addtask('do_generate_initramfs_setscene', None, None, d) +} + python do_rootfs() { """Virtual task""" pass diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 7fd67f8b..e4a3813c 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -16,6 +16,8 @@ ROOTFSDIR = "${WORKDIR}/rootfs" DISTRO_BOOTSTRAP_BASE_PACKAGES += "locales" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = " gnupg" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = " ca-certificates" +# install early, so we can stub the update-initramfs script before rootfs install +DISTRO_BOOTSTRAP_BASE_PACKAGES += "initramfs-tools" inherit deb-dl-dir diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc index 2c07d098..ea3755f6 100644 --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc @@ -13,6 +13,8 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_core}/recipes-core/isar-bootstrap/files" DISTRO_BOOTSTRAP_BASE_PACKAGES += "locales apt" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = " ca-certificates" +# install early, so we can stub the update-initramfs script before rootfs install +DISTRO_BOOTSTRAP_BASE_PACKAGES += "initramfs-tools" BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir" diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc index 6478d3a6..98e427e5 100644 --- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc +++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc @@ -55,6 +55,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \ SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs" ROOTFSDIR = "${SBUILD_CHROOT_DIR}" ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}" +ROOTFS_FEATURES += "no-generate-initrd" ROOTFS_POSTPROCESS_COMMAND:remove = "rootfs_cleanup_isar_apt" From patchwork Wed Apr 9 11:28:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4147 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 09 Apr 2025 13:29:17 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f60.google.com (mail-wm1-f60.google.com [209.85.128.60]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 539BTG51005644 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 9 Apr 2025 13:29:16 +0200 Received: by mail-wm1-f60.google.com with SMTP id 5b1f17b1804b1-43d5ca7c86asf43278585e9.0 for ; Wed, 09 Apr 2025 04:29:16 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1744198150; cv=pass; d=google.com; s=arc-20240605; b=lVIG+lF8rXIVHY6LK8CEUI+kp04t2iGIWi+mkpYLQbxXmSoeaLF62hzDkPUglQKCk2 UCH7LLR5h1oCq/NDDGc1AwDb2UxxgN/06kaj1ej94tAhuO0vWmJFsI2j+tlvV9/H2ke7 Mp2Z3spoJNVUuBCZzDs+grpMqU+cOs4ozGkviP0gBWjdMDIUN/1qtfw32n0mDWNP9rWK 9HWm3nupnBfe9mjNEcrRTHcLDlcIfFD1Zmr8dtJJpKQCEE1e+G5BbxcD9w9Us90P4waI mOAUtXVUHZrwiYZwdmA1ms26ELvJWQvBPLScx2wiS58/sNMnhS0sRnjJB1b5jqRiOmY8 a4eQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7AZk09mok4zorJSt5eRhzaCMYCWDcEUc9hSXzaqMong=; fh=0P6cJcjTlswStPQlKQM52kexZntOeOmvlgi6ZFkgHQY=; b=FuPUsgSKfwgSKS1tx1lRYnEEtKsaSpZ0rglGa2BPS1GiUEpIz8KbnukBGTkmMLkihU zLFSs8OhrReCgDJUbddRQzqoo85MmGg5KrYbZGLd9vlm/Ie3URISX7oQLufsQ+LpdD47 XT1ckdaVocslXXKjI+K3qoZ0wywjIOvn8yv/853cimS2BI5mrUlCjSe1T7UJjCnfr5VB 5Dg1rppZp6UZRhL3OXOdC1ppoZUUfNTDuOBzZ7baJEwOJnEY1+UvRh67wTFmuCWnmDT8 Ty6QWV4+mqb4Fg57Ha7c8k4BgO4YMdWjzSURNIvspP25/wZ2MIayt+4CJpSZrNPdgU7C LNzA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=KuhENiOW; spf=pass (google.com: domain of fm-1321639-20250409112906598b5dd74b9f5b3224-xz8tmk@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1321639-20250409112906598b5dd74b9f5b3224-xz8tmK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1744198150; x=1744802950; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=7AZk09mok4zorJSt5eRhzaCMYCWDcEUc9hSXzaqMong=; b=W0u4hEkyF9dkgQ0GjJpEUFkieaPBw5xxgwE5GXryKsC4jrP7UPZvFWklHkYGhVzwie j/7AVIoNUjzMPaLa/1ahjNfSpFFepI6GdQrWIDM/AZMxmH1HhIm/4usv17d0s5ZsKqoQ JeTvp4d8+V1ZaX9bC1eAzv7lk6Ds+F6EOHb6jfYQL+kNMY7tW7V+Z+xJV/oMHDQ7V08u ZSAdF1X5B8QB/zvoBPAKEAUD+TgYj19YhldPil/svS2mPGDOHsZH2YMMQqzJC4pWo9Pl /gRfTILRJdqd/W/RL5iG7c0QV0c6HFNazh9jt9XBzKCS6KK0CyB2XGEdXz5A8972xrSb Qn/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744198150; x=1744802950; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=7AZk09mok4zorJSt5eRhzaCMYCWDcEUc9hSXzaqMong=; b=O2/LefEOm3gJuB9M8FmD1ZUf1zQ5na6XRy4cv+sinFjFRm6njbwL8ZxiXwMtXZgBsj PylpuMz6VHvanGSqQ5XCtlu4JFAwdr3JYl8v49rHveHyLeMpdQAY0Vb8I7hoYZU59qWC JAVfbTbXYu/JXa6so0L9xqXRlHULm83q0tG25P52lZsxqyOk6DtGRyxPZlHoVnPJvBYF R6z2S2O5jKOaOV/PcPL0GHDE8EIaTtSVMDD00TCzhHaa1CWVy2//gnYP7AkCBQ2muFDC aPVrneGLwS1sRJAATUjagz/2hsIKDs0WbQOmEs/Aep0/FhqT2bH6VyWNUd1mH2T3tYnZ cLHA== X-Forwarded-Encrypted: i=2; AJvYcCVyoAstCllHss5eIQ+LTi5NBoPR3Dbw34yo5/1JdAsM+ppQEhmVYlAaUKs2eGQp7pMfA7ckbAo=@isar-build.org X-Gm-Message-State: AOJu0YwnsxeiJ9IDdMq0+8oyMAbBonK7ZFweLzM/fkVjhPK8ZmwcpMxM GG1cLTYe12bwSVN0hqZJU/UqAX5tv73rRlU6xHCMT7pj6ZT1IPVY X-Google-Smtp-Source: AGHT+IEbPbYM/wzBG3tM5VHZPOPmXyKTeOKXH7P4gcFuSQnxcFtysl+9Sbgtx3BuEqHw9ISFsQx0Rw== X-Received: by 2002:a05:600c:1f89:b0:43b:cc3c:60bc with SMTP id 5b1f17b1804b1-43f1ecd6b3amr26578575e9.15.1744198149612; Wed, 09 Apr 2025 04:29:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAJYmpKByR4XWKGeoa8BpYNQHRFKKl7hCSzo65cW4waTLw== Received: by 2002:a05:600c:1911:b0:43d:40b0:33 with SMTP id 5b1f17b1804b1-43ebea4f8eals8802735e9.0.-pod-prod-01-eu; Wed, 09 Apr 2025 04:29:07 -0700 (PDT) X-Received: by 2002:a05:600c:5486:b0:43b:c0fa:f9dd with SMTP id 5b1f17b1804b1-43f1ed4b4a3mr18999235e9.25.1744198147005; Wed, 09 Apr 2025 04:29:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1744198146; cv=none; d=google.com; s=arc-20240605; b=cpV8mgwo99Voa+BG4FcaoJmNNZjhojbvcubxFaRPlT8K2JffjI+DESpR8tBm8Jfku6 fTJ+hhzbg4aPOmwNBqnxcFHcT6+MFftrhNQGA7Z/2oNP5MSg8YqOE4HAAoAk7JX7c3x/ pg3nhc7wwQdKiUDeQIW3/trERH2bAwhoXDIuyr4JMeQrG28evfgNhs7GlUFkCmwvJ4nU MTGr6hiCq6Y0aEhECVS1QNk+C7GKT8hdWrrazpaKCXeeWLYuiHoiEzZbwditV7fQhOR6 QY6P1NboLl3PlGT23q48F8SRCAp5GXMqC+hn3dWPQxa9VHjDkRiI31SbBzSmKDJQWJ9Q mYeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=2WjB82fWDFzXzerXyqysNj8YxnwoitJ9rx7WC+MuApM=; fh=ihiGjvo7NUvpz5Iu2nNH4iKRnWbkT1eQ96FbYrmsmqY=; b=ZSFh/pUk+5fch4ZWeuLQC+s7obrUJ0hmZ3SIZIYBORg0scWd8lbesqhKQJOWrJZCCm oVgbvPu5MByubCsa4ucsssfqoyywgpahyVijiSq/r8hXHEcqttMjQnTX1uZU+m3EEKwU /Bz0mT/siZFp3W/6EQOaCzAtTcaGatGx58UIUbk51/rvRuYgoQjW5ZIr9/J6w/B3s0Uo 6Z97sOZQF9nMNqsGFGCljBHe8hmT1f//qVBICRsZ+gpeK4ghf85rKCz2wjQBTEPFKaB1 I+85QU1067suYoCmbDNwi2Y6O0K8jeuI1YEnql9Ni5hecTcBnN9twp07qyQD/FqxEne9 eMvQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=KuhENiOW; spf=pass (google.com: domain of fm-1321639-20250409112906598b5dd74b9f5b3224-xz8tmk@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1321639-20250409112906598b5dd74b9f5b3224-xz8tmK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-43f233b7d71si209695e9.2.2025.04.09.04.29.06 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Apr 2025 04:29:06 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20250409112906598b5dd74b9f5b3224-xz8tmk@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 20250409112906598b5dd74b9f5b3224 for ; Wed, 09 Apr 2025 13:29:06 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, cedric.hombourger@siemens.com, Felix Moessbauer Subject: [RFC v3 2/2] report approximate progress during initrd generation Date: Wed, 9 Apr 2025 13:28:47 +0200 Message-Id: <20250409112847.3395116-3-felix.moessbauer@siemens.com> In-Reply-To: <20250409112847.3395116-1-felix.moessbauer@siemens.com> References: <20250409112847.3395116-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=KuhENiOW; spf=pass (google.com: domain of fm-1321639-20250409112906598b5dd74b9f5b3224-xz8tmk@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1321639-20250409112906598b5dd74b9f5b3224-xz8tmK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= On non native architectures, the initrd generation has to be emulated. In combination with kernels that provide many modules (like the distro kernels), this leads to long initrd build times. To give a rough estimation of the duration, we add progress reporting to that step. As we always build the initrd with MODULES=most, we know from experiments, that ~50% of the kernel modules of the rootfs are included in the initrd. We use that number as a guesstimate, as we don't have precise numbers when starting the task. Signed-off-by: Felix Moessbauer --- meta/classes/rootfs.bbclass | 6 ++++++ meta/lib/rootfs_progress.py | 33 ++++++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 8ab44472..adf1b2cf 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -454,6 +454,10 @@ do_generate_initramfs[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" python do_generate_initramfs() { bb.build.exec_func('rootfs_do_mounts', d) bb.build.exec_func('rootfs_do_qemu', d) + + progress_reporter = bb.progress.ProgressHandler(d) + d.rootfs_progress = progress_reporter + try: bb.build.exec_func('rootfs_generate_initramfs', d) finally: @@ -468,7 +472,9 @@ rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHand rootfs_generate_initramfs() { if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then sudo -E chroot "${ROOTFSDIR}" sh -c '\ + mods_total="$(find /usr/lib/modules -type f -name '*.ko*' | wc -l)"; \ export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ + echo "Total number of modules: $mods_total"; \ echo "Generating initrd"; \ update-initramfs -u -v -k "$kernel_version";' if [ -n "${INITRD_DEPLOY_FILE}" ]; then diff --git a/meta/lib/rootfs_progress.py b/meta/lib/rootfs_progress.py index f808852f..1cc70c87 100644 --- a/meta/lib/rootfs_progress.py +++ b/meta/lib/rootfs_progress.py @@ -28,14 +28,16 @@ class PkgsProgressHandler(bb.progress.ProgressHandler): self._linebuffer = self._linebuffer[breakpos:] if self._stage == 'prepare': - match = re.search( - r'^([0-9]+) upgraded, ([0-9]+) newly installed', line) - if match: - self._num_pkgs = int(match.group(1)) + int(match.group(2)) - self._stage = 'post-prepare' + self.process_total(line) else: self.process_line(line) + def process_total(self, line): + m = re.search(r'^([0-9]+) upgraded, ([0-9]+) newly installed', line) + if m: + self._num_pkgs = int(m.group(1)) + int(m.group(2)) + self._stage = 'post-prepare' + def process_line(self, line): return @@ -65,3 +67,24 @@ class PkgsInstallProgressHandler(PkgsProgressHandler): progress = 99 * (self._pkg + self._pkg_set_up) / (self._num_pkgs * 2) self._progress.update(progress) + + +class InitrdProgressHandler(PkgsProgressHandler): + def __init__(self, d, outfile, otherargs=None): + super().__init__(d, outfile) + + def process_total(self, line): + m = re.search(r'^Total number of modules: ([0-9]+)', line) + if m: + # in MODULES=most mode, we install ~half of all modules + self._num_pkgs = int(m.group(1)) // 2 + self._stage = 'post-prepare' + + def process_line(self, line): + if line.startswith('Adding module'): + self._pkg += 1 + elif line.startswith('(excluding'): + self._pkg += len(line.split(' ')) - 1 + else: + return + self._progress.update(99 * self._pkg / self._num_pkgs)