From patchwork Tue Apr 2 11:50:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 3461 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 02 Apr 2024 13:51:01 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-il1-f192.google.com (mail-il1-f192.google.com [209.85.166.192]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 432BoxxR018096 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 2 Apr 2024 13:51:00 +0200 Received: by mail-il1-f192.google.com with SMTP id e9e14a558f8ab-3689a9427e2sf18302775ab.0 for ; Tue, 02 Apr 2024 04:51:00 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712058654; cv=pass; d=google.com; s=arc-20160816; b=KQcrEwGCTBLmHkpqx7M6PgWj33VbFdCRp2+tq1l1TrY6j+399Q0J54MFfKVh9+7YRQ 1D+UyJ05tjOVWscl9DzvxvHQewWvKHw/3SE4gKISCZEirrEEpTnCIcVyIEbYU9309Q51 ztFH7M3Uc6ATp5e7cYlTIdyAAtn4i5dj+u4DnN1q0H153N54H+i0HaqaKZMzT3Nre6sf oBzZX+q7gy7693R2l88FNpPoMzyxmIAt2yIE2Dbza552RIizsUg1ecvPrAkAsGX2WajJ KMyfibrqaF+rQm92ZNovp4z9Qi9LMsAQkyaY/Zmtf2fl6xY4NYoHe4RlnbCL9/+qT2+J +m3A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HQU0dATTJzlCtO5U6kyiGTukhKED0IdL9opLFNB0ynw=; fh=v/NqrZ3YYRUmYSPJA9219YjIUfZysze82i1+Byng4O0=; b=r2bUTIjeo+O9GH59ENqzDr9yDnv3emtWZhD7gzSenldE7KVo2iB5pNZ/nHkTz/E93W MIEDuQkkiyCfljWqmYjfJbcktAjLGhJIvK4PWmcqP2kS/OY7cQ8ljn4S89HyHC/RK0PL 158EbhNc6mifSGXV0ySplj5bNwbjcmtbsN43Vg4viBTFSWd31ecvWB1NldT03PZwzV9D T9SPw0CJTZBBQPAH+vq6ULmlRGl8WHqaPa1cYQC/nM09W/khDTh1+lGtudsDD5N/vPYI EQJfRykRRHMYHXNF7ELtRzclTbQTQq5Cn0hiBFyEFQtOme7coXNcIR1Lh06vzeBaZ1LL JFZg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ke2Cvddm; spf=pass (google.com: domain of fm-1321639-20240402115051ea02c2efe24bf6a314-x9aw3_@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-1321639-20240402115051ea02c2efe24bf6a314-X9aW3_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1712058654; x=1712663454; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=HQU0dATTJzlCtO5U6kyiGTukhKED0IdL9opLFNB0ynw=; b=ou9akHIFpQmczw5WpL1OIhHxDsYD9ups8V4zzUYOWqsfYc+EsghrmLkpY1h0d9wlxN TCJy7RJi3Bm10kw9UP1XUcniCTjnssbGbI4JzoF96Vlq3g1Ii3kvN1YwuBzxHsroy18h K8e9uUQcv9KLPWPDFHU4qOzGI3tHU/tjsJQSEdJpUymnOUI46g1bHCwu8967fKiJLkD6 7u1xxYUVEvtyRICWBDZdoUSp+E/AbxTgd+NP7GxAhd5Ww9ZOsMZPxWXIwQQOYAqMkXop A125o+Yqs42EbNVrHxE8+0lqWNyiAZ6u1ziMTt/ravFlP6UyQEK8mA8+tu4lfF1ZmQvB Wtlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712058654; x=1712663454; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=HQU0dATTJzlCtO5U6kyiGTukhKED0IdL9opLFNB0ynw=; b=iiomgtCoLnMNslSOKhmbOmA4N/E6vP0G8iG5wLpdN/hc9CAo7qr+6TWVmHgYhKnh9a Df8LL2UWRutO+yKt7O7vpP0lFOdZDiJDg0gNZaNpJatkMjjOlwCWVK3lQpUszLpUQgQ9 ms0SCqMzYtZ4XEtmf1uUShssaC+etRDWahL6eBhL/t+wHCzYFT3TWn5gRnWFsI057YKP GJC+M5+T0F0cGFxYrbcfPgt2O23KQiFfjNtxopmF0bCiAfsNSnrgOpMwC3fyBYETu21K ep2wYs5dOmusGnSIUFs7rA5iuQQFZH0lYOcY0zOMFndwaRxSzyPpZDdc511o6RSPTF93 CTCQ== X-Forwarded-Encrypted: i=2; AJvYcCVoMe5s2V/DdObIp3EyARm0BsoNNt1a3gMHAWJVW/GbbHfpxDXRkinrc664S5+rsfC2KRZ0hYf769NF13vgdMgHKenPdBU= X-Gm-Message-State: AOJu0YyC5dRxqWyEfzyldrM69+BrQL+y/BHlfAw/MA70ImwzYSdNLhEX 2EEX2R777VRpIGwLIyLj6yezttF7XNOLfW5V0yx1ooJtLOf85B6S X-Google-Smtp-Source: AGHT+IFB1wcCCq5+j/92bw3NBVeNHsBT+0pO+CglaVNXL8ZP3e6INwjInBo0ICWWuLfbbQpTkQ24pw== X-Received: by 2002:a05:6e02:221a:b0:368:b52b:b449 with SMTP id j26-20020a056e02221a00b00368b52bb449mr15897336ilf.0.1712058654300; Tue, 02 Apr 2024 04:50:54 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6e02:13a1:b0:368:a946:6501 with SMTP id h1-20020a056e0213a100b00368a9466501ls3811321ilo.0.-pod-prod-06-us; Tue, 02 Apr 2024 04:50:53 -0700 (PDT) X-Received: by 2002:a05:6e02:13a1:b0:368:83e0:34c6 with SMTP id h1-20020a056e0213a100b0036883e034c6mr15242628ilo.26.1712058653190; Tue, 02 Apr 2024 04:50:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712058653; cv=none; d=google.com; s=arc-20160816; b=cuHI1MP4b1Uu+4hfp9TbALkQkFs/hNLvOm6rAp389e2hj6foLDPZ1AP5TYkEpVs6wn AVQzSzCkqKaDJ05Q1HpiZNJMlaxUE2s0VDXfe+uM5q/lPkUnuKehKw6Ld7cWLK8D6v/c p7b/TF6BU0sjZMmlrxAAUg0nOPGhUiG3CNBDQfsFwHLoTW6yB1O/h0MNw+ndQvjWRTqr cjuOF3ZCJ7DK6rAEbYorw6Jq1M7OwI99uobNfpQ6UeXSbrV/AHqmwqbgBWDy2z8OWgmO SOSUOGImiljD10kgfozsrCHQKqxNKNew977/gFUVS6HoTWmaZyEn8boxdL7Uru0GLSzJ qS1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=MsxO2s4n28cdnWR+N6JuqJYVTLBfhrQaNtfSCJQ+w7o=; fh=N+fZ9sr4kgf5JXCC9K+ELSxRDZ5FuP3OY2Uk+J+GHUk=; b=edBAvzeI1vM4ell81K0NAICDpr0cUReFbp8+1yHz491Ez8pOpU6hrKLmr4hMOpKvrz jYnvU7iFuUnlYxp/mhqDfXoRGrWLPd+fwhiyouiOcPT0t2xb8a4/GZH1sr+PKy7LlllU U8cJ/qkXRfQhx/MABH1z5trVgG3osmhlAh9b9Ve/z4vg0iNyWW5Zr0WyFsxH0pgrkYpC /89w3q5FUn1qlPrrITdK9ngc6chB1GlpyvmLBhiSwPB5s1XqO/bP+PZVCPyXgAbrprDe VHt7z60JdSKz/MbXwgUk5xvimPPl3d6sEec1acUfvae0J2i+SZA0bioTGxFLU7aqfyyQ tfsg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ke2Cvddm; spf=pass (google.com: domain of fm-1321639-20240402115051ea02c2efe24bf6a314-x9aw3_@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-1321639-20240402115051ea02c2efe24bf6a314-X9aW3_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-225.siemens.flowmailer.net (mta-65-225.siemens.flowmailer.net. [185.136.65.225]) by gmr-mx.google.com with ESMTPS id bp25-20020a056e02349900b00368b3f7f8b8si818706ilb.3.2024.04.02.04.50.52 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2024 04:50:52 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20240402115051ea02c2efe24bf6a314-x9aw3_@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) client-ip=185.136.65.225; Received: by mta-65-225.siemens.flowmailer.net with ESMTPSA id 20240402115051ea02c2efe24bf6a314 for ; Tue, 02 Apr 2024 13:50:51 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, Uladzimir Bely , Felix Moessbauer Subject: [RFC v2 2/2] delay creation of initrd until end of rootfs install Date: Tue, 2 Apr 2024 13:50:39 +0200 Message-Id: <20240402115039.854878-2-felix.moessbauer@siemens.com> In-Reply-To: <20240402115039.854878-1-felix.moessbauer@siemens.com> References: <20240402115039.854878-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ke2Cvddm; spf=pass (google.com: domain of fm-1321639-20240402115051ea02c2efe24bf6a314-x9aw3_@rts-flowmailer.siemens.com designates 185.136.65.225 as permitted sender) smtp.mailfrom=fm-1321639-20240402115051ea02c2efe24bf6a314-X9aW3_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch solves major performance issues around the initramfs creation by ensuring that the initrd is only created once. This is implemented by stubbing the update-initramfs call during the package installing. After all apt operations are completed, we manually trigger the initrd creation. In case a custom initramfs is used, the creation is completely skipped in the image rootfs, as this would anyways not be used. Before that, each package install that made a initrd relevant change triggered the update of the initrd. As we have multiple apt calls during the build, this step was sometimes executed multiple times. In addition, the apt install step is emulated, further slowing down the initrd generation. On some layers on non native architecutes, this summed up to over 10 minutes of initrd generation time. Signed-off-by: Felix Moessbauer --- meta/classes/image.bbclass | 4 +++ meta/classes/rootfs.bbclass | 32 +++++++++++++++++++ .../isar-bootstrap/isar-bootstrap.inc | 2 ++ 3 files changed, 38 insertions(+) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index b537dfa9..513d7389 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -67,6 +67,8 @@ inherit essential ROOTFSDIR = "${IMAGE_ROOTFS}" ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache" +# when using a custom initrd, do not generate one as part of the image rootfs +ROOTFS_FEATURES += "${@ '' if d.getVar('INITRD_IMAGE') == '' else 'no-generate-initrd'}" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}" ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" @@ -475,6 +477,8 @@ do_rootfs_quality_check() { args="${ROOTFS_QA_FIND_ARGS}" # rootfs_finalize chroot-setup.sh args="${args} ! -path ${ROOTFSDIR}/var/lib/dpkg/diversions" + # initramfs is generated outside of the image rootfs + args="${args} ! -path ${ROOTFSDIR}/boot/initrd.img*" for cmd in ${ROOTFS_POSTPROCESS_COMMAND}; do case "${cmd}" in image_postprocess_mark) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 498fbfd6..fb14f3ca 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}" # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR} # 'export-dpkg-status' - exports /var/lib/dpkg/status file to ${ROOTFS_DPKGSTATUS_DEPLOY_DIR} # 'clean-log-files' - delete log files that are not owned by packages +# 'no-generate-initrd' - do not generate debian default initrd ROOTFS_FEATURES ?= "" ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" @@ -117,6 +118,16 @@ rootfs_configure_apt() { EOSUDO } +ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation" +rootfs_disable_initrd_generation[weight] = "1" +rootfs_disable_initrd_generation() { + # fully disable initrd generation + echo "replace update-initramfs with stub" + sudo mv "${ROOTFSDIR}/usr/sbin/update-initramfs" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" + sudo chroot "${ROOTFSDIR}" ln -s "/usr/bin/true" "/usr/sbin/update-initramfs" +} + ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" rootfs_install_pkgs_update[weight] = "5" @@ -318,6 +329,27 @@ rootfs_cleanup_base_apt() { EOSUDO } +ROOTFS_POSTPROCESS_COMMAND += "rootfs_restore_initrd_tooling" +rootfs_restore_initrd_tooling[weight] = "1" +rootfs_restore_initrd_tooling() { + if [ -e "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" ]; then + sudo mv -f "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs" + fi +} + +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', '', 'rootfs_generate_initrd', d)}" +rootfs_generate_initrd[weight] = "10" +rootfs_generate_initrd() { + if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then + sudo -E chroot "${ROOTFSDIR}" sh -c '\ + export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ + update-initramfs -u -v -k "$kernel_version";' + else + echo "no kernel in this rootfs, do not generate initrd" + fi +} + do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" do_rootfs_postprocess[network] = "${TASK_USE_SUDO}" python do_rootfs_postprocess() { diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 17f19fd8..f548e202 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -24,6 +24,8 @@ DISTRO_BOOTSTRAP_KEYFILES = "" THIRD_PARTY_APT_KEYFILES = "" DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales" +# install early, so we can stub the update-initramfs script before rootfs install +DISTRO_BOOTSTRAP_BASE_PACKAGES:append = ",initramfs-tools" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}"