From patchwork Mon Dec 4 02:36:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 3220 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 04 Dec 2023 03:37:32 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-qv1-f57.google.com (mail-qv1-f57.google.com [209.85.219.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 3B42bUML013283 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 4 Dec 2023 03:37:31 +0100 Received: by mail-qv1-f57.google.com with SMTP id 6a1803df08f44-67ab20c5b98sf24136586d6.0 for ; Sun, 03 Dec 2023 18:37:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701657445; cv=pass; d=google.com; s=arc-20160816; b=00khKj6LrDkvL4FAxDk1yZxJdgg/7ZkUknjBsU4eyU/kfm2es4Fs6A6zNNopbMXAcw WlM4Rc76T8iqP90zXc+TjxlfR+h13H6WxZsGq6E15YwHL7Pt8YKnEr7Xt++Al2TeoKb3 xyP3GcENL0c6yrXhNPbsnxerAP1yJYW/t7xlO2BpxYosZ4ZZWWBEZs54kj17FDaGs4aM yHUNDUyxeBSvwmRr0ckYbt2pXqbZHQlx7ehL1fpcqqmtQqeQ6Neiqk7LQF3y/i0/d+dN RJ1GhIQTGy3Ek20IcFPrvSxtfMccEj3VrCq/gQfdJ+KdnB2omC9nkg7S+MydqKm6vQ7p 5ERw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=coIgeu8Ao7eqaO1+DLKHl4xwQL0rfuG0UE6AHYVVCZI=; fh=cYOfP7mHP3HOQz2FrofLMAijdwCZnujT5yZbx4t+io4=; b=nSMjosE65LrFX/NqP3/pqxWt7vV/41hsSeQ5gK85TGat/GQb+XSbVoKs3u+2ooHNSG n+erobzClaIuqijtaDCv3FtAT9jxCd0cZjMmrHWFQfkkdw9Ytrpna32a2ZkLdzB/f9KQ xs7pQrTW1tf9DXuWjHLLe6c76IZveqaBrtJvvAorWjZBvtHYzdJBvznhhOHxE+KMnUy6 vy1hKTk/7ggW5U0uj5eI0webY81/XUsFlo2TebzUQS21mFxtZ8GATyaMYC3InlhnQTf3 sH/2YQVI1VCumafiZnUhHPA6H3TVZgYCu0KnXCv219aY4sQoeaRRnxfMUUpNEhYItPlg SErw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="ie/dSDpo"; spf=pass (google.com: domain of fm-1321639-20231204023722d1a0a9545d815dd5a5-qfudxf@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-1321639-20231204023722d1a0a9545d815dd5a5-QfuDXf@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1701657445; x=1702262245; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=coIgeu8Ao7eqaO1+DLKHl4xwQL0rfuG0UE6AHYVVCZI=; b=X7wrIghpnYA5ut1fOvglD/eTUDhOIKZyZtx5ARUezJphxibGWTkAbnK2j+DVk7e/pT zH0JAgnw28/cK25PvX5N9p8KPAe4pdgkVtEAtZVEs6UCHI/hzzUUOadSqCTfS9hwf84e Ux3kYV5HnV8mJhlLrurr8wOAEUdKlc5bk+mZTyV1fOlZQ5MvbVpuR8w299PhJ8qNRzvo QHtMIGQ1iLe1uJP5/OoXNUI45O3Ygprnb1ve2Z36F3JkvJKJQi/snMup52XZJkoxFZuj o0YrtDc3iKZsfpmjZjMsAAS/rJ7um8F+D5sUnetPGt4SOkXtwpPBO9c3H5YypIde6XOb tqKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701657445; x=1702262245; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=coIgeu8Ao7eqaO1+DLKHl4xwQL0rfuG0UE6AHYVVCZI=; b=U0WvmdqVnmVwQjveYsemwFMnf/26CluGiV4HCrw/PoNn2kCCuhs3uUl+CBa1RhpiMd z/KOCayeION3XIg0xXYlL6CygIjmVY8ZV2hyAD7lbM0Us/nNbfVgExOHCx7Eb3++B1Ry 9RkqoXyF8CG4wOLbzGmfZd5r+OltiCcQ4ZQ+/z4aIAgOG9Qn80h6nocTDmeMG3Y0oSX6 4ZPYPdYcsdTTX66pu9X0i6beRCYZUcPILpWUC+lN4mVed2C5zFY6S6Rf44BPE4kf8dO1 HQkKr3whNwWsOG8rPP6qZpR8T0fLLv0TgKjU+6WNpy9V+HoRJfD7mT9DD0Z97nkJ8Lj4 2KHg== X-Gm-Message-State: AOJu0YzwNLLQBLKQx3G29jasQOhr1j9Hde+nqVB7Epi36kJggfUgRBlX XubAi/QH14yof6L1S3uFotQ= X-Google-Smtp-Source: AGHT+IEga5RyWAn70YfmUjHjZddAWMMumde78kMNF5W88MQ1VrDU0NSYsFJWUlEWwab8zgOGbFM+Gg== X-Received: by 2002:ad4:4e06:0:b0:67a:a947:72b8 with SMTP id dl6-20020ad44e06000000b0067aa94772b8mr3910957qvb.55.1701657445028; Sun, 03 Dec 2023 18:37:25 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ad4:4af2:0:b0:67a:bf7a:7ba6 with SMTP id cp18-20020ad44af2000000b0067abf7a7ba6ls891209qvb.1.-pod-prod-01-us; Sun, 03 Dec 2023 18:37:24 -0800 (PST) X-Received: by 2002:a0c:f78f:0:b0:67a:a10e:9187 with SMTP id s15-20020a0cf78f000000b0067aa10e9187mr4064891qvn.14.1701657444083; Sun, 03 Dec 2023 18:37:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701657444; cv=none; d=google.com; s=arc-20160816; b=JgCWKfIVtPrZxuD6oV5WgRcnCnVvFpQ5Er3L+4/V1xOA3ILQP7zj7+VuWrQxn5xaEP FIpBq0Glqmp0PgVA0W3vAw5KykA3a23v0S0nGKESXfde24Z1GRZMM1gOr1AzpKXgbD9O K2trnmlDxJoTO6jrm4I+FwR3PmIZyxFr+rzdOMFFA6/mV89T/3zFdd0ypjvFU3pE1lyO A6nXSWtiG8p0doLZY5t5bQskhbNA8nBFOkd2PIRLoO7Zx2YZL/SSbQ+ETfKVz/jtt5i5 33U47fS1Gvywivrx2b6GgF/fZ5tuCfSIrk2W9QOofaXxRoWI5GUirIQGvTTMOOu5vvws BOQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=jS675G72o/9Tlb/tn0H2nD0ZrR2jiMefQZ88vIkojjE=; fh=cYOfP7mHP3HOQz2FrofLMAijdwCZnujT5yZbx4t+io4=; b=DoMx6Dhm+gwO3Vy/eSjqgrbFi+8WflqkkdkCThimsBfDntrOfP86kq4V5f8hRLV13B zRbAvcDF+jBAVupm4PslaN9r6CGLUsNDaQJ67Pd4vHzbi2XfHdpTrVm8rVLYztlpoRg3 QdGHaxvAVVaoW4DmlnL46OsuUl43iPt2MJtrNX2Kb7+3m2iOWyBKpilMNnKliM2Dadru XuOp1JFKgICM3eYBRCno3WEUwjieUm4LemlgkmdeLxG67h7bIn6ZDvQ1BQQw0549vA7Y 2FXDybhSB7Vkk1q4ZHqAbJ9WpPz6hLBtOVAHFnurElMQKbDxxmF0/jX/qy1jfMvDJhzD HozQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="ie/dSDpo"; spf=pass (google.com: domain of fm-1321639-20231204023722d1a0a9545d815dd5a5-qfudxf@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-1321639-20231204023722d1a0a9545d815dd5a5-QfuDXf@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net. [185.136.65.228]) by gmr-mx.google.com with ESMTPS id dm6-20020ad44e26000000b0067ab24a47a9si358243qvb.2.2023.12.03.18.37.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 03 Dec 2023 18:37:24 -0800 (PST) Received-SPF: pass (google.com: domain of fm-1321639-20231204023722d1a0a9545d815dd5a5-qfudxf@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) client-ip=185.136.65.228; Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 20231204023722d1a0a9545d815dd5a5 for ; Mon, 04 Dec 2023 03:37:22 +0100 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, baocheng.su@siemens.com, Felix Moessbauer Subject: [RFC 1/1] delay creation of initrd until end of rootfs install Date: Mon, 4 Dec 2023 10:36:50 +0800 Message-Id: <20231204023650.1912972-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="ie/dSDpo"; spf=pass (google.com: domain of fm-1321639-20231204023722d1a0a9545d815dd5a5-qfudxf@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-1321639-20231204023722d1a0a9545d815dd5a5-QfuDXf@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch solves major performance issues around the initramfs creation by ensuring that the initrd is only created once. This is implemented by stubbing the update-initramfs call during the package installing. After all apt operations are completed, we manually trigger the initrd creation. In case a custom initramfs is used, the creation is completely skipped in the image rootfs, as this would anyways not be used. Before that, each package install that made a initrd relevant change triggered the update of the initrd. As we have multiple apt calls during the build, this step was sometimes executed multiple times. In addition, the apt install step is emulated, further slowing down the initrd generation. On some layers on non native architecutes, this summed up to over 10 minutes of initrd generation time. Signed-off-by: Felix Moessbauer --- meta/classes/image.bbclass | 4 +++ meta/classes/rootfs.bbclass | 32 +++++++++++++++++++ .../isar-bootstrap/isar-bootstrap.inc | 2 ++ 3 files changed, 38 insertions(+) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 73f1d52c..39addc59 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -72,6 +72,8 @@ inherit essential ROOTFSDIR = "${IMAGE_ROOTFS}" ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache" +# when using a custom initrd, do not generate one as part of the image rootfs +ROOTFS_FEATURES += "${@ '' if d.getVar('INITRD_IMAGE') == '' else 'no-generate-initrd'}" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}" ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" @@ -488,6 +490,8 @@ do_rootfs_quality_check() { args="${ROOTFS_QA_FIND_ARGS}" # rootfs_finalize chroot-setup.sh args="${args} ! -path ${ROOTFSDIR}/var/lib/dpkg/diversions" + # initramfs is generated outside of the image rootfs + args="${args} ! -path ${ROOTFSDIR}/boot/initrd.img*" for cmd in ${ROOTFS_POSTPROCESS_COMMAND}; do case "${cmd}" in image_postprocess_mark) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 1b95115a..69e38dac 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}" # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR} # 'export-dpkg-status' - exports /var/lib/dpkg/status file to ${ROOTFS_DPKGSTATUS_DEPLOY_DIR} # 'clean-log-files' - delete log files that are not owned by packages +# 'no-generate-initrd' - do not generate debian default initrd ROOTFS_FEATURES ?= "" ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" @@ -117,6 +118,16 @@ rootfs_configure_apt() { EOSUDO } +ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation" +rootfs_disable_initrd_generation[weight] = "1" +rootfs_disable_initrd_generation() { + # fully disable initrd generation + echo "replace update-initramfs with stub" + sudo mv "${ROOTFSDIR}/usr/sbin/update-initramfs" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" + sudo chroot "${ROOTFSDIR}" ln -s "/usr/bin/true" "/usr/sbin/update-initramfs" +} + ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" rootfs_install_pkgs_update[weight] = "5" @@ -310,6 +321,27 @@ rootfs_cleanup_isar_apt() { EOSUDO } +ROOTFS_POSTPROCESS_COMMAND += "rootfs_restore_initrd_tooling" +rootfs_generate_initrd[weight] = "1" +rootfs_restore_initrd_tooling() { + if [ -e "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" ]; then + sudo mv -f "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs" + fi +} + +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', '', 'rootfs_generate_initrd', d)}" +rootfs_generate_initrd[weight] = "10" +rootfs_generate_initrd() { + if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then + sudo -E chroot "${ROOTFSDIR}" sh -c '\ + export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ + update-initramfs -u -v -k "$kernel_version";' + else + echo "no kernel in this rootfs, do not generate initrd" + fi +} + do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" do_rootfs_postprocess[network] = "${TASK_USE_SUDO}" python do_rootfs_postprocess() { diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 3477c2fb..4c6011bc 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -24,6 +24,8 @@ DISTRO_BOOTSTRAP_KEYFILES = "" THIRD_PARTY_APT_KEYFILES = "" DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales" +# install early, so we can stub the update-initramfs script before rootfs install +DISTRO_BOOTSTRAP_BASE_PACKAGES:append = ",initramfs-tools" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"