From patchwork Sun Jun 25 05:54:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "MOESSBAUER, Felix" X-Patchwork-Id: 2920 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Sun, 25 Jun 2023 07:55:13 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f57.google.com (mail-wr1-f57.google.com [209.85.221.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 35P5tCJf025272 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 25 Jun 2023 07:55:12 +0200 Received: by mail-wr1-f57.google.com with SMTP id ffacd0b85a97d-3127a76c56bsf750912f8f.1; Sat, 24 Jun 2023 22:55:12 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1687672507; cv=pass; d=google.com; s=arc-20160816; b=aSawZEs1g9FZ0tbeDwxQVgrQIYvb7jaxhBnUV5HhPtXT5ahzOfUwtGuRYwjC0/Y7pX 7g02Z/XdH09mDgVuyMrfqZq8e4SEklgmEKbx5H+IJgk6TosQwQsRengeCb1U2ctg+o0c ccYD4d18AWxtP+H2rdqGpHXATfrQKvn4AT8suCleMY6xztvqQX7oMkr0fiFyvnJ40XUF qnI7aFgIgsq9V8mRKVfZd/sj2eoY0je1nY8sYSpiOV3UPxjh5hJ3nuQA3rPoWBeR3eCL DdTFEub34Q4OQJ/rXlSj27J8vagQMFMvzgnjjoaYOFNhiDxnPKEsDj8qn9/UhF7HTY3y qZGg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=w6casltvGSu0B/b4XVbYiTBddI7X15xs1a3kPDdmrVo=; b=y71ylqpLPQNBLdlFbgHiVWTxQIeLIr+EfFT0HMPYRhu7hfFX1LiOG/I5rCCRfUKgHH 9p9l1+iKKh40m2xxgVr8dENOEURF0S2d7a3GQFWb3SynkRfKXzxoHADsWvAxhKkjnSx8 GkxNVRTXlX51xBBcRQ/ZAxw21hbdur+3CTtZHR7uYFmtr9Bt2dIERVE8txTTmeeuKgR1 pxj2LPFytSAiEtHLE++nenNL2jlOU1N7rNKQpHn/pslF9GKyy6Ql/EqJIQ6N0/WfAAH7 me3pD1acdOCsHRqC4l5cTmcsdjnEE6g3IbKFLhpAhvJCWIyH3ChHggvcJINoEjeTl7VI 5D5A== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=dONzkBul; spf=pass (google.com: domain of fm-72506-20230625055503cea601ef847cf23692-gwsbbw@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-72506-20230625055503cea601ef847cf23692-gWSbbW@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20221208; t=1687672507; x=1690264507; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=w6casltvGSu0B/b4XVbYiTBddI7X15xs1a3kPDdmrVo=; b=RJbRTHB5NjvtXdShxYsbZgJosLKNyJcaN2+E20FbfnIox01j1rcJgj6miAwO7FWgmD 5lFRFPkgoBYlXkSrn6g6mVZQTwxVijAQiMZQGvOKY0afhixVbKoICk4ti8A1kXiwikhg X7S9EIYoPteCmSKLPWftQuyFf0Y3+UGodkHNcFLYFxLu7DeUbqp4gRll2kfPlpY0YZzX F30psWIl2KRa0Kcsz8NP62wOFIc8OQbFSDHJioFXEXJrDmFUtlVYWl+C1JEoc0MPXGif y6abF0skr9zlywv2ctn1mBLErb4it4oCUWnOUhQDgpjkW4hd/1gsrus2aw5S0fkRwTaw oN2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687672507; x=1690264507; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=w6casltvGSu0B/b4XVbYiTBddI7X15xs1a3kPDdmrVo=; b=XPafre21vHKF3BuxIOzy3dpA4DyaSa7+lRrgGLW0Lw0gzFJPiWfn8FiCir3J3R+hhl kZ4tfjOomLjxjMb/Iv9ZhYWspgxMosANI/uTmnRSoy3bEQXimWhDOSjgdvf3aMs93MWh uhJbQT+T2erFmWBtY3NzTEc9JsPEVSqJaZwvLfJLX8Y3Y+lv2AQ8hu3NmFJGT2bB8u83 WxsTFCu+IFHb2fLwNSYE69F4kMRplP7zcD4nXuzAj4BWTQwmqhYDGYof+W8uHHnObEhX asnf7V3aTV/frFRekBpG18QbmrJW9GQss45lNgo3QQXt+hjvld54/dGMkI6yQKsNnBON PBiQ== X-Gm-Message-State: AC+VfDwUi/rNWkLvwfuZWYyLI/+ujJyPXIPSJRd7NITMvi04yE8ZWXR7 f0bjRz8S3PSz6HW+fzHry1s= X-Google-Smtp-Source: ACHHUZ708ADghZvhzOoF05UF3j6KR64Ibz1DpMTOTNxsiPDbm0k+e63TDFpHzWlZ+crawk7ww+9ojg== X-Received: by 2002:adf:ee88:0:b0:30e:55ae:c69d with SMTP id b8-20020adfee88000000b0030e55aec69dmr15055257wro.45.1687672506537; Sat, 24 Jun 2023 22:55:06 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:7114:0:b0:4f8:58d3:b79f with SMTP id m20-20020a197114000000b004f858d3b79fls530512lfc.1.-pod-prod-06-eu; Sat, 24 Jun 2023 22:55:04 -0700 (PDT) X-Received: by 2002:a19:e348:0:b0:4f8:48f3:f06 with SMTP id c8-20020a19e348000000b004f848f30f06mr14391318lfk.48.1687672504228; Sat, 24 Jun 2023 22:55:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687672504; cv=none; d=google.com; s=arc-20160816; b=f7TOpeHYrXy3L0hWsa7YsjpaeuIo2qUtT9BnuHWS7JYpOK9EX03HUPbGXbneTR+Knc FXmtAfiDzXBZ6xQsccYslalm0mQN/9IsIyAbfyWi5DbM89uVOCFJKgOzQikk5R5+CjBg mMR47PaCjjJ+0wW4ZrLNUsUXXq9vJaEph6YGZFl2nDBAxovlLU3OkqaZghiN/LRx7JLk 0yANbwW/aU8J3legGoPs7DrJsekfjmdW3LTz+mzg6eW9ZwqB/DVIz2y/efpA6Fre2GY2 LU4aDVlZaZ4XjWiQ8OpLGgxt+V56b2MT6ArpYzZNjUAJ2U+4B4sfLRFPj4a2EeYmKTJP SpVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=dK751GRscJzHo1kdeFLpfBIh2WTeYOqJG6S4uFqNdRQ=; fh=cGtWllDED0w5X6kg5KaY2TuUOGUxlC1C5M/bnQ8ze7g=; b=KBhuDQVsocBYv335UMPgOdlNmAYN1cEF+VHJ4BrWvShsDUShrv05fgH+/cLc7otco2 9Ban3b9Z1sotLRZj/lYCg0Ns5LEcyXzPmblYa1Wh3m6UB/y4keORCg8DoEUpg2I67gSb l4Kg8rPx1Y7sY3s1d/zaYsfbMD50EuFqKeSkB4gBpWOWYBGT64c0CIiu1TMz1HtLy0Ou BlOq3UNl6kf1ub6PHnQ5nyw+GtRaiQ9KdKRgjGF2gZmWpioCkrGJ+CRWsK8XEvBuMfW2 1iM8CNwzR36fl5kot8apEbzsg1fq6G0SqHtEDLrUGnMvb192Q0tHp8mbEsDxaDTz7A+E mVqg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=dONzkBul; spf=pass (google.com: domain of fm-72506-20230625055503cea601ef847cf23692-gwsbbw@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-72506-20230625055503cea601ef847cf23692-gWSbbW@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id b5-20020a0565120b8500b004f86d47501fsi140349lfv.4.2023.06.24.22.55.04 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 24 Jun 2023 22:55:04 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-72506-20230625055503cea601ef847cf23692-gwsbbw@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20230625055503cea601ef847cf23692 for ; Sun, 25 Jun 2023 07:55:03 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: "MOESSBAUER, Felix" To: isar-users@googlegroups.com Cc: henning.schild@siemens.com, Felix Moessbauer Subject: [PATCH v2 1/1] add support for derived sbuild chroots Date: Sun, 25 Jun 2023 05:54:42 +0000 Message-Id: <20230625055442.1569823-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-72506:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=dONzkBul; spf=pass (google.com: domain of fm-72506-20230625055503cea601ef847cf23692-gwsbbw@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-72506-20230625055503cea601ef847cf23692-gWSbbW@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch adds support to create derived sbuild chroots to speedup the build process. For packages that share a large set of common build dependencies, a derived sbuild chroot can be created to avoid the overhead of installing all base build-deps on each sbuild invocation. Signed-off-by: Felix Moessbauer --- Changes since v1: - dropped example - add variable SBUILD_CHROOT_PREINSTALL_EXTRA to add packages doc/user_manual.md | 26 +++++++++++++++++++ meta/classes/crossvars.bbclass | 11 +++++--- .../sbuild-chroot/sbuild-chroot.inc | 9 ++++++- 3 files changed, 42 insertions(+), 4 deletions(-) diff --git a/doc/user_manual.md b/doc/user_manual.md index e07b76d8..106d2dca 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -1406,3 +1406,29 @@ To enable it, add the below line to your local.conf file. ``` BASE_REPO_FEATURES = "cache-deb-src" ``` + +## Use a custom sbuild chroot to speedup build + +### Motivation + +There are use-cases, where many packages need to be compiled but all of them +need a similar base of build dependencies. In case the baseline is quite big, +this adds a significant overhead as the build dependencies are installed individually +for each and every package. + +### Solution + +By creating a dedicated sbuild chroot for this use-case, the baseline can be installed +first and then all package builds of this type can use it. For that, create a +new recipe with the name `sbuild-chroot--`. In that recipe, +define the following: + +``` +require recipes-devtools/sbuild-chroot/sbuild-chroot-.bb + +SBUILD_FLAVOR = "" +SBUILD_CHROOT_PREINSTALL_EXTRA += "" +``` + +Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = ""`. +To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`. diff --git a/meta/classes/crossvars.bbclass b/meta/classes/crossvars.bbclass index 201d460f..8d1da09c 100644 --- a/meta/classes/crossvars.bbclass +++ b/meta/classes/crossvars.bbclass @@ -10,22 +10,27 @@ python __anonymous() { d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name) mode = d.getVar('ISAR_CROSS_COMPILE') + + # support derived schroots + flavor = d.getVar('SBUILD_FLAVOR') + flavor_suffix = ('-' + flavor) if flavor else '' + distro_arch = d.getVar('DISTRO_ARCH') if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or distro_arch == None: d.setVar('BUILD_HOST_ARCH', distro_arch) schroot_dir = d.getVar('SCHROOT_TARGET_DIR', False) - sbuild_dep = "sbuild-chroot-target:do_build" + sbuild_dep = "sbuild-chroot-target" + flavor_suffix + ":do_build" buildchroot_dir = d.getVar('BUILDCHROOT_TARGET_DIR', False) buildchroot_dep = "buildchroot-target:do_build" sdk_toolchain = "build-essential" else: d.setVar('BUILD_HOST_ARCH', d.getVar('HOST_ARCH')) schroot_dir = d.getVar('SCHROOT_HOST_DIR', False) - sbuild_dep = "sbuild-chroot-host:do_build" + sbuild_dep = "sbuild-chroot-host" + flavor_suffix + ":do_build" buildchroot_dir = d.getVar('BUILDCHROOT_HOST_DIR', False) buildchroot_dep = "buildchroot-host:do_build" sdk_toolchain = "crossbuild-essential-" + distro_arch - d.setVar('SCHROOT_DIR', schroot_dir) + d.setVar('SCHROOT_DIR', schroot_dir + flavor_suffix) d.setVar('SCHROOT_DEP', sbuild_dep) d.setVar('BUILDCHROOT_DIR', buildchroot_dir) d.setVar('BUILDCHROOT_DEP', buildchroot_dep) diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc index fd8bb648..c572456e 100644 --- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc +++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc @@ -11,6 +11,12 @@ PV = "1.0" inherit rootfs inherit compat +# set the flavor to create derived sbuild chroots +# this maps to a schroot created by a recipe named sbuild-chroot-- +SBUILD_FLAVOR ??= "" +SBUILD_SCHROOT_SUFFIX = "${@ ('-' + d.getVar('SBUILD_FLAVOR')) if d.getVar('SBUILD_FLAVOR') else ''}" +SBUILD_CHROOT_PREINSTALL_EXTRA ??= "" + python() { distro_gcc = d.getVar('DISTRO_GCC') distro_arch = d.getVar('DISTRO_ARCH') @@ -36,6 +42,7 @@ python() { SBUILD_CHROOT_PREINSTALL_COMMON = " \ ${SBUILD_CHROOT_COMPAT_PREINSTALL} \ + ${SBUILD_CHROOT_PREINSTALL_EXTRA} \ libc6-dev:${DISTRO_ARCH} \ fakeroot \ build-essential \ @@ -52,7 +59,7 @@ ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}" # We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild ROOTFS_CONFIGURE_COMMAND:remove = "rootfs_configure_isar_apt" -DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}" +DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}${SBUILD_SCHROOT_SUFFIX}" do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}" do_sbuildchroot_deploy() {