From patchwork Thu Feb 24 21:40:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Mikanovich X-Patchwork-Id: 1531 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Fri, 25 Feb 2022 08:40:59 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f192.google.com (mail-lj1-f192.google.com [209.85.208.192]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 21P7eweS001580 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 25 Feb 2022 08:40:58 +0100 Received: by mail-lj1-f192.google.com with SMTP id h21-20020a05651c125500b002464536cf4esf2190203ljh.23 for ; Thu, 24 Feb 2022 23:40:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1645774853; cv=pass; d=google.com; s=arc-20160816; b=qs/Q+tv6HHN48NvDj9QqY4DG4kPoBwCf9jO+BN2Fv2RU/ddRdE64Rm9qOA/gQl88ze OOImZu7SsRvoyKJ39p0BrqfWpjy1vp2Xba33d7gMZ2wH04C/Dq3UR25SSxCSMysvA/i/ ac4DE2a3LOXb/mjmh0QzOHn+bqMKD3cBVgs6tc1FoPPu+L+UMdKRB2mloBiaOFg5rCvr lKasN5x+uKi9IYdRA9huC+BvetA/V42CNN+JLFd7YjXUxKRMeTO2TL8vXS1OlQAN0p2n jojzPpOyoNlg1DDmHIqeZtdPKlTaciwRxN2CPxusgnlzquCypuf3N7IT9JwxBUOsMu2Z Up6Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=owJYtCs27qeSBwhbe/X8pHJK7rmjzosUia+KmC1Z4gw=; b=ep0aHzrOz5yTD/GWTF6479LIGyogKPfMbS5MFohl33Ho9yQniLHZdAQn0CpVFoIrk6 DY8i9oEMnP5uKB0E2YA2zfZxwSFQH6kRFzbsdpwUoDYaLPbz9OSCY0hHhFHtPbtoeQiG fRuHzCgtiaCNZT9eVPluhn+JSAukhb9SdS4IjfhYq2Zee6h6psDSCE5y/znALHoX44pL okoPItwF6l8tN9x0AgP5OrLi45QeZKwGD29cWjsABWs36koF/EKf80pAJXrY+f85hd3y Zg/iYSG1kX+lBhdbjpvrXa6O73EUc2AsliSuEfA8EtjHFteXugbRvcejoYGUu/CpwtsM bl+g== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=owJYtCs27qeSBwhbe/X8pHJK7rmjzosUia+KmC1Z4gw=; b=oj2czKdWaBlKHITEsvA2CGSQyiBOcjjh4lt8PGXSHY0nBGiNQ87Eonu9pUmtCLFpxj UjyuxmIx4qh9ECGUyycNUP91U2KzEfnPPjk9lDY3oBaWe5jvdj6xgvCCHqmNljqrgyhg GfoNirM7/sTRim/NonFOMe84cfg2AarmjdabNVGPAQn/9NVj0Kopdl4Y05zYBaTsWWL2 nhHoL7SqijSp2FvCg4koDsyRv7N0rR/KZl95AV6iL3tejlce87ZqnDIXQA7TnHCaeOw3 EC6ynyaFFM4hnrUGOLhYJWmnw6h37w7JMexr1JgRd0vUtnTPTiOVyWOwspkm2r0AwdYd en9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=owJYtCs27qeSBwhbe/X8pHJK7rmjzosUia+KmC1Z4gw=; b=fIMOwIiLLtOwN1RhyjD+UORNZTbfeSYTCQuMB3tRAsFlc3dySKPk7AaDBzT4U7IaJC NtFkkLyzRXinvI5oKWJHhRVLn1hTj6I0ehMjAYWLJDCRPGdV/5h6VanRO/nLWlLWE7Dm gQEiTCNPyt8PW3fTHe9+XNwcQclVDuZ7xA/De/f25XpCIM4wwFQBfhiuSjArBhCT8tRd amLF4K56xHcGWfZpslU9xUEbR6A72IMORy4e6SNpI8Ls2NHxijh+lD5kj/1nTAj9HbLi 5V4yeSP0eFvWts9LxZVVFRDWu43+PPkuuR4SabdG6UHddSQcbRIaqVFX8iv+jOIUX+BL 5TOg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM532eT8Dv1KkKdrWay/6XJBmBRsIjkMl/PFPRdNYLXNm9Dg+PPHqn n7SuYJHsjtT/wSEKqqiAKzw= X-Google-Smtp-Source: ABdhPJzNVRIPUlmO6JWAG6Z4BpASEJzWherT3j/rx5qh3mu8+V9wf+5ZLZKp7ZVb9R+dk2rNBBmf4w== X-Received: by 2002:a05:6512:ac3:b0:443:d3e3:db0a with SMTP id n3-20020a0565120ac300b00443d3e3db0amr4259671lfu.298.1645774853104; Thu, 24 Feb 2022 23:40:53 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:15a3:b0:443:7b15:f451 with SMTP id bp35-20020a05651215a300b004437b15f451ls146537lfb.0.gmail; Thu, 24 Feb 2022 23:40:52 -0800 (PST) X-Received: by 2002:ac2:5963:0:b0:442:eff6:4219 with SMTP id h3-20020ac25963000000b00442eff64219mr4192029lfp.56.1645774852188; Thu, 24 Feb 2022 23:40:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645774852; cv=none; d=google.com; s=arc-20160816; b=z0B+3ykBsPz0mqRFGjGBOBxKZWyJQEfHL1JWBT6KStQVnXdmtXJq+E6vB8NjxhU99H XIWodA8e1M4CMNCi5wGzAZqdIGFbCXYqOMyVpqZlElFWppTyJZF6ZiuZ1hC2yM062ioD XdTrT61hALey52tSeV/8C8SZo3L+43DqNByIFroBycGtPn4iKpzA/tXp2slZ2yw63UTa hVi/4cl9BhW3WpO4vRk3Au5I8z0ld4oYEQumulmHk3dyWNTD5g0mn6Sajw2QIT6AU9OL eIfMBU/8ucs3Gl36pYpBxUuDtf9voIPEUMmj9GjiBrwdztmZ/sdp4+SGFapaIfkzW/NL X8vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=t5dJzx+egQ1wtk667RJp5DbnFahR/p2vyY4v7NQjgXE=; b=IQWteqj001FAFyjpDzk0VsY59GUszECyWS24dEGOIKX5NVwGRcI+Wcz1rSC9/NXGxq GI/DoJBqy3g/9dLJvkFnPkYgljpSB/I3SJh7hxOzgfRw7oVx4D+ij7mn42cvrGluzzB1 dLnTTrfTJYaSPBMraDomck6gmQ2zGikCVOchGjKNi48MoVxjnXFZIM9zpdRUvPrPeHPB pdkPW4Fs6RsVezoRYEOz9t+RIcMvJD1koXen+4Y6iXy63D/CC65YM8j1xBw4jZ4VBY3x XYFIXxRESN5n0mvNHT33GaPTZia0reSSCtRViwFqpUMTo+prES5AvnC8YZbH5iezDawv ar4g== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id w24-20020a2e9598000000b002463b72fb7esi94355ljh.5.2022.02.24.23.40.52 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 24 Feb 2022 23:40:52 -0800 (PST) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (mm-5-27-214-37.mgts.dynamic.pppoe.byfly.by [37.214.27.5] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 21P7emqq001558 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 25 Feb 2022 08:40:51 +0100 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [RFC 2/5] meta: Implement two stage build Date: Fri, 25 Feb 2022 10:40:37 +0300 Message-Id: <20220225074040.20975-3-amikan@ilbers.de> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220225074040.20975-1-amikan@ilbers.de> References: <20220225074040.20975-1-amikan@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: amikan@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: INBOX Split package build into two stages: building source package and building binary package. This allows to obtain Debian package dependencies before any build steps will actually start. Signed-off-by: Anton Mikanovich --- meta/classes/dpkg-base.bbclass | 34 +++++++++++++++++-- meta/classes/dpkg-gbp.bbclass | 6 +--- meta/classes/dpkg-prebuilt.bbclass | 7 ++++ meta/classes/dpkg.bbclass | 21 +++++++++--- meta/classes/rootfs.bbclass | 2 ++ .../buildchroot/buildchroot.inc | 2 ++ .../buildchroot/files/build-source.sh | 16 +++++++++ .../buildchroot/files/build.sh | 2 +- 8 files changed, 76 insertions(+), 14 deletions(-) create mode 100644 meta/recipes-devtools/buildchroot/files/build-source.sh diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index e3ec133..916bae7 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -75,7 +75,7 @@ addtask adjust_git after do_unpack before do_patch do_adjust_git[lockfiles] += "${DL_DIR}/git/isar.lock" inherit patch -addtask patch after do_adjust_git before do_dpkg_build +addtask patch after do_adjust_git before do_dpkg_source SRC_APT ?= "" @@ -175,7 +175,7 @@ do_prepare_build() { true } -addtask prepare_build after do_patch do_transform_template before do_dpkg_build +addtask prepare_build after do_patch do_transform_template before do_dpkg_source # If Isar recipes depend on each other, they typically need the package # deployed to isar-apt do_prepare_build[deptask] = "do_deploy_deb" @@ -238,6 +238,19 @@ python do_dpkg_build() { addtask dpkg_build +python do_dpkg_source() { + lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", + shared=True) + bb.build.exec_func("dpkg_do_mounts", d) + try: + bb.build.exec_func("dpkg_runbuild_source", d) + finally: + bb.build.exec_func("dpkg_undo_mounts", d) + bb.utils.unlockfile(lock) +} + +addtask dpkg_source before do_dpkg_build + SSTATETASKS += "do_dpkg_build" DPKG_SSTATE = "${WORKDIR}/dpkg-sstate" do_dpkg_build[dirs] += "${DPKG_SSTATE} ${S}/.." @@ -300,6 +313,21 @@ do_deploy_deb[depends] += "isar-apt:do_cache_config" do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" do_deploy_deb[dirs] = "${S}" +do_deploy_source() { + #TODO: dsc_clean + find "${S}/../" -name '*\.dsc' | while read package; do + repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \ + "${DEBDISTRONAME}" \ + "${package}" + done +} + +addtask deploy_source after do_dpkg_source before do_dpkg_build +do_deploy_source[depends] += "isar-apt:do_cache_config" +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" +do_deploy_source[dirs] = "${S}" + python do_devshell() { bb.build.exec_func('dpkg_do_mounts', d) @@ -327,6 +355,6 @@ python do_devshell_nodeps() { # devshell may be placed after do_instell_builddeps in downstream classes. # devshell_nodeps will always stay right after do_prepare_build. -addtask devshell_nodeps after do_prepare_build +addtask devshell_nodeps after do_deploy_source do_devshell_nodeps[dirs] = "${DEVSHELL_STARTDIR}" do_devshell_nodeps[nostamp] = "1" diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass index d956e8c..b02aed0 100644 --- a/meta/classes/dpkg-gbp.bbclass +++ b/meta/classes/dpkg-gbp.bbclass @@ -12,8 +12,7 @@ PATCHTOOL ?= "git" GBP_DEPENDS ?= "git-buildpackage pristine-tar" GBP_EXTRA_OPTIONS ?= "--git-pristine-tar" -do_install_builddeps_append() { - dpkg_do_mounts +dpkg_runbuild_source_prepend() { distro="${DISTRO}" if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_DISTRO}" @@ -26,9 +25,6 @@ do_install_builddeps_append() { sudo -E chroot ${BUILDCHROOT_DIR} \ apt-get install -y -o Debug::pkgProblemResolver=yes \ --no-install-recommends ${GBP_DEPENDS} - dpkg_undo_mounts -} -dpkg_runbuild_prepend() { export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder=" } diff --git a/meta/classes/dpkg-prebuilt.bbclass b/meta/classes/dpkg-prebuilt.bbclass index c32224b..c5d645c 100644 --- a/meta/classes/dpkg-prebuilt.bbclass +++ b/meta/classes/dpkg-prebuilt.bbclass @@ -18,3 +18,10 @@ python do_unpack_prepend() { deltask dpkg_build addtask unpack before do_deploy_deb + +deltask deploy_source +# dummy task for 1st stage +do_deploy_source() { + true +} +addtask deploy_source after do_deploy_deb before do_build diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 320102b..e95420a 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -17,15 +17,17 @@ do_install_builddeps() { fi deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}" - sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ - ${PP}/${PPS} ${PACKAGE_ARCH} --download-only + sudo -E chroot ${BUILDCHROOT_DIR} sh -c "apt-get update && apt-get \ + -o Debug::pkgProblemResolver=yes --no-install-recommends -y \ + --allow-downgrades --download-only build-dep -a ${PACKAGE_ARCH} ${PN}" deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}" - sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ - ${PP}/${PPS} ${PACKAGE_ARCH} + sudo -E chroot ${BUILDCHROOT_DIR} sh -c "apt-get update && apt-get \ + -o Debug::pkgProblemResolver=yes --no-install-recommends -y \ + --allow-downgrades build-dep -a ${PACKAGE_ARCH} ${PN}" dpkg_undo_mounts } -addtask install_builddeps after do_prepare_build before do_dpkg_build +addtask install_builddeps after do_deploy_source before do_dpkg_build do_install_builddeps[depends] += "isar-apt:do_cache_config" # apt and reprepro may not run in parallel, acquire the Isar lock do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" @@ -42,3 +44,12 @@ dpkg_runbuild() { sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH} } + +dpkg_runbuild_source() { + E="${@ isar_export_proxies(d)}" + export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}" + export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}" + export PARALLEL_MAKE="${PARALLEL_MAKE}" + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ + /isar/build-source.sh ${PP}/${PPS} ${PACKAGE_ARCH} +} diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index d578dd3..c5ea748 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -88,6 +88,8 @@ rootfs_configure_isar_apt() { mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d' echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \ '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list' + echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \ + '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list' mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d' cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt' diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc index aa190e9..2872680 100644 --- a/meta/recipes-devtools/buildchroot/buildchroot.inc +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc @@ -9,6 +9,7 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260 FILESPATH_prepend := "${THISDIR}/files:" SRC_URI = "file://configscript.sh \ file://build.sh \ + file://build-source.sh \ file://common.sh \ file://deps.sh" PV = "1.0" @@ -58,6 +59,7 @@ buildchroot_install_files() { sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}/isar/ + sudo install -m 755 ${WORKDIR}/build-source.sh ${BUILDCHROOT_DIR}/isar/ sudo install -m 755 ${WORKDIR}/common.sh ${BUILDCHROOT_DIR}/isar/ sudo install -m 755 ${WORKDIR}/deps.sh ${BUILDCHROOT_DIR}/isar/ diff --git a/meta/recipes-devtools/buildchroot/files/build-source.sh b/meta/recipes-devtools/buildchroot/files/build-source.sh new file mode 100644 index 0000000..a327c9b --- /dev/null +++ b/meta/recipes-devtools/buildchroot/files/build-source.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# +# This software is a part of ISAR. +# Copyright (C) 2022 ilbers GmbH + +source /isar/common.sh + +# If autotools files have been created, update their timestamp to +# prevent them from being regenerated +for i in configure aclocal.m4 Makefile.am Makefile.in; do + if [ -f "${i}" ]; then + touch "${i}" + fi +done + +${GBP_PREFIX}dpkg-buildpackage -a$target_arch -d --source-option=-I -S diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh index d4709cf..c19221b 100644 --- a/meta/recipes-devtools/buildchroot/files/build.sh +++ b/meta/recipes-devtools/buildchroot/files/build.sh @@ -14,4 +14,4 @@ for i in configure aclocal.m4 Makefile.am Makefile.in; do fi done -${GBP_PREFIX}dpkg-buildpackage -a$target_arch --source-option=-I +${GBP_PREFIX}dpkg-buildpackage -a$target_arch --source-option=-I -b