From patchwork Thu Jul 25 15:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3722 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:25 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f57.google.com (mail-wm1-f57.google.com [209.85.128.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAOwH007138 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:24 +0200 Received: by mail-wm1-f57.google.com with SMTP id 5b1f17b1804b1-427df7c3a2asf7680635e9.1 for ; Thu, 25 Jul 2024 08:10:24 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=Ma2qh70vN8jSxcWsaKCktelPOyIaHT/TA7dsTrBhq0IKgXmoIyFyShEimBUYdkTUSV rKWm1gQQPBm1t4c36hy+qm3BjCQz8QCw8UXiiSCazBcXTp9BoS4Kd4g3r0x05jjpfhJg 2/QB6v6TYEm/FPJ1LCE4tagswlBhgLMG0G0knVGPzu4HFfs24O/uBsIob/MIOoobnx87 spHRIKE8VpPo7AVbAfpFfj10bkbN0jNRiN2khXsRCWVqWilD7zUyWomqRLMBK9w9rWEX RezepqSve/je0afT0HBvguJ/KXbXtQsQpq5i6NRYaL7bo0aT6QNhXa5/8e1WAUWr+X56 rvjw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=r/xxjf4FajxD4+sAk0cVvTezHgBf0oz62I8Eq5Q/9Ug=; fh=jpxsTVOPPlwlNSS8DWqlj9cO5dEp8dT98NgooChUZ5k=; b=TbadNYkUTDMSuQAVIMTaK5uZJkhxVHhze5X46RT2sidvOSBRzB4+AXZgv/jin7f4FL kayAlBnpWvravrEtx/obOWIpHoDIgN209LjFMuEhqEamFGJ33YTH1OHSm44Mdi1yKr+3 /HjFrH0cRDnRNXuU9WlJr6Ctb30Q3cbCCX0ZQc9y/dDYphuUkDsjKzayZ1WhE/EiHY7H uMRMnpKYQn78LsY0VHOy9N3zHerWSd6Bpdl8WK7FOoafxCCAqdSR6GF3a89DG/pROUxA oTmkLsFwkkmZT4oj9ErAUYO9FyMGDQOEA+pL89DLsgwifDZmVWosCGsF/utk/XX5DwfN No1w==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=r/xxjf4FajxD4+sAk0cVvTezHgBf0oz62I8Eq5Q/9Ug=; b=cjD3rrmqlkbMd/sztusIV+Ca6ractwaKxj4YoxI8JD6szgLYoCbkgAFZPA1I8Dcb5X 0/CUKGt3mEVTC3XDHT/XZ4oyPaO93C+r9E+/7DWf4j6kY//ysHsu/NNlpYmg9Hx8QwxP XgAd+VIRsmP0pYcMDAv5NGTX3McbiQ670581Alj6vNNtlPBUXmf3PVDn+I3y6hv/WZmr u+nttc7JB32ea1zECDnqkxlrpTCX0VlBESo8035bbQsMfV9+PM52z87iV4kkhuq0YCph 9wVa99XWZXIPLRNPpAq6cpLFF0+METELwWSr+RHMsrJChGFZBrqFY90LHh3i9QV6/H7q MLgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=r/xxjf4FajxD4+sAk0cVvTezHgBf0oz62I8Eq5Q/9Ug=; b=YD4kQocqnzd+c9dN4Ea//SLS7V6+U0USVztWXr1Cfpoza5j+UgCji9xSNUO0NHv1K5 RmdAa2Nhy7y0gi2SJm71hRe07w1LBSKKmPf8J7/6jVrmAMtL4Z/uHGU4vQ5cHjkrWqV1 K9EpD0jbMzf8LM+xn25ejLnInks2CS2ULFCaVxO07OGsXiUbYemEaNHk8K+UtKuZp3mo f0Sw4XbUgFmFEdMOa7d7wnUKFKa2se7Nzd4900xoSPKu6rRYXSCINac83UaNEKoihcOa bW2r6TZVHic4Wtn5arr+OhDS7P2mupPn5gIbIEHgZenZ8ajnXRPZfVUnJVJTnsik58r1 qysw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXjsDY2li26b4j54FhRhnar+rp6bYKnCMJc8Ig6UUUrbCyXdALHrObObTOCc4DoAIernFiS8zJ79jcNkGJna0NHvK67XpI= X-Gm-Message-State: AOJu0YwFHyt1TbR7udaqDNtC9s9PKXD+WK45XfDHiW4vUgduk0LUiBTI 95H7dsq5pJti/6IB8HKHtE8T1WInPMkUKNhL4aYq3jIwq2I8y+u0 X-Google-Smtp-Source: AGHT+IEc9e4YeleRRqRWo9kLvaVoSKK2GRJGF4uNbxe4BG931bDzHjxBF+TAk80qz0MbJIr7gypd5Q== X-Received: by 2002:adf:e84f:0:b0:368:4e98:6e9e with SMTP id ffacd0b85a97d-36b3644559emr1597219f8f.63.1721920218050; Thu, 25 Jul 2024 08:10:18 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d12:b0:426:5e93:d7ca with SMTP id 5b1f17b1804b1-42803b7b796ls4662975e9.2.-pod-prod-06-eu; Thu, 25 Jul 2024 08:10:16 -0700 (PDT) X-Received: by 2002:a5d:5491:0:b0:367:94a7:12c8 with SMTP id ffacd0b85a97d-36b3637efaemr1647987f8f.6.1721920216123; Thu, 25 Jul 2024 08:10:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920216; cv=none; d=google.com; s=arc-20160816; b=jzxiLD+HNmuFACIRJ1qaM8Fhvi8xFUj/K2GCLpb8X+XVW1VcyoVwIQs1QzFrQhYGJF hWlIu/pJlcSqqEDTEbTnH2REJmuzLKJl+ewkDsT4IF7kwuuVa8+mHSHz6mQJL0zp/Nm6 iHwillqMsLtQayDhq2+8aWAXQHSzFk8jRouD1U60K6uE9EwDqmaeGujxyq6AhAZaW12q ToZX9juPhVBn6BOlroVSRosnHJqU0MEthoGQNUdbXGHErthyWvrGkh8NZqsKac1TUvNU Sb5R9SMxPsfWcNnAWrp3zwj2Gxma4kFzPNWWTw4hMmC3XhleWwGJH/gU4cxhViIRagJt b28A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=NFA2f6aW6K5nCB+gVMdmUGt6fT62XLvTj0WWYhI8rQw=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=UV6BJzOz6pp5a6RBTglUC38EtnoTZy8B4aW8k9qb32Yvcb4MiSaHzyCKBKwplX1o5+ smVpNTDEEflF8dxDucamEKYT0viMwmPGjGbQ56Fi6UGczo3QARuJRwM05f4Cb4G0150z g+pwjbti9/HYd0LXzqQKhs7UzHKAbe3YhoSa8Cly/s3UkiP2rlpyagXX4xtvKE0k62CQ ixaJC7fnAmxzixcudR/kj2k4M8uw4Gvw9UJ79uBCGGzA6Rdt1O+itlKwDNaOfRB6B0Yr sS+Hr7Hw7SrrJNYPLsLDlkaOtT2aIIVqXTkTywRcmf2otOg/eou+PdtV23DNwemV6MRB eBqw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-36b3685c516si37702f8f.8.2024.07.25.08.10.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKb007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:14 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 05/10] base-apt: Predownload packages to base-apt before install Date: Thu, 25 Jul 2024 18:07:37 +0300 Message-ID: <20240725151006.2129-6-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch uses debrepo script to predownload packages to base-apt repository before they are installed in rootfs. Signed-off-by: Uladzimir Bely --- meta/classes/crossvars.bbclass | 1 + meta/classes/dpkg-base.bbclass | 27 ++++++++++++++++++- meta/classes/dpkg.bbclass | 8 ++++++ meta/classes/image-locales-extension.bbclass | 5 ++++ meta/classes/image-tools-extension.bbclass | 13 +++++++++ meta/classes/rootfs.bbclass | 4 +++ .../sbuild-chroot/sbuild-chroot-host.bb | 2 ++ 7 files changed, 59 insertions(+), 1 deletion(-) diff --git a/meta/classes/crossvars.bbclass b/meta/classes/crossvars.bbclass index 00326c9a..03a74ebe 100644 --- a/meta/classes/crossvars.bbclass +++ b/meta/classes/crossvars.bbclass @@ -27,6 +27,7 @@ python __anonymous() { schroot_dir = d.getVar('SCHROOT_HOST_DIR', False) sbuild_dep = "sbuild-chroot-host" + flavor_suffix + ":do_build" sdk_toolchain = "crossbuild-essential-" + distro_arch + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR')) else: d.setVar('BUILD_ARCH', distro_arch) schroot_dir = d.getVar('SCHROOT_TARGET_DIR', False) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..367ea52d 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -11,6 +11,7 @@ inherit terminal inherit repository inherit deb-dl-dir inherit essential +inherit debrepo DEPENDS ?= "" RPROVIDES ?= "${PROVIDES}" @@ -114,6 +115,14 @@ do_apt_fetch() { trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + schroot -r -c ${session_id} -d / -u root -- \ rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt schroot -r -c ${session_id} -d / -- \ @@ -136,18 +145,31 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}" # Add dependency from the correct schroot: host or target do_apt_fetch[depends] += "${SCHROOT_DEP}" +# Debrepo context is created by target bootstrap, need this dependency too +do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap" do_apt_unpack() { rm -rf ${S} schroot_create_configs + session_id=$(schroot -b -c ${SBUILD_CHROOT}) + echo "Started session: ${session_id}" + schroot_cleanup() { + schroot -q -f -e -c ${session_id} > /dev/null 2>&1 schroot_delete_configs } trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT - schroot -d / -c ${SBUILD_CHROOT} -- \ + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + + schroot -r -c ${session_id} -d / -- \ sh -c ' set -e for uri in $2; do @@ -157,6 +179,9 @@ do_apt_unpack() { dpkg-source -x "${dscfile}" "${PPS}" done' \ my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${SRC_APT}" + + # End chroot session + schroot -e -c ${session_id} schroot_delete_configs } do_apt_unpack[network] = "${TASK_USE_SUDO}" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index bcc3f828..bf3994a6 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -111,6 +111,12 @@ dpkg_runbuild() { echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) + debrepo_parse_dscfile "${DSC_FILE}" + + locked_update_cmd=":" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'" + fi sbuild -A -n -c ${SBUILD_CHROOT} \ --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \ @@ -122,9 +128,11 @@ dpkg_runbuild() { --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="mkdir -p ${deb_dir}" \ --chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \ + --chroot-setup-commands="${locked_update_cmd}" \ --chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ --finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \ + --finished-build-commands="mkdir -p ${ext_root}" \ --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE} diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass index 9149d643..4dd93f1f 100644 --- a/meta/classes/image-locales-extension.bbclass +++ b/meta/classes/image-locales-extension.bbclass @@ -6,6 +6,8 @@ # This class extends the image.bbclass for setting locales and purging unneeded # ones. +inherit debrepo + LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\ en_US ISO-8859-1\n" LOCALE_DEFAULT ?= "en_US.UTF-8" @@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download" image_install_localepurge_download[weight] = "40" image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" image_install_localepurge_download() { + debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge } diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index e8ace8f5..ddb046a8 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -6,6 +6,11 @@ # This file extends the image.bbclass to supply tools for futher imager functions inherit sbuild +inherit debrepo + +python __anonymous() { + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR')) +} IMAGER_INSTALL ??= "" IMAGER_BUILD_DEPS ??= "" @@ -40,12 +45,20 @@ imager_run() { echo "Installing imager deps: ${local_install}" distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + debrepo_workdir=${DEBREPO_TARGET_DIR} if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi fi E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${schroot_dir} ${distro} + + debrepo_add_packages --isarapt "${debrepo_workdir}" "${local_install}" + debrepo_update_apt_source_list "${schroot_dir}" "base-apt" + ${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <