From patchwork Thu Mar 14 07:27:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 3411 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 14 Mar 2024 08:32:03 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f60.google.com (mail-lf1-f60.google.com [209.85.167.60]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 42E7V6wZ021620 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:31:06 +0100 Received: by mail-lf1-f60.google.com with SMTP id 2adb3069b0e04-5133f2e0485sf472991e87.0 for ; Thu, 14 Mar 2024 00:31:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710401458; cv=pass; d=google.com; s=arc-20160816; b=BktBJQjq1Vs4N9MHsLKIQuFZhE1+C9o+FGA8N6b4u68iUa8AnUv7q6LFy8oj0C/kd5 BCMK/bao2ex38Rcz4xvhwQrkauTm+gswrS6NeNhH1OglZ/y+cv4rHsFk2Hyuv2f/ibh1 4ebbO6J3hH0luk60QGdB5HgcdrclxtTOBBXhpzBjxoUKlcEmyG52Jj3+XlbMqt4IY60O U78PUAX6isPAdGFQKRMT1+wwTnUFVqJa7rbkTd3zA1yIZbtzKfecCc+ogQP6iZdfUeDJ xzo+zhsKmL4Q52FR/EJ/eWkkQRoZq+tqetmVpHUswrKFois1uCB58wYXN5b5TP1o+1Pm IXJQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=/Kd32Sh/p1wBKuxeYTryl/dHVZXrZ+JbgCOUf0ggJLM=; fh=EJCLMzL4QrRA1ga/MeUP8dBrLHjR4MUnsCuVqN8Ww3U=; b=Rg2Q6oCwSlxqUMzEjZC9KCkKqp6iEGSWXhmdH93qA7zxhUJQW7hk8/ka9l+nu/g54C cs1fxBHd7/t5NmR5A86NkBAVKfsHEMg4hG7iiOu46hPLCcNAOXpWRLsWz2WaYB9jb4wn vonCHriSlF43/JahAT1CmHxx/7vXFTpTOSTWi/vJY8ZiH7/dNVwMgWXoeKQblWHh/cDl RmwOzvl/xaBCyzRQhjJKL3QUAj7+04+3vp/h+4FWJoUsxKX3XJMXWNcUF9+f8KmY27qH KUOdVdLmnspkFTI8O3IvK/jRQkw8MO1a5Qcr7c8Qc7pFPI989DJVQnJnJBUWAH3q2kQW VRqQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1710401458; x=1711006258; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=/Kd32Sh/p1wBKuxeYTryl/dHVZXrZ+JbgCOUf0ggJLM=; b=i73/dPnT9hZgtsZqKvIggLPGH+VIrNlnEvwEqr3moKPqTES9iCdL5os/vlpo5d94lm EvUILg1V4B4tru7xv6Lo4vJnKETvSIZi7OVupyOwxteI2QqK1RQpzc16Ug/ChURj+rrw xnZDwOGcGLpPk6SjJ9nkHy4YxTiSkjP4lRLPu0MS6aDWJhxcXFAijVi6TS6T4KquKl/r 9K7+ehEX+p+CgnYi8VSB42qyE8YNaQOqGzpuRNEXadmHSA5KfK9Uv70GC2Km8b3/xm0n RknNisk32cwqnNDJYnv8rCv5ERbHQozADswrtGAF4GAwJ8Jn8Cp58hAJYKeLM0YcOJUM 8e1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710401458; x=1711006258; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=/Kd32Sh/p1wBKuxeYTryl/dHVZXrZ+JbgCOUf0ggJLM=; b=FzPSJI22hQwWicwQwtiO1sz4vCTnSjUt18xD/qqbymbpZ0ru/eXFhyslfwh89se2z9 ab54J1fIRpj4bM5O1g3rA1DZJT07Y+NVHyA+Pz8/Wqk0YG9lsOyrbrAJaFbHWJIC35wn Vsum4qkKUKwrra/Bjb64l4HfDWuyF8P6lqs76eW9xLvh3yOlOeH38YTAWbvVFI6Y4A3I 6OBGWElRvYHHcc5fY/E0cEPJhFz/8vn7+fyvj/OR2fJlA0zpxqYA3rYtWzF7/lPdwGG5 E0Cgu6jXOh60xsdn/8QTaHp2n/2DAB79I0/HibU4JdW5+HyFSEcBlMQgd03q7ilT0ckz IiyQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWKO0iJah4k7gxMOm4tMMgcssck7hU9nWQbbVrTAxvSPjRzw19g1LPt75nSLtUURWjlKkFC1qCC0v/nLTx/j6FK/xE6gYg= X-Gm-Message-State: AOJu0YzHljEsNCcA875ClrcHhVymHvzwO8Oa5HUP9hHEdvWx+K3Y+voI QbbvQ/W43MidfJZeY00kZ9VU57lo/Vl0DDR0NaFCBTQAC2AdjhYg X-Google-Smtp-Source: AGHT+IHdcvq37oh5UqSrywAsTRjN9l4Eiqp3hPXtUqTp1msZ5HyiSLLTVftPfK5yJ5b7X3Sub/1u5w== X-Received: by 2002:a05:6512:3084:b0:513:c97f:6ba6 with SMTP id z4-20020a056512308400b00513c97f6ba6mr2006615lfd.15.1710401457556; Thu, 14 Mar 2024 00:30:57 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:b041:0:b0:2d4:6c0a:be15 with SMTP id d1-20020a2eb041000000b002d46c0abe15ls171859ljl.0.-pod-prod-00-eu; Thu, 14 Mar 2024 00:30:55 -0700 (PDT) X-Received: by 2002:a2e:9295:0:b0:2d4:6d7b:4b68 with SMTP id d21-20020a2e9295000000b002d46d7b4b68mr419566ljh.6.1710401454920; Thu, 14 Mar 2024 00:30:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710401454; cv=none; d=google.com; s=arc-20160816; b=cHpuwazE/hkdbsUdjvDQML4mJQkLZXsCGG0dSXreUPMKEZALptTFrejwtvXXqun6My RUxmEXVja0gJ+pB2y+d9ZocQjhRfeQUqnC3qDkZOmTrcGnEWp/7cPTqZ7sIG10zgVqeF x9WgPiiMZsZzTIIwo+pZvO8FoBTVveUU0RPx0lilkb63Wwg48HXyoVKC7ONPFuOCfVFi XDMdvt/NPBdbEWQHIOZCGx8xoJu5gk9xxfYwe1DOHPaiBbwU8jxRbcxe1LRnTZ7l4yFx 4sqGB1Ie9NSkVsUNV/aqzboaNLHavPY7HcjCbyxIXQi/QVEzo6OiGmELEVIjQnDEYIbk CoVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=EJJJARxNXYE8s9kG74Vkd+z9AVmvWaTcsF2BWxhXv0I=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=tf688zhmclKmpyGdxJqR4vTHyzW9091I+FAP6UOOe6iyGyPyLMM1KR/Bn+Wy+Y2W+H 2r8rohpmrMbd4pAtrH3TIa4sT9r8iay2w/h+o3Ecbzjh+DqWpH4QFJNxiNKS3rMpVV8v ORmHYiVWioTWU7iZoPTryoeVljk4rzFnTn/36NzS+5vrS+1SKFC8cxj1nCYyz1O/HYSd WyFPNxn8hl0sNFT01A/KWHZDnFoa5tHjMsPab/Sa437sY5J6YYxqV2DocsT8VZOgwLVk 9/PX5BwJzo0H0Fstg/MF6jmW/Hvy22LJNUkJ2eMd7D/1Jtye4S7Om1/NAyzbBlNa3CfO qZ9A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id t10-20020a05651c204a00b002d471e919d7si6891ljo.7.2024.03.14.00.30.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Mar 2024 00:30:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from hp15.localdomain (mm-209-185-213-37.mogilev.dynamic.pppoe.byfly.by [37.213.185.209]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 42E7UoA1021482 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Mar 2024 08:30:53 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v6 06/11] base-apt: Predownload packages to base-apt before install Date: Thu, 14 Mar 2024 10:27:23 +0300 Message-ID: <20240314073047.29465-7-ubely@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240314073047.29465-1-ubely@ilbers.de> References: <20240314073047.29465-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch uses debrepo script to predownload packages to base-apt repository before they are installed in buildchroot/rootfs. Signed-off-by: Uladzimir Bely --- meta/classes/dpkg-base.bbclass | 27 +++++++++++++++++++- meta/classes/dpkg.bbclass | 8 ++++++ meta/classes/image-locales-extension.bbclass | 5 ++++ meta/classes/image-tools-extension.bbclass | 9 +++++++ meta/classes/rootfs.bbclass | 4 +++ 5 files changed, 52 insertions(+), 1 deletion(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 361a7e9b..0df47529 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -11,6 +11,7 @@ inherit terminal inherit repository inherit deb-dl-dir inherit essential +inherit debrepo python __anonymous() { distro_arch = d.getVar('DISTRO_ARCH') @@ -125,6 +126,14 @@ do_apt_fetch() { trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + schroot -r -c ${session_id} -d / -u root -- \ rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt schroot -r -c ${session_id} -d / -- \ @@ -147,18 +156,31 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}" # Add dependency from the correct schroot: host or target do_apt_fetch[depends] += "${SCHROOT_DEP}" +# Debrepo context is created by target bootstrap, need this dependency too +do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap" do_apt_unpack() { rm -rf ${S} schroot_create_configs + session_id=$(schroot -b -c ${SBUILD_CHROOT}) + echo "Started session: ${session_id}" + schroot_cleanup() { + schroot -q -f -e -c ${session_id} > /dev/null 2>&1 schroot_delete_configs } trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT - schroot -d / -c ${SBUILD_CHROOT} -- \ + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + + schroot -r -c ${session_id} -d / -- \ sh -c ' set -e for uri in $2; do @@ -168,6 +190,9 @@ do_apt_unpack() { dpkg-source -x "${dscfile}" "${PPS}" done' \ my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${SRC_APT}" + + # End chroot session + schroot -e -c ${session_id} schroot_delete_configs } do_apt_unpack[network] = "${TASK_USE_SUDO}" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 3fa9f604..9aba205a 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -43,6 +43,7 @@ dpkg_runbuild() { export PARALLEL_MAKE="${PARALLEL_MAKE}" rm -f ${SBUILD_CONFIG} + debrepo_handle_controlfile "${WORKDIR}/${PPS}/debian/control" env | while read -r line; do # Filter the same lines @@ -111,6 +112,11 @@ dpkg_runbuild() { DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print) + locked_update_cmd=":" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'" + fi + sbuild -A -n -c ${SBUILD_CHROOT} \ --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \ --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ @@ -121,9 +127,11 @@ dpkg_runbuild() { --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="mkdir -p ${deb_dir}" \ --chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \ + --chroot-setup-commands="${locked_update_cmd}" \ --chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ --finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \ + --finished-build-commands="mkdir -p ${ext_root}" \ --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \ --debbuildopts="--source-option=-I" \ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE} diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass index 9149d643..4dd93f1f 100644 --- a/meta/classes/image-locales-extension.bbclass +++ b/meta/classes/image-locales-extension.bbclass @@ -6,6 +6,8 @@ # This class extends the image.bbclass for setting locales and purging unneeded # ones. +inherit debrepo + LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\ en_US ISO-8859-1\n" LOCALE_DEFAULT ?= "en_US.UTF-8" @@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download" image_install_localepurge_download[weight] = "40" image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" image_install_localepurge_download() { + debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge } diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 46bdf78b..4e5bd550 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -6,6 +6,7 @@ # This file extends the image.bbclass to supply tools for futher imager functions inherit sbuild +inherit debrepo IMAGER_INSTALL ??= "" IMAGER_BUILD_DEPS ??= "" @@ -40,12 +41,20 @@ imager_run() { echo "Installing imager deps: ${local_install}" distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + debrepo_workdir=${DEBREPO_TARGET_DIR} if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi fi E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${schroot_dir} ${distro} + + debrepo_add_packages --isarapt "${debrepo_workdir}" "${IMAGER_INSTALL}" + debrepo_update_apt_source_list "${schroot_dir}" "base-apt" + schroot -r -c ${session_id} -d / -u root -- sh -c " \ apt-get update \ -o Dir::Etc::SourceList='sources.list.d/isar-apt.list' \ diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index ddc5d12c..13e06ae1 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -2,6 +2,7 @@ # Copyright (c) Siemens AG, 2020 inherit deb-dl-dir +inherit debrepo ROOTFS_ARCH ?= "${DISTRO_ARCH}" ROOTFS_DISTRO ?= "${DISTRO}" @@ -153,6 +154,9 @@ rootfs_install_pkgs_download[weight] = "600" rootfs_install_pkgs_download[isar-apt-lock] = "release-after" rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" rootfs_install_pkgs_download() { + debrepo_add_packages --isarapt "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES} }