From patchwork Fri May 26 07:00:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzimir Bely X-Patchwork-Id: 2846 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 26 May 2023 09:00:45 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-pf1-f184.google.com (mail-pf1-f184.google.com [209.85.210.184]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 34Q70h1C013388 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 26 May 2023 09:00:44 +0200 Received: by mail-pf1-f184.google.com with SMTP id d2e1a72fcca58-64d1c53cad8sf721282b3a.3; Fri, 26 May 2023 00:00:44 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1685084437; cv=pass; d=google.com; s=arc-20160816; b=HJfTGvYclwtkqogTIUB8P3T2kLmuVEjGsojRUudecYRwzGKQDZ/JOhBmearIHePtHG a4VAuImkgdS+64RsWLnVeu1SYbO9HeAbS2hRf2IZonMpDxNJeluM5ivf/Ad71oe0QPUV KUYFoQHbeKKuwAs171DJ2X05it8iRhtUD8OU7S6aS71sNSv7VtLAoAmg7Gzz/83ypXu8 NhKKeBPHMMW9W2E4ZbFXcj53+ZzfqNwZ1iXbUYrR0ZuzFeTiQNckKVt6cZLcczYMDR9Y COFTCdFVJ82Lx1GoJhhNC9OwJ87RYo+VMFMN/a0LNr8uOCh3H3C3RTn6gN9AO8cDa6tA ubKg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=Nk6KuRptsxGojLLiJJjnVRmoHvx+/Iv3x48VUesX0OQ=; b=FRktt7seyq9tFD2ybKNJPbEm1N4WmIAJjz5Sp+2DeEzcHca8DYejgPkLP+aFzipGXt TcEIIG8kfe4yhTm158rnv2wfPJ3AGfc6jMFZSMAYjpFm3NcgKaWBN/bvzrnXTuRQ9F5O CbDMuu0O+aBy1tuat3mnUm2ekhvkkS2ipG10yx1gmBA2LkQwkqqCbK6kjOVwj3HL6sfK G8DPeXgFtmQ2AAi1c6fHppWmozdYPpbw71aH4QNGpJ3wJBmCtMry1hgVo1ahqSbMrmNQ OV4jnp/lobyf5nB+Zs2DAHzCpLfG1/7S6Rw3OrSEojHN4mMCbz6MeSOyZSF5M04AYaf2 H8cA== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20221208; t=1685084437; x=1687676437; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=Nk6KuRptsxGojLLiJJjnVRmoHvx+/Iv3x48VUesX0OQ=; b=kB/10HnSNR2Y30VZ6BGYq592dJzWs6U6gyh8vkBAk1zlo/AiWwHcNMfhDM+xv5at5Y c2QO0pEGOoPpwc3igEMsQkN/m5W5RCginkZAPrinLShujp9Kc4H2uQ/UbLHxXK5bXVtq sbHb3fcs44j/K4PIx+QO6u2xdxfJbDzBoRfnVwoX4liN3+jdKzd/O7lHxfv+IznEVSpP Q3ERXG2/t1a/gFfuwZUaXXdQcCId80eL2PQldlwJEJUcaP8pC78rpuhIn2Cxp0pdyzUc Bjp0zAzY4trQU9LI27mUCcklD8VCThcq07XFDtxlY6lbyxxk6DEKeGKa8QTbuf5dhbdl JznA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685084437; x=1687676437; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=Nk6KuRptsxGojLLiJJjnVRmoHvx+/Iv3x48VUesX0OQ=; b=Z/I+Bs6dnvGpZqeQ59RQ0SMvs3q/J+osyXqsfRr55v9uEsz8P2G/k6nrFsdsQaEmL6 9Z0I3qPHktPw91BPC3fFSe3sFgFo0IejJtk/AIZ0LpG/q19GZ/NuVDJ3u/vzPAEU44Ue uzzu4oxR/spqxDfDe4xy5ZLyrzsr/6uWQC5g8DQ0ZSoDcYCcdIYipF9Dqfzyfo842UYh Y+8w9ohmRxmsZnUnCzNVygMG7KdFwBgTzfZR4KCcyTmcb/z4FzjcZwPe7/bLZl4RY1vi 7A8Rbj5JXLBZ6T/xTUaAbphoVSb2SzY5pR7wRbaqbH+gL0qtvfaoGgFZ8twEMawDNtSg wCwg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AC+VfDzJ8AQVGNSBk6p1297gBrbTFJ2mv4xl61Y594fygtmgfqDFkhN5 SEoZPyrWau8NzdXVaj2P2l0= X-Google-Smtp-Source: ACHHUZ7nmiBtuO2FLizYVLVYqV6gb/gdmghVjnyJzz36XXjltj6nxH3v0sF85f91DeeJu0YCMsSGfQ== X-Received: by 2002:a05:6a00:15d4:b0:643:a029:5b3f with SMTP id o20-20020a056a0015d400b00643a0295b3fmr530488pfu.2.1685084437338; Fri, 26 May 2023 00:00:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a62:cdc1:0:b0:64d:6575:dc3f with SMTP id o184-20020a62cdc1000000b0064d6575dc3fls1928607pfg.0.-pod-prod-08-us; Fri, 26 May 2023 00:00:36 -0700 (PDT) X-Received: by 2002:a05:6a00:1809:b0:643:aa2:4dcd with SMTP id y9-20020a056a00180900b006430aa24dcdmr1943830pfa.16.1685084436478; Fri, 26 May 2023 00:00:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685084436; cv=none; d=google.com; s=arc-20160816; b=Rxpdz2hZklj6/iM9D5GDifor6RJVfYuA7CYZReU7fWuZWHm/zwOWHmEvuwZ4kd/nis fFqGseeN/MlKjHe5pcfFSxyz+P83lkS8sXp9Y1XpW7L34hjnNtJTCVIFlc+q9z63ktR2 EUfKR8NBmG4YQgiCp42QkS8C5ReOe//k1MPCtPLjb0z2RT8EPmMOnwrHEkWdIrmspG+t 8ur0LyZbsJ5tPxxaO4f3Rm49Lk9ZQ81BgpsuMOsvDb5PtOAp6y3sfTzutNGQTkglm6gj lr7r8m9c5FW6eE5LfgRN9PLl7PmFcFKLhrIc37jq4UAsz6Q1R6hyNw7nj5TzudK8cVNx w18g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=IlQXBCUH1GZmpqnuzrWVS/60kim65u/urNsIsS0ZRRI=; b=OyHlO/nPynSE03ptDPBn6CrDxzuMmLQtMeKQyD/+w6Q3wE2Vn15WGWTEqf+YtC67jX M24d8Nznnpt1jH/9vC5ae7EsekJRK8sq/0zbk7QmethbCVsMeYy5CHCbUvGyUXVhdOk3 oj7uOU2tGv9w4Iy/H63YSlKnm/rzdu1SkDcNYW4HSOWdx6N8ecUxSdTJ3T17fQ9ecGUK zjjYHn+5G6hRie6STxBq1nQi8NjBmCMzcoeAOISZmi2mVWjcs1L6y1Y/ShnfZ2Ea15/3 g7RfINeRVOlXEtBtCb7IppmuhZvAOp+A4Z5i4PfX0NzUy52FEtX2aICyuHL2ooM+lEur e/kg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id o9-20020a056a001bc900b006438069d21bsi293196pfw.1.2023.05.26.00.00.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 26 May 2023 00:00:36 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 34Q70R3c013306 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 26 May 2023 09:00:30 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v5 08/12] base-apt: Predownload packages to base-apt before install Date: Fri, 26 May 2023 09:00:23 +0200 Message-Id: <20230526070027.16890-9-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20230526070027.16890-1-ubely@ilbers.de> References: <20230526070027.16890-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= This patch uses debrepo script to predownload packages to base-apt repository before they are installed in buildchroot/rootfs. Signed-off-by: Uladzimir Bely --- meta/classes/dpkg-base.bbclass | 36 ++++++++++++++++++-- meta/classes/dpkg.bbclass | 8 +++++ meta/classes/image-locales-extension.bbclass | 5 +++ meta/classes/image-tools-extension.bbclass | 9 +++++ meta/classes/rootfs.bbclass | 4 +++ 5 files changed, 60 insertions(+), 2 deletions(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index c3e47a69..faf2ec77 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -23,6 +23,8 @@ python __anonymous() { d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True)) } +inherit debrepo + DEPENDS ?= "" RPROVIDES ?= "${PROVIDES}" @@ -126,10 +128,25 @@ do_apt_fetch() { trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT + # Begin chroot session + session_id=$(schroot -b -c ${SBUILD_CHROOT}) + + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -d / -r -c ${session_id} -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + for uri in "${SRC_APT}"; do - schroot -d / -c ${SBUILD_CHROOT} -- \ + schroot -d / -r -c ${session_id} -- \ sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}" done + + # End chroot session + schroot -e -c ${session_id} + schroot_delete_configs } @@ -139,6 +156,8 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}" # Add dependency from the correct schroot: host or target do_apt_fetch[depends] += "${SCHROOT_DEP}" +# Debrepo context is created by target bootstrap, need this dependency too +do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap" do_apt_unpack() { rm -rf ${S} @@ -150,8 +169,18 @@ do_apt_unpack() { trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT + # Begin chroot session + session_id=$(schroot -b -c ${SBUILD_CHROOT}) + + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -d / -r -c ${session_id} -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + for uri in "${SRC_APT}"; do - schroot -d / -c ${SBUILD_CHROOT} -- \ + schroot -d / -r -c ${session_id} -- \ sh -c ' \ set -e dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")" @@ -160,6 +189,9 @@ do_apt_unpack() { dpkg-source -x "${dscfile}" "${PPS}"' \ my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}" done + + # End chroot session + schroot -e -c ${session_id} schroot_delete_configs } do_apt_unpack[network] = "${TASK_USE_SUDO}" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 5c8996d6..ea26b530 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -32,6 +32,7 @@ dpkg_runbuild() { export PARALLEL_MAKE="${PARALLEL_MAKE}" rm -f ${SBUILD_CONFIG} + debrepo_handle_controlfile "${WORKDIR}/${PPS}/debian/control" env | while read -r line; do # Filter the same lines @@ -101,6 +102,11 @@ dpkg_runbuild() { sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}" DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print) + locked_update_cmd=":" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'" + fi + sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \ --host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \ --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ @@ -109,8 +115,10 @@ dpkg_runbuild() { --chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \ --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="ln -sf ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ + --chroot-setup-commands="${locked_update_cmd}" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ --finished-build-commands="cp -Ln --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \ + --finished-build-commands="mkdir -p ${ext_root}" \ --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \ --debbuildopts="--source-option=-I" \ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE} diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass index b48bcec8..aa4874c1 100644 --- a/meta/classes/image-locales-extension.bbclass +++ b/meta/classes/image-locales-extension.bbclass @@ -6,6 +6,8 @@ # This class extends the image.bbclass for setting locales and purging unneeded # ones. +inherit debrepo + LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\ en_US ISO-8859-1\n" LOCALE_DEFAULT ?= "en_US.UTF-8" @@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download" image_install_localepurge_download[weight] = "40" image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" image_install_localepurge_download() { + debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge } diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 7c84505f..f98cd759 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -6,6 +6,7 @@ # This file extends the image.bbclass to supply tools for futher imager functions inherit buildchroot +inherit debrepo IMAGER_INSTALL ??= "" IMAGER_BUILD_DEPS ??= "" @@ -21,14 +22,22 @@ do_install_imager_deps() { fi distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + debrepo_workdir=${DEBREPO_TARGET_DIR} if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi fi buildchroot_do_mounts E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${BUILDCHROOT_DIR} ${distro} + + debrepo_add_packages "${debrepo_workdir}" "${IMAGER_INSTALL}" + debrepo_update_apt_source_list "${BUILDCHROOT_DIR}" "base-apt" + sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ apt-get update \ -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \ diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index cedde239..47f07129 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -2,6 +2,7 @@ # Copyright (c) Siemens AG, 2020 inherit deb-dl-dir +inherit debrepo ROOTFS_ARCH ?= "${DISTRO_ARCH}" ROOTFS_DISTRO ?= "${DISTRO}" @@ -148,6 +149,9 @@ rootfs_install_pkgs_download[weight] = "600" rootfs_install_pkgs_download[isar-apt-lock] = "release-after" rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" rootfs_install_pkgs_download() { + debrepo_add_packages "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES} }