From patchwork Sat Mar 22 06:15:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cedric Hombourger X-Patchwork-Id: 4123 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Sat, 22 Mar 2025 07:15:47 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f61.google.com (mail-ed1-f61.google.com [209.85.208.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 52M6FkbC012071 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sat, 22 Mar 2025 07:15:46 +0100 Received: by mail-ed1-f61.google.com with SMTP id 4fb4d7f45d1cf-5e6a64bd1ecsf2403839a12.2 for ; Fri, 21 Mar 2025 23:15:46 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1742624141; cv=pass; d=google.com; s=arc-20240605; b=g/uFodLixRQvG+Z2PhCxZ7vy07mk7ov18fd8FrEEJCNNGEVw/xEh54kMe2YNNZcSzu kSDFhI7oSGAsd/MIYuacsgmRAJyDJFAje5Ss0gYD1ItVeWp3h20Vx7cJFHhYvO96hh84 PlBK2sBTuLYhlUpVum0Q829Ya/4msKc8rML9XWS7OyZclW8IwTnTDTk27a9EGYxFSNkM /cr5rKRhbeA6ITRMZsAzW13ncL45wp/btVGyKGeWBVjLf/lAnlT+PusQyKdfwB54fRKS mVC7XvqL+XSYY681imBBo2BGUhl/U1WZ2aPDBxXUslBv/m+ps+MLNORt2B9k1hnCpAax VL9Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0TXHR1k2WNnQCyTEOL6CnUXKYtyjKSUYYPvKQr0cW44=; fh=EPy0eKUJlANEkBgzexGByPcqL1akRqZiUtKp3GggHNc=; b=V5uSzJP+kG3ZLgTgFd58Z97fGfmdtRUcFlmdquwfr5bCk+0koNRBXBCSWM03g/Qe7i ZWxEBu3PVAJzmeGgdwMpNwSweSUQxFsPfaW5em9+bR9JxVrobW3/Pd0vMoczO284367l mpttpXETifQqNgKWVYxKQm/xKT4eSQZ6aMxUYTrg3FgFIuulEj7exTt3qRb4g1UHpPPi 943wHFKG3sv0FizKUHqH9fVjv+KCQcWIf53pibqNwBfIda+P4XkW950jYZ43uvdzaXaY tTxxvD8NMKC/MZNht8UcrJIsNxth6VGu9iiQI7j9al3erCboI94P1OvKJEGOg4I1Eniy GYWQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=TqKX+pn8; spf=pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1212295-2025032206153659e06e379c321ff20c-Vp4v_P@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1742624141; x=1743228941; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=0TXHR1k2WNnQCyTEOL6CnUXKYtyjKSUYYPvKQr0cW44=; b=ZsXXgvm9VbvRkInvJMBgYf9Fm+XpAAVqp6jiZ4PXYhKKt66MxH/iOeHJiFw1HTKxfv kk3Q4pJRzIqjYNINfDnZvKoBUk8nVVY+s79TYjLTSO1JAjXp2Tbey1RM9Fpcd/Uue81S Z+5sD4bIm15Qvx42js5efkI5WGpYpr4HyrC+yuyahQqgegYvak4IDAh+zlanHLE9ZuSx RD/+sbo5ZdMcNXwfFczwOfhkqsEeoQhkcOyHpVyRtLawODT5ia4UbzpizEvwZowrRI4t tgJKUJuTOb8/nyrtjiXF/3UenXmk8qKZdJcOedQG6ZYOGwhN+JmqsOKTMd3ca1OAECl2 40Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742624141; x=1743228941; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=0TXHR1k2WNnQCyTEOL6CnUXKYtyjKSUYYPvKQr0cW44=; b=W5LVyzOZcj8WXbKlAiK/a1WjcnoxiSfPxd6LKe2095ZaZ763N0LoDPVEFChqhVDDfR Eo2CX6d5dj7Kmo7DvTdKuxg32bmx2N1c8iZLsAXND4EH8UFk+PpkXg3CoRnKmYfp5hW/ AHPss+EEee4uvId2vE576lZIWzmGuEeD186Z3hLFf6poSouDLk6AsdAN9IzZxrYXB+lU qLPP1Tby+4otSABtlSsXQPUx2KU4Jkat1zOZJSNZZV/AHcwau2fN5+P8LDvmIK+hBHuU qXio5FbnY3OzKC5fHwPs7lclhL1i1aBZ/dUpVU7Mwya7+6kDdYkmfSyHvsw9EpdLkQcE jwxw== X-Forwarded-Encrypted: i=2; AJvYcCXMAT7qasBX2fGdVRGLKi5zs9yq/aOIz9rq5m71LrixS5/4XhXsfc483AHmJ2LSZTuUzqFkTgM=@isar-build.org X-Gm-Message-State: AOJu0Yx+Q9ymljF2UEb5wQFR5eWsobIm0PWVWIhagcZAT/PnEwqxDeQ8 VZgTUDNxnW9LyutiTTvejwM8CLyxdp/fY57rzHHbQPJ8vIxRCnqr X-Google-Smtp-Source: AGHT+IG3zbJUbWJscgRgylx1sz5CdLFusx3Nggtn/xrRMi8fduGA0VUW4bhMRmE9y60lcrseh3k1oA== X-Received: by 2002:a05:6402:42c6:b0:5e5:9a2b:167a with SMTP id 4fb4d7f45d1cf-5ebcd468ed3mr5484722a12.17.1742624140115; Fri, 21 Mar 2025 23:15:40 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAI1LAJbTtEbvEoM6BHldT3xKU8YPt/NNs6anEKr5A5siA== Received: by 2002:a05:6402:184b:b0:5e5:b9a7:2c27 with SMTP id 4fb4d7f45d1cf-5eb9c4bcc0els267152a12.0.-pod-prod-02-eu; Fri, 21 Mar 2025 23:15:37 -0700 (PDT) X-Received: by 2002:a05:6402:1d4e:b0:5e7:5b30:3c4f with SMTP id 4fb4d7f45d1cf-5ebcd433b34mr5166353a12.14.1742624136720; Fri, 21 Mar 2025 23:15:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1742624136; cv=none; d=google.com; s=arc-20240605; b=H0dTlpIs1STLo/J3XjKeAraN/5udHFbw/xOb9Yu5impTMfpBM8znXvO12WfXka0uFu AIpHeugizaeofGDDriQw6Tq+alyaO8yrZLvVroY3/rduEqg5GywmBnZuTtRMgV+Ck4bq qhm06nYgJnygSH8W28GApA5QQB3v+cLpX5FEHM4l8SqMJJ2YqCODLn1gISDYVSrWGGd5 HPDSxUWii+J3iimz3h9gU/tMb9uMDgf3ZmvrDVsFV0a+e7scWC/oCO8fff6IIThgWs8y VwZ3b79jJYoflU6hzjy38VxJnWHSELYNWZF5wd6Zhm+awVTE/N2yqZsjGlA4G38dqOaq 1ZCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=lLJio5i0tNngSt1Yjd7FH08MYQ8vTvR6B+69Gmw9R74=; fh=yuBAxkaMYSnaF7r+lbEZdWO7eG7htBqkQ2A7TcD4+Dg=; b=Vk+tcPCQ8POG7ykZr7RlHew0gsIrBfJjgQKr1qde5IKUEnkUsbJyU8ujUFhoYuT00w 2EJPcofGy3jgeaOyDXbwUZLmpzb12rh+4BUkDfCNQLoEX7LYiFLE+Abx+LWmKkYwjD3v 8eiKwaDiIzCIsoXT3Ccs7EgDINH2kom1+Sh6L98r6C+CqF+KytGC1SiODwpb/4uoXPp5 PTIFFMwjSso1MKbTMQl/PSLk2G09gXYCVVITpD8fBYJbyIb0nm2mxzW8JjKTQFfsrN/c ks1LimgPLpWUC2/wU2LyaW3kO0p8KhJunu7ShG/P9DnooLUGZBWoTHjSwrxjqMI/7aEv iM/A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=TqKX+pn8; spf=pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1212295-2025032206153659e06e379c321ff20c-Vp4v_P@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net. [185.136.64.228]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5ebcce05169si30181a12.0.2025.03.21.23.15.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Mar 2025 23:15:36 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) client-ip=185.136.64.228; Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 2025032206153659e06e379c321ff20c for ; Sat, 22 Mar 2025 07:15:36 +0100 X-Patchwork-Original-From: "'Cedric Hombourger' via isar-users" From: Cedric Hombourger To: isar-users@googlegroups.com Cc: ubely@ilbers.de, Cedric Hombourger Subject: [PATCH v2 1/1] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download Date: Sat, 22 Mar 2025 07:15:19 +0100 Message-Id: <20250322061519.2522459-2-cedric.hombourger@siemens.com> In-Reply-To: <20250322061519.2522459-1-cedric.hombourger@siemens.com> References: <20250305131142.2717692-1-cedric.hombourger@siemens.com> <20250322061519.2522459-1-cedric.hombourger@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1212295:519-21489:flowmailer X-Original-Sender: cedric.hombourger@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=TqKX+pn8; spf=pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1212295-2025032206153659e06e379c321ff20c-Vp4v_P@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Cedric Hombourger Reply-To: Cedric Hombourger Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Several calls to dpkg-deb are made for each single .deb file found in downloads to parse individual fields. This approach is terribly slow when a large amount of .deb files are found. Use apt-ftparchive to produce an index of packages that were found and a simple awk script to produce a (sorted) list of source package names and their versions. "apt-cache dumpavail" is used so we only attempt to download source packages for versions that are known to apt. Also avoid using sed to remove Epoch from the version when we are trying to determine the name of the .dsc file: we instead use a simple POSIX parameter expansion to remove everything up to the first colon Signed-off-by: Cedric Hombourger --- meta/classes/deb-dl-dir.bbclass | 78 +++++++++++++++++++++------------ 1 file changed, 51 insertions(+), 27 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7ebd057e..75877750 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -5,23 +5,6 @@ inherit repository -is_not_part_of_current_build() { - local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )" - local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )" - local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )" - # Since we are parsing all the debs in DEBDIR, we can to some extend - # try to eliminate some debs that are not part of the current multiconfig - # build using the below method. - local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \ - "${IMAGE_ROOTFS}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )" - - [ -z "${output}" ] -} - debsrc_do_mounts() { sudo -s < + # Version: + # Source: () + # + # If Source is omitted, then = and + # if is not specified then it is . + # The awk script handles these optional fields. It looks for Size: as a + # trigger to print the source,version tupple + awk '/^Package:/ { s=$2; } + /^Version:/ { v=$2; next } + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) } + /^Size:/ { print s, v}' \ + | sort -u +} + debsrc_download() { export rootfs="$1" export rootfs_distro="$2" @@ -54,16 +55,39 @@ debsrc_download() { ( flock 9 set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do - is_not_part_of_current_build "${package}" && continue - local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )" - local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )" - local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')" - local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc") - [ -n "$dscfile" ] && continue - - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \ - sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}" + + # We need temporary files for our lists of source packages + # trap exit of this sub-shell to remove them (this script may exit abruptly + # since "set -e" is used) + avail=$(mktemp) + wanted=$(mktemp) + trap "rm -f ${avail} ${wanted}" EXIT + + # List all packages known to apt + apt-cache -o Dir=${rootfs} dumpavail | debsrc_source_version_filter > ${avail} + + # Use apt-ftparchive to scan all .deb files found in the download directory + # and get the pairs that we wish to download + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \ + -a "${DISTRO_ARCH}" packages \ + "${rootfs}/var/cache/apt/archives" \ + | debsrc_source_version_filter > ${wanted} + + # We now have two sorted lists: source packages we want and those known to + # apt. We will only consider source packages that may be found in both. + comm -12 ${wanted} ${avail} \ + | while read src version; do + # Name of the .dsc file does not include Epoch, remove it before checking + # if sources were already downloaded. Avoid using sed here to reduce the + # number of processes being spawned by this function: we assume that the + # version is correctly formatted and simply strip everything up to the + # first colon + dscname="${src}_${version#*:}.dsc" + [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || { + # use apt-get source to download sources in DEBSRCDIR + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \ + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}" + } done ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"