From patchwork Wed Mar 5 13:11:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cedric Hombourger X-Patchwork-Id: 4075 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 05 Mar 2025 14:12:13 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ot1-f59.google.com (mail-ot1-f59.google.com [209.85.210.59]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 525DCBFY016724 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 5 Mar 2025 14:12:12 +0100 Received: by mail-ot1-f59.google.com with SMTP id 46e09a7af769-726cb11e531sf5426313a34.2 for ; Wed, 05 Mar 2025 05:12:12 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1741180325; cv=pass; d=google.com; s=arc-20240605; b=Vqwclh75N8g466WVFEIE1qw3J/h7piwiTJY5MLyp9C04nRpxobzsItx3fpI0/veBDb 4G28VvutSB61FiAxsOcTWfbc4p1vWa5UPPYvzsfbhX2dsjimBa3dnC5ZeuVneK6qmY5W VWgMc0AJyZOXQ5eg0uXhgbpKvGFUfRFU0ujEqj2F+Bs+eSLkzhGDzRfgybmfLLpAU2eg KZ1PwLmrOOMi4YLElKUDAaXG5cvrHrQAjpYkG8pbdBdOuFctv/RR0K9q2eqzaBnqWu7F QK5tyzRZYEyakdOIXtWTTOTXq7jSTFAJoAHnLnbP3B6FvlWZyvpIn04cq0y5BdWcKy8m S/5w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=SqJHpdua24r6VIJ5pUeZm20iLMb9elVe0uNYLfTYn1I=; fh=jUmV3bi6YeCzYsVa5wRuf2nRFFFxjB1RtkKekrjcd1U=; b=EuSVZkyjboJvtBrXX8gxfVXmALAQSF5EnNzpPWGjK0ZZroixZh97TfWw0CfgePu5qd G5Jtw4jMzn86rP8DdiHrdQs1a26VjaRKkeSA1pG16MlV9ssyITdSD1ZurrA596vfIMn9 HC+RbHQOU5qMkLLAL4/ndKlMWxWGYIb8rHZXRKFCZ+9miZBYMAbZh1lvnQwJ4JFM6CPY hJHyPnp4okyfYAEzKcfH1CZgG4B8LpZG1ap51+pmLmJBNptEiU+NLG2GqFIG6Apy8eat BcDXPN2PUkaSID5dOHkQ0Y2Qi72pDhgtDjTJC5Z3T4clqH1PcLkDezQXxLfwbBwtSoV+ T73A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=LjFIfQuj; spf=pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1212295-2025030513115927fdf11129f6fce058-n8aP0s@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1741180325; x=1741785125; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=SqJHpdua24r6VIJ5pUeZm20iLMb9elVe0uNYLfTYn1I=; b=RxVGVvXUE8LQAUSiAFRtQ1AjDsjp6wEwbBxiUqRZmPRIIlyHVlRTgPTXZgNKKAIGS7 24ewrqBrPs29wdCbrplN+s1771NUAQ5Dc5PC9e9h5/uU+uBE3Z35hqc8spLDYDslv4Sx VhF1QYQM1XA06u1QYPgYKkETIqgVvH7unC6M+rs3sa4qA/56vj88389jZctl5Pp0z8Qm QFeDekLtd9H5LMZYBg26veLNFstxt+aJq8atYSlgun4+Xfupy93Dr4OMXD3F+43/E9zZ IxRqYV1ERYmOl4iKXYhNWjaOiD6iPLbK5eP/MUDTQh+3PfK0N4ZO7Fpbguxpf9xhYp7c K6wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741180325; x=1741785125; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SqJHpdua24r6VIJ5pUeZm20iLMb9elVe0uNYLfTYn1I=; b=bwl/qZtPnsqzRQDfCYXvAi7/N8pwoL2qLBNgioEBQx4p0Z3GzwB0h85Iq1kuMdlGff 5mVkoas0qFcjz6E9+kBxHEDysgy/xJBDzP2oHbT0y1MRW8kNL55sbLqVXwYBXsPqHawn Rts//wv3TBduQ/STgO46lAq6He5z0HT/wvmpEDO69gFGHZuTLz+PECTTGhkTcTmMCBkc PSZP/Gov5Ev5OGPDi4rZgbcMSPmFtLcNVHL6O4HHNfC3rtWgnKumLBZ8qJYG8o/xIvID 1jB0s+lVmKRHC7VVrTgJ0CufSEEIUHE9Qg6SbuuI3mZc+gtC1BvhMFM4D9Pw7j6RvPwH o5KQ== X-Forwarded-Encrypted: i=2; AJvYcCUqIz3xsfg/7TwmMUJ9ul+iD4bRCbFe3/rpy7LlrNNsuFya75kdnjskxn+z8UezbyMI7BoadLo=@isar-build.org X-Gm-Message-State: AOJu0YywEXRdQxIz5cJ8q8zL4n3xZMOUiB52O0EMBY7K1KTQDBziEUoq +kBmstnuVonaYuW0cb9VQLOzf8Ed/JaHHc+sqfDd7YEdtbcofFvV X-Google-Smtp-Source: AGHT+IErmgQFnQc8I0k8MOu+dJgrWKu864A+lQWjcn6v9X8MADLPQdQ87lkiuv2q9qxze1bPLGe4PQ== X-Received: by 2002:a05:6830:7004:b0:72a:b17:6570 with SMTP id 46e09a7af769-72a1fbf8e51mr1691682a34.11.1741180325350; Wed, 05 Mar 2025 05:12:05 -0800 (PST) X-BeenThere: isar-users@googlegroups.com; h=Adn5yVGsf0OsFXk/2Nx9HLg0Ujz9E0dIIGsQGGzOxVX8YHwxuw== Received: by 2002:a4a:c70d:0:b0:5fe:b6d6:7500 with SMTP id 006d021491bc7-5feb6d676b5ls1458134eaf.2.-pod-prod-07-us; Wed, 05 Mar 2025 05:12:04 -0800 (PST) X-Received: by 2002:a05:6808:2396:b0:3f6:6d5a:8df7 with SMTP id 5614622812f47-3f6831e27c9mr1684702b6e.26.1741180324264; Wed, 05 Mar 2025 05:12:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1741180324; cv=none; d=google.com; s=arc-20240605; b=gAtsUNYf9sbk/rOEHnFtfyOvcxgylMiGhfdGR8fq8dXZ1n5sVZczZdfnKrMniaNbok t7zSyENj7HRYrwueRmIQgfkMyxkv30exXXmhz0QlfZqB5mANBUFGhiJrEiYyHtAlQ1/e cmT+Ca3ueMY477NYcYpYiqxc2M/pwhGiP7+VcJTo1ogs0UJCxcMDZhK/c3RjIz+Ris3H rQ1njkJvg3Gph8FNkXgOESMSUO9gNC1WhtMNgNGOxstdRlatdoO9ypWfgWcEBtNvC45s rbNljqp1iv+ir1AFQqNJ/fH4TtWFn+aCSfwLYzDFmu6aclHxNqI5ivv6+VMeL740RBdk EN4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=sN+xxxbr8E441BajSuylkncXPAfli8OKo7mAO4Z4URs=; fh=yuBAxkaMYSnaF7r+lbEZdWO7eG7htBqkQ2A7TcD4+Dg=; b=d1uXRaxCJe9tw+XV/kSG0Q834g/YINwpEaZBafj03eAnT5JXLgVCIGyM+yVEsqWgnR n1BLFCyrWXqauSWiHKDUrogk+Ytp4LtI15tohcxwXGn+/w1qzl/FhDwFHa26B1YOobBO EQTGZL4tWQmf+Qfmf8KaK7eTDkSlyRAPp8T5WFxwhZ/q3MJmmwW+0m1Y7GNepyUtPECw XAPcpI469kf4aeb5xmrqfWeX98xcPN75FwYJRFaSs5z3pE/MfqGMGQyFZwra1QjKRMnC yFDTzcY9ZiQZZEKVRWRBfRnWKLEBksicMxMjtDB2NQfhv+9BKNi1eFRhZxGQTvXgb1aU AF7w==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=LjFIfQuj; spf=pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1212295-2025030513115927fdf11129f6fce058-n8aP0s@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id 5614622812f47-3f5506c4cdesi754398b6e.2.2025.03.05.05.12.03 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Mar 2025 05:12:04 -0800 (PST) Received-SPF: pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 2025030513115927fdf11129f6fce058 for ; Wed, 05 Mar 2025 14:11:59 +0100 X-Patchwork-Original-From: "'Cedric Hombourger' via isar-users" From: Cedric Hombourger To: isar-users@googlegroups.com Cc: ubely@ilbers.de, Cedric Hombourger Subject: [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download Date: Wed, 5 Mar 2025 14:11:42 +0100 Message-Id: <20250305131142.2717692-1-cedric.hombourger@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1212295:519-21489:flowmailer X-Original-Sender: cedric.hombourger@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=LjFIfQuj; spf=pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1212295-2025030513115927fdf11129f6fce058-n8aP0s@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Cedric Hombourger Reply-To: Cedric Hombourger Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-2.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,RCVD_IN_PSBL, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Several calls to dpkg-deb are made for each single .deb file found in downloads to parse individual fields. This approach is terribly slow when a large amount of .deb files are found. Use apt-ftparchive to produce an index of packages that were found and a simple awk script to produce a (sorted) list of source package names and their versions. Also avoid using sed to remove Epoch from the version when we are trying to determine the name of the .dsc file: we instead use a simple POSIX parameter expansion to remove everything up to the first colon Signed-off-by: Cedric Hombourger Signed-off-by: Cedric Hombourger --- meta/classes/deb-dl-dir.bbclass | 62 +++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 27 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7ebd057e..53ce4538 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -5,23 +5,6 @@ inherit repository -is_not_part_of_current_build() { - local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )" - local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )" - local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )" - # Since we are parsing all the debs in DEBDIR, we can to some extend - # try to eliminate some debs that are not part of the current multiconfig - # build using the below method. - local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \ - "${IMAGE_ROOTFS}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )" - - [ -z "${output}" ] -} - debsrc_do_mounts() { sudo -s < + # Version: + # Source: () + # + # If Source is omitted, then = and + # if is not specified then it is . + # The awk script handles these optional fields. It looks for Size: as a + # trigger to print the source,version tupple + + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \ + -a "${DISTRO_ARCH}" packages \ + "${rootfs}/var/cache/apt/archives" \ + | awk '/^Package:/ { s=$2; } + /^Version:/ { v=$2; next } + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) } + /^Size:/ { print s, v}' \ + | sort -u \ + | while read src version; do + # Name of the .dsc file does not include Epoch, remove it before checking + # if sources were already downloaded. Avoid using sed here to reduce the + # number of processes being spawned by this function: we assume that the + # version is correctly formatted and simply strip everything up to the + # first colon + dscname="${src}_${version#*:}.dsc" + [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || { + # use apt-get source to download sources in DEBSRCDIR + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \ + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}" + } done ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"