From patchwork Tue Feb 7 07:37:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2613 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 07 Feb 2023 08:37:56 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f57.google.com (mail-lf1-f57.google.com [209.85.167.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 3177btTU009265 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 7 Feb 2023 08:37:55 +0100 Received: by mail-lf1-f57.google.com with SMTP id bp18-20020a056512159200b004b59c4fb76bsf5982977lfb.2; Mon, 06 Feb 2023 23:37:55 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675755470; cv=pass; d=google.com; s=arc-20160816; b=lbWm7mzBf2U5E8QBUfk1+zoWxTZ0HDmMI/FnSYWKwtcoZSifhZ3hh78dtubXHIiM9t Zx7stzgDXq4olA1jUEo+9hg5upGv/4e1U0UG600pZyaN0uvE7J7dS34/o+UFf14E2txG LIxXCEhLtXnPj20Pmr1nj1wMnQZA3n8sGjvFfsRQLLCOMqnlxKF8twaueyprArngLWlh apa2F7ncpTKGPxPNm5JOgUlOUZR4rhvpcwLKG3FBTjVF2fEUbxotJySNrmJjI8SHyg2b s9oJ/tkKQYXnpbyXxrghftZRIqP2aFA/cISZ1cMne3B50+thGDCngp/h5zXGrdz4RGke sFRA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=8nB5TqNZhWufF3kRUlgVqUDdNfGRtZY6f+ZC1/LN4zw=; b=yrIMd5qEv93lGysqwqH0wSgvZwG3YReCBrfHOlHSAfVyYObwQV4bKU6WU60HiSSej2 LrJtkeiEldqMXDOVjxIsJwfmUgBQqq2LcbOsjUK/E2JRtdim4q/VQT7dAB40j1POUxH6 0FZZTTsz8CdLcp0pARKNCLyvqQkwQK1Z1ZNAsFkIoklEzF/L7hA8O4buqbaYM/otCI6P oAvl/uUnL4Q4h8ryn4O6A0J+VXjlP58GEKeK8NqLx9byFLiB55qvw9XsgOLOHXGT4g+V 5iNbf4ai01ioan97pA842JImv8qn2/O9RW8z78liWUdu6vZpZkH6ITKidZG1gdc3QUE9 817A== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::169 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=8nB5TqNZhWufF3kRUlgVqUDdNfGRtZY6f+ZC1/LN4zw=; b=rjjVPTkcXhKYrvx7OqxS7uTZkszRKoDUMdxrCoL75L4Y4FpiIl8ITtk1qHRPKtvDpC /6IEn+6inOOr+J957pXIo8yZnuOCQYT9zXNVHrxqM2RzjofoqR9jzkHRrBu9sgTmU29d Evbxq6Y7BlhByGkvMJllsB9RKkEtqQh3xVzZEYLkivTP6nHIQnVwTy6oQBOTmsBTO0Tc A5x29vr/bdPhE6fz40bpJXj2Epoos/9hJaV/T3GqWLrZO5hNI5mgErPc1WPViCNFcXx0 f97g8n8PO5+JHGxSJOFSAmzpo9uC5cyKZ6omBi+1i5tu4WlTaADsFm2OuXX/OPXQt7wy lZKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=8nB5TqNZhWufF3kRUlgVqUDdNfGRtZY6f+ZC1/LN4zw=; b=64ak6+7inWLj2LTZkMpfV/NzOt6p4mH1prw0HX+MjYpsVfCf7UETRGtAv8/SqZQqyn fA2U6ibKYkz0DUd1MY688NMxQiJiXmSJJ9qzGwLK+H0I9dTBMFhEZEHuWOh6Yt4Db/Jl jlApqZPX1rX1chZF+9gOMug3jdcnCoTq5gspkyjK3ZjWIWmy0BKXN5bYOsH6MtaQ/ozF z6pHCj7WMUh7DdwcpHMDrQGH1AaS5YNMl29yyj5Ey4D+ioG8pc7ux0XyELRBZYaI+HVD EDS5NDUqrlAUCrBrhwtQpz91FSCd2MTawiqa7JU0FlzFVZxg6oAeehxU+4OZYhe5IW5C 5c4A== Sender: isar-users@googlegroups.com X-Gm-Message-State: AO0yUKUkG9j71IROHJsSAWVUATd/4UQNrhhnGzY+I7GByrRbqQa+KduX JSZSY8HK+wECNt7GE1Hi/Uc= X-Google-Smtp-Source: AK7set9LQoQoV9XhWY85xVuRcbwGjzHu6dIfpsLHghsGv4Yg4WS45xY9JfPkeDqLyZM1CGPuVCNMSA== X-Received: by 2002:ac2:5dd4:0:b0:4cc:8589:595b with SMTP id x20-20020ac25dd4000000b004cc8589595bmr365700lfq.0.1675755469851; Mon, 06 Feb 2023 23:37:49 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3d1f:b0:4d1:8575:2d31 with SMTP id d31-20020a0565123d1f00b004d185752d31ls2281516lfv.0.-pod-prod-gmail; Mon, 06 Feb 2023 23:37:48 -0800 (PST) X-Received: by 2002:ac2:5e8d:0:b0:4b5:aa85:3874 with SMTP id b13-20020ac25e8d000000b004b5aa853874mr545077lfq.68.1675755468199; Mon, 06 Feb 2023 23:37:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675755468; cv=none; d=google.com; s=arc-20160816; b=RwrRDA5sOcpaX6EHcM6JyEyDHg7RctND96SPNS9P6VZkqR0qTrAO9IoRuRfG5fVj8H gOYluwqsKNRgnmvBMQ5jIx8KdxtrFKCjIPuMGwZxyPpOfg+0HYNKYulDFuwSR5gWPYd8 9O9Ua8Nii7f8cc+FzYDtG7lwA6SIkO1DACZOhGKT2gbkNJevfQZREWvypjOYGXG8Yw8E 7qUglmlOTQ9RWoybgYCkBwQrpx9zKuEDjFFwdBnnr3I7umEBvrWdfZ74WAfYFGCHH5Iu veJf7z0C2dZTH2Q6aAwh1sGrO0XRrSJ5BsfxN8RP3AnllG+XG2u+WgMP1CFrjBeFdHOZ hXVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=ZDiAA8F8XVGqDzTZubEttHLM/8H5GUNzWaFEH756/l8=; b=WygCUW1wKo350ycOYCO3NG0OotKeI3JX6qe0hP9elL5iyJoFbXn3GZwk7ytcYNGeF1 IdEEcLMCRzyzbZd8U9D8UJPS7h2lE/bfzmTJeEVc0qdEtYG1qFFDMLhEiPHU4SCVGO91 746RNP1Xv4E6w3Ude+zU3fo59V3xxJoexzdSXakvcyJmn2isorcAnIwbn19Bgfy5OfdM t2sFBqFkv4DIDJIm4XTDeTBBcV8KoDntr+lCV8N596VoHL6Bz/uH+7LyQQ4bKm32q1fI jqYjhWosB2jegprTu8k//Hu/VWOoPUz8/aNfIQKaI/nGZyUsOxg6NKGu3XzF/bkFzDXy ZRKA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::169 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from relay08.th.seeweb.it (relay08.th.seeweb.it. [2001:4b7a:2000:18::169]) by gmr-mx.google.com with ESMTPS id k31-20020a0565123d9f00b004cb0f0982f3si605756lfv.4.2023.02.06.23.37.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Feb 2023 23:37:47 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::169 as permitted sender) client-ip=2001:4b7a:2000:18::169; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:fccb:83:3c8e:9265]) by m-r2.th.seeweb.it (Postfix) with ESMTPA id 9C7413F1EB; Tue, 7 Feb 2023 08:37:45 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v2] deb-dl-dir class rework to use faster ln -P or fallback to cp Date: Tue, 7 Feb 2023 08:37:40 +0100 Message-Id: <20230207073740.389604-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::169 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp The original class functions deb_dl_dir_import/export were using cp to copy debian package to the target rootfs but this approach is quite slow while using hard link does not work if the destination and source dirs are not lying on the same filesystem. Thus, ln -P should fallback to cp when it does not work (which is different from complaining on stderr). Moreover, these two functions have been reworked to reach a straight forward and more compact form. In particular, export function was using bashism to do some kind of comparison which after all is useless because copying back without overwriting just fulfills that part. More rework using sudo in a different way plus a corner case addressing, in case the spia file exists for some other reasons. Signed-off-by: Roberto A. Foglietta --- meta/classes/deb-dl-dir.bbclass | 53 +++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 3b1517dc..1fe9d40c 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -78,39 +78,46 @@ debsrc_download() { deb_dl_dir_import() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ + export sc="${1}/var/cache/apt/archives/" + sudo mkdir -p "${sc}" [ ! -d "${pc}" ] && return 0 - flock -s "${pc}".lock -c ' + export tf=$(ls -1 "${pc}"/*.deb | head -n1) + [ ! -e "${tf}" ] && return 0 + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - sudo find "${pc}" -type f -iname "*\.deb" -exec \ - cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} + - ' + rm -f "${sc}/"$(basename "${tf}") + ln -Pf -t "${sc}" "${tf}" 2>/dev/null ||: + if [ -r "${tf}" ]; then + find "${pc}" -type f -iname "*\.deb" -exec \ + ln -Pf -t "${sc}" {} + + else + find "${pc}" -type f -iname "*\.deb" -exec \ + cp -np owner --reflink=auto -t "${sc}" {} + + fi +EOFSUDO } deb_dl_dir_export() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" + export sc="${1}/var/cache/apt/archives/" mkdir -p "${pc}" - flock "${pc}".lock -c ' + export tf=$(ls -1 "${sc}"/*.deb | head -n1) + [ ! -e "${tf}" ] && return 0 + flock -F "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - find "${rootfs}"/var/cache/apt/archives/ \ - -maxdepth 1 -type f -iname '*\.deb' |\ - while read p; do - # skip files from a previous export - [ -f "${pc}/${p##*/}" ] && continue - # can not reuse bitbake function here, this is basically - # "repo_contains_package" - package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) - if [ -n "$package" ]; then - cmp --silent "$package" "$p" && continue - fi - sudo cp -n "${p}" "${pc}" - done - sudo chown -R $(id -u):$(id -g) "${pc}" - ' + rm -f "${pc}/"$(basename "${tf}") + ln -Pf -t "${pc}" "${tf}" 2>/dev/null ||: + if [ -r "${tf}" ]; then + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec ln -P -t "${pc}" {} + 2>/dev/null ||: + else + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec cp -n --reflink=auto -t "${pc}" {} + + fi + chown -R $(id -u):$(id -g) "${pc}" +EOFSUDO }