From patchwork Fri Feb 10 12:42:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2630 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 10 Feb 2023 13:42:20 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f62.google.com (mail-wr1-f62.google.com [209.85.221.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 31ACgJlG024209 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 10 Feb 2023 13:42:20 +0100 Received: by mail-wr1-f62.google.com with SMTP id g8-20020a5d46c8000000b002c54a27803csf355067wrs.22; Fri, 10 Feb 2023 04:42:19 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1676032934; cv=pass; d=google.com; s=arc-20160816; b=I7uQ4k4k4qYCzv4ibCGvfopMbprau6px8KIA+tXqSWSew30bHhw3M0dn6P2W9LmaDJ lElz5TE4Xxve8zch6c4xhAbT596dKR5Lup9OXUgd+MvbXQtAw9Nc11J976GevY8T+EiA TkzJJlRTqm90DSxSc1cbheN1F2VcNJiY6xLLfGdN9C/R0LL+hmgDDY7GFrlZ9n+gRlKT QAk+8gHqCfj30zgryJqY3zxX3kHKpJQDYA1TuO1OhpZmubfmxi5yXijXUKPLFf8edDqw 727hpIFTjwyh3j4c5CsqnjnOgjL+1Sug4LlCz7mkEOTF9nvneFXy9hAMqk1IvBz/ZDsB m1MA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=ZmcnJ2Siz7ldYqwILpiQIQuK3OSZOXd44EG6rR4lQus=; b=OndU1YoywZMqNpqmW5PCku00nHphC8v1SZi/1hGWQMEB8CwiTOncdv1Z/xQGVwR/dE 4mTgCHh0+OJjmeUDw+agBIFUXs1s3zc2CjNC9gXnhWGi+z3SGvmU53jBlgbVYovbUKzh lkrUJiZVGGSGC7rlN77A6JyRghyOhPl6iYmB9WDJi0ZDBt0v1J0AThFsjLPPoYhZL7P8 PDIkHeVmyPaWsJu0igT+VqGppy9bhWNGZypweRKXICWcJSAP74G0BfYueIn4QQm/E4IP KjVqek6+RJ/RInPUdjRohAyRxXBLGmAzxXyI6ngLWsVObkT3U8nlYlWgW5DANj5jCDEg cK4Q== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=ZmcnJ2Siz7ldYqwILpiQIQuK3OSZOXd44EG6rR4lQus=; b=jcmPVecjjI45o11oJYcIX450uOfAWSNLwqQ7MYz3YVvt2A7N+4HVuSIg4wrA79RdbH 1218EWwJc7k7i885RvLiSN4VySMZCTcyBh8u+O+YEg4jLJJ5kN+X82hO0E6zYmCi4y+3 eov5+Uuuy++5ZohEcUMbgzWwD9qohOG48TkpLuBa9yxBpB1TM35mLs6zYJE4I0A9eN2a zEAkHlzKAB2sv08pvLiZxY57hDvd201Zn146Z10dCOPLrSr1fZPo/vi28Ois/dw+dBbR 7+FjY4GSmL7/yFr/sdQheluywwvOTVLiBAYll/jiHIuTy5KBTonuwjTwCJ2QHI6g1joh fABg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=ZmcnJ2Siz7ldYqwILpiQIQuK3OSZOXd44EG6rR4lQus=; b=srmy7RHOtVm0NZiCPWzkOv/Dbuw2pZ1328WRcGwKyQkVRuxSoENdAt0lGW8ZgJ1Nll PK67uTPyhXRyEI0XlUBIdLf0pDbZuGkODnWd5D99Fmfu68KFW43ihewX2eurAKf2YZ1Q pbCwgAQabJ91oVAOj5IA8bElA0QAu0bdwBO77fCuxi8D12QodEvYgOkRzhe7aC93DaDA /kGoWgnqiTK7temEe4X7YWXZIgesUWY2q7avlq0gnnMKZdixkeiuaz6z2phLkIdj15JC SmarEYYmvI6otNavSmrA69iNOQXqSAZ7SPqwwTJQzUevqSDUzgfazpQ8zHe4c2Qz1Afx ewJw== Sender: isar-users@googlegroups.com X-Gm-Message-State: AO0yUKX9KacoasB0qv8oZn4YF+eJQSlgGTEn9NTMKgXuTV40Kk5beXdN Ag6xO3Cr06zgiUskxPWQnCA= X-Google-Smtp-Source: AK7set/SmfkVAlcp5J45Rj3pw2EgUom1GpVT48hUg+nuv1mP6ym6NA0bQ1uG7mW2xsttMMjRf4EUoA== X-Received: by 2002:a5d:5645:0:b0:2c3:e34b:5355 with SMTP id j5-20020a5d5645000000b002c3e34b5355mr693830wrw.301.1676032934414; Fri, 10 Feb 2023 04:42:14 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:58cd:0:b0:2bf:ccef:53a6 with SMTP id o13-20020a5d58cd000000b002bfccef53a6ls228434wrf.0.-pod-prod-gmail; Fri, 10 Feb 2023 04:42:13 -0800 (PST) X-Received: by 2002:a5d:4fc3:0:b0:2c5:4a89:f60b with SMTP id h3-20020a5d4fc3000000b002c54a89f60bmr1449197wrw.38.1676032933116; Fri, 10 Feb 2023 04:42:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676032933; cv=none; d=google.com; s=arc-20160816; b=wDdTWJeorkKi6HdXhzH9PRnOGdKF77SbtPgZxY3LrbJAWjKi8x1k/6LOdJPn8uF0GE fWSDxXLiusm8fm6+Z2VQUUvy86ooz2h/XKDG4cGSyDKqUCa6ROkO4JVn6qafbkqlNLxt du3YMvnqLlhuRPdhjaiMpDO6euBQNIcXBbkFXDIltOPdypcHInRDFUWym//uY5y7w3KU s1DQvRxbmWj3IUkEcxzN2PGvE7jEonPK53hy6Fm/66xMpDwKJi9g+pol5HtmEYD8MAhY hjKjkllAmDhPUTk83NIrTYOWHoRnH9fqGZfqTmXZm6FS/0+6iDXghSHx41Vbn1inBkFZ EKSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=RBXCOTC4TduH4xhi45tf+rYieAq0UrdXMg3sTsboY0s=; b=WA/WhZiTD6YXWvluU0lx+BBeLkAbN2rmhfBrB4cdQFtFa3V9JnZvUKSSpFfkrqTi5I pd+ucsb822aT9jiu0KA6nPUu9EVXYAjUG3ZZ8crof2TRfg6yH816VjBQ04lJbEpnm3L6 sfQc0QYbPCmoBbFoHmszpr87mRPk2tjioCt4EwVBQAa40ia74eH4pM1J76mfXyWu6VJL r7CA8RL+wIfkQelGPrv76hzHdNlbA077rYMXonXAw5ZUUR0SM7ahvRv7ePYCQKtDZFYs jYNAZq2JPiNlt5HHspi1TMR7yeUk+I6r6+idmbNcDCbwy5ZQGt88fk7raeXDuiFv7QeH zNkg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from relay04.th.seeweb.it (relay04.th.seeweb.it. [2001:4b7a:2000:18::165]) by gmr-mx.google.com with ESMTPS id bg10-20020a05600c3c8a00b003db0d2c3d6esi435424wmb.0.2023.02.10.04.42.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Feb 2023 04:42:13 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) client-ip=2001:4b7a:2000:18::165; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:e47:5fa:2d03:1a15]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id 5859C203F9; Fri, 10 Feb 2023 13:42:12 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v5] deb-dl-dir class rework to use faster ln -P or fallback to cp Date: Fri, 10 Feb 2023 13:42:07 +0100 Message-Id: <20230210124207.1609382-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp The original class functions deb_dl_dir_import/export were using cp to copy debian package to the target rootfs but this approach is quite slow while using hard link does not work if the destination and source dirs are not lying on the same filesystem. Thus, ln -P should fallback to cp when it does not work (which is different from complaining on stderr). Moreover, these two functions have been reworked to reach a straight forward and more compact form. In particular, export function was using bashism to do some kind of comparison which after all is useless because copying back without overwriting just fulfills that part. More rework using sudo in a different way plus a corner case addressingi, in case the spia file exists for some other reasons. Rebased on the current next and bugfix about checking the destination not the source file. It does not delete a package but touch a test file to link. Signed-off-by: Roberto A. Foglietta --- meta/classes/deb-dl-dir.bbclass | 55 +++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7db25251..21925c9f 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -78,39 +78,48 @@ debsrc_download() { deb_dl_dir_import() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ + export sc="${1}/var/cache/apt/archives/" + export tf="ln-P-write-test.ok" + sudo mkdir -p "${sc}" [ ! -d "${pc}" ] && return 0 - flock -s "${pc}".lock -c ' + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - sudo find "${pc}" -type f -iname "*\.deb" -exec \ - ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + - ' + rm -f "${sc}/${tf}" + touch "${pc}/${tf}" || exit 1 + ln -Pf -t "${sc}" "${pc}/${tf}" 2>/dev/null ||: + if [ -r "${sc}/${tf}" ]; then + find "${pc}" -type f -iname "*\.deb" -exec \ + ln -Pf -t "${sc}" {} + + else + find "${pc}" -type f -iname "*\.deb" -exec \ + cp -np owner --reflink=auto -t "${sc}" {} + + fi + rm -f "${pc}/${tf}" +EOFSUDO } deb_dl_dir_export() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" + export sc="${1}/var/cache/apt/archives/" + export tf="ln-P-write-test.ok" mkdir -p "${pc}" - flock "${pc}".lock -c ' + flock -F "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - find "${rootfs}"/var/cache/apt/archives/ \ - -maxdepth 1 -type f -iname '*\.deb' |\ - while read p; do - # skip files from a previous export - [ -f "${pc}/${p##*/}" ] && continue - # can not reuse bitbake function here, this is basically - # "repo_contains_package" - package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) - if [ -n "$package" ]; then - cmp --silent "$package" "$p" && continue - fi - sudo ln -Pf "${p}" "${pc}" - done - sudo chown -R $(id -u):$(id -g) "${pc}" - ' + rm -f "${pc}/${tf}" + touch "${sc}/${tf}" || exit 1 + ln -Pf -t "${pc}" "${sc}/${tf}" 2>/dev/null ||: + if [ -r "${pc}/${tf}" ]; then + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec ln -P -t "${pc}" {} + 2>/dev/null ||: + else + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec cp -n --reflink=auto -t "${pc}" {} + + fi + rm -f "${sc}/${tf}" + chown -R $(id -u):$(id -g) "${pc}" +EOFSUDO }