From patchwork Tue Feb 7 07:52:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: roberto.foglietta@linuxteam.org X-Patchwork-Id: 2614 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 07 Feb 2023 08:52:33 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f190.google.com (mail-lj1-f190.google.com [209.85.208.190]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 3177qWOA009356 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 7 Feb 2023 08:52:33 +0100 Received: by mail-lj1-f190.google.com with SMTP id a38-20020a2ebea6000000b0029065ed6963sf3296455ljr.16; Mon, 06 Feb 2023 23:52:33 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675756347; cv=pass; d=google.com; s=arc-20160816; b=GU9ASRsLrg0qmyM5tOe6E+GVMyjoyQ66PEryQB+g+2wSsYoEKwWSImWARgJXEkvLBS dIhwCrJ7t1nnpVJRvsTAl1N1gpgqJ4IqTQk7UiDUUZ2+PmeWYmo+75ENpKfWsmf2wH0u ApW1rhKWy92YydDHSh1H8bi9X2PLUE7xmtKCvCY7Tge4yz8gF6Zz8IktKX2Qy8BNQCpz dr8l+km3t8rB+iQtGD/KAFrqCwYiffg+7TtLMJ8+jbo0Gi1JPD3E7+ml7Po+wxqnmss2 k6WAFku5RbGHe5ARn/aOQav2HZv2Et4e0+HpZuYzj88GpEpxwtnV3OytDnixaZ1DukAX 7V4w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=QVV8d5x+0mIJLKLvAU9axE0WPK6VqWPOUVTe0JoVuMQ=; b=q4vQ/exZ2uVxfTvimO9qFGC2UqqL+NQlFhHrlfTz05f9dQNTtNl8xn5YUbdZYdQW2Z O7KbGIEFyyv1cXhR7xxB/Z2lDwRoINrP/8vNswNa2YJ2kpqY+u79yg2NWhjbJ7PG0CdN XjhVSiH3Ve7MsmCpvTkYCL/Fvqw9sMj41XPS+WwAyvxnqsUo6e8XVKWk+ZXuDTyCbnS/ 5ngvm6HsJq4WDgQI2H6FFTsd4OfaIwi8K54j1CSYD/PgRQjphO4JsLdPL2PoRY7ZVye/ qA9IvrCqBKCxew6L2JroEmRLrqjAybqqGA5kTwNUuOu3LYz3Dw3BC2DnOa5RSl5mioHr KkEA== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::171 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=QVV8d5x+0mIJLKLvAU9axE0WPK6VqWPOUVTe0JoVuMQ=; b=rUdP8pbAompDYra/CKMCaaMM/M9qYp+c3wArhiNmoHXhR5rPuU8ALDC6tMYuAoBI0d 5rZ4pEZz2ePynM1pPfCM9za1xR1WDt4Fa2Zt6+gIkcN4tzWyeQv5ejr3neev6e6PiMcS zb2jkoWzsy7g4b+fuGkYfmvjJMieX8m+YZif74w/YLAl75/oawcxC646Mw7L8GM6Ap3k Bjqyh7grqLGK0tJtkudilAF+8w0/fpHE86zB/X+z5G3IU/7m6EUIQaRJMS8sqYuDeB9d dgh6D3RcJ38OPMVxIaOTk5lOU7rIAPHv4bRRVDaMM7uNt8i2jwtPg+mSpNABs5Ui1Dz2 KIyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=QVV8d5x+0mIJLKLvAU9axE0WPK6VqWPOUVTe0JoVuMQ=; b=3B0Slygd+Ay/St9S/V0MJtQyzlgiDjYAKgePhPCZxFFGQ/CbyCXjuaR320dKd8ZjCE ngrkr6RMWTPl61d7iCAFBk/COcmU3xoGIJpJlwHfmVwGSdCr1ATdDvJRXRDSbaWyKKmr Eq7EInJR5Je0Vl71h4RiR60MjWx6E1xCh/b9r/s92xf9EXef9B9ddhBrg7OiFiZ1V+FO TaKYs8t9UIp1fJmeAoPoOJzzGxQx61kADJRUdqap6IHk/V6bGWphZykvcyo6vgBMWXLc /NbHro+NuBsBarmjRDfJMbFq8x2ZUFzGMrS9DQebO501LgNt+QYrkq01gpxjcdY2jVTT P5kg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AO0yUKVWSiOYvF0j6uEni6SqH1u6UTOFcLwoqY+05k1p2vFiCGT5EThA ri0Y6VKrzYi6R9rAZ9ULzkk= X-Google-Smtp-Source: AK7set9h857qPMTzNVIg3cBXJnid864ee02FKJhMcY5UsudP44lhmS60BgX7oT+bqFnzES1wItl04Q== X-Received: by 2002:a2e:9608:0:b0:28e:4072:ac5d with SMTP id v8-20020a2e9608000000b0028e4072ac5dmr353333ljh.36.1675756347449; Mon, 06 Feb 2023 23:52:27 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:d01:0:b0:290:51e7:45ed with SMTP id 1-20020a2e0d01000000b0029051e745edls1966046ljn.8.-pod-prod-gmail; Mon, 06 Feb 2023 23:52:26 -0800 (PST) X-Received: by 2002:a2e:8396:0:b0:293:fca:ffb7 with SMTP id x22-20020a2e8396000000b002930fcaffb7mr665547ljg.10.1675756346015; Mon, 06 Feb 2023 23:52:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675756346; cv=none; d=google.com; s=arc-20160816; b=myoOpkzSjuD8FkmqASw9vNYRXsCID3s65wfBdpyiK0f3z/vBJIfxcSbBqVY249JT6t uV3TBNj0fDO9TcsxFizjqF58zzT2pG7ysZ+EFPYRjINuNPPth1VXTM26FgpvhByx6SWO igJsW4LmZK0Ib/fG61TIEjmAde2Yrluvhg7jCfceOnG4wrPoen0qh8pwK1wGc8Sc2/pq pP42BJWL7IG6YmfZlDIpfanqtMrAwwjALsbhhTvdw4RvHHemaXp4A0jZiFasLWAzM0FK dVVcqe7e4bR6HP6fF6rc+04GECvNfSFlyqOR3n9su8oRa/pahLnUjLiGnxd+7zrPgXnx r0oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=Tu9n7DL6Lec5AHMmYdJsEZgyHgxHlhV1m0er18AaLdg=; b=CvW+OCF44AlmO2yfm7MS35Ip+OCWQ27ryQuG6emp6rllfOuvws9pMt+DH3b16h6dAG 0cvLA4/aJLAaOQMjHQb/0Wnd/VqOcUrwbLKHT0ZdTp1z7rAng/8AENKrmifhfZYxFYKF HkVessyHmqTcK2XW6Vjyqrr2ljqVLSG5yHWR8rSKefCCBTaLbI4Qa09PmJVNVikLosV/ qzdatLVI1AmHwFLfYVC2bYClbk5VfUrA4AZPkE1pJ9melirEMK3SANBOhWK0ClaJJ4RI 4TGAnJeWipbGyuObNiJbk5jvUkV1Itkbqq0La+7yj52ASv+HAMBUq9wd+VvSNRBl7w+V qrzg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::171 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from m-r2.th.seeweb.it (m-r2.th.seeweb.it. [2001:4b7a:2000:18::171]) by gmr-mx.google.com with ESMTPS id p22-20020a2ea416000000b002905672e241si428349ljn.5.2023.02.06.23.52.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Feb 2023 23:52:26 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::171 as permitted sender) client-ip=2001:4b7a:2000:18::171; Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:582:9dd8:9cc7:d7ab]) by m-r2.th.seeweb.it (Postfix) with ESMTPA id 05E363F409; Tue, 7 Feb 2023 08:52:25 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp Date: Tue, 7 Feb 2023 08:52:22 +0100 Message-Id: <20230207075222.390954-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Original-Sender: roberto.foglietta@linuxteam.org X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::171 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: "Roberto A. Foglietta" deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp The original class functions deb_dl_dir_import/export were using cp to copy debian package to the target rootfs but this approach is quite slow while using hard link does not work if the destination and source dirs are not lying on the same filesystem. Thus, ln -P should fallback to cp when it does not work (which is different from complaining on stderr). Moreover, these two functions have been reworked to reach a straight forward and more compact form. In particular, export function was using bashism to do some kind of comparison which after all is useless because copying back without overwriting just fulfills that part. More rework using sudo in a different way plus a corner case addressingi, in case the spia file exists for some other reasons. Rebased on the current next Signed-off-by: Roberto A. Foglietta --- meta/classes/deb-dl-dir.bbclass | 53 +++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7db25251..1fe9d40c 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -78,39 +78,46 @@ debsrc_download() { deb_dl_dir_import() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ + export sc="${1}/var/cache/apt/archives/" + sudo mkdir -p "${sc}" [ ! -d "${pc}" ] && return 0 - flock -s "${pc}".lock -c ' + export tf=$(ls -1 "${pc}"/*.deb | head -n1) + [ ! -e "${tf}" ] && return 0 + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - sudo find "${pc}" -type f -iname "*\.deb" -exec \ - ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + - ' + rm -f "${sc}/"$(basename "${tf}") + ln -Pf -t "${sc}" "${tf}" 2>/dev/null ||: + if [ -r "${tf}" ]; then + find "${pc}" -type f -iname "*\.deb" -exec \ + ln -Pf -t "${sc}" {} + + else + find "${pc}" -type f -iname "*\.deb" -exec \ + cp -np owner --reflink=auto -t "${sc}" {} + + fi +EOFSUDO } deb_dl_dir_export() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" + export sc="${1}/var/cache/apt/archives/" mkdir -p "${pc}" - flock "${pc}".lock -c ' + export tf=$(ls -1 "${sc}"/*.deb | head -n1) + [ ! -e "${tf}" ] && return 0 + flock -F "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - find "${rootfs}"/var/cache/apt/archives/ \ - -maxdepth 1 -type f -iname '*\.deb' |\ - while read p; do - # skip files from a previous export - [ -f "${pc}/${p##*/}" ] && continue - # can not reuse bitbake function here, this is basically - # "repo_contains_package" - package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) - if [ -n "$package" ]; then - cmp --silent "$package" "$p" && continue - fi - sudo ln -Pf "${p}" "${pc}" - done - sudo chown -R $(id -u):$(id -g) "${pc}" - ' + rm -f "${pc}/"$(basename "${tf}") + ln -Pf -t "${pc}" "${tf}" 2>/dev/null ||: + if [ -r "${tf}" ]; then + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec ln -P -t "${pc}" {} + 2>/dev/null ||: + else + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec cp -n --reflink=auto -t "${pc}" {} + + fi + chown -R $(id -u):$(id -g) "${pc}" +EOFSUDO }