From patchwork Fri Feb 10 15:34:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 2633 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 10 Feb 2023 16:34:52 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f61.google.com (mail-lf1-f61.google.com [209.85.167.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 31AFYpeZ024820 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 10 Feb 2023 16:34:51 +0100 Received: by mail-lf1-f61.google.com with SMTP id x7-20020ac24887000000b004cb10694f9bsf2394104lfc.6; Fri, 10 Feb 2023 07:34:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1676043286; cv=pass; d=google.com; s=arc-20160816; b=mVEacGQYgx7kdGutjEnAq0DkZKCrHwTxg6K3NN+hH/jpMFRKNWdLQbavLSraJ/j1Vr XrYgV6DVE7f6YSj3vuEduaTU9BbNChFXbDyrmTGVHGRkXEbfTDFXqsDt2Ict7lQdoGoC Zw70PnhUlZSa2cXEc+pBU1qFRpjSlfxicXOVgfFyRF7/CF6Ys0I/JCyWSaYhzZc2J0lo hHCvchxXPZ8d7SZxZOygf09fDBL97K9GMuKmubaBfFBl9kOvnejANM87MBIvKjfi3lHB LDet3QC87iE2wYKwbedCirOs/HA4h8lzlXdnzdufJz4QX5A1P7HbONwkVhE0xk5GjVjs gAZQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:feedback-id:mime-version:message-id :date:subject:cc:to:from:sender:dkim-signature; bh=z6xlGd9+/0qS9cshwy+mXR7VBoDqJNgf3Xo2r3QM7cs=; b=chMCkDVvD+sQX/uG+LPjeE7wEr0Lazx+VoGpmU/EMbVy8Ql9Kf+NzmbyzttRqCL4Ll oifgv96HRa9PKQ8iVkof4adjcyqjIYwfRG/CfFfWSKheoJu4flMv3NBT+0QM9yDfqX04 +ekNekXxGgky0ZBzXQBPt0q7k9NowphKZqlFYLhfYNVEccbgBH0cdgQ02z0uxAuEwYbE yWpxNmzy0zwKTMPvUciTfIKt9lYk9rSIW6O9xfq54iQWJpfiCZxn3xb5bU0IRIOaiBIy RuPee/je6H++rYEy9/1L+obzj/xC/60OSRlfBvOIHmlmsdwFRWWIgLuBnEE10a/Wz0Si uMrw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lpeKwdfs; spf=pass (google.com: domain of fm-526363-20230210153444b39bd6e0cc7877d3a5-_b_ctu@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-526363-20230210153444b39bd6e0cc7877d3a5-_b_ctU@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:feedback-id:mime-version:message-id:date:subject :cc:to:from:sender:from:to:cc:subject:date:message-id:reply-to; bh=z6xlGd9+/0qS9cshwy+mXR7VBoDqJNgf3Xo2r3QM7cs=; b=BCl35Jy73jkvbR29xPOcf0Z8cEtTMSp+IYxDgpQBQHcaOH/lNcyBUvsk7ZLjeEUeDW pw8mg4YwKnHsTFghmFreh0A39dntZkyc/1vtpt33toZLU7GiRCnlN6UO7obD/qvmFnJJ /t0tUt4DE3s00bb9ji0FTrRKz0mOJlN8x4FrWvrCPhiviVunKp+m0eT+W+ZlKyOBcbFV bFOqpE1kxmC1hmwnQkDdxkaDh2amoT5k74foWBJhglrGVP/9ndx+2Go+GVQLV7ZJFOyK 01aTCJ0LUWKY5QXkIAyizjkSWut3pPPsJpWky/8KDssJ7Yng8j+jeZV3jw2Xpe3ac4W6 VfqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=z6xlGd9+/0qS9cshwy+mXR7VBoDqJNgf3Xo2r3QM7cs=; b=3wcd+Hvk3xJBrJo1mFFxk0g3Zb0rMb97duIgpXKeZfORiutDAZfg4XkdZheEf7zoay TGoz3vvPhOnpvaCjTfwcfobvjrnlA1W1T/NIVfoUXAR5GLP5ihx7zWstuqNAZFr2en/X pNs2IsRwsb6ySYUgV1CWFRKQ+pUf3KVLcf3Frz/JEZRiTxeTIM3QkWnd5XKWnQEaY02W vQx3Z8Yvm81TF2jozNiD+DRL03Hd3HmfidA6HScMeobkISes9x07BNQfQAJez8JecVt7 yBYfaxjprPprG8r1aIbM30ZTFGg6JALkuarub2HgjKWGkY+cCdYsVEvEbF9BnjE0Tvv+ PC+A== Sender: isar-users@googlegroups.com X-Gm-Message-State: AO0yUKWovBZu79/pc7J4c0n8VXA8kI+3VqyVZI2mraYfbTldE5ebZsh8 7MjrtUiyPm8snkZMOBS+i6I= X-Google-Smtp-Source: AK7set9dmxQY4PRwtLzIw/qjjDz0g7pWvf0WZfZ2TjGj6X6hc3I4rPB6emimO5+a4n7FAMKGZVEr4Q== X-Received: by 2002:a2e:7c0e:0:b0:293:2b48:d2a9 with SMTP id x14-20020a2e7c0e000000b002932b48d2a9mr1816344ljc.74.1676043286194; Fri, 10 Feb 2023 07:34:46 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:2526:b0:4db:2bf0:d4b5 with SMTP id be38-20020a056512252600b004db2bf0d4b5ls929464lfb.0.-pod-prod-gmail; Fri, 10 Feb 2023 07:34:44 -0800 (PST) X-Received: by 2002:a05:6512:21c8:b0:4d8:6540:a72c with SMTP id d8-20020a05651221c800b004d86540a72cmr5283191lft.46.1676043284718; Fri, 10 Feb 2023 07:34:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676043284; cv=none; d=google.com; s=arc-20160816; b=IcBnDPtqstTdv+3+9uyjdLmbN5osy5ktMSRCmW131PrVN2bWqV9rVlo0u0zrzlN7oc /HLXiO8zdxb4GPdjyt235NqY/WrB0MtbA+89qzdsmHWtZV8WJEnVhc46/YfMI3spKhMb txEbp7eQj+JxEJLp+lN4sN7fPN2rkO5jsVZi1+gJPQ3HzCFixV7f2UXlM7ttQ2IQEQeq haijT0xHWQx12VsMWYO4uXRk7ZgMEddx8zYyjmACCsAbsMYmVjVqunLJo5zvriInmfCZ mjBaBBUcRaZ3pMuVQpiaJrXuG1p7fppUVBCQBbH1vZR0wWx5or2Tw5fkufUaDhDJpCJs 8RqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=XlqzTjJBED2bihT07eNPe22HlTwnidJqD0sLtwHnLrk=; b=bRr6uzSe2DpBFM6TGKO/+b+dG3QV4/Do16s+wbwjnwZTu+fVvjklnhhCiBDertiSIo J1v1A4M+Rnhwg+Qbxc2mb/O2yLfEYviYMRAHaKFWLBB6C+QvgZNx/+t5QZrqPSmxO++x WL3r9o++X76qLpLYBU/RxOYoyk6fO8dvudHuyIyBjPuzHbfFSmRf/PDzmYvnd8V/lc0o INRL6XpwElYbjd2Eh1T3Sroua0jjZrgJCFqlJMiA1UjpZvVoWHLiMvU9rjBgBGJOmir7 0l8tYTrR4Ua+7HaktFRb1rkk2LQ4isswnHQh+NjtwXwYC3TOwCwy6/a9U3qoPn9qY/gT lz7w== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lpeKwdfs; spf=pass (google.com: domain of fm-526363-20230210153444b39bd6e0cc7877d3a5-_b_ctu@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-526363-20230210153444b39bd6e0cc7877d3a5-_b_ctU@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id p8-20020a05651212c800b004d579451cc2si264980lfg.12.2023.02.10.07.34.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 10 Feb 2023 07:34:44 -0800 (PST) Received-SPF: pass (google.com: domain of fm-526363-20230210153444b39bd6e0cc7877d3a5-_b_ctu@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20230210153444b39bd6e0cc7877d3a5 for ; Fri, 10 Feb 2023 16:34:44 +0100 From: Adriaan Schmidt To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: [PATCH] fix(isar-sstate): also handle zst files Date: Fri, 10 Feb 2023 16:34:34 +0100 Message-Id: <20230210153434.1024604-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-526363:519-21489:flowmailer X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lpeKwdfs; spf=pass (google.com: domain of fm-526363-20230210153444b39bd6e0cc7877d3a5-_b_ctu@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-526363-20230210153444b39bd6e0cc7877d3a5-_b_ctU@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= With bitbake 2.0, sstate artifacts have changed from tgz to tar.zst. Our isar-sstate script needs to scan for those as well. The implementation is backwards-compatible. Signed-off-by: Adriaan Schmidt --- scripts/isar-sstate | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 53d0541f..c14c2843 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -40,7 +40,7 @@ followed by one of `w`, `d`, `h`, `m`, or `s` (for weeks, days, hours, minutes, seconds, respectively). `--max-age` specifies up to which age artifacts should be kept in the cache. -Anything older will be removed. Note that this only applies to the `.tgz` files +Anything older will be removed. Note that this only applies to the archive files containing the actual cached items, not the `.siginfo` files containing the cache metadata (signatures and hashes). To permit analysis of caching details using the `analyze` command, the siginfo @@ -576,7 +576,7 @@ def arguments(): '-v', '--verbose', default=False, action='store_true') parser.add_argument( '--max-age', type=str, default='1d', - help="clean: remove tgz files older than MAX_AGE (a number followed by w|d|h|m|s)") + help="clean: remove archive files older than MAX_AGE (a number followed by w|d|h|m|s)") parser.add_argument( '--max-sig-age', type=str, default=None, help="clean: remove siginfo files older than MAX_SIG_AGE (defaults to MAX_AGE)") @@ -664,21 +664,21 @@ def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): links = [f for f in all_files if f.islink] if links: print(f"NOTE: we have links: {links}") - tgz_files = [f for f in all_files if f.suffix == 'tgz'] - siginfo_files = [f for f in all_files if f.suffix == 'tgz.siginfo'] - del_tgz_files = [f for f in tgz_files if f.age >= max_age_seconds] - del_tgz_hashes = [f.hash for f in del_tgz_files] + archive_files = [f for f in all_files if f.suffix in ['tgz', 'tar.zst']] + siginfo_files = [f for f in all_files if f.suffix in ['tgz.siginfo', 'tar.zst.siginfo']] + del_archive_files = [f for f in archive_files if f.age >= max_age_seconds] + del_archive_hashes = [f.hash for f in del_archive_files] del_siginfo_files = [f for f in siginfo_files if - f.age >= max_sig_age_seconds or f.hash in del_tgz_hashes] - print(f"INFO: found {len(tgz_files)} tgz files, {len(del_tgz_files)} of which are older than {max_age}") + f.age >= max_sig_age_seconds or f.hash in del_archive_hashes] + print(f"INFO: found {len(archive_files)} archive files, {len(del_archive_files)} of which are older than {max_age}") print(f"INFO: found {len(siginfo_files)} siginfo files, {len(del_siginfo_files)} of which " - f"correspond to old tgz files or are older than {max_sig_age}") + f"correspond to old archive files or are older than {max_sig_age}") - for f in del_tgz_files + del_siginfo_files: + for f in del_archive_files + del_siginfo_files: if verbose: print(f"[DELETE] {f.path}") target.delete(f.path) - freed_gb = sum([x.size for x in del_tgz_files + del_siginfo_files]) / 1024.0 / 1024.0 / 1024.0 + freed_gb = sum([x.size for x in del_archive_files + del_siginfo_files]) / 1024.0 / 1024.0 / 1024.0 print(f"INFO: freed {freed_gb:.02f} GB") return 0 @@ -716,7 +716,7 @@ def sstate_info(target, verbose, **kwargs): for k, entries in recipes.items(): print(f"Cache entries for {k}:") for pn in entries: - artifacts = [f for f in all_files if f.pn == pn and f.task == key_task[k] and f.suffix == 'tgz'] + artifacts = [f for f in all_files if f.pn == pn and f.task == key_task[k] and f.suffix in ['tgz', 'tar.zst']] print(f" - {pn}: {len(artifacts)} entries") print("Other cache entries:") for pn in others: