From patchwork Fri Apr 11 11:29:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4167 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 11 Apr 2025 13:29:38 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f64.google.com (mail-wr1-f64.google.com [209.85.221.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 53BBTbOL017846 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 11 Apr 2025 13:29:37 +0200 Received: by mail-wr1-f64.google.com with SMTP id ffacd0b85a97d-39c184b20a2sf976605f8f.1 for ; Fri, 11 Apr 2025 04:29:37 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1744370972; cv=pass; d=google.com; s=arc-20240605; b=kBI5zpMB+CwPCpfhFOG0Iu9O4d95YCh/Ic4wyWUdPU+ef8olFAgxUO4d6ZYjZlhrVQ 2C/VgQXNi3mpKz47bHwSPPcNVQ2Ab2tLvMoHX9OSvXSellghu0PeSnb8uwo01W7gE1V2 hTOtWCA1S64RjW9izaoNkknsfOqb0n1pBfaz1WMGlIscGTuftJCrdxLdaPVsN39LHnHd koy89ZXW7BvGMobGpRIXuEsoUxEKeuiK9PF3xAVXCKchqMip17PS11buKk7Tb9DvbpXN k8fBHkQQiRJuRAytEvDngFdDK5nvs4L43S/GODCBVfMf660R/U/ujQoembF9VDY3Qfog Ofaw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=KXPYoX4jCXM+soR6GeTkF6B4PRd5EgOmwASl8qsEk3M=; fh=EbYpB9KYHeEbZxLPs9FeQbLAzZkGr0T10sAuoLtI7M4=; b=SmhZCiRUZAg9Gf6/nGPDy1oyn71OrCZ1+QTHxTfy8nQ7K7OGGbt9Cs8qpVFFJdO9uF UT90Ms2Nz0nzhb8ApUn2cjL7y5fYXuVC0jCQ2CDl6YnpY79UVD2C56y8rf0/qdMvT2gY rvThHqlQiJI0B6LeChkLEzY+bWyG/i/AW/WfEAo1XuRTL/Kj0uAOeEreeGczoNeMGAib NkTSAgG4TABmb2LtAao9cNMKpcl0eDyWef1tqMjW3aNe9X092ftuK4yFEOOur+jLIt9G tLVlECPKi14b304dbOS4tL2WAnf/ETFn5jiUCG+0unNxy2v1943wOLsJpUWMXA4pMhtD jYjQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=OUEYk4SG; spf=pass (google.com: domain of fm-1321639-2025041111292708ff1ed24cdbf860b0-8ei7hj@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-1321639-2025041111292708ff1ed24cdbf860b0-8eI7hJ@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1744370972; x=1744975772; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=KXPYoX4jCXM+soR6GeTkF6B4PRd5EgOmwASl8qsEk3M=; b=ervNooXRKP2Zjo5ow2l40x3uDgv4k9/ZX1A26Nuxl+D4NuF496FwSQOG0feFwpRF80 rgCpoRW6WANKeW6gQFUgw+whRQak2lupWsCI+zHMPJ9AoLfTO7r6Ups1l9vpflhy+Urf erq6U/mytpsG4wReQV3ALrgS6b5O4UNCizTcUO4rBXY8fdvZ1dmM+PfJJRwy1VdWzeu6 z8qACjvA9nHqCWbdyVuopMMyGHWw5ZzWjr9l2yonMG0wY7VYCRRcTkqDtmrGAXLrmRO4 mFWr4pKxTi9ekavXusnIfwkvgT47J5JfMRycO4M/7ZGCdzoymRbQuwCJa/noZ8zDoDVh nlvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744370972; x=1744975772; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KXPYoX4jCXM+soR6GeTkF6B4PRd5EgOmwASl8qsEk3M=; b=sRe/rFTgW6qzMciIYZCBkFU4/SaerFlDRrPX/cFjsGjDTHnzC1yqsjYRxdu4ZpBLLe Kiv5NUfIdoyTNk/WhEgY3Kf3dLE+U8RL8NNFOpYC4f0pV4ubgHJbJN5K+gLuFmBexz18 5yG+VdCX9eNRJF5q3zWmchihlK/3S+lVGSSohvEYaZ2+Z7ysI+EtSH4drRDi6wlowqO8 gzHVsPjxCwgktBbcmSCZ9JAwZjBi0dShL9JbB6AllQp6bu5AISL9vd5zEW+jPpqqXQ4M NJZM2qgAsTMPQe93rKN5YDNA6RhnGUUGlY3D9khWcMTcH+iloAH0EwGjVwTRDuzi6r9j VDiQ== X-Forwarded-Encrypted: i=2; AJvYcCWaW2oXoEJfPVFByY094O0xAwb+/OHc3VFFcv9jZz0BRuADCrkn05lKhrelryXi/bVNZJveLsM=@isar-build.org X-Gm-Message-State: AOJu0YyjuOecfhQRNSOBK6nkIPJEa1wX5YzZ48VjLJc8DfbSlO8o7PJ0 RNNIcKrXjwfLMAC1C+2/VfpquFnwTdTH0xzTUuoi5hZIvH5C61Jy X-Google-Smtp-Source: AGHT+IEEykbowBvRiPIlE6ZS/ug/SU9mrcFpDCHwbSTSFpOvRR1djYVNCDoQCh+MdiNofr7TQrrBvQ== X-Received: by 2002:a5d:5f92:0:b0:39c:266c:d82 with SMTP id ffacd0b85a97d-39ea51ec9cbmr2012922f8f.10.1744370970569; Fri, 11 Apr 2025 04:29:30 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAIuKJYlyD51jZQZ+/tnFMp4Zfsd+BjfA/AvmBbp9XRSRw== Received: by 2002:a05:6000:2a2:b0:38f:21ce:c3c8 with SMTP id ffacd0b85a97d-39d8df966adls724575f8f.2.-pod-prod-06-eu; Fri, 11 Apr 2025 04:29:27 -0700 (PDT) X-Received: by 2002:a5d:648f:0:b0:38f:5057:5810 with SMTP id ffacd0b85a97d-39ea5215119mr1724778f8f.25.1744370967519; Fri, 11 Apr 2025 04:29:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1744370967; cv=none; d=google.com; s=arc-20240605; b=B55/6GLq+BvycRFBNxF5poO2jNlr5TRrqLGArczYhusDOY/vXBNRI2liBjBoe7irP9 iczD982RfAxWhMgwcxuZd1eo+pA0ShADppPRKb1/zJWXF3bgiaz3+L5VUVQprCm/Dx/M 5XJ5b+4Pqsma9vRBShDLjKDg8RQyHcwI9zreeoZza8avr4MNP2UgxsxYLk3WuDfbUsJm qYCvOQLRyU2/MLejQ/S05+cPsCf7Gk1ZZ5jO6gqRwmU0Xe3OgJ01wRrxRcuxwAEXA1KF JIj+hBPWTyLMdGS5TXGFfTN4pfS1eE7iMcU0HpMnRT/zVJ/ceJ25bnN34yYxgfY7bru7 +FGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=UYTLzOgpYgLgZTjEyMF/Nb9g8riQI6zeEW+OnDpZ2wI=; fh=Xwx5KAQd7M5XPJvQVz6xz/z//08kOEttrlFeLBySLJ8=; b=MLLAbZ6mXBlV0YJ6NOPukQs7tpJ4AJFtOL0jzDj7nC0DJyuFeYSYLLN443/oXh6VLG /+9l+qXQP9h3AlsC0t+8es1j8VzSwHqzV8q2Qn9tCJgwQ2W78vKCKBrHjrv+QffKiniU 64BOpqsYWm4lDwXwfWllbWDK8KeUCD1ke9V6f73DXWdahrJSYDw9p8kdQkMTjDRu3zWJ ybbAMDjWuUL02krcULiAPrkMVyqb6OfzkVdcYWpSgbCsY2nR0PfeMlhtcg9uGbGCJUs4 RpAFMInZIyKoUGCoWx3jBhvLBJY2ooDU9vhFKCrKCC7nISacFaX8iSO+He4pQQLbblbO i6dQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=OUEYk4SG; spf=pass (google.com: domain of fm-1321639-2025041111292708ff1ed24cdbf860b0-8ei7hj@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-1321639-2025041111292708ff1ed24cdbf860b0-8eI7hJ@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-39eaf43a6a3si24042f8f.8.2025.04.11.04.29.27 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 11 Apr 2025 04:29:27 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-2025041111292708ff1ed24cdbf860b0-8ei7hj@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 2025041111292708ff1ed24cdbf860b0 for ; Fri, 11 Apr 2025 13:29:27 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: adriaan.schmidt@siemens.com, Felix Moessbauer Subject: [PATCH v2 1/1] isar-sstate: add filter parameter Date: Fri, 11 Apr 2025 13:29:22 +0200 Message-Id: <20250411112922.3992939-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=OUEYk4SG; spf=pass (google.com: domain of fm-1321639-2025041111292708ff1ed24cdbf860b0-8ei7hj@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-1321639-2025041111292708ff1ed24cdbf860b0-8eI7hJ@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Currently all sstate operations are always performend on the whole cache. This is problematic if some files should not be uploaded or a clean operation should only be applied to some files. We now add the filter parameter, which can be used in combination with any isar-sstate command. By that, the operation is limited to the files matching the filter. Negative matches are also supported. Signed-off-by: Felix Moessbauer --- Changes since v1: - fix help message of filter option: make clear that it can be used on any command - move filter option up, so it is before the command specific options No functional change since v1. scripts/isar-sstate | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 64511c44..adf5dc78 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -31,6 +31,7 @@ and supports three remote backends (filesystem, http/webdav, AWS S3). The `upload` command pushes the contents of a local sstate cache to the remote location, uploading all files that don't already exist on the remote. +If some file should not be uploaded, use a negative filter like '(?!sbuild)'. ### clean @@ -602,6 +603,9 @@ def arguments(): parser.add_argument( '--max-sig-age', type=str, default=None, help="clean: remove siginfo files older than MAX_SIG_AGE (defaults to MAX_AGE)") + parser.add_argument( + '--filter', type=str, default=None, + help="filter tasks by PN (regex)") parser.add_argument( '--sources-dir', type=str, default='/work/', help="lint: absolute path to sources folder (e.g. layerbase)") @@ -632,13 +636,15 @@ def arguments(): return args -def sstate_upload(source, target, verbose, **kwargs): +def sstate_upload(source, target, verbose, filter, **kwargs): if not os.path.isdir(source): print(f"WARNING: source {source} does not exist. Not uploading.") return 0 if not target.exists() and not target.create(): print(f"WARNING: target {target} does not exist and could not be created. Not uploading.") return 0 + if filter: + reg_exp = re.compile(filter) print(f"INFO: uploading {source} to {target}") os.chdir(source) @@ -646,6 +652,10 @@ def sstate_upload(source, target, verbose, **kwargs): for subdir, dirs, files in os.walk('.'): target_dirs = subdir.split('/')[1:] for f in files: + if filter: + parts = f.split(':') + if len(parts) > 1 and not reg_exp.match(parts[1]): + continue file_path = (('/'.join(target_dirs) + '/') if len(target_dirs) > 0 else '') + f if target.exists(file_path): if verbose: @@ -664,7 +674,7 @@ def sstate_upload(source, target, verbose, **kwargs): return 0 -def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): +def sstate_clean(target, max_age, max_sig_age, verbose, filter, **kwargs): def convert_to_seconds(x): seconds_per_unit = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400, 'w': 604800} m = re.match(r'^(\d+)(w|d|h|m|s)?', x) @@ -695,6 +705,10 @@ def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): print(f"NOTE: we have links: {links}") archive_files = [f for f in all_files if f.suffix in ['tgz', 'tar.zst']] siginfo_files = [f for f in all_files if f.suffix in ['tgz.siginfo', 'tar.zst.siginfo']] + if filter: + reg_exp = re.compile(filter) + archive_files = [f for f in archive_files if reg_exp.match(f.pn)] + siginfo_files = [f for f in all_files if reg_exp.match(f.pn)] del_archive_files = [f for f in archive_files if f.age >= max_age_seconds] del_archive_hashes = [f.hash for f in del_archive_files] del_siginfo_files = [f for f in siginfo_files if @@ -712,13 +726,16 @@ def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): return 0 -def sstate_info(target, verbose, **kwargs): +def sstate_info(target, verbose, filter, **kwargs): if not target.exists(): print(f"WARNING: cannot access target {target}. No info to show.") return 0 print(f"INFO: scanning {target}") all_files = target.list_all() + if filter: + reg_exp = re.compile(filter) + all_files = [f for f in all_files if reg_exp.match(f.pn)] size_gb = sum([x.size for x in all_files]) / 1024.0 / 1024.0 / 1024.0 print(f"INFO: found {len(all_files)} files ({size_gb:0.2f} GB)") @@ -753,7 +770,7 @@ def sstate_info(target, verbose, **kwargs): return 0 -def sstate_analyze(source, target, **kwargs): +def sstate_analyze(source, target, filter, **kwargs): if not os.path.isdir(source): print(f"WARNING: source {source} does not exist. Nothing to analyze.") return 0 @@ -768,6 +785,10 @@ def sstate_analyze(source, target, **kwargs): key_tasks = 'dpkg_build rootfs_install bootstrap'.split() + if filter: + reg_exp = re.compile(filter) + local_sigs = {k: v for k, v in local_sigs.items() if reg_exp.match(v.pn)} + check = [k for k, v in local_sigs.items() if v.task in key_tasks] for local_hash in check: s = local_sigs[local_hash] @@ -828,7 +849,7 @@ def sstate_analyze(source, target, **kwargs): def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, lint_stamps, - excluded_tasks, **kwargs): + excluded_tasks, filter, **kwargs): ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() # only list non-cacheable tasks here # note that these still can break caching of other tasks that depend on these. @@ -844,6 +865,10 @@ def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, li else: cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + if filter: + reg_exp = re.compile(filter) + cache_sigs = {k: v for k, v in cache_sigs.items() if reg_exp.match(v.pn)} + hits_srcdir = 0 hits_builddir = 0 hits_other = 0