From patchwork Thu Apr 10 09:16:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Moessbauer X-Patchwork-Id: 4150 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 10 Apr 2025 11:17:40 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f63.google.com (mail-wr1-f63.google.com [209.85.221.63]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 53A9Hdpg010450 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Apr 2025 11:17:39 +0200 Received: by mail-wr1-f63.google.com with SMTP id ffacd0b85a97d-39130f02631sf204368f8f.2 for ; Thu, 10 Apr 2025 02:17:39 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1744276654; cv=pass; d=google.com; s=arc-20240605; b=g77D/K5FpAhUFhHIY6fqpocH60jlSH5+COAQkHS3dLTOfYIpYKjlgs0jmdbltoTl2d bIZVzH9Cgw5QRRq+3mZnPQjmudiIGr6XaUoWvAUcMLsLun8a9cY+x84Ak5C5iTW8FW4N 2ZI2YuUzp+E/mjyIbE66tTdKAaH0cKF8X/52ViJnwGbKTrOLvA+O06q3Ph0YEX1s2pK6 dP8+N+AoNdlcTtawhGRzZ4+9hVRr1/8Vk00gAE9GoMAyuiecIRq4OO27qhSH1zDq5h4i LpjlB8gu6HoIMaRodBZ5UFQk24pd/k/qeXyvtq7G2oQ99RuJrfnkXwBT/Z9QfkiSge8k dvkA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=4iFpelOkEqvmcz3vYcn9jTb9tnaYovBUpGFiR3BAbVw=; fh=89+8gfHVVKFp9LRkFBMgVqcLrbfJ5wo4wRtr3Qkdxwo=; b=kmgWUtOnw2Vw9c3Sa+PROB8sIdkGwuTda29QUAsxyp3LuLPOyPQztdmGo4wTaL5LGW npYomKSyvN3jy+g5QN8B8F9NQm88hUKbMmO3A+S0VNLbWCazVqHQwRJglvvvP4Ch3pce 4Xr0/aqo565y1iSiiapO8LCLmxoTiGPvjD8rzGDCyTi7D6n5RAwUlDpXKWnQ5h52ut4M 6xUWKixdsSGHszd/Au7wlOHND4jB7lffqO0SF9c9GTfoHvQCEOQ7HAZKqMqvxv3wmO2I dr4bAGgIfljg+PKYZQ39Afym6VIt43Qb0NJ7xWX3/B0RFPdLf8rlxCfjpEp5/QNFgiPe 9wSg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=MVV8y2Vk; spf=pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250410091729601c087d8b3d26f65a-_0sAeK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1744276654; x=1744881454; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=4iFpelOkEqvmcz3vYcn9jTb9tnaYovBUpGFiR3BAbVw=; b=XiWrYoV6MeCx+VIzh6N82hXTEpi4YKBy1Xznrbr3/hzZcv8Nj/yDkjWVNWFhcGSDh7 V2PlFq5jGZ62f604tAGwLj6Z+GoCAkQ+3JRiDsMjfBl7a++GjXPng18+eG7waz0kJ+eD 5WQxo04l94HS8O/XCY92pGOE6RxUzhbQ3Gk9jINLstEmlZjFf/lj7H36ga5uLZeE5a5T 2DS10LAwXw1nRQTmRms2woDnWUkYvYw2gRQ/UY40C0v0Z8RVkUFDeEZUk8uWDSnerEId aF4bSV+fN+kwcKC2MOFnrI6Tnk5tlfrb7tLQPtRIQfSMAENWPWrkHF1dhYI2XSl7puOK abrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744276654; x=1744881454; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4iFpelOkEqvmcz3vYcn9jTb9tnaYovBUpGFiR3BAbVw=; b=oe8r2880V7D3PNTyPL/W2l9FxkZYYEZ86/YDlfaDlx6ki0bEcaszJfmOuYlJoTk0J/ GCvavccw/gyuN20KzPWUPDM8wUGi/m4bQggxykxQGMz6N5ROPJmoqzOnFeFpfo5uI0Pi YZ2ni43QtgP5FvQT4smdXYHUMaASI7JwU76e7a91YpC2C3+vQnYX13QPx/FbKMSJWOyJ 5RJBNdPDDacm93PwSk3QjKUxC4p42nk2/rU7QgW1JfXWTFf3cE+bUagh8J/Z7n7T1RO0 v/FT+V18Ukba3d4O26sAyz0bZsWZqNMpCbvGCVOEAIJ/VuwoUaF2rglELckmNvWfVo2Z Veiw== X-Forwarded-Encrypted: i=2; AJvYcCV59RcxM5vGxjhWcuG7LcdeGDdA3Rx2xAAnyawu+pWbMWZplP+5++yPKm5iMNfzxvL7yn2Ibqg=@isar-build.org X-Gm-Message-State: AOJu0YzZraD3v9nHD+IV0f7gfael6LVsZcaABHcGPE5STuZrFZKZTARx E1HCQ+IqFrKlyHS7yMBns1pi0a+I5Ae9wZ2wITUE8E6d1mRAGBWj X-Google-Smtp-Source: AGHT+IEH2xKciOBRxX2XQlIkG+9jp+p5v6sxErC9RvjLu40gv6GaAqcsUSxNFt0Rfb43KmwBKRQgqA== X-Received: by 2002:a5d:5983:0:b0:38d:cf33:31d6 with SMTP id ffacd0b85a97d-39d8f27612emr1651199f8f.3.1744276652996; Thu, 10 Apr 2025 02:17:32 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAKqfLgTBjs7SMFESNlgIbeqs4K9HovkQbPomHYo7oB/kQ== Received: by 2002:a05:600c:1d23:b0:43d:40b0:33 with SMTP id 5b1f17b1804b1-43f2c24f86als2056715e9.0.-pod-prod-01-eu; Thu, 10 Apr 2025 02:17:30 -0700 (PDT) X-Received: by 2002:a05:600c:3c91:b0:43b:cc3c:60bc with SMTP id 5b1f17b1804b1-43f2d7d6f10mr23615935e9.15.1744276649921; Thu, 10 Apr 2025 02:17:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1744276649; cv=none; d=google.com; s=arc-20240605; b=M3iZEDsHx/38FizU34wmzo5aKQQdEh00FrbaiH5kV/IraYfpRNIajqYoCnz0zkCLf7 bcZY9Rg3u5mMhBOE3OMyWBhuzZGOLJp2QEQ+Hd5MtcC9GwberrH77o9/+5xXoa+eyHST COmWnrGnG9pwbM/uLhogdCnN1Dq0UnGU8z0RjdtTae2JY9lbcuuQIqUxE+/jMcv1cX6S BjVAFT2ATc56AmVhVbzPybIDh+fjcjDbMRZSzg0vhEz8eKnKNq5CAgFvG6lMDod1NR4e Jv3NwSEQuGHxMEZSSap1I+ZMqK1HHcJtEnTJL61eS9RJJ5dG0Ws4fJG+bWeDyhUBTlpY kFpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=MaNfNMIBig9Ef9e6U7xXxSwvqtqyzJ8O2h9d6hryhcE=; fh=OYA7Xj8Fgh6stKMJ6XJH0QE9OZQModMu4f2HyOuPUjs=; b=X4RA8RXRK8B/nE/8AlVIl7TJb1rXAYHghJIs3FAuxRdh2VwVT2nLs/d3d465BuHMGB b3l/hRtZ8sja0xrCmRaU12T9+4PHm8jhEIUsfvH0CSsazvuGtYBZ1ClOe7hJJ3idb5HC LOKsRz6vJbgRQIEmakvtPYBVOFexuekYNOZw+vV6jDRP/FWtdSgtyOfjAyf/W3lbl4kA znbryCx9zG6pGkFHIuad4yQF3EGH1OJt3LsBcyiNnaid5h+yjMJD3mT0ZzefGeeMiZK8 zzdj6pmIqO4WOMg2mc+ERr/3Yir1v2RYwQBZMq+KBwCRe9FZmfwTL+QUMLRGMsjlbb4d 1t7Q==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=MVV8y2Vk; spf=pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250410091729601c087d8b3d26f65a-_0sAeK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-43f208e8436si2042235e9.0.2025.04.10.02.17.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 10 Apr 2025 02:17:29 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20250410091729601c087d8b3d26f65a for ; Thu, 10 Apr 2025 11:17:29 +0200 X-Patchwork-Original-From: "'Felix Moessbauer' via isar-users" From: Felix Moessbauer To: isar-users@googlegroups.com Cc: Felix Moessbauer , adriaan.schmidt@siemens.com Subject: [PATCH 1/1] isar-sstate: add filter parameter Date: Thu, 10 Apr 2025 11:16:45 +0200 Message-Id: <20250410091645.3840431-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=MVV8y2Vk; spf=pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250410091729601c087d8b3d26f65a-_0sAeK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Currently all sstate operations are always performend on the whole cache. This is problematic if some files should not be uploaded or a clean operation should only be applied to some files. We now add the filter parameter, which can be used in combination with any isar-sstate command. By that, the operation is limited to the files matching the filter. Negative matches are also supported. Signed-off-by: Felix Moessbauer --- scripts/isar-sstate | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 64511c44..8206a28e 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -31,6 +31,7 @@ and supports three remote backends (filesystem, http/webdav, AWS S3). The `upload` command pushes the contents of a local sstate cache to the remote location, uploading all files that don't already exist on the remote. +If some file should not be uploaded, use a negative filter like '(?!sbuild)'. ### clean @@ -620,6 +621,9 @@ def arguments(): parser.add_argument( '--excluded-tasks', type=str, default=DEFAULT_IGNORED_TASKS, help="lint: comma-separated list of tasks to ignore (default: %(default)s)") + parser.add_argument( + '--filter', type=str, default=None, + help="lint: filter tasks by PN (regex, default: all)") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -632,13 +636,15 @@ def arguments(): return args -def sstate_upload(source, target, verbose, **kwargs): +def sstate_upload(source, target, verbose, filter, **kwargs): if not os.path.isdir(source): print(f"WARNING: source {source} does not exist. Not uploading.") return 0 if not target.exists() and not target.create(): print(f"WARNING: target {target} does not exist and could not be created. Not uploading.") return 0 + if filter: + reg_exp = re.compile(filter) print(f"INFO: uploading {source} to {target}") os.chdir(source) @@ -646,6 +652,10 @@ def sstate_upload(source, target, verbose, **kwargs): for subdir, dirs, files in os.walk('.'): target_dirs = subdir.split('/')[1:] for f in files: + if filter: + parts = f.split(':') + if len(parts) > 1 and not reg_exp.match(parts[1]): + continue file_path = (('/'.join(target_dirs) + '/') if len(target_dirs) > 0 else '') + f if target.exists(file_path): if verbose: @@ -664,7 +674,7 @@ def sstate_upload(source, target, verbose, **kwargs): return 0 -def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): +def sstate_clean(target, max_age, max_sig_age, verbose, filter, **kwargs): def convert_to_seconds(x): seconds_per_unit = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400, 'w': 604800} m = re.match(r'^(\d+)(w|d|h|m|s)?', x) @@ -695,6 +705,10 @@ def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): print(f"NOTE: we have links: {links}") archive_files = [f for f in all_files if f.suffix in ['tgz', 'tar.zst']] siginfo_files = [f for f in all_files if f.suffix in ['tgz.siginfo', 'tar.zst.siginfo']] + if filter: + reg_exp = re.compile(filter) + archive_files = [f for f in archive_files if reg_exp.match(f.pn)] + siginfo_files = [f for f in all_files if reg_exp.match(f.pn)] del_archive_files = [f for f in archive_files if f.age >= max_age_seconds] del_archive_hashes = [f.hash for f in del_archive_files] del_siginfo_files = [f for f in siginfo_files if @@ -712,13 +726,16 @@ def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): return 0 -def sstate_info(target, verbose, **kwargs): +def sstate_info(target, verbose, filter, **kwargs): if not target.exists(): print(f"WARNING: cannot access target {target}. No info to show.") return 0 print(f"INFO: scanning {target}") all_files = target.list_all() + if filter: + reg_exp = re.compile(filter) + all_files = [f for f in all_files if reg_exp.match(f.pn)] size_gb = sum([x.size for x in all_files]) / 1024.0 / 1024.0 / 1024.0 print(f"INFO: found {len(all_files)} files ({size_gb:0.2f} GB)") @@ -753,7 +770,7 @@ def sstate_info(target, verbose, **kwargs): return 0 -def sstate_analyze(source, target, **kwargs): +def sstate_analyze(source, target, filter, **kwargs): if not os.path.isdir(source): print(f"WARNING: source {source} does not exist. Nothing to analyze.") return 0 @@ -768,6 +785,10 @@ def sstate_analyze(source, target, **kwargs): key_tasks = 'dpkg_build rootfs_install bootstrap'.split() + if filter: + reg_exp = re.compile(filter) + local_sigs = {k: v for k, v in local_sigs.items() if reg_exp.match(v.pn)} + check = [k for k, v in local_sigs.items() if v.task in key_tasks] for local_hash in check: s = local_sigs[local_hash] @@ -828,7 +849,7 @@ def sstate_analyze(source, target, **kwargs): def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, lint_stamps, - excluded_tasks, **kwargs): + excluded_tasks, filter, **kwargs): ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() # only list non-cacheable tasks here # note that these still can break caching of other tasks that depend on these. @@ -844,6 +865,10 @@ def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, li else: cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + if filter: + reg_exp = re.compile(filter) + cache_sigs = {k: v for k, v in cache_sigs.items() if reg_exp.match(v.pn)} + hits_srcdir = 0 hits_builddir = 0 hits_other = 0