From patchwork Tue Apr 2 17:28:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Larson X-Patchwork-Id: 3466 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 02 Apr 2024 19:29:09 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-pj1-f64.google.com (mail-pj1-f64.google.com [209.85.216.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 432HT7ck019592 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 2 Apr 2024 19:29:08 +0200 Received: by mail-pj1-f64.google.com with SMTP id 98e67ed59e1d1-29c7932c5f5sf62041a91.1 for ; Tue, 02 Apr 2024 10:29:08 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712078942; cv=pass; d=google.com; s=arc-20160816; b=mbOU/2JCHvr1+uX7yJsYWk85bsfRe4M4KxiGc+/320zVm9MamuWckyXtSZqqI/EIAI pBWjrz2WQ+2HdkncwVR1J9wVLZpNzYyArBz9XxhRoAnAxqzcAg8uTXtY4hhI9eUQnHO+ 5NGmMA8MwZFqThXJAAZU9/5/L6sEJ3n1sZnB7g6O2v7Gm3CBr3OXnCSz41FRI9cKke78 woaUWdMm4XWqP3/Ba2004zjruAyovQNIDEKHKvEml2PNfrKYbp1hBFVbDK1zjTbi1IS+ 77Q1kUZlsgqOQddtatM/edVCahJjRRyCLUz64MksP3+lae69BsmDI0wGgEoyjXZp51e7 lM9w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature :dkim-signature; bh=OfYwjejDKGoavg856QNdJQSdmejoLoLchd5KT2EsLYA=; fh=rUKRF4S6heonBiPHp8NfbHFWhTTl+vxSVuaY8W4HSpM=; b=QSQ2rI32fuUHypFRvrygkYYYDQVoboB1BPzHHLL/MPInQPikoDhj32gPPLkv036OXe UJpQBZudDqF4kCHPyYWs6nO9wgPzFgPwskgLvppHSgspJzMpUH2dTVZjzc+HFjWkHXpe 5pCRp7nviWeF0S7lkQLSTozdffj6yVLLaOWxm1V4PhygiTydNAcP0wnjzdBn8tu/oKmR FyW+tjywzWNCwLbzxj39G4z4vjoOxO4H9M0x3stZ2pWqcosAxXASR1xhoMuC+iyoCVJr 28hfXmO0umBT3mqloVWnWiLoWts48OIufqGnwUdSuB+/Czs4+Tp+lj5nNsuMEy8LzjMN N1rg==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=gZqLPNCU; spf=pass (google.com: domain of kergoth@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=kergoth@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1712078942; x=1712683742; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=OfYwjejDKGoavg856QNdJQSdmejoLoLchd5KT2EsLYA=; b=ltVwDW7yHAHw3cZwFeq7MuPDZaUrLhrT5tyYTopygkVfkTy5EVCl3+2x0dyFknke+R sdeoxzfZtClBj+/7eGdU2b91qCeAbQe1MeKxyWQbItItEp+Xt22nNwtuuuAu4atpXD0j 6zpQKe2IvbtVbI9ciFfWnrS2DUEs/ya2+PDccfHOUc5A4bOAHfNDF13fivT6Fs+kPkZ1 JDORFNoYjunINlsAl9Z79p68OdTOUPvDQby4n23rnUmEg4T9myndyVOUXbL/fglc5LpY PwmFLuJF2TAdk0pQj+TkVFD/4HsknhLP34lTnJ8xG0K2m+azGImPaWJOeFNIoAWpaeA0 os5w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712078942; x=1712683742; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=OfYwjejDKGoavg856QNdJQSdmejoLoLchd5KT2EsLYA=; b=MZkd0mN4Wi5KUmaY2JnbwVCGvGgYDBLsTtVvXk+2lxduqpkRYMyTu2qz/OYMqFbETc ThgF9OJD284HyfGSI6yKuM6GRZH/bZq4wiqkEaKMxdF/OzVj6U+12OazGkY0D0G3j3sn m7ww+jK+zJS6ROeyR7lTAFiC9PO2Xi5CKE+fxgFTQihMEyAiLra9Bl/OEQ2HJVqmpYCb sJzXI9rSfOpi8+xCtd23SGDW8pB+7ga42giNqhpJmsPxNJgAd/t+nS1CDj0iwBG5Py7D XRK7phUFQjz3ebeH7pf9A/1HMKCXoakvE4QpeGcau6kSiBBXfUyofKcAqbxYcWV0ZYX8 QrDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712078942; x=1712683742; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to; bh=OfYwjejDKGoavg856QNdJQSdmejoLoLchd5KT2EsLYA=; b=ipTTRBEmTWY1TfiAiQ+GKr88eqCQ287TqD82CKuOYhbCznHo3SmS2AxQNBQb9uOFwj dNq7/1wZJe1n8s4nLQYmwUp0r0wblHnDf4A0FbUjnXdiL6gs1LDecw9sManCWc4PUdkh oXnY8DE1ruT7RH/N1zjpTQRRU6Sg8mal30d7T08GJZ6pP0HrkDduVvIuO08mRtsmpBfE 1rZZKmZQOEE6Do4CBLmvaEeySjnOJV86qP1iMUYP3+0W5RBTTt2G7fkl1xElx98atMwt kfPUyVYkj7VpQ5Y3MW56bvxXUrnDCUUNlA8IQgXnB5xAQQ78ODtZTbQfRF/EQU2FGqjf edRg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVgqoFzT0xDeH+1W8JEc853MIumJHWFaAsMEWUQkAfAjPgLN8fBiHTAB1I81/6VXioyAs627WsvLEc3tOSOpkyhXU90rik= X-Gm-Message-State: AOJu0YyPCdAKbYCgRxMF7krNQ+G+EF63hoQDCbVXN64J37zb262b1qaR PyUPHQ029edO2zxLwV+wZVGtaKeFTP4/uYetLuwdhqaGlT80ei+F X-Google-Smtp-Source: AGHT+IFb5skojo6LYPMn8w0VlVA+Vk+wy0vfNkYIzAvcf9ddZvFK86dtwTFoTgNBQcIQuFJgzGBhIw== X-Received: by 2002:a17:90a:558e:b0:2a2:79cc:5eb9 with SMTP id c14-20020a17090a558e00b002a279cc5eb9mr217120pji.7.1712078942262; Tue, 02 Apr 2024 10:29:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90a:7564:b0:2a0:4708:fd74 with SMTP id q91-20020a17090a756400b002a04708fd74ls29321pjk.0.-pod-prod-00-us; Tue, 02 Apr 2024 10:29:01 -0700 (PDT) X-Received: by 2002:a17:90b:4ccd:b0:2a0:78f4:2dc5 with SMTP id nd13-20020a17090b4ccd00b002a078f42dc5mr180371pjb.22.1712078941177; Tue, 02 Apr 2024 10:29:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712078941; cv=none; d=google.com; s=arc-20160816; b=kHfsSKibzFun9/PwlZll3QtoVjqIc9N0CiCbTLbUcZVDRkeMSLVbs0siP9lqqrpJqV INq0NdQ9XJS+9XA3hSdYlNzRLJ5LD2Mut9OfmiOn4GlQmy91ZxMs1J9cqEQlZIOVHfMi 4JI0RNge/9PPL1xfpzfyzYkKPFXvYVoXCOqQRLcetBknszncz0hSP1uefMeFsrgigoLv JuUP6ghDR5Kn3nyTkN5B/ogrmIECnBgKlfjXiPdS1JHUDNegghuszlQDwgUQ3nVOQ7pA 8u5TW/BBS2iAVOM5rEnTaYkCP2vr+uSAC+Dcmfc8YnvP6wD5AH4FAH4T7x3vW8qxy11P BqQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=pCp2/5apqyEvfPvSZ2O2t8CM9o6Ve9WNpL+a27IeD3k=; fh=oQsVBfLFKOCDpQI4v78XDUCp+c0vYPEMW+xCOFIgP4c=; b=ghj1ip9g3Ge8Qe1ZeTIoNISwZlTzeRF4mvQtj6e75YmqHvizko4Q3r8jolaoR6b5YO YM1QFy+qLq2wNwyHk9n3MqMvdR5b1lTquU7lRmMGYsuQOcRTm1bNLnIv+uVTBzSx3EgV B58WEAxLxrL91JvjEv6HFBaEC0BAm/nSsYTSRMIfmiLv4rnrIwh2KgL54LafMo9rQINm l2muG5c71ZK88gjY+xZYRngpTd7Tf7S+izCNz/YWYFkgtHnMWPka45K4hWmJ6dKW2biF /Q7RqZ6hBG8hZ8hLQKHUIpsXxfbCyZnlWQc8ozjGjhi7kt/76c/yNHOOu3VEX4qhOV0x +nEQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=gZqLPNCU; spf=pass (google.com: domain of kergoth@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=kergoth@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com. [2607:f8b0:4864:20::102a]) by gmr-mx.google.com with ESMTPS id d126-20020a633684000000b005f0a5113629si500562pga.3.2024.04.02.10.29.01 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 02 Apr 2024 10:29:01 -0700 (PDT) Received-SPF: pass (google.com: domain of kergoth@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) client-ip=2607:f8b0:4864:20::102a; Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-29f69710cbbso70557a91.1 for ; Tue, 02 Apr 2024 10:29:01 -0700 (PDT) X-Received: by 2002:a17:90b:3712:b0:2a0:76b1:896 with SMTP id mg18-20020a17090b371200b002a076b10896mr224404pjb.12.1712078940778; Tue, 02 Apr 2024 10:29:00 -0700 (PDT) Received: from dc4670097af0.alm.mentorg.com (nat-mam.mentorg.com. [139.181.8.34]) by smtp.googlemail.com with ESMTPSA id bt19-20020a17090af01300b002a2313f62dcsm66705pjb.0.2024.04.02.10.28.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 10:29:00 -0700 (PDT) From: kergoth@gmail.com To: isar-users@googlegroups.com Cc: Christopher Larson , Christopher Larson Subject: [PATCH 4/9] isar-sstate: lint: add support for checking stamps Date: Tue, 2 Apr 2024 17:28:08 +0000 Message-Id: <20240402172813.418770-5-chris.larson@siemens.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240402172813.418770-1-chris.larson@siemens.com> References: <20240402172813.418770-1-chris.larson@siemens.com> MIME-Version: 1.0 X-Original-Sender: kergoth@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=gZqLPNCU; spf=pass (google.com: domain of kergoth@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=kergoth@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.0 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Christopher Larson Bitbake supports writing signature data directly to the stamps directory without having to build, so we should add the ability to lint this signature data as well. This is useful for checking for cachability issues without having to complete a build. Signed-off-by: Christopher Larson --- scripts/isar-sstate | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 9b20cb8e..b77f73eb 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -154,10 +154,12 @@ SstateCacheEntry = namedtuple( # "${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:" # This regex extracts relevant fields: -SstateRegex = re.compile(r'sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' +SstateRegex = re.compile(r'(.*/)?sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' r'(?P[^:]*):[^:]*:(?P[0-9a-f]*)_' r'(?P[^\.]*)\.(?P.*)') - +StampsRegex = re.compile( + r"(.*/)?(?P[^/]+)/(?P[^/]+)/([^/]+)\.do_(?P[^/]+)\.(?Psigdata)\.(?P[0-9a-f]{64})" +) class SstateTargetBase(object): def __init__(self, path, cached=False): @@ -288,12 +290,13 @@ class SstateTargetBase(object): class SstateFileTarget(SstateTargetBase): - def __init__(self, path, **kwargs): + def __init__(self, path, regex=SstateRegex, **kwargs): super().__init__(path, **kwargs) if path.startswith('file://'): path = path[len('file://'):] self.path = path self.basepath = os.path.abspath(path) + self.regex = regex def __repr__(self): return f"file://{self.path}" @@ -334,12 +337,13 @@ class SstateFileTarget(SstateTargetBase): for subdir, dirs, files in os.walk(self.basepath): reldir = subdir[(len(self.basepath)+1):] for f in files: - m = SstateRegex.match(f) + relative = os.path.join(reldir, f) + m = self.regex.match(relative) if m is not None: islink = os.path.islink(os.path.join(subdir, f)) age = int(now - os.path.getmtime(os.path.join(subdir, f))) all_files.append(SstateCacheEntry( - path=os.path.join(reldir, f), + path=relative, size=os.path.getsize(os.path.join(subdir, f)), islink=islink, age=age, @@ -592,6 +596,9 @@ def arguments(): parser.add_argument( '--exit-code', type=int, default=None, help="lint: return this instead of number of found issues") + parser.add_argument( + '--lint-stamps', default=False, action='store_true', + help="lint: assume target is a stamps directory (target must be a local path)") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -798,7 +805,7 @@ def sstate_analyze(source, target, **kwargs): print('\n'.join(out)) -def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, **kwargs): +def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, lint_stamps, **kwargs): ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() # only list non-cacheable tasks here # note that these still can break caching of other tasks that depend on these. @@ -809,7 +816,10 @@ def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, ** print(f"WARNING: target {target} does not exist. Nothing to analyze.") return 0 - cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + if lint_stamps: + cache_sigs = {s.hash: s for s in target.list_all()} + else: + cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} hits_srcdir = 0 hits_builddir = 0 @@ -891,10 +901,12 @@ def main(): target = SstateDavTarget(args.target) elif args.target.startswith('s3://'): target = SstateS3Target(args.target) - elif args.target.startswith('file://'): - target = SstateFileTarget(args.target) - else: # no protocol given, assume file:// - target = SstateFileTarget(args.target) + else: # Either file://, or no protocol given, assume file:// + target = SstateFileTarget(args.target, StampsRegex if args.lint_stamps else SstateRegex) + + if args.lint_stamps and not isinstance(target, SstateFileTarget): + print("ERROR: --lint-stamps only works with local file targets") + return 1 args.target = target return globals()[f'sstate_{args.command}'](**vars(args))