From patchwork Fri Apr 5 16:31:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Larson, Chris" X-Patchwork-Id: 3481 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 05 Apr 2024 18:32:35 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-pj1-f56.google.com (mail-pj1-f56.google.com [209.85.216.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 435GWXOZ003064 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 5 Apr 2024 18:32:35 +0200 Received: by mail-pj1-f56.google.com with SMTP id 98e67ed59e1d1-2a2c80fb803sf2070652a91.2 for ; Fri, 05 Apr 2024 09:32:34 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712334746; cv=pass; d=google.com; s=arc-20160816; b=wRQGimfLRXqwwwa1Ge0TwELsvJMT6rOZLnLt7kjEaypYfoVc9nmnxvauAcB/hOWIRE 7q7nH3OGP+J+uisByprhh5Udc/BCKrRlxTaGaxJQwjoyaYnuOmL4Pw7Lau+H9IXj6Rfw 1+Q4zqmVXiSNDxB9fwdRMngg4rGZqoxLlH/WoWwUZxSG8mQKVa4thD8dZylwigHfSfQ9 BbZUb9InkDzou/58+H06RwLZvemllKdNwPhIgK8NAOXrwjdOynLpLD2CfhUR9pCy69g7 0vE/LFXuTt63wJbqTLB1llFBGk+P7qDqw2Zc31gQ7WzjGuXCcKAhMV8HYaoFWhi8q40S OX9g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sZZoK/Zn9kbrKaG/t9DOn2rwd4BY0vv4hcmYd9PPtJ8=; fh=deZRGzUM+lXZk52dz8Ej+bfybTglxHe3hWGnZNGJ6vY=; b=X+X2xrytT26cNz27V2cUzQl51lJV1kHvZHr15FnKC+U2X0UE0IApTLPrPr2/8TAUXM +b8sDjLahyOtjopyXd4j3NBcbElePUVKV3xFOp5bDhtHiZdxXWPwMdWnQvrSNdUa6xhp oFCZUvT503QTnLWhkJVnpgS+IYhQ16otw9rm8MBVGOblfEEKs0DQNOv2WiNqJ75LNx7B BgiUJj4ShPKTSzOmkLR3bxY613WIWQpZm5CtLc1SnDxyW5EKYShDtSlwDNhceJZdK5MJ KQVZ0cMYN1ZKBapQNid0rdwn2bij/sQqf6qRcGSQ8O4FIrjGrkEHojMe+wR2I7mtyzz0 HkTQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lCH2YKi5; spf=pass (google.com: domain of fm-1325773-2024040516322481f15bbe14d093a36e-wq6imq@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1325773-2024040516322481f15bbe14d093a36e-wQ6IMq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1712334746; x=1712939546; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=sZZoK/Zn9kbrKaG/t9DOn2rwd4BY0vv4hcmYd9PPtJ8=; b=LV1wo8sESS8DSAEpPKE+iQHwvCz6CqwmiPvRr3iQdTZ2IYuGNz6+fkOzS8Qa60Te/J QdrWYA2Yxb1zwHivYaAofQAr3udGTMm1Q/9XYyfg3XF4LJPaegZjMeJ7t+9Oyp9TrihK VZK5dkCWiwV/TVraa3lC64trUrMQrYGjPchkyRKDvo/29TDHvmqWKrHWEwnA9ABncorQ 2zT7VLTJj1FeikiVAXJp+c2VAvzT2sOugJnQzKL8ecXbGAqJuyFs/kEorevenkwBPr7p wGfNDRCKUzK6OzPTleSDcSYmUBVvrjNYlzvb9QhZNrPT05LV7cFyiEamnDjGUK2uFJDW GYXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712334746; x=1712939546; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=sZZoK/Zn9kbrKaG/t9DOn2rwd4BY0vv4hcmYd9PPtJ8=; b=ahRSdkDcT1d116eNxSFLhMN61BUAUMH0iHreUT73iStgLMm28z8B+6uIzPZEgoLK3h eeavH41PNCYyPFVVOKtDPFc748/Aeae3QPBaLTcmPBGNzeAG3jdhLLrFxU/CzYEO4yjX WoUtY07RxpR61R1JRXOrIJ06J+z5wbCbU58wV/GjcVTq/yl1NbH4O2CQbbApJNZPPixM AxxwnV4GnGOCG9y3gjB9ItJmGKWDtlQS0AZc3SuvrrkHPOQb1TezDgZ/1xRf5LmRHyVd 2Mq39sMh/ETV6nCDa816hcwgfgDBkQ2i/kpv3UDv+dB9v6Tg5AmQTMlvCbPTT0FEKfPl G3tw== X-Forwarded-Encrypted: i=2; AJvYcCXhHhNBM/CivSOTIjtvkN8DWj2b5Z75INRCxi7JB/324iyNJpB/EbVklVGoD8ZmNi5D1Z9502KqeictE4+z5C9y1dJ9aDA= X-Gm-Message-State: AOJu0YzU1WKrT1blf2cIdgY4NhJU5f8pZT+Gs7ufk3a/Bom/Kp+3o8QP fEUED304atjtzaABnC5U8vHho5dZ9+/ll5Hsb3wtzSd65mCqt7JE X-Google-Smtp-Source: AGHT+IHbOe1YvA6H/3v/T1CV5kHgPyVRECnKG4kSsPcHaK5Qd6MJYMpDAnLB4sHH1fySrEf4kLe57A== X-Received: by 2002:a17:90a:a005:b0:2a2:ea20:2074 with SMTP id q5-20020a17090aa00500b002a2ea202074mr1988320pjp.23.1712334746628; Fri, 05 Apr 2024 09:32:26 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90a:c20e:b0:2a2:9d0f:c728 with SMTP id e14-20020a17090ac20e00b002a29d0fc728ls1362034pjt.2.-pod-prod-06-us; Fri, 05 Apr 2024 09:32:25 -0700 (PDT) X-Received: by 2002:a17:902:bc46:b0:1e2:3db0:1d84 with SMTP id t6-20020a170902bc4600b001e23db01d84mr1806315plz.32.1712334745446; Fri, 05 Apr 2024 09:32:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712334745; cv=none; d=google.com; s=arc-20160816; b=t+4QURxjPXn2YGFsP5Kl8K5b1qcl/SkcKHh732OB46X5FkosUCXEMR9zyIXb6+Z1/y j8Q/jw4AQwb+rsPV1/FwtBh73p7P0H54e5LBc8sRv+v620O8qQFfRFwDSG5kb275Jg3J NIfL/QtPJ2HZluuXSp8Nw2ehKv/Ze51N7K/a5ltV+jjtyE/5E2gbAGvLuzztGLGQyryh 4JdWWJ80f5ulycaXbjW3PcP4Dnay10VFdR9n0QG8jML7YZgb8qPXMt4HFnupB2Y9C1Rg xqCccIrSYHbWpX5TvN3WYB8XaQJTrEmsKhlw96U8y0/Ut3F2s/XJZ1b0nTKz91uGAc/G i+IA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=lHo56J9PyTOzEGKYUsZ78jlLZiPMDdfI577GnBCiu5k=; fh=3bh9z2BWAzGAc1byRhMi0K+arMZmX5BQHE0sOxINdyU=; b=myyJkOFpUt/V7q38T4BnnTWmxjfJtAuMQyZT8XvjVNEzd5aPAI+on+EUczMnZXnsD3 dG1etMUc2HZCc79bo48LkJgqFEolv95XsJlHJUmJ8/RqLXNBPElWTconrO/TMm38Bsiq wO0+gRwU8GV0ws2KnUyQ5UkX/XRqYDPBecrzEBaAIcSwFO3UrgDZBi64B6Qz5aQQPxH0 B7lmuSUdZyfsr4+T3JqFLvtAN78aIT6DQMZgNHFISyEB4ESxZJBJVbQc/kJW+h5MEgpx sxuwfVrzT2NbyXrOEK/PyGoZJ9ubK9TK7KsY5Fzey0zvvSqa63UBKs8/hHahksF8MkoX 1f/Q==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lCH2YKi5; spf=pass (google.com: domain of fm-1325773-2024040516322481f15bbe14d093a36e-wq6imq@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1325773-2024040516322481f15bbe14d093a36e-wQ6IMq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id jc18-20020a17090325d200b001dd61b4ef8esi97963plb.12.2024.04.05.09.32.25 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Apr 2024 09:32:25 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1325773-2024040516322481f15bbe14d093a36e-wq6imq@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 2024040516322481f15bbe14d093a36e for ; Fri, 05 Apr 2024 18:32:25 +0200 X-Patchwork-Original-From: "chris.larson via isar-users" From: "Larson, Chris" To: isar-users@googlegroups.com Cc: Christopher Larson Subject: [PATCH 4/9] isar-sstate: lint: add support for checking stamps Date: Fri, 5 Apr 2024 16:31:30 +0000 Message-Id: <20240405163135.2987489-5-chris.larson@siemens.com> In-Reply-To: <20240405163135.2987489-1-chris.larson@siemens.com> References: <20240405163135.2987489-1-chris.larson@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1325773:519-21489:flowmailer X-Original-Sender: chris.larson@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lCH2YKi5; spf=pass (google.com: domain of fm-1325773-2024040516322481f15bbe14d093a36e-wq6imq@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1325773-2024040516322481f15bbe14d093a36e-wQ6IMq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: chris.larson@siemens.com Reply-To: chris.larson@siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= From: Christopher Larson Bitbake supports writing signature data directly to the stamps directory without having to build, so we should add the ability to lint this signature data as well. This is useful for checking for cachability issues without having to complete a build. Submitted at https://groups.google.com/g/isar-users/c/2NB-PXyswq8/m/GMuGk61TAQAJ. Signed-off-by: Christopher Larson --- scripts/isar-sstate | 51 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 9b20cb8e..d68b8938 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -66,8 +66,16 @@ format as `bitbake-diffsigs`. The `lint` command searches for common flaws that reduce the cachability of a layer, e.g., signatures containing absolute paths from the host -environment. Issues found this way usually indicate errors in recipes -or in Isar itself. +environment. Issues found this way usually indicate errors in recipes or +in Isar itself. + +The `lint` command may be used to check a shared state cache folder, or, +with the use of the --lint-stamps argument, will check the tmp/stamps +folder and expects this as an argument. The signature data files must be +populated in the stamps folder, which requires either a completed build +or a `bitbake --dump-signatures=`. The value for this argument may be +`none`, `printdiff`, or others defined in the handler, but `none` is the +simplest to just generate signature data. ## Backends @@ -154,10 +162,19 @@ SstateCacheEntry = namedtuple( # "${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:" # This regex extracts relevant fields: -SstateRegex = re.compile(r'sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' +SstateRegex = re.compile(r'(.*/)?sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' r'(?P[^:]*):[^:]*:(?P[0-9a-f]*)_' r'(?P[^\.]*)\.(?P.*)') +# The filename of stamps and associated signature data is defined in +# Isar (ith an added suffix of `..sigdata.` by BitBake): +# STAMPS_DIR ?= "${TMPDIR}/stamps" +# STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}" + +# This regex extracts relevant fields: +StampsRegex = re.compile( + r"(.*/)?(?P[^/]+)/(?P[^/]+)/([^/]+)\.do_(?P[^/]+)\.(?Psigdata)\.(?P[0-9a-f]{64})" +) class SstateTargetBase(object): def __init__(self, path, cached=False): @@ -288,12 +305,13 @@ class SstateTargetBase(object): class SstateFileTarget(SstateTargetBase): - def __init__(self, path, **kwargs): + def __init__(self, path, regex=SstateRegex, **kwargs): super().__init__(path, **kwargs) if path.startswith('file://'): path = path[len('file://'):] self.path = path self.basepath = os.path.abspath(path) + self.regex = regex def __repr__(self): return f"file://{self.path}" @@ -334,12 +352,13 @@ class SstateFileTarget(SstateTargetBase): for subdir, dirs, files in os.walk(self.basepath): reldir = subdir[(len(self.basepath)+1):] for f in files: - m = SstateRegex.match(f) + relative = os.path.join(reldir, f) + m = self.regex.match(relative) if m is not None: islink = os.path.islink(os.path.join(subdir, f)) age = int(now - os.path.getmtime(os.path.join(subdir, f))) all_files.append(SstateCacheEntry( - path=os.path.join(reldir, f), + path=relative, size=os.path.getsize(os.path.join(subdir, f)), islink=islink, age=age, @@ -592,6 +611,9 @@ def arguments(): parser.add_argument( '--exit-code', type=int, default=None, help="lint: return this instead of number of found issues") + parser.add_argument( + '--lint-stamps', default=False, action='store_true', + help="lint: assume target is a stamps directory (target must be a local path)") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -798,7 +820,7 @@ def sstate_analyze(source, target, **kwargs): print('\n'.join(out)) -def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, **kwargs): +def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, lint_stamps, **kwargs): ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() # only list non-cacheable tasks here # note that these still can break caching of other tasks that depend on these. @@ -809,7 +831,10 @@ def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, ** print(f"WARNING: target {target} does not exist. Nothing to analyze.") return 0 - cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + if lint_stamps: + cache_sigs = {s.hash: s for s in target.list_all()} + else: + cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} hits_srcdir = 0 hits_builddir = 0 @@ -891,10 +916,12 @@ def main(): target = SstateDavTarget(args.target) elif args.target.startswith('s3://'): target = SstateS3Target(args.target) - elif args.target.startswith('file://'): - target = SstateFileTarget(args.target) - else: # no protocol given, assume file:// - target = SstateFileTarget(args.target) + else: # Either file://, or no protocol given, assume file:// + target = SstateFileTarget(args.target, StampsRegex if args.lint_stamps else SstateRegex) + + if args.lint_stamps and not isinstance(target, SstateFileTarget): + print("ERROR: --lint-stamps only works with local file targets") + return 1 args.target = target return globals()[f'sstate_{args.command}'](**vars(args))