From patchwork Mon May 9 02:15:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1772 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:36 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f55.google.com (mail-lf1-f55.google.com [209.85.167.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGZ1X003563 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:35 +0200 Received: by mail-lf1-f55.google.com with SMTP id w34-20020a0565120b2200b0044adfdd1570sf5610968lfu.23 for ; Mon, 09 May 2022 03:16:35 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091390; cv=pass; d=google.com; s=arc-20160816; b=t20jUSStxmMGejeq1aWbV36TtkVf8Nfh+2BL2j6NBPrOx77cyjbO3oFuqahdese0NJ F2vmo5WQpg3S+lFfqWLMDGOG4IeAPk4VkAZMuowavNbAB3QDquWgH4vnWfvotdgA3Cid ypgPvBcxiONZRHuq7WQ9KUZczJz5LU5c1F9luEdK5MpbCjaHIxSAgF+CNRaW9PFaimCG 125JNQOWNWocHgvzUvgps2VgE2w87mCyski0WiMazOiex7z5C5rk4JPiIb3iIZy6ebpP X+ejq5JYvOZPHE5UXM4NE/tBFckNaJpPJs3rw/3N3ZPT791Rwd3iUQ+V/t0Z09zn2EO2 VZeA== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=FkkyiJyWwtGjWXGVe7mAVsVTbWOeb+zEwGx3TSjZebY=; b=1LJ5luGsSq9spM/f4S7RrEhpdo2PzGkSuxmvdl/muo1xilHgqE1dUmM6Lyw7yaC3WR WUN2UqTzakndc5UazQNKA6th9iMPevQEwyCmD6//hJ01SOBsvNOo61Z/HiHgQm7fizWm HSi74R2kERyhhg2aDT88JjjLzskOH6UF8QXBA4cztrg3uNrN8NpB7BBAQYC20gT0MBYn /OlZJSfPFb3UTsbWhQOfrOKSJbV8sYfJd/v/u0AznMLXtEaNX5VSnpqfCXfABv608HzO SyGwGsXV6r7JHydg1NRjIdmy9pYvvJSOaCyO5uKOVweXJtoS1WwrPyC4JKqrHQ8bShxc 9WvA== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=GHDrniOf; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1a::630 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=FkkyiJyWwtGjWXGVe7mAVsVTbWOeb+zEwGx3TSjZebY=; b=k8a9moZMO7baqcHGrlIDkdxEyom1AmjVBwO4uzHgfeuCFA+AziK9XI453jN563wSj+ xj10F56eq5eHuDchL+bZksk33nb8RRM0aEgL4geHvtFR7kaULiRVGd+4M3b4xW+t+I3g IDKGRC7Xuxp5xDXgt5kvFazlM9jXWZB9VAY/VM5tPTYakg8zefsdLqTPU4ygQ9dgdx/X NAYP3+mbAi9pP0Jw7rw5htTBwEtf6Y5jtftTb/cZtZ/Ty/pG9CtqwbQUQdztwKRyqBOw v6LS2Nwe3FQhtX/t9Yw7iKSRqZ+eQ7w07sxHO04DscYMAOGuPSK91HB3Yy8vf44DCNbC xABQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=FkkyiJyWwtGjWXGVe7mAVsVTbWOeb+zEwGx3TSjZebY=; b=z2nvZ43yfHohzgTI8zdQT0NxNyCdlgrKLXLxXrTDdm5deR4DZ7ms7B7IoMF0UNepad GH5mA5M5uiSwufDp8aonN/8WRjCd7YcSi5qoq0/gAGp72hVscaxxpaS9phdCh401ophm 1wRftPuP16xt+IOdsOZFgERKo5tuxbrJQrh7j6hl914nC+MC8LSMwjUp3Wna1UKxL8Nf BRcoigxeLu4q67wq/nvyhbsxkhlsreHVwK4o6eyqMtE+bX8hSTbBeXgzgQLal78g6SsH w8aDP/fTxLlwlRgFWXKnWTOYfdq2Ig3hRxcFx5Ox3CUsZBFHfQ4W1AlDXioBlzoobiZJ Lo/A== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM532Hb1w+Roo7LpVboC4Ctlls9pWLIMPDpL/yLMrI2nVGdq006tIu T6+DYVenU9ary/fWI1c/OHw= X-Google-Smtp-Source: ABdhPJxb9FNke52RpL2jD5G3/qooiAAiakKSI7XInnTGtbJmV0Cqd/0I8j6AEPzDpz4AMZjVlR+csw== X-Received: by 2002:ac2:424f:0:b0:473:f8c7:1726 with SMTP id m15-20020ac2424f000000b00473f8c71726mr10551220lfl.74.1652091389839; Mon, 09 May 2022 03:16:29 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:4ca:b0:24f:f52e:16e0 with SMTP id e10-20020a05651c04ca00b0024ff52e16e0ls2778235lji.3.gmail; Mon, 09 May 2022 03:16:28 -0700 (PDT) X-Received: by 2002:a2e:9953:0:b0:24f:2926:6a23 with SMTP id r19-20020a2e9953000000b0024f29266a23mr9958307ljj.312.1652091388750; Mon, 09 May 2022 03:16:28 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091388; cv=pass; d=google.com; s=arc-20160816; b=OPanQpwFOeKsCMgRqwUxGVxxbrprILfLePMiA09B1taq2HvmsQ6V86IjYprKZ8MVyK 63vhUG7r7WcuBCcSk1R83RwqqcV54eWaOk2v0Sd1dMgha3l9ZK3+p/0N+48G1W/A+Qrv k2Pqt9DQVzwWPI+suOM1bNIf0P1jEAYa68Tr+UMojbFj99C5a+tNqdF7Nq+XlK6oVLBZ Nxf+GuV9/rVloOIab5w00zcU6K2Jm4WRCeKlVGqG9Uhl3leWhnzo4UcbzthW3YK3JYzM UbPdyhBzxSQGgKSa8uoBVZYDtq2RoIOKIuuxA+Tn5QGkr7SmYoNdm2XeOWfcnic60UE6 Ql2g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=LukLnJJ8bdM3RtizZ6zN9vcty59nn+4XHvtEn077ASw=; b=n5q7i5Ovzyw8EYBU+hoMunJbkNnW0tkjQkHRz/OdkViLJzr7gEX7+7UaXkzfxJjaZG XQTrSJU26Es8mgJ0NA+XHquGE3Skp56lVC+uwt0MFamvnLAd1/6bKCU4mG/I/u17+Zv6 16+tu0Cl5+Lj0LGbLvEfWdk7fiDpIbp52GLB4zYm3GoZ5CN+aVr/pkTDEIhmzNyBvVsT U5iodfQJcqsPS+fQbVb9PYxdTx9qUUowOIo8L71o2VHLKg+SESaHfJNhgWbKwEtntUDa cKUEGXRRM955r5QuuhYQy9J7/Xc3W3DVi4A3hc54zq281CNswZP3XyGtRyjKXJmMcPtf TNvg== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=GHDrniOf; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1a::630 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on20630.outbound.protection.outlook.com. [2a01:111:f400:7e1a::630]) by gmr-mx.google.com with ESMTPS id j25-20020a2e8519000000b0024e33a076e7si465430lji.2.2022.05.09.03.16.28 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:28 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1a::630 as permitted sender) client-ip=2a01:111:f400:7e1a::630; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SOYc5vrADTnoJQCouYOrH4s4/vrKC2jgnK9gWGKjivCJf0EysXETOUo1VZp1wk/gVb2Wm19ytjblbFQOXjMNF4O1QbZ7mxumVWyPDKGynNXjBf8uJ2hvcSbTS+CSfegIihrEl7w5sdWo7CS3D1xcu6dGGsb3lH6acHKZAePE8s99MqQyPbcLNa1FJ0yX4Gsz0cnt4XExvZH7bMV7DUm34hAx31MVfDlQ72PM9VTmhTRF5Abl2VWcrtTtxLuQwnXbmU29nyKBbGrfdb9me4QRMr3E9VkZJwsZdBltCq6D3Z3bHQ1G0xE08U5oJCR1RHbYSkrO+IiFA/f942NgGdga6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LukLnJJ8bdM3RtizZ6zN9vcty59nn+4XHvtEn077ASw=; b=cN3V8iy51L7MmhLozbicaUaOUvsKgHf10mxrzh9ZkpDthQ8hNWzRA0KP6qdlvoGE0I2M+Bti2TNQkYg8ZQkfepPgQItVIEGjD4/Tmxl1qdbqezdUvSeUx/9SQHI5bviRLiB+yLQdxcnVVWoc458xSaYeBqQ0PXu0AkxSx2PY7rax+V7SU3wc2+7YrgJcJo0FWWfagNP0UVMRme+s3g6yk62FOayEYnUoBOFL64HgbOlOwhAvZwziZNUzyX/RkBDffk8dMl6h6o0Zd+QevHGxsQ2LS92kffqeSBGCAFt4ZWHYflMZxnU/nEgHvVXzwtyHQnLUHIXmCX1V8ShiWOhLGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.72) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from AS9PR04CA0090.eurprd04.prod.outlook.com (2603:10a6:20b:50e::11) by AM0PR10MB3090.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:15a::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18; Mon, 9 May 2022 10:16:27 +0000 Received: from VE1EUR01FT070.eop-EUR01.prod.protection.outlook.com (2603:10a6:20b:50e:cafe::92) by AS9PR04CA0090.outlook.office365.com (2603:10a6:20b:50e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23 via Frontend Transport; Mon, 9 May 2022 10:16:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.72) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.72 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.72; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.72) by VE1EUR01FT070.mail.protection.outlook.com (10.152.3.47) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:27 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SMA.ad011.siemens.net (194.138.21.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:26 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:26 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 0/7] Sstate maintenance script Date: Mon, 9 May 2022 12:15:57 +0200 Message-ID: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 84b338bb-2589-4e03-f448-08da31a4fa6d X-MS-TrafficTypeDiagnostic: AM0PR10MB3090:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GktIj/wXDLrmjU6cC6hHNX34mlUVGnbStsY2wRxKtcX7QdqnGOtKD96urbp7YoF2QaezlO/XJyQK/RSPzlhRllmGxNT0VSwYQj7t3y9SYiYLeKxY71xkXhO6Nkg3800KIMO+QD+kfilvxFr+V9fTi/7QddGH3L4y9nvShS3vZJ1YnMN/IMMlyt/FqX+awSJNx54A6IIZRpWrpcNIXtFrj81hodpLhHAeVKjUHo42X6+KoiFJXDZOdX8OvWmUMB/IKBDLPbn8R/wrSW6J5E8OOMMW356zEQCF+mNrew/OG5GSUcmPi2mG8neu/gVjbtDDv+qB9NpcX9xlHEHBYjdEEWbxukSMa7SYLmjhs2Pm6WSbghlEmLY38Zy5I5ssuFPAqHkfrSdmCs7xVD1eWeJfiQJEtXvoaMEFar+200GK1ImQ3OhXhQtYkVyX/iO3ShjTDFZK2wgVJbhIJYwg4NdfOGkjDBARgt+gSBjCkGFU4wFC7PvCSmIIsx9TKMM46yXLtVRre1uMfiT4R1kPKAJmaFo0zMYiL4uPmuAK8wXbrVwr64BzyULweQTo+ET4onynhA5JXNMO+oIhxZUtLohNrAlrf3GTGBxskEuZtOXqxZ7+C458crJONi7nI3eSzBDAp/EWpow29lv4TKfEqSoHq6wtZpoarvBHxczkA91qKxnTsIUytSZ4eCyHIFAf9Upr6eIEFMn0kVSeAM8zkpfABw== X-Forefront-Antispam-Report: CIP:194.138.21.72; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(6666004)(2906002)(82310400005)(8936002)(4326008)(36756003)(8676002)(40460700003)(956004)(5660300002)(86362001)(26005)(81166007)(44832011)(70206006)(2616005)(70586007)(508600001)(47076005)(107886003)(1076003)(356005)(336012)(82960400001)(186003)(316002)(16526019)(36860700001)(6916009)(83380400001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:27.1785 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 84b338bb-2589-4e03-f448-08da31a4fa6d X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.72]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT070.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB3090 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=GHDrniOf; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1a::630 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX We have been running CI with shared sstate caches for some months now, in several downstream projects. This is the cache maintenance script that has evolved during that time. Detailed documentation is in the script itself. Main features: - upload cache artifacts to shared caches on filesystem, http, or s3 - clean old artifacts from shared caches - analyze in detail why cache misses happen (what has changed in the signatures) - check the sstate signatures for absolute paths pointing to the build host The last two are especially interesting, and have already yielded some improvements to the cacheability of Isar, some already merged, and some more in "[PATCH 0/7] Further improve cachability of ISAR". p1 handles another absolute path in a variable (LAYERDIR_isar). p2..3 are minor patches to bitbake (both already upstream) that greatly improve accuracy and performance of the sstate analysis. p4 refactors handling of the apt_* tasks. This was motivated by the sstate analysis, but I think it also makes the code cleaner. p5..6 add the sstate maintenance script (2 authors, hence 2 patches). p7 includes a signature check into the sstate test case. This requires the changes from "[PATCH 0/7] Further improve cachability of ISAR" for the test to pass. One issue: testing! This is not easy, because it involves infrastructure, and artificial tests that provide decent coverage would be quite complex to design. If we declare that we sufficiently trust the sstate code, we could add a shared/persistent cache to the Isar CI infrastructure. This would further test the sstate feature and all steps involved in maintaining such a setup. In addition, it would significantly speed up CI builds. changes since v1: - generally improved script - analysis and cachability improvements in bitbake, dpkg-base, and meta-isar - added "sstate linting" to the testsuite Adriaan Schmidt (6): meta-isar: improve cachability bitbake-diffsigs: make finding of changed signatures more robust bitbake-diffsigs: break on first dependent task difference dpkg-base: refactor dependencies of apt_* tasks scripts: add isar-sstate testsuite: add cachability analysis to sstate test Felix Moessbauer (1): isar-sstate: add tool to check for caching issues bitbake/lib/bb/siggen.py | 11 +- meta-isar/conf/layer.conf | 1 + meta/classes/dpkg-base.bbclass | 14 +- meta/classes/dpkg.bbclass | 2 +- scripts/isar-sstate | 863 +++++++++++++++++++++++++++++++++ testsuite/cibase.py | 8 +- 6 files changed, 884 insertions(+), 15 deletions(-) create mode 100755 scripts/isar-sstate