From patchwork Tue Apr 12 22:35:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1676 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Wed, 13 Apr 2022 08:36:01 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-pj1-f56.google.com (mail-pj1-f56.google.com [209.85.216.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 23D6ZwoI001724 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 13 Apr 2022 08:35:59 +0200 Received: by mail-pj1-f56.google.com with SMTP id 38-20020a17090a09a900b001cb95d9ea46sf629708pjo.7 for ; Tue, 12 Apr 2022 23:35:59 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1649831752; cv=pass; d=google.com; s=arc-20160816; b=lWXvjJUkV2A7ljlP3+6o3eLGWdJOVa1FEhlHf0mZiwrleNXkD9FzAwyUi4knW2v2qF GWnKwOHZQVwaAP5l7ZbUQA0KmKn19kGVkWWXiuHOatkYyqyLFsYuPZwZ9PFRle8cdngd wAaYFxqHhKLb/8H7kgwJ5MlEuE+qzYclFzfhaKZ7k2mIIHmn8lJXpiW0nCoRIl8HH0Mm 09BPsaAbuBzgpZ4mG28mUXkq6aUpEVJX3f7uateoLicDSkr4/xIXgImuo8vcfSnHR+qh CUIJHHd5QQIyWVPZaExhjfDiLTzeayI0v0Vs59endxHTd65W2/bfHDBk3tT4lP3khjN6 3pXg== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=qG4CiqP/daRympq4+3iOaQxBrRFcXRojX7HP9Gun52g=; b=VwTdLkNL47tTbjqMu28OWj0s0bEJaQiySlIBLLMAx+lKz0OVNaML/KQAgze4UXhcLB IQgh6Blsp0IHcEDgrPcl9TVQXvhzsAPzXLnp5OskWFSvdh/oQo8vismGkO9qZMohLLhs 0/QwUCyZSxUFLfMzfvRyx6xS2cnmLOh6cjhs/ccR17epgYtZUBiu4I8Cy8Nkbe4hJWxS y7wuwPd7LGGyy2Fpb3KRcoCiOgCgEGNKnS8N78YOZ2lujUh2AAf1iY2h93RnSXlt10fM YyZDIVlF//zNZITeNSupV6AjQCd7tsu2DW85QrJrmczBJOAAISzn3wRTuVp7Cwoj5sDt BN4Q== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=KkjQoOw6; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0e::60a as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=qG4CiqP/daRympq4+3iOaQxBrRFcXRojX7HP9Gun52g=; b=KjnIfG6U38qLQJh2eYkx5sgDczJkMrShNRs3ISff89HenwwA562LzzBnSGPxBFdeE/ vhU5F+nQl2EP8Aw6dCtpmJoHB3T7HG/ncYVzhunnHjRfqtSxmZvYcZfiF0kQgikO3J0K v82d/xUPFDUaZuLdAi2VLu+kKEkTNWmWgx+Wgde/1ardHRJsdgzU42dlqp5AmwpIMXXA 8EP4xh7wYd62D/SWSMM08wGmBMOmPLmNUA021Xsx2Ro/bbFmcZZMignKBZMoXyNseTJJ lu2JLsZsAW7r7fMMe8tAYABvwByr6yRb91402D8oTWlVdbNe50+f9qOl8pEYStzQgt2S sghQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=qG4CiqP/daRympq4+3iOaQxBrRFcXRojX7HP9Gun52g=; b=ATbYlaoFBr2cDnSVkzdcmnprRZHaU96I2LOn2Q2byVsLd7hdiGb3DlbCUwvqvsOwx9 1OOpKQWzy4t9VS6NmjBAPbTxp6kDF+JhxbTsBpd+pKKBPXCn8hvj8+nO2/+ZQ7qbrL5H mTTNVhPHZjYfRaYLyInFmoAJ6HTao2WToYIu8VZI0fz7oQR21Li3AJATsPeFajYZL3tJ zMA6oVTgrdAgzuml+ob1l4dg7/JdVXNlufrkZXIvTshyNjXNUE31sT8RyAgXxufbnvVQ SSynw/QyS725cjx6mxLfa2z35lmliGZHBSGM5FYFFi979VCFzNZe5WF3YH57FE6RHh1u yJBg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM5320JXN+nvtd5U/oGevNGfYoBRoi1V2A4KUQJtRuZNmFvKSJun0+ L0tDi1snJB1NntVubNwJ9Dg= X-Google-Smtp-Source: ABdhPJxsloQlvckZ8IHzDVdjJPqyd/77xdYg6/o29BJHNGak3Za3tJWXS6N53Z/6IVex4utRpcaKYw== X-Received: by 2002:a17:90a:8591:b0:1b9:da10:2127 with SMTP id m17-20020a17090a859100b001b9da102127mr9143508pjn.13.1649831752480; Tue, 12 Apr 2022 23:35:52 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6a00:179d:b0:505:9501:adc5 with SMTP id s29-20020a056a00179d00b005059501adc5ls952131pfg.2.gmail; Tue, 12 Apr 2022 23:35:51 -0700 (PDT) X-Received: by 2002:aa7:8888:0:b0:505:a3e1:d246 with SMTP id z8-20020aa78888000000b00505a3e1d246mr19053592pfe.76.1649831751781; Tue, 12 Apr 2022 23:35:51 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1649831751; cv=pass; d=google.com; s=arc-20160816; b=nvupHwVNhScG2NimfC+M5H1T433SevWRpbHE1stCMhhyzPE0IMEcC1AYQOkXLfpjJ9 spMd687yj/5uVY4ev/QODGJSHGU9c/8Js7WlBNL4sKFHgAwna0xAA86nUoy4EcdKpq/C dw291aZ8ejt/DP78Dvo6HTVnhxSXiGzVOAJp6L/ZK92aI/CXIIi/8frlflDZ4ZsKBS3W sdrMUiH+i3tVehTpm5lzuM+/2dOgJ8RkQBi5V8uCvX9hKU0xvf8HT9u7g5K31VY5v5G7 3TTvuSpdfK/uu1+IycG65rAnINun44svqInQ3xgplMIgBWWwJHidB7uE3MC975K1iprY LxrA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=E0CaKH8+c99DFRZd6PHg2ah+/UymOSYz4rDwXKqgaXc=; b=JaX9wVrkPeNjXxtZWhECv31XW/fQTb7aFL/fcCW9SMIqbJOkVM6runJIdhJSxF4GML j8j+mnKptwYW2hokrItMLXX0EXUYKVkbxPaTpozQadz91jS2tdXwwJTn/pVajUN0/jgm Z6lbQD70kVWZsd0xVrXGpMW3xPohhgTuCR6Ga9kDZI62J6zzPo5awzCK8IE1faAL0Wmt E8c6t5fkSW513LWgsPRBcCyhZWDcmCfkZpw5dLqZMPBbjJrw1FqirsRewypwBcamU/Da XCOSA/KD3rQ/iYNp4D4hhhQQOtmUuX4PP3zZ45T3RgSfXbrHpDcLaUx4BIcNJ1SYWNRJ ivlg== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=KkjQoOw6; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0e::60a as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04on060a.outbound.protection.outlook.com. [2a01:111:f400:fe0e::60a]) by gmr-mx.google.com with ESMTPS id m8-20020a17090a34c800b001caa0d88ff2si803526pjf.2.2022.04.12.23.35.51 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Apr 2022 23:35:51 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0e::60a as permitted sender) client-ip=2a01:111:f400:fe0e::60a; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZyrNIkF+D6m+rNsvAE5kT1cPkkFJWFo7m+Ww8aSuu7k6mHC4Dqciz7ZGV5zQ6ojxf2g4OdgrDTtIxBZaRMLGWo9ZUweKRJWUzp3TKGreFyFs1OeT2uk0lALBFZKE8fMN5JP6BEj9arw6nR5rp6ZcMa5mddAVtBTZr4RopuEv0KmdwdqNneQpzVMpkkJqbrIUIV+GsPyihxiGHO5SOQ0NXBPR5fRoOHK6eym+lCayjUoqvoS5OomkmyLi/sSv3GD/fWt0tjOUnWYS2ajP5ic1i6Y5kRg7a4JMCv8WzngHncrrjd/JeQHc02ukOdDvTqoW1/6qnCts70IF3EMXgE2EKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E0CaKH8+c99DFRZd6PHg2ah+/UymOSYz4rDwXKqgaXc=; b=HoMr7NpX8ORxweTumQsvgpI8Q5q7YS1LH+kzFVquZFC9pxdoPQOqklC1luwKShm0BOEgvzI9PT/4Lm8C85JKB6PYkGAjOSlRTvcc1DOxAjzP539bmg4F2tSPgJloh+t8wg/gAdqQZDTn/MHeozcVUafUbzr1JW+q5oXWCGceCnJixRBLOIGCKdsVoVg4+yJD5pK4ZJV1Oh2fUl6D6qlsAeLpQOmm4FHWixUTGYvcPPASnmozH05h2oJgK/n6e1n73wAKulUcKSPmdbfEhARbO9cNFgE8FPRFIyzfMVu5Ghzk8yYproFNB9s51IGec/oS2jAQUPIU9Gxl/K6OuPGIGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.73) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from SV0P279CA0043.NORP279.PROD.OUTLOOK.COM (2603:10a6:f10:13::12) by DB9PR10MB4716.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:255::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5144.28; Wed, 13 Apr 2022 06:35:49 +0000 Received: from HE1EUR01FT107.eop-EUR01.prod.protection.outlook.com (2603:10a6:f10:13:cafe::81) by SV0P279CA0043.outlook.office365.com (2603:10a6:f10:13::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5164.20 via Frontend Transport; Wed, 13 Apr 2022 06:35:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.73) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.73 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.73; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.73) by HE1EUR01FT107.mail.protection.outlook.com (10.152.0.244) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5164.19 via Frontend Transport; Wed, 13 Apr 2022 06:35:48 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SNA.ad011.siemens.net (194.138.21.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 13 Apr 2022 08:35:48 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 13 Apr 2022 08:35:48 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH 0/2] Sstate maintenance script Date: Wed, 13 Apr 2022 08:35:32 +0200 Message-ID: <20220413063534.799526-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bf9d30c4-6762-4017-cef1-08da1d17d8f9 X-MS-TrafficTypeDiagnostic: DB9PR10MB4716:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FGINQqzOBAQFmVuuvGRzRVgSDkCfJ16yWej9zLvbYB4Lb3HAds8toYPJxCqz5jPzC5Og90ziRJNkimds6NjLx3kJ7oz9w2DqrvtDAZarT5nsUZu7IDPpbpTfodfJ2Pg/HlxonkwE2z1pmBoj0Xd+DFm0YPZ/+6CQ59GtLyWjH2J1MrN6LYNpb8uUbfxaGX0ixakumslTom0uJsSkuO6Y89ew2vJ5FhRTDs0JWQirvATdcRoyHIGCWuTjK+2/gLtplAfVPwPOd34L2IGfgIbSaGCFENreE5HLl7musqZhxbTwykNvVa3kT+dctPaizM8mqZN3UAebcM4fHiht8NmKHQZqKJ6rJCSl46zqjtoqzGGf+SiSQJrn+FvqS2fISf8mGQj/AAHtvpNewe0uet/+yhmAZC/sxDhTXZaqYKp+79F2Pwz8PVRu2n/2NRROsn2jNg3LqIdI333lfH3mnsBvvZ9yRS1QcAb3/AdFf+WUBqEXeqLRUgctHwukQ5epxU12zBHoPE6VtJ9VOZmE9j5JNMc59ZENJceYP87BWShec0pmRl/fljAVOfF9A8jrbPxV/Ab8rWlSCL+v5LRLdpwo1G+Ck+L+0S70J4bYwzMrMQFcDTlkxUQGy07DOm20gyksIbP9Guq8KCzTuqAA+hRjYAJH0l+JVAus1wJBNtST41G0Yp/V7zh9TDps3znn87DzmfaFBI5uwgw4MdIrl3TvnA== X-Forefront-Antispam-Report: CIP:194.138.21.73; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(36860700001)(47076005)(6916009)(5660300002)(956004)(26005)(336012)(186003)(1076003)(2906002)(2616005)(107886003)(44832011)(356005)(16526019)(81166007)(82960400001)(36756003)(83380400001)(86362001)(40460700003)(82310400005)(498600001)(8936002)(4326008)(8676002)(70586007)(70206006)(6666004)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Apr 2022 06:35:48.7268 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bf9d30c4-6762-4017-cef1-08da1d17d8f9 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.73]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: HE1EUR01FT107.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR10MB4716 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=KkjQoOw6; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0e::60a as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX Hi, We have been running CI with shared sstate caches for some months now, in several downstream projects. This is the cache maintenance script that has evolved during that time. Detailed documentation is in the script itself. Main features: - upload cache artifacts to shared caches on filesystem, http, or s3 - clean old artifacts from shared caches - analyze in detail why cache misses happen (what has changed in the signatures) The last one is especially interesting, and has already yielded some improvements to the cacheability of Isar ([PATCH v2 0/4] Improve cacheability); analysis is still ongoing. This feature becomes more robust with a patch to bitbake (p2 of this series, also submitted upstream). One issue: testing! This is not easy, because it involves infrastructure, and artificial tests that provide decent coverage would be quite complex to design. If we declare that we sufficiently trust the sstate code, we could add a shared/persistent cache to the Isar CI infrastructure. This would further test the sstate feature and all steps involved in maintaining such a setup. In addition, it would significantly speed up CI builds. Adriaan Adriaan Schmidt (2): scripts: add isar-sstate bitbake-diffsigs: make finding of changed signatures more robust bitbake/lib/bb/siggen.py | 10 +- scripts/isar-sstate | 743 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 748 insertions(+), 5 deletions(-) create mode 100755 scripts/isar-sstate