From patchwork Mon May 9 02:15:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1773 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:40 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f59.google.com (mail-wr1-f59.google.com [209.85.221.59]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGeuM003567 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:40 +0200 Received: by mail-wr1-f59.google.com with SMTP id t17-20020adfa2d1000000b0020ac519c222sf5556012wra.4 for ; Mon, 09 May 2022 03:16:40 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091395; cv=pass; d=google.com; s=arc-20160816; b=yzCYLYp1JmeqJFKxfADL5msyT1GUQa55IOthdhL+2/0g5Ga6git92rOL7ji5knPdTV 1gfGFP0+fuED22QjlWslUK8nuxJieYJTQa1UR8wEP+yvEJXaz7XCcBVwHmG55ug4gspQ 1rYBp9rJTXu+ihUVddPM+LzOb0xlOzGSuP6lFTKZRov5+p5uQakj0wrxIXqpgEE5A9tm 6wZAb12cixkBjijP6aiiYEuXjfvhe9V+xVOB09fcVPgT4fQvYC0H2NdzYuJzbEagvfDH MBHJYGfhlMhBGjxNzeOr1zk0G46k0lJTQkSrxm9QL80cXUsvJC6LjD/iYJtMld1FW97B q34Q== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=gnYNuTvO0MQ8KVMqXJuIbQEM3SUi/fpA4v1aX2XHkh8=; b=xVq8n1+ALDuy1imx/y5NBb9UQk8OpDw6otTQ3pNLKYWAyWwWzpZ7yIYEUutxP4puzN G5dXHyutYuxRREwqNoAo4Usw1eHiiSh4pkTnU3ytWTfcjOJJ++cWWLjJa493o6OobUYj blj7yoZWRASnWSvsSLL8zW7SexV5rJtToaUcrpQgcRQ5tZx6IInUi7sF1JWWKW4v/n47 nD/mzRv+hEwVhy+rolyDCj9SuJGA+G/aMt6SehpQ5gG42WdiX14XVBYUWcuRXkX1us7L 7zhFijv4EYipoLQON9RNIqrVZnsqb0H6DL2AWAKLEI8IExG+QY7oV+TvEHP2lnC9vfCu 6Mqw== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=l+bhYkvi; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::610 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=gnYNuTvO0MQ8KVMqXJuIbQEM3SUi/fpA4v1aX2XHkh8=; b=XxMLsVKrqXa/lYGJMt6/HuJYOZdM4YmxW4zIRXO311mvRPIPcyNe67/AfO3Ry4dkt5 Sbg6uLtA7e7i2Rz/BlgCNU/AjhdZrTpkKOxMHNJxaDEQbCGYhtn2KIspvOp0KNYNpJLC QmifsvKc3dNq+dDL91Ilz4GDmN1hPTfhAq5ksGRGA8qealgf+KIrc1KGPdnksdjPblVp mLjJzBnWG/xt6HZYo0+5DrcofwTROJETmbA+FNIGmtWLzhkhq70oMrIBFvvzseMa/57s Jxl8oDF8vH9MXUzI+bNBYIdGLs5B0UDv22b36xtTtb3F/bz+Jskg7fQIimCcK1PB18/N 1h5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=gnYNuTvO0MQ8KVMqXJuIbQEM3SUi/fpA4v1aX2XHkh8=; b=k7LEmBoPK2vUBsHHLhRxY4id5svk/QP/upGA+WyvazlpLDre/zUq1HdkORxbthjD9b /aJyooxXO4Q/vu95yAQa1nwFknbl9ZQk2bwx8ZqQYTRp5DXF/A63mSAub+IoAwLZ0pqe eURzo3NjD7ItkNjdpmLzoJS+PsCRD0FsCfDHFI0Fu2XINDh09lvZNfWqPernwsbEA8Ob oexD/O857JaszSbZ2N8nsE4kCGecwAgcgXwo2KqaZU1D/jewlUtLA1nt59f9vf2LjbZ5 cHqzkAWLwel9Tj0xkAmIq+QjXOzSG/z0lK2lSNxLxpsY1R4/uY6dB65N2eRckBik0bnG +x1Q== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM53091kjTml1JRqkyWYRPZbzwV+WMdBDxLDX9KNReiB2yrgsqfw4i 1/V7YFESv0A5aCL/6ZxPFBo= X-Google-Smtp-Source: ABdhPJzQSBOOI9NjB2Tdz6f7Lbp6T9B3ZiQgFSpXtUYwr4Mi6M3bMXzU+9xSGPvNVTvmHNyAzUCr8w== X-Received: by 2002:a5d:4205:0:b0:20a:e23c:a7f4 with SMTP id n5-20020a5d4205000000b0020ae23ca7f4mr13520796wrq.576.1652091394743; Mon, 09 May 2022 03:16:34 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:2747:0:b0:381:80e8:be59 with SMTP id n68-20020a1c2747000000b0038180e8be59ls3662726wmn.1.gmail; Mon, 09 May 2022 03:16:33 -0700 (PDT) X-Received: by 2002:a05:600c:1c95:b0:394:709f:a0db with SMTP id k21-20020a05600c1c9500b00394709fa0dbmr15592811wms.82.1652091393859; Mon, 09 May 2022 03:16:33 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091393; cv=pass; d=google.com; s=arc-20160816; b=Eshnq/Vnn67gryAljctzf7OdQ/qXGnf5k/7JqehzOb6kQHF1ulCDHbCcAWUGSAsk1t qxaJ2y9PMwKLdIQUgLzKmQGqEcmTaWaxPeuQmSGVaKoVkm+T8t03mTs8jrtChLcVV4XH 5ywnbHRSjhQAZZ4OVzyhXWXEqh5VRBvMexXyAuH5xjlv1ulU/3ZPwg7Dg8rCT1KTz4MS R8P0+TDTrmdKd1PpNkUv6YJFtjesYaF4NaypazS7Uk+NamYy5xVITiY8JieqmGsvkdu9 K5fa4XOhHqkOT+xc9A/uJapozilEng5V3+f0gaMN/BOYaX9A7diI2KuOeCIpbmifl7jb Ierw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=mNb/6XY30UEXCxAXJZK9KImxEiEIhST41B8zkT4GvKU=; b=UDqLF6QIOgWpJPlVdr7TY3joDpE1K66K0jJZCZAsvbiKI48toML54IofETw1ui/QT9 XyMLbfp9jIxsEoQ+Zxg6DgG37m6s9H4SYAXa/tiQDF9/jiyRKmYHZBT9g9tvGWqIpYo6 qDPeAT2M6C5/Is4BhKqw/aAaulmYOEBRJvEo68vifDqlvVE8cdmrPpvfxqu1/B/cnhdH 4W2/Y4y+ycwD6oUEl4lhGyGrRQ++fGBwdkY/NMYV4wybchViRYEFp+OAmEFnoMCHPXq9 MKFccXZlTUWuPQdHj154K2H4evwL6VpXRpVYZYlA4+fPufw6DN/6a8Z0yf1C0nzM8+Qr vxHQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=l+bhYkvi; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::610 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on0610.outbound.protection.outlook.com. [2a01:111:f400:fe0c::610]) by gmr-mx.google.com with ESMTPS id n2-20020a1c2702000000b0038e70fa4e56si506192wmn.3.2022.05.09.03.16.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:33 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::610 as permitted sender) client-ip=2a01:111:f400:fe0c::610; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P2N2dSbofK/6jl/V6kFt9aIAfDBxCq7oL9z4wcmCTF8SMpvlHvRrQKSsSsAS5PRvZKgUkQt7HF47Arb6ifIOBRg/DI9majRkeexTUnUIwgQ9xR8c6IXPVvnENvUZHrWPuc1eDyezzvjrYyirJUN07xrrhTXMC9MNSWssRjpndsjKwfK3qq0XomycnsaiLpXqA3c8XftiFMjVV6aFxuExQiiZyKoCPky0QuLVS2mw6hurblRAJRzlZyScP4V038tTtAkck3oCVKk06Tp0uEaoJF9KEHwLeBLi/Nij0fA82wDP9BWehKfhZ7gqBkRXNUn4VQYFiAkzXOelD54mdGiJiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mNb/6XY30UEXCxAXJZK9KImxEiEIhST41B8zkT4GvKU=; b=GFv4aeUjcqjyfCeaQ69qJSwvo7aFxl7pMK7TP/qZGOlHgJxFfH4vZfT7Pg+LQgDyUQCZt4SUz4VP2wrkeT3Q3Ug44Zn8tWQQlGWVjbP6iGYppHe02jotYUP1ko1NMoue78ZlnJtQYaf8wHlPR8yvIMeFJ+x/CLKM3xP7EYcSR2GRyoya+B6ZjaBmJNhoHEvwJz1yHg/jhLo4LGlDZG7o9+lprUuPpatcRcV1Cv0AMBtslA2i20P82vDYS16PCQ9E8aXtwCXYxCgUJc9Els5383XMNp1e8Rql/pNQbzycB07KahLRUHX2V0+Pu2VuGUlH/0RvzJUlOw33aGTyjva1kw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.71) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from AS9P250CA0002.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:532::6) by PA4PR10MB4528.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:107::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18; Mon, 9 May 2022 10:16:33 +0000 Received: from VE1EUR01FT034.eop-EUR01.prod.protection.outlook.com (2603:10a6:20b:532:cafe::5) by AS9P250CA0002.outlook.office365.com (2603:10a6:20b:532::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23 via Frontend Transport; Mon, 9 May 2022 10:16:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.71) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.71 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.71; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.71) by VE1EUR01FT034.mail.protection.outlook.com (10.152.2.235) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:32 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SKA.ad011.siemens.net (194.138.21.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 1/7] meta-isar: improve cachability Date: Mon, 9 May 2022 12:15:58 +0200 Message-ID: <20220509101604.3249558-2-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: deb15a83-f625-4479-e635-08da31a4fdaf X-MS-TrafficTypeDiagnostic: PA4PR10MB4528:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3jnIyBK2yBp/HxAlTgRm13m6mu/utlbGQ9/srD3rI7IYq6x6wymmCqbxS5h0CGFL1R3ipcZ2JAXGfHDV1myqN2ThXPFA7zcxd6YZPtztK/5Oo1+nbTMq9Bkf0nG42US6AIQ+eB2xVpEAdzxbaQ93y2BHEQal0OrUnnQV6YrVZ3ZjdmyNjFOkBQ/FicvOjlky6ccWBOaeuGsiVn9Z8WZ7tTWjUJlKKpVgK1ZQMjd44RJXmP9VemTPc2nHFvvXWrP/d9jnWy7xXU6TS5DJSZ6HWuUXqYsTmZFrUnyOlo0BrEUIxj1IrvtzU7jAAAbUCmzRR86GATTUE5ZcXLrN94DroomNxLNMf3qrjKkFLv+W3h6R1t88Fmy9ouSdQBrU0n49qdIXsjcAB7gZF1bdV/HDNOvIlO48soIdBXB/xkL35os+vJTdCOO4h06gqVP7W0LSVNAq44rUWjLZczChvdheU4OVhBV2023vZ1pG839Dbjxkm0sVMuY4fpNsGEeTVLGd8YeoWLXM+0xsCdILyDzIJ8afFZjE2XdvzZvy66IDwa6vlJ294TGeU7iXJ4t4FJ0eEW1E0RW5EAwhmHF67G/JKOUiDX24rfuFgu+06xuT+paG/EhA1gE+Ig+8zxjfYdUu1fv8aGSJfSJGplf1kdPmgiEdBsoQG86Fgid5RSW6/BzQ/HZ20Tv1J8yu8nDpxZpPPSmVuPBZ8zvRBkUxwzWSKw== X-Forefront-Antispam-Report: CIP:194.138.21.71; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(47076005)(356005)(2906002)(336012)(8936002)(36860700001)(26005)(16526019)(36756003)(44832011)(40460700003)(4744005)(81166007)(6666004)(508600001)(5660300002)(86362001)(1076003)(956004)(82960400001)(2616005)(316002)(4326008)(107886003)(6916009)(70586007)(70206006)(8676002)(82310400005)(186003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:32.6438 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: deb15a83-f625-4479-e635-08da31a4fdaf X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.71]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT034.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB4528 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=l+bhYkvi; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::610 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX LAYERDIR_isar contains the absolute path of `meta-isar`, which breaks cachability. To resolve this, we override the value that goes into the signatures. This also demonstrates the recommended way of dealing with this in downstream layers that have `LAYERDIR_*`. Signed-off-by: Adriaan Schmidt --- meta-isar/conf/layer.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf index 9939bdc1..dec2658f 100644 --- a/meta-isar/conf/layer.conf +++ b/meta-isar/conf/layer.conf @@ -17,3 +17,4 @@ LAYERVERSION_isar = "3" LAYERSERIES_COMPAT_isar = "v0.6" LAYERDIR_isar = "${LAYERDIR}" +LAYERDIR_isar[vardepvalue] = "meta-isar" From patchwork Mon May 9 02:15:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1774 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:40 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f57.google.com (mail-ed1-f57.google.com [209.85.208.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGes4003568 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:40 +0200 Received: by mail-ed1-f57.google.com with SMTP id b65-20020a509f47000000b00427b34634d3sf8037332edf.20 for ; Mon, 09 May 2022 03:16:40 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091395; cv=pass; d=google.com; s=arc-20160816; b=rXI42x6VU3aw0vMKXvPwNmcITuad1DiH1ieM/Xp95lJhKnTpuoIdza4MKJRbZUMS7l 5rqHK3hh1KchBbjHKUE4KRN5i1v3ovyZK4bu9W20dWF7JYwg+e4L8MR8lMgLmUaA0m/5 +c+QsPaiL26FS0gT959SnrG93u/uJYZoQ5BjgrtvVWD/AJLhGX9flwwAK8bkvcnUyxc4 +GxpqJjp3SG60kUheH2vva4mPGHJFdNOjgmbTn+X3AxZv7b79fXsMtwD+XYkkQ4G3x+X CcJvZLz5k6jwT0T3lPPNSDG9jZ17oc6gJyhYW4H3t1DReLNzqYIkdmFFG+2HlOdDu4L/ vQ7g== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=tSHL7dq89xxV7qR5sHPU/fTXXA9HEYe77zG9sBwztLs=; b=O3UnCmtnq5BHXe6skTLJ9NC3twGFjQf022qTBMXpPgoHWbolEzVkjXqA4bEBBBMYN2 mRUWRXhA1CEAeTYu960iJZrfEoFTr0KWjIKoxWFXl3nY8TEFBYBlYyFXzb6jcGaDWr++ LlhKhQ8RTaMAzTYOBGMH6bihMbhHS3k7lSdCYJlxdoLkX+/sgZj/gnvAcNsFVDvOv8UW oYx7qnG98g63U0s59MHtHbIYf6Zoetxn85H73P3kJWMEGvfVS2ZIenYMHECVR7hWINhC w2O4wtGWfkKrH57twRC0ap6dR5hnx7D6iCdR0dMUdzlCe3UyMvIhv7pBD+XQuwn8PDI6 L/Hg== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=h0j+SgTv; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1b::629 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=tSHL7dq89xxV7qR5sHPU/fTXXA9HEYe77zG9sBwztLs=; b=aB65oquYnZqWhaKhtkSBjPHbMvOKlyZQ6/z6OWGLcmk3u1aAOIXa6PSM/d75C/y1ih XN9BJfUJ1YLpN4SGSIwIEiK6c61DZRjI8ytI0EnghbQu0mdwddfzJ205ZbIgXDjII+sD EmKYAN32IK2ixVDbVnQDPoHiEkNXQxFmw1BZRJSlOs7qTCsLuZNaWzpcwCnzZLsJ70oh eKw644tQA7EajubYrddbsVEraHuA8nYe3nMe9mWioUSc+yza9kT8VOktIkTK/INwwlL3 tt3vT1XwN+VjjEo444x7QCxauEr0byrYUMguoqrbomUtc9l+Ws9YomhJ2aeEc5H48PnZ efSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=tSHL7dq89xxV7qR5sHPU/fTXXA9HEYe77zG9sBwztLs=; b=zpm9ES2/DnD4Jo4K6Ix2CnzeQG3ZoScRO8gF0F6Vzc7E3mOBkPVjt8dGwVSyvTOWSw xpnG3mTvLA+CcbjZHuZhQiYnDIHyJMJPVZSiq4kEkqG8R3+5RIwaKi94CgcDPIO7td7i 6FLEORk15OV9MsXfYy7SmtpIzlqLeyE/OoUPZR8n3AvL/dBy/is/4tGWFdfmv0LBWxY1 g4Sa0B8mQS/BDBYbWnuxExGY96m7DbhX4922/uGi4zzEejy+ohEBIR5KmDTFqsuGILXg ex4c+AUvKIYxwYJDlm8heBMqH/y+8CrFD9IKAy8JJd99A7FPBgMHP6PGJjKVG8wnRp7h v4eA== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM532cuSozlQK6eWMB82U1bh9NiOSBgaLjJc1uRevkmXRpoNbFT0Mw VTe9BsUVTVV/oIqk6l4Dmn4= X-Google-Smtp-Source: ABdhPJxGC3Ej3MUx7+TeX7oqowMAZSQJiDtkdpo2l+rr3GE1gse3ZOlAZ3yz9gEeQRIwq/bUUHPOnw== X-Received: by 2002:a05:6402:51cb:b0:428:2617:64ba with SMTP id r11-20020a05640251cb00b00428261764bamr16537812edd.135.1652091394856; Mon, 09 May 2022 03:16:34 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:907:3e11:b0:6f5:1321:37ec with SMTP id hp17-20020a1709073e1100b006f5132137ecls4119022ejc.11.gmail; Mon, 09 May 2022 03:16:34 -0700 (PDT) X-Received: by 2002:a17:907:2394:b0:6f5:108c:a43 with SMTP id vf20-20020a170907239400b006f5108c0a43mr13828529ejb.28.1652091393958; Mon, 09 May 2022 03:16:33 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091393; cv=pass; d=google.com; s=arc-20160816; b=bSelfYUSIXUvja08nJBhy3xAfsceKTCvRswWNPsli69UmH+gpB1xysZhaYJRK0RXLb THLG0MKeB1B4gxiCPiiDBfyJgTJKApZTz6/wmmtoHfBaTTcm5a4uKYTk0Ejnm2N9ouJm k9n+058a5YvZl6S2Z9d6w7tQIFWTxbufaP7Q4bVZW12OKRFAmOy7zuAbclfAIqalFwjU YsdGu5BbITwunabgzVJnOvKBB/DTSKny5aJwtTfnI/NLewjBfyOPL2WAHmmXwSBBnyp8 ax47yzVS0vfcsMcMgefen77v1Z0kWkxjaJv75BonwqXjg1enFv8bg4OCYpmdBBAHd0Yt ngnQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=Ge02Nudm5zzdK+VXUw1/C6pTnZ+FXc1MQU53qtSZKvI=; b=S4hzkcSDviASXIVNSp5Qp2sNJ8+DphpXOTbFdYS8/O0HC/XfkH0I95GX9SoL2CJadc EKOjw65YW/JD2NnvoOJIn5m2Y9GoRG7p5A1QmUj3dDawo10Ny896gW5t+Ky2xxGpSVL3 EmeJAaUc3VXqA6Cj+9hLmKfKQQ+RxKdzwA+ZLXCMbcbc4qoCE3naDCpDySrU8Xzz4lq5 iVI7828+0lWMtmRP7R90HdurjEZ1dc+mzzaCj94N60VRSzwrPxIP7vRB1mHZnzBaip+O Nexkl8ZTz5UgFe+V+QeG20j/7OjhFYsLaoNgczQGqyS/EwLT3FVo4/snP/PtQ3d/aAmT ANLg== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=h0j+SgTv; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1b::629 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on20629.outbound.protection.outlook.com. [2a01:111:f400:7e1b::629]) by gmr-mx.google.com with ESMTPS id y4-20020aa7ccc4000000b00425b0722545si722856edt.3.2022.05.09.03.16.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:33 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1b::629 as permitted sender) client-ip=2a01:111:f400:7e1b::629; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XaPQUI5OGluXJiXnlFFj7sfF5+hiBz1id6FK3Nf9CKxSGKPHcm6ObFPB/MdxTf1VJ74lSnLOVs+SniCX0ZcFcun9BFl4WFlDiS4SZbM6omzqz+FipDsL2/Miv0yRynM5gDKsWAUCPk2fFfo5a9zM4BrQcN/WJqI+q0EnJv6epnah4T2rqAJBuZH/u30MZ0kyhFHovkoWtCG1ZHL/CL8eAG/0HF7oeiZ3IHTEUyz4rxLcUZ3yuerbFiQas78W/Iz16myNRRrDMA0LvjVVRBIk5yfbqTeipwp59rIlrZe6svcQxpoIqxz10wBwGB8vq0HR+rlTMPeW7arUli5S4Wa1iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ge02Nudm5zzdK+VXUw1/C6pTnZ+FXc1MQU53qtSZKvI=; b=QSrQNIxrPaDF2EFRDuGzsCIXQjpGJ2zeviXFpaVbetpeHhR1R5VOnKqweNcbLuYF3gA4lxI5jUY2Q0MTBc3/wnvGVP+74mk5BHNgRuUXcd4HC0QVpBVB8XLYZRmTDzMxhucvYAqgaOOxZUl7CeQozxlr4cK54X3G91QBIUzBKkbx5VciOBWJwIeIhqU79CtbMa0IOZ0ufxx8lXF9tAMLL/7sLd+oOpf7MKmnIQGYLbe2+gYL3Y4RsjGlxiiVd+wxktHkakzJeY03yzPXjFCSpyLl7cw8EZrfl06p6XVNVTdsJ39b+ZAe04XOSu4Tkg5IL2XoW/nF/SuuG2dLz7SBIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.70) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from AS8PR04CA0166.eurprd04.prod.outlook.com (2603:10a6:20b:331::21) by AS4PR10MB6136.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:587::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.20; Mon, 9 May 2022 10:16:33 +0000 Received: from VE1EUR01FT058.eop-EUR01.prod.protection.outlook.com (2603:10a6:20b:331:cafe::24) by AS8PR04CA0166.outlook.office365.com (2603:10a6:20b:331::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23 via Frontend Transport; Mon, 9 May 2022 10:16:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.70) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.70 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.70; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.70) by VE1EUR01FT058.mail.protection.outlook.com (10.152.3.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:32 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SJA.ad011.siemens.net (194.138.21.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 2/7] bitbake-diffsigs: make finding of changed signatures more robust Date: Mon, 9 May 2022 12:15:59 +0200 Message-ID: <20220509101604.3249558-3-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1389eb25-7dae-4bd6-76ed-08da31a4fdce X-MS-TrafficTypeDiagnostic: AS4PR10MB6136:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: d+h2dV7VSuNm7m8sFi6FDW6nve/FKK0d66F6ifEu5LJhU5Y5SkmSYLEa1OXgJeXILDTC74pBGsKC1j5b/1KSkAo7ctiNaI6QeR0pXesVPsn/bQ1MILxHNQQs1HxwWHGo5oMJ1l2QW6jjEEtVxF0wrN2gXiNcmeVPOYQfdWow9J4/p1Sq33MrI5aDqI4F41WYSQrNkge37zGqjr5WnXprpIe0+/Wvx+fO66UWnQV/yt7s9t7qkQ6Xn5dPJD8BC1CHrYXoOBt4SAkiSVuuxoZ3PtMUa/uokyUMRxsMS2TaKspyIdLMVxY3SpQ+oEr/xPXFa7H9Wm8FBaTNEliQ7WpbfN8kBioMmzDCbkjZkO099weKYIR2SOnYUZVtWqupn2bWS44UL7Px0uHf3DaEr2fOL/g+M4OTr0aeQxTXQycCKKwbjQ5xUcUaD2vfc40I7haK5VKE+Kelem47+PpzoEn4tcVi10SjSavW98lvZ3FLEAvw0MAMYIO9QI0a9Dyxb1EHrHCj6QNzbgBM1LVfyBP5twlSH5+sIU43E13osiupWTjdhSUnck0EQfvhu3eHdfBLGxNaEAZ+TSAd2LXSeaXW9Qm6/O5dkvFZ9vMAuvx3DmhxY938vy7Fh9sfMPHuzD2Wsyxb5DrrvsoKJGEuwYeMA+kGOWJYQlfSqryh+SQdoXv682k2a0TKgGYr1Eb7Igce1LJVRm8jevDt3olQhzKReK1ALZ4TlzwJVK00OaBp+hFvTgWu8o6sieyEIXzRtewz37jIxOhL1i3K2Pqg9YKQxw== X-Forefront-Antispam-Report: CIP:194.138.21.70; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:hybrid.siemens.com; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(47076005)(2616005)(186003)(508600001)(1076003)(966005)(26005)(336012)(40460700003)(956004)(86362001)(16526019)(36756003)(6666004)(36860700001)(70206006)(6916009)(82960400001)(44832011)(316002)(70586007)(7596003)(8676002)(4326008)(107886003)(5660300002)(2906002)(7636003)(83380400001)(356005)(8936002)(82310400005); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:32.8285 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1389eb25-7dae-4bd6-76ed-08da31a4fdce X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.70]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT058.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR10MB6136 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=h0j+SgTv; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1b::629 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX In `runtaskhashes`, the keys contain the absolute paths to the recipe. When working with shared sstate caches (where these absolute paths can be different) we see that compare_sigfiles does not identifiy a changed hash of a dependent task as "changed", but instead as "removed"&"added", preventing the function from recursing and continuing the comparison. By calling `clean_basepaths` before comparing the `runtaskhashes` dicts, we avoid this. Backported from upstream: https://git.openembedded.org/bitbake/commit/?id=7358378b90b68111779e6ae72948e5e7a3de00a9 Signed-off-by: Adriaan Schmidt --- bitbake/lib/bb/siggen.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 0d88c6ec..8b23fd04 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -944,8 +944,8 @@ def compare_sigfiles(a, b, recursecb=None, color=False, collapsed=False): if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data: - a = a_data['runtaskhashes'] - b = b_data['runtaskhashes'] + a = clean_basepaths(a_data['runtaskhashes']) + b = clean_basepaths(b_data['runtaskhashes']) changed, added, removed = dict_diff(a, b) if added: for dep in added: @@ -956,7 +956,7 @@ def compare_sigfiles(a, b, recursecb=None, color=False, collapsed=False): #output.append("Dependency on task %s was replaced by %s with same hash" % (dep, bdep)) bdep_found = True if not bdep_found: - output.append(color_format("{color_title}Dependency on task %s was added{color_default} with hash %s") % (clean_basepath(dep), b[dep])) + output.append(color_format("{color_title}Dependency on task %s was added{color_default} with hash %s") % (dep, b[dep])) if removed: for dep in removed: adep_found = False @@ -966,11 +966,11 @@ def compare_sigfiles(a, b, recursecb=None, color=False, collapsed=False): #output.append("Dependency on task %s was replaced by %s with same hash" % (adep, dep)) adep_found = True if not adep_found: - output.append(color_format("{color_title}Dependency on task %s was removed{color_default} with hash %s") % (clean_basepath(dep), a[dep])) + output.append(color_format("{color_title}Dependency on task %s was removed{color_default} with hash %s") % (dep, a[dep])) if changed: for dep in changed: if not collapsed: - output.append(color_format("{color_title}Hash for dependent task %s changed{color_default} from %s to %s") % (clean_basepath(dep), a[dep], b[dep])) + output.append(color_format("{color_title}Hash for dependent task %s changed{color_default} from %s to %s") % (dep, a[dep], b[dep])) if callable(recursecb): recout = recursecb(dep, a[dep], b[dep]) if recout: From patchwork Mon May 9 02:16:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1776 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:41 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f58.google.com (mail-wr1-f58.google.com [209.85.221.58]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGfh9003576 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:41 +0200 Received: by mail-wr1-f58.google.com with SMTP id e21-20020adfa455000000b0020ae075cf35sf5634438wra.11 for ; Mon, 09 May 2022 03:16:41 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091395; cv=pass; d=google.com; s=arc-20160816; b=RA8Y44iWd8K4bQ2sSiz4qZqxwRKEaXBTdF9/UofWokrCI6aQRj21SudtvodaVfZAOE i2dPXEn1n00TZ7Cdor+3msjG+KFx0AHviDGHmNLYKj7V4XavEpXlJ4aD+DX+lVSwc0Gf 2qeYZwxe7Ygwlwc533WjhdKZnbEgrHczaA3Qcg+DbZZjWWcWt6tFnlueLMKVJf+ocplN LoxFga5ykeIpVQ8L35t9nUCMCG36jQlCRzmwVFptx8kA+sBi52bk7wBtqh0qNyuymZpR NEc69moMKMIv4CZdhC39vIPiYE9XFguKax5LPgRHkLCU11OTkS0a8wUlmxkqPnySuigt zqLw== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=ZL2uqbe2rYaMuk+DI03C4187lUjCh1GkIHCp28WhTGk=; b=AmIcuPQm8OndVXWSp8IMgY0yRrCSv5awpiTwS4oxKhQuV5DoPNSm5RvbV0ridsqWxW msJVi9cEPPGm1pinA5eh3Vyx1gQHnDaJ7Be6NOfhVQjxTJY2y+A60IV8ULMho0qaVLma qb6K9TVPJu4FZtxwNk6io1mVCzvzTU7ujYvjFaLJH/xgfllwLuWIcywMgTbTfXCvnY0X Wo8yGIAe0aP0UQxkX54dD7MW/oPCW775sx8RY7KcQQDx1cjDJfpXA+Aj7w1R8gPNrh8M TfsYM3aDui6Xe6QoJsINze3JN44DM7iVyfs68pM01MXEkSR8Hjam45UsCoXYD1ZDEXmL a/AQ== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FvDu3oeJ; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::603 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=ZL2uqbe2rYaMuk+DI03C4187lUjCh1GkIHCp28WhTGk=; b=UmMjYu7SaZ0irpli7j4Th2Ll67/alwVzRw8jp3iCwptO9yTCttF8ayh5dl7dbEv+xK QnZuuq78P1NNyNPWhFG51hH2n0bfV1LhZbHZlKGQy5TmF+WJO4EamgWmsG9Ief25QCvI 1AyZEW8D9hn9eqvIc1f+dfDjtqPfpP4IdvxIfYWrJWYeOIePeDmUrajY0Hq5yW88hqvx QfXFGhdFH0AYUo9Cja8vKcWgXbgHENEYIC5ZWXu15UzN0pifMGzwYZ1E2EcMoot/wjJ8 gRX4E9e1ca138WQ+F7OFOjG9rJFn1kSCzo8hODgM2Riccv4HtLM5yGkkHDtSFMd7RxKA Hivw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=ZL2uqbe2rYaMuk+DI03C4187lUjCh1GkIHCp28WhTGk=; b=YruztPtiWCXc6kYmfbXflhmaSrj1LDCM0I9NJbzlSOBBtrBLtR11xE2j/rbJy7nikh vwGor+t5bL0ou6Cbz3MKojdUf2a+qBC1AtvFated2mclvj7o5/zdtSALt/pJ8xH6FRpE bg1SR2edGhzaGZuV6NbS5JSjhIB3gC310EIj2AaY2szMeFZTeArl8p6/e3Amf9YCjVnq rEDMvYDdMU6wcH7jUXJX/I3PQcnU/8Iz35vghyFKG5rpnzCOEEZ2c+17GvkRIESAqOMY TlbE0XOCAQhdpt2u/4KxtRnsq8MBkCX2w6TGOmm3uaXWlHXcauanH+4McFHkeCtfy7mF yqog== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM531+fjNZVHkolMWlp1Orbw55F64JH9X4kycq2254nbEqcbHwygHE nj0mnunYUwoL4U88UG0KkaU= X-Google-Smtp-Source: ABdhPJw3fvbmqS9JzsNuAWPzoIcbYHq0LQ5JTR/GcGQMka6tA+rJpDTvJ8uBKEXcbUsro89kKhCjrg== X-Received: by 2002:a05:6000:18ab:b0:20c:8d82:52c3 with SMTP id b11-20020a05600018ab00b0020c8d8252c3mr13172707wri.701.1652091395726; Mon, 09 May 2022 03:16:35 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1f11:b0:394:7a2e:a847 with SMTP id bd17-20020a05600c1f1100b003947a2ea847ls3666698wmb.0.gmail; Mon, 09 May 2022 03:16:34 -0700 (PDT) X-Received: by 2002:a05:600c:4f53:b0:394:6a35:79ac with SMTP id m19-20020a05600c4f5300b003946a3579acmr20149165wmq.36.1652091394699; Mon, 09 May 2022 03:16:34 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091394; cv=pass; d=google.com; s=arc-20160816; b=ZeXddGnpckZriScRJAEQpj465FHexbdmU6KlyVLqnct4WUrrsKQMPXKaGj1Y2DbDVM g/sIZZpbckrZwE4IJnxMmxdS88hzrotBzo1BKSAwU338gFNDm/8RnYpIOnp/+LZk8u8v XyVz7hxJGa1t9fm/1MUJoaOWmtWHW4jQW2g0lxfsor63lp+hnXZCsfakFVnwDvq+pLdc fRuPHyP5Q+CQIR+dh9shcUKTp2fMpa594kTEVVdcsg2ZyxEDuaVQpP5RwDvyKJlTmjGw Ik7lfJOnO7k/JBkFt43SkBY2Ch6qP67yefgiIf097PZpMRkTJobbzeOb7KOITobeP5Ka 7XKQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=SykF9y3JfQciTZ12LDItLf9tIkNUOJl8u6ITLYQcKXA=; b=QiEODYcgcmk8a0+//ZZy44NH9IyKQaFnQUKvYevKTRdQqzRxvtG55qD8s5RKffUY1m 3ckaZyS91miHAy1PkdbU/zCuqsUp1Li0/5pzH/bkWxU1RcU2Fp5TNvAIHXTDN6NcmCaT pJW9BSPKZdCnR0aIFFhUpKZgvY8Em5KYWNaPLappWPn4fqkN550VSnQM4YrLmvKqCSU1 O/PMiiEbwruBjNWVaPr4/F7JnYr0OsdEcYRVx/W2MUzwYZFoXhCmvfbJqFeBFZom7e7l 87Lkqy3oMRlLZqvlMMt4YsSClA9EE8aHG70eumgU02AUNtBfQgeFR1sHzqyAywpf+x7Q NOlQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FvDu3oeJ; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::603 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on0603.outbound.protection.outlook.com. [2a01:111:f400:fe0c::603]) by gmr-mx.google.com with ESMTPS id n13-20020adfe34d000000b0020c67de496bsi567464wrj.2.2022.05.09.03.16.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:34 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::603 as permitted sender) client-ip=2a01:111:f400:fe0c::603; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GF8cN9mzqYpIecykKRe6VXswHYGsqBjpx8UwIYK0GvQJU9yga0wKenDb0cIX2v2wT6Z9YRiVmEcPUdlc76fBq1OxGwErZ0ZpOXTwnc150y/r0n2d73b1R+cGsgzwvG9c497Qd/w7dOotBXwySNV1IcNKSR3jVxuNtNefGPusvzBoVEVW7hD7gA1fD7gWDEJ5HlW26aEucv/hcRibogGA1UXG5/Z18SKRtpSeHA3E4zFS3ROdz0rSHsqcQ/MDXh780mwe5PsOFmd93S+z+Q6HDbyAArQ3ha9+FB2xQ93CT12301Z9i2jjVgi/eBvHPnabiWGBbFdFxxF8JR9NfLO/RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SykF9y3JfQciTZ12LDItLf9tIkNUOJl8u6ITLYQcKXA=; b=TWn2iACgeYYvFslWQBZz+ZLNhR9yiwWpLKjTeg4bAf441Nf16dFQuWGubSBZalXQwQEEYeafDRgTV+rciBp/J7BElnjavzQWAFKzsaJR5Ix4vCRZhrJCAjPayBu1dVhkrcMWN+ZjPLyR0x/RYkFr7N6SSlsZnOcaN+9uGR13RWCBiULqTyNnMEqFAZQUWnnelD1HkY7FUMwrNS/TN/DZ2RNrsXSobDxx/lIxrEDSTjdotjFXWJjoTzE2iFVs10w9AgPb6PjLqpNB1ob4Fo769DiXMqjNSbxHDxy2WmoTsMTKqXJ9bx0TwdrYpUlDxkVUktofJasu2e4sfrfkWxI2Jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.73) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from DU2PR04CA0351.eurprd04.prod.outlook.com (2603:10a6:10:2b4::10) by DB7PR10MB1913.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:5:11::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.22; Mon, 9 May 2022 10:16:33 +0000 Received: from DB5EUR01FT061.eop-EUR01.prod.protection.outlook.com (2603:10a6:10:2b4:cafe::30) by DU2PR04CA0351.outlook.office365.com (2603:10a6:10:2b4::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23 via Frontend Transport; Mon, 9 May 2022 10:16:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.73) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.73 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.73; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.73) by DB5EUR01FT061.mail.protection.outlook.com (10.152.5.197) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:33 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SNA.ad011.siemens.net (194.138.21.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 3/7] bitbake-diffsigs: break on first dependent task difference Date: Mon, 9 May 2022 12:16:00 +0200 Message-ID: <20220509101604.3249558-4-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6cf646a7-7720-4ea7-d9c7-08da31a4fdf1 X-MS-TrafficTypeDiagnostic: DB7PR10MB1913:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DE6b9FzhDitQG7ESAVaJFf0WnTUy5anqjYnxK9rsbIj1JGDtUzHJ7ksz7gEbLkK+iTnr7/7gui9la+VOnyW1j/3L58mtBYDENCiMpD4wwuIiDmf0BarJsRPX4FQ1qRs2yCilXYy1m8aIArekzAdtqW70CXzvbQZnHQKxEkZTjKwpOoo9D+sIyfUJn28IbTSG9sDecEMeBfszu7o1DT2gRx1tgtYS3q8e5pmZd2B4E4KaKu1/BA3Yd9wxDTKUGT9pACSy4Wao1DcW2MkkJLS/Jwnb5HKJiiXYWWow4jlreDQ0TB09VpGMa7gveN+9oDJeDIZeUIa4ysMXhQGYZnJ+p7MHWLO4Nx89t4V4BuBMwWwVrl2kmlCxr68UQvdSYuCWbU+/wg47KJ5N3C1gycaDvhMSeXONZQbqyuIlRaCEulWZD6c6Hv3LuMJbmiRSfMYetkpEjD50sX4IGLLf5cizs31Q7SYPBEpq7lQA33pCG7mNqrFsYSP3l8F3Zd9EEy/gC3sW/Aly4m3v1GwSjz+EPMJU9JlZrsQVE3o7+5Atp+Xf0w5fCp2FVFIEshkJH4BTCLm/6sH8gcWv6m08I7jb/u2hktgCWHLsT3jL89OsQpAC6WD7YUms4lQirM3Pi3hisYx2lkMNvUEZnYDxgdseAPWzpqvCb7Z1LkeTU1LQJPuxcQHcmtsZ6n4i9UUptnAUGqHfJQl/RWi/5S1RofZQFotvArOXqNZgfJCgVjqb/2IJ7owcdPZPevF+yrRDq+dRTGsyI9Zy9cAHx5xKq/toULTzsY+/Can1fZ6+4U5aUGU= X-Forefront-Antispam-Report: CIP:194.138.21.73; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(6666004)(2906002)(966005)(86362001)(8936002)(498600001)(5660300002)(44832011)(81166007)(83380400001)(356005)(26005)(82960400001)(956004)(107886003)(2616005)(36860700001)(40460700003)(47076005)(336012)(186003)(1076003)(16526019)(70586007)(70206006)(6916009)(36756003)(8676002)(4326008)(82310400005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:33.0604 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6cf646a7-7720-4ea7-d9c7-08da31a4fdf1 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.73]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: DB5EUR01FT061.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR10MB1913 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FvDu3oeJ; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::603 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX Currently compare_sigfiles() recursively calculates differences on all dependent tasks with changed hashes. This is done in arbitrary order, and only the last of those results is returned while everything else is discarded. This changes that to instead return the first difference and not calculate any more, which significantly speeds up diffs of tasks with many dependencies. backported from upstream: https://git.openembedded.org/bitbake/commit/?id=8ed7722865d2dcfda1697aaf4e12b828934bf527 Signed-off-by: Adriaan Schmidt --- bitbake/lib/bb/siggen.py | 1 + 1 file changed, 1 insertion(+) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 8b23fd04..767aeb0a 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -980,6 +980,7 @@ def compare_sigfiles(a, b, recursecb=None, color=False, collapsed=False): # If a dependent hash changed, might as well print the line above and then defer to the changes in # that hash since in all likelyhood, they're the same changes this task also saw. output = [output[-1]] + recout + break a_taint = a_data.get('taint', None) b_taint = b_data.get('taint', None) From patchwork Mon May 9 02:16:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1775 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:41 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f61.google.com (mail-wr1-f61.google.com [209.85.221.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGeAt003573 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:40 +0200 Received: by mail-wr1-f61.google.com with SMTP id v16-20020adfd190000000b0020c8fb5106dsf5085585wrc.19 for ; Mon, 09 May 2022 03:16:40 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091395; cv=pass; d=google.com; s=arc-20160816; b=xIY8KnB29DkRDyW7Hc+lRPtX6F/yT7G6xntpq4hB7di2+jhm9DFJGp+4E6jBxkOVWt agx/so657N0dsVF4XWHuScRr/y7ETuSyV7Ie4QBTH9//e/eCa6+q3du1VqDKe8R0FtJL JIWJLe9+EMFWMLQokAaMj6wU9SHKF8IoFz+05rwooTec6q1grSeBqdSkj0Z/xJNw9vvb OG/hr2MW49E/wispBxhOF/UV+FSF7ATN0jIMIUJzj3c7c+sR8bVmeFRv3JRxAvLyuDaf +C1YfoIbxIAO/AoWVTP3lUuMAnGQf45b6IhAVX9xXbZmcTY+fce2s70d+QD3fb00j8bU LSaw== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=4dmgM89FS2Ck2Ysq5wJVOQg8mU4MHonJXHnSSm3W4g0=; b=qm6fQhpJjwUf/CX31epfP/OMvKl6mwzaK05VAis4QQ8IOT+UfeIV55i6qLAh47pLKQ aZE19YYn7mhVYPeWcuZxziKe1qNvXC5x6wYAcOPSS/0SVc1/HAfVKUAE6MW9XErujnfb WnPXhNars7WlI3BhKeFof1H+o3I/UcX05rb/9rLMEwHgHi306yEWTo8F80j+PRhmq2/0 QcQr/QCaJUkUI/3VjxY8Kc258i8eHZoFhzDm9PTsoLTTQw7UgQc+Q65IIbZmyTYcX8e3 AdB7e5vOkU2vyxVSPgYAzk4GNNWssg1aGO52VBh8Y/mHd0+esb55RrvL6IG8Z7b5Ct1d mUWQ== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=D9KKrKRR; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe1a::62e as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=4dmgM89FS2Ck2Ysq5wJVOQg8mU4MHonJXHnSSm3W4g0=; b=HGoXLll2GwHTRPqGkTPMddFaR0s/OH+JtQzpfjnUvtrEuvcEMPK55jWacs9NVyFe74 REHO7eqhfXS6873/2N96JwjP+N0N+q2ed03EVvsH29+Vw4ZnBb932ycVLtX4KsFMF7zu x50lcUk4NeekSq7AVS3BzkAU7sAL+bqbnz8RhXecEUAz9o6eBj2h+K6cKHB1BG50SPRH SBsGwYxPRMIX3Rt/jiZaKtFmitcWSo/73jlDStJnl+XuhLewt/JLeljsgHE+bUSQuIZV 4ILb0M/xZR5AMgY/arPmOw1xTql/oIKVtJaYJyhafq/Yk2LD9YkiVHISR7AkaSBmpWqM dBqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=4dmgM89FS2Ck2Ysq5wJVOQg8mU4MHonJXHnSSm3W4g0=; b=HCh60pbLf6ZICUG+FL1oT7utqckaiZPGNLnKRkKu4cgn6QwgTl4vFoHHzVxLVO3Tf3 d3Ahve+OVvrfyMXL4dQOuMFnWz/9JfCwRaxOoMDqkfWlAtWtIS4leMnuiB71Ri7pHbKp adAq6ARw63FX0sVU70XLV0G5uLrBDJYll9E3XG54nVksGFcnQmZltKi9I8SeScycs7Qo d64ugHv6cnCi0bagGMZowShsykx+typ+2Ob7uOsFaTDXCaDQp/lbkkV3OSL04S6NQejt /dvVI2HsQqFiqFDsOPJ/oLHuxHcOBilVcIKwc0NZT5/QGzz5bx8tGnfIZCOFGXszHlkw csAg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM530Q8QD+1oISGaEYvqEYQNtXwIsCvC55dIBUYcXzIyOD5W7JOlzj lC5s9LxUNXDuuQfJKfnWb2Y= X-Google-Smtp-Source: ABdhPJzdfNU+r+cSBdmnhwqmMFrSlucmY8mJCZVnRoHEQfuSJPA1YkAjc+rEOwndyLMXegCn5cgYPg== X-Received: by 2002:a05:600c:3b98:b0:394:6a82:8d59 with SMTP id n24-20020a05600c3b9800b003946a828d59mr19967657wms.126.1652091395452; Mon, 09 May 2022 03:16:35 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:500d:b0:394:77fc:d6b8 with SMTP id n13-20020a05600c500d00b0039477fcd6b8ls3665652wmr.1.canary-gmail; Mon, 09 May 2022 03:16:34 -0700 (PDT) X-Received: by 2002:a05:600c:4ecb:b0:394:5be7:791d with SMTP id g11-20020a05600c4ecb00b003945be7791dmr21453216wmq.151.1652091394542; Mon, 09 May 2022 03:16:34 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091394; cv=pass; d=google.com; s=arc-20160816; b=X2DlS9k3Qy3075un6oPgUw0fd2m/gPDvEt41omR+mm2yq95BhOT0ZbGZSEpPUA8hjb ic+9jXmQVdaipaZMbU7f8ditMPN/VaoNXFfoDTe021SZqf+4QJwQVqv0fy+xUXqJ/jR1 QaK4WX6LHw6oKdhG+vDbMiT+gHeX4UNaqnVWt4aI1NEQ/w7543LM6mscN1XI6sKn2Kvd l+zlUJyJgj8RD6VojgmyAtGl/q0yHAA1pG/5X8PZoGx9p6dDZWXQMC19LIw80OCxMeGa IRXw+aUCom4c1LyIIxsb6Zd9TD0V0FugK3XOd0TiiOpAMa448dic0O4FTroasqXGzBbX 4iRA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=edMSbgG37op+oRYu6k3XGgg+sBW5TWKc5O79lNuK8dc=; b=FhMCM+RqTzXZDjJOlVwXTHPvthy2g8yMv1DiDiIVyYxm8xzJ0Lp1X/QsHP3E+n4Sts TBm/El8D11y9SwjAFD52PqT0Dz9jB92uGQ4FrKcDAYdf1c5n3n/opPHUfqMBn2bu+zxU MwgD78a1OFg50H6un8uKsVpP5ii4r+Wsut1DzXy7efCc7og5GA8UsHcNHxHHTaOFUbX1 wkAE4zIkqG80xYnmknDMtF9J7E4iITlUSHkzVoObbkDIc3Zv4N/7/r5WeM4HRip5EQsN kETScbFePmshyXdOzMvOd+V8MEgUlNTwdAuNQJWiC+EygMm1pU9+JAomW23JGaya+SqW +UMQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=D9KKrKRR; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe1a::62e as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR03-DBA-obe.outbound.protection.outlook.com (mail-dbaeur03on2062e.outbound.protection.outlook.com. [2a01:111:f400:fe1a::62e]) by gmr-mx.google.com with ESMTPS id bg21-20020a05600c3c9500b00394803e5756si309193wmb.0.2022.05.09.03.16.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:34 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe1a::62e as permitted sender) client-ip=2a01:111:f400:fe1a::62e; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DpApfoYdkAJ5vjkvTbBEdDckgx85ju7U1iHaY+FL4Mp8jYEmzlE+AREcwjg1ma+CJuof/CH75LaJF1ERY2wFxtQ4pWWVFO9NY/Kip/loxRTVjbPJ6RTdhKUAmzDS1TUCnf8XqJmzMKBsmA3zcQ6Jk/jTolHDwK7MfIAwRjdz3P+1sXFC6L8gwe4phWDMHytEgXf7xkFkvy0KnpcVUPLIeWAAyE2r2efgOXg3m+W8pkyA+mi0VTFCxY/s8PHBhSk2IfLjI6YeAsRJhHwjRrhrhC1pqXB9K3WvcpbrI4FzXWfA60NBl3QAkZosivdrotcF3p3bn6FTzky5nzEK8kQBJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=edMSbgG37op+oRYu6k3XGgg+sBW5TWKc5O79lNuK8dc=; b=RwDHURWsJA15/CbWNX5VOs0kYcmOTWNyDmnpEqQhmvP3pBCTWKB0oiOOKSM3/RocKNqim4cYPGZNTtj/hnD5M5PUKpaKA9llAb5H20X36CFgR23TFC+J2FLSFazh207HhzRE/w8U6g9PHd1PQgr9wZWQgVdCYcXXm3Dgo5LKK1Fx+mU7cfPzFIfq4Iiw8wkIOuiz4hms7znUHsR5v0FACZqPuRhRFwvRoXWjsqeFh/C/cNfI/IzDgP6CSfBnzDOygwN8T4jEtP9ATPaAaY8TrfRpHkm6FzDtTnd0erLupS1MZLqamFiOfUrAjaeO3dWz2M/QQbZCrBcjg01K1FA06g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.73) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from DU2PR04CA0339.eurprd04.prod.outlook.com (2603:10a6:10:2b4::18) by DB8PR10MB3580.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:138::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18; Mon, 9 May 2022 10:16:33 +0000 Received: from DB5EUR01FT061.eop-EUR01.prod.protection.outlook.com (2603:10a6:10:2b4:cafe::6c) by DU2PR04CA0339.outlook.office365.com (2603:10a6:10:2b4::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23 via Frontend Transport; Mon, 9 May 2022 10:16:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.73) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.73 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.73; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.73) by DB5EUR01FT061.mail.protection.outlook.com (10.152.5.197) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:33 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SNA.ad011.siemens.net (194.138.21.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 4/7] dpkg-base: refactor dependencies of apt_* tasks Date: Mon, 9 May 2022 12:16:01 +0200 Message-ID: <20220509101604.3249558-5-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 563d26af-2c62-4e30-5f5d-08da31a4fe39 X-MS-TrafficTypeDiagnostic: DB8PR10MB3580:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: neRdVuhKDEQybUGtxH6oZyJYirGq0Pa/c2bNusUTFEzM3vNUUmaTI1Sas6ZNKFAJvYUlulrDMAysmFxpdLMCBr52rk8kfSaLOWEt7plxcYDCcXZ8r7OFx4PLFk6T0D9PwQkvGiXQYDM0A/yOxq2YW5AnBCwf/O6VSpetmXU9XfDUtAsevaeomqDFqGubBgqhjaK/aUy3OYW7Fp6pzieArTfLogomcGSE7aa/FqiHfjlS0gypvy/+DjX00hZyUShZG5sniPNmu2JinMnlTnZONFU4UGhF2EJSqZ3blzAC85gXUh74ZFnEw3mBWK98ht8h1N20Fm0D4/sNsV/LQZ5tuBE5nY9lvgreIwQ1j2Js/4dpfMd0hRicmlGk8o/E1O5n8xxAG+uuppxoSCnXGbeh1jAGr6l7MNuTzueGopOekcy0XhMmRqSRwR52l8RPcdiHKor88eITLV3CfAgZBAd/zwQx23ic5Xb9cKOrBLz4NSjeFuQIIYYvUhOoKSykaGHHfR1CEDb9tgsUKa4X4QulWvulOAYn/MP15ZLVzE2z1kMw+Z/nTtYGZxFJIIMYm3ad9lD5gunHATnwAXcf+nvXFYZYwi0ErlZeSnl3R+4lgHYFPPJQdRNwF6uigJMH5itgPfO4jCiUQB9BWo12oaOw3hp1qzQwfFx7emvYpFUC8we41u4nKHdEEohekTvvT2qS6zBpwEQ8F9PKDlbZcdNMmg== X-Forefront-Antispam-Report: CIP:194.138.21.73; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(82960400001)(82310400005)(6666004)(81166007)(36756003)(356005)(336012)(508600001)(47076005)(40460700003)(36860700001)(2906002)(5660300002)(16526019)(6916009)(70586007)(70206006)(26005)(8936002)(44832011)(1076003)(4326008)(8676002)(83380400001)(86362001)(316002)(956004)(2616005)(107886003)(186003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:33.5135 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 563d26af-2c62-4e30-5f5d-08da31a4fe39 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.73]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: DB5EUR01FT061.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR10MB3580 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=D9KKrKRR; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe1a::62e as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX Only recipes with `apt://` sources in their SRC_URI run the three tasks `apt_fetch`, `apt_unpack`, and `cleanall_apt`. The current implementation creates the tasks (and dependencies) for all recipes and sets them to `noexec` if they are not needed. It turns out that bitbake doesn't generate sstate signatures for `noexec` tasks, but carries them as dependencies to other tasks, which can break analysis with sstate-diffsigs. Also, I suspect that `noexec` may not have been designed as "optional", to be added in this way (OE never does this). The new implementation only adds the three tasks when they are required. It also: - removes the dependency of `apt_fetch after do_unpack` - makes `install_builddeps` depend explicitly on `${BUILDCHROOT_DEP}`, a dependency which was previously only given via the `apt_fetch` task. Signed-off-by: Adriaan Schmidt --- meta/classes/dpkg-base.bbclass | 14 ++++++-------- meta/classes/dpkg.bbclass | 2 +- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 86933c57..fe6d46dd 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -95,10 +95,9 @@ python() { d.setVar('SRC_URI', ' '.join(new_src_uri)) d.prependVar('SRC_APT', ' '.join(src_apt)) - if d.getVar('SRC_APT').strip() == '': - d.setVarFlag('do_apt_fetch', 'noexec', '1') - d.setVarFlag('do_apt_unpack', 'noexec', '1') - d.setVarFlag('do_cleanall_apt', 'noexec', '1') + if len(d.getVar('SRC_APT').strip()) > 0: + bb.build.addtask('apt_unpack', 'do_patch', '', d) + bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) } do_apt_fetch() { @@ -117,11 +116,11 @@ do_apt_fetch() { dpkg_undo_mounts } -addtask apt_fetch after do_unpack before do_apt_unpack +addtask apt_fetch do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" # Add dependency from the correct buildchroot: host or target -do_apt_fetch[depends] = "${BUILDCHROOT_DEP}" +do_apt_fetch[depends] += "${BUILDCHROOT_DEP}" do_apt_unpack() { rm -rf ${S} @@ -142,9 +141,8 @@ do_apt_unpack() { dpkg_undo_mounts } -addtask apt_unpack after do_apt_fetch before do_patch +addtask apt_unpack after do_apt_fetch -addtask cleanall_apt before do_cleanall do_cleanall_apt[nostamp] = "1" do_cleanall_apt() { for uri in "${SRC_APT}"; do diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 320102ba..af833536 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -26,7 +26,7 @@ do_install_builddeps() { } addtask install_builddeps after do_prepare_build before do_dpkg_build -do_install_builddeps[depends] += "isar-apt:do_cache_config" +do_install_builddeps[depends] += "${BUILDCHROOT_DEP} isar-apt:do_cache_config" # apt and reprepro may not run in parallel, acquire the Isar lock do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" From patchwork Mon May 9 02:16:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1777 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:43 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f57.google.com (mail-wm1-f57.google.com [209.85.128.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGge7003593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:42 +0200 Received: by mail-wm1-f57.google.com with SMTP id c62-20020a1c3541000000b0038ec265155fsf9684177wma.6 for ; Mon, 09 May 2022 03:16:42 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091397; cv=pass; d=google.com; s=arc-20160816; b=TQ9C+4a2s0hU8123BP3JPy1ccekjL1S/D33b5QSuq0ABnhTy5EqAmOTaz7YrVpKh6D R9wnnFRMu13PgFLoyDUNvO2lNbpNMR7rzRFdw+xz5YmOhPPnNW93mhkJfH7js5ugJWIC vpE3ZvyUj7xgmsuLIrq2kgDAEeMEbpM8mxRlMgMWCQq47tg5MCnapyViU9O31PISX581 BfQhcPpqEVqgeN9i5f9Nsc37E6VD43VYEw5Esplf83hnL3mSMj6wdE8eXRRgFRuyRFVG VBC6eIaNa65PZSuQXCCDb9iJfJMIBAZexZ0sm6kWbH4kbaMsgzzSxLBK9eenWIGeFgBj t8xw== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=oGZzf7WeX4y2JDzkBC9FT7LD0lO0lBdgpbZPQz3swNM=; b=zFKFd4bczSXBJtpUqto7NZcU5+Vn+HINvCOG4+2TYynE1HJMw99Ro6eYl8RTfvQQAC WRTqmzfChkWHhxOwWlLelyCU7IBgKQHlI/ECq9rPgRkbMsZZMe6sRtI3lsRutKucXaAe x2l6eRtzD7CccT0sc1a/hjnAPcy8ZvGXviCkcC9MPTIPv/owQdjDoleYMuP2fgHHy0Zs yycW7CZ31cFFqqF/9ib9XfMBgUt9/oJaes8tuW++iiWARx0els1O15g4FXlqXUQPiC4t /4wvpMmRSHthp/aI+2WQTpVHYR0nY5sG0P72xQuTLiHjIZRDchhGXtMGYeCmq8pwac/4 6Bzw== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=jtpoRWYv; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe08::601 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=oGZzf7WeX4y2JDzkBC9FT7LD0lO0lBdgpbZPQz3swNM=; b=Qd9UiWRmN9fWPl2iAbtWLiLi9zKJeuWGpMyT0EoXIkFkgcjVGfVNOyF/DCxNTSOhm/ cyn88/e3mpfC9bKhm77DKUfEhkGqzUAxKSf48Q/xJRPzgkjG4BiuICgzUkdi0wPoJ6tI dTdHfx/gM2sKH3A0QNU4kGqTZ3tGgLnweM1pnVK8PZp4cB+NJ6nZba8o+I4Qu8Ct1Rj+ gHXzbUTvVwtcw1cgxRm307IuYQvy53ITwnamYOaj5BIuNS3Jbdv87fm05ZvEyMn9JHsh 14lg9P070Vx9VGxZr12Ik0sxcyOVAiJScm4it4+fyP+UIAL/2ONlxXq3tu9fGw9DQPlc hhew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=oGZzf7WeX4y2JDzkBC9FT7LD0lO0lBdgpbZPQz3swNM=; b=Bl5b9leERFyMk9hEF5knuJ69FD6TUMifHkjLfuoI2dUG8AivTqqfPdi+dQ2W9N/YAe 8pW637STmLdlMmqoq6mRMK/8zbhMYH0iBPMmyo8EtUHOphxqXpRUlq9GCQxJfDiOXMLg o7xmdxsWq3Mtd6WKug/6onm1lGcvnt1CUtMk91jba6WBXd0ZDJI+A3Vka9yxuvRA3EDh 6/2WwcrRbO/hBHBhMVSGjjUnmHPY3jF05lx6iHV2xOj7EtWr9+FgW4VeWC4va8+N6aWB o5qZjI9Utbh5hqt+HZUPg8D5Ac9UvQJm8lrEw47FwTFBGeAifzNeiDLOGwErgkaBJOOU aR1Q== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM532H8X77C6IWo/xacgsew1Xb6cYk7/FTfw7sJtDjEz4mp1gw6cMU TzPybhhCx0o4yRiFg81EBbY= X-Google-Smtp-Source: ABdhPJy8h5WpTyyTMQvCOLYk8Kqicb/zUVFzmJWcyUhQ/qg/xzMzRS0nr9pqIjIHAULiVyTxsy2+Ug== X-Received: by 2002:a5d:64a6:0:b0:20c:64ef:c9cc with SMTP id m6-20020a5d64a6000000b0020c64efc9ccmr13698060wrp.190.1652091396922; Mon, 09 May 2022 03:16:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3c9c:b0:393:ef44:c49d with SMTP id bg28-20020a05600c3c9c00b00393ef44c49dls3662845wmb.2.canary-gmail; Mon, 09 May 2022 03:16:36 -0700 (PDT) X-Received: by 2002:a05:600c:ad1:b0:394:1585:a164 with SMTP id c17-20020a05600c0ad100b003941585a164mr21874958wmr.101.1652091395947; Mon, 09 May 2022 03:16:35 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091395; cv=pass; d=google.com; s=arc-20160816; b=dW1RoUsJRzl/q2zRHOORxAGYFuu5fsRQWpSyYTWpFQpnMUSxMsr1XC4TjZrC4ei03K UEd8lVfxZ+RAcbvZTn32RWjUmkD8Tny8VWbvRnR5M02O0oAg12JTwNploM4FrZ9pCftT SdUZrZ5MT1Ono28lVQU1PGonYbXROeyS51Y1KSqneddeQyK5P9n5PjQQxE4D/CDgCSW3 uuitTAycGwkWHX4t79z9BnOk0LFBeql2j/NSuCZEulNa5XyJGdaaUu2BIo6GH7bCq/72 jodK15UcQvlVHC27v0bCU30QvXNnFjgSpvD1a8ozY2SyTCErGqc8D4Env+3COlWiYJBc L0bw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=Ui6oeJEk+W/LgmqaQmHzKHZVpIrCJ99LFcojvPmlaf4=; b=uQ7d3uvuecp7S4JRGGLcWgFH9tFFGWSoLzqUe/z4FPZptENmdrstuKhlarVxbUHcH9 Pw+PqKDwSH2cyaLFEXwhX82iXpzJJeTCM+pqeubSCz7+mvmVEjBem5DSJw9T+QDP9nqz AHIu51nZNULN2ne7rQa16T0GOrzrtlRrspJmD7P53qUu3B87nop1EqqnkNS3C5VWtkhH 2pPzG4nzGt/zWMjydD2ilcaW75qIZioNGL0iQJlAsveT1/rkzQnzsdXEx2JT1CwrRu5Z 6GC4h1KqXpvNbKzXaJPzihFyDEVF5ttYoiobfvDyTy+m7smAPUn35Qv3NFDAGU61QZSq X7TA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=jtpoRWYv; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe08::601 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-am5eur03on0601.outbound.protection.outlook.com. [2a01:111:f400:fe08::601]) by gmr-mx.google.com with ESMTPS id f129-20020a1c3887000000b003943f888697si440269wma.0.2022.05.09.03.16.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:35 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe08::601 as permitted sender) client-ip=2a01:111:f400:fe08::601; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UyumMGkC7SegIgv9MVJ7ykCPLXf+NN9DfBcbZi/C3DcW3cMLYPr7QNHV2qzoFz3H0B/nseippeutOE2kSe4cXZfrHbApy28ZGgNfsNapMfgBjtbgNtRUP4ONKEjhl6oVkQGvM0AGH4lBfChjhYhsBIc16DJ/gj4z7FyJ1vX818HPx8uHMy9SM9/3/yFBFe+/bRhPzaou4sUky54FdogUdODvAXi2z3CLJ4Wy8F6ixr0/fhCFr+dy++wsyqRundL49B3klFQ9U4TEiunXPCbTs0vt91S8VUUdAuamesV162QZYOUpiux+DtBZF9m+3JOLrVhB18/TQbt/H0cKmgFrdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ui6oeJEk+W/LgmqaQmHzKHZVpIrCJ99LFcojvPmlaf4=; b=T4YIGC2mtNnwFBVwYNynXOuAPOhe6u2rqnD0UsP1BWSj46xXqVDPa7j/q4PAx9CF3o/KR0iGT2RsZPILyEhcvarKKJ3wh9K0P4zHvcLRHl87LjQyfjQzbQkHcwhAInPZru3BsLEQs6BXb//qKxfrvCFWaU9EcOt7MVYZaLlzZ2rzZNMrjfM5tLEDcuDH1vypOJFkjHt1Mt0gJ7tANnIC02lxOyddflLDrvQJj+2FPvLvEOT0PQvfV9Kxt4M62Rvp3Fslw5VEYsY5K/A3MkrSOAysulABT8pEmOMqdVs5dfmX7vYaGqJzVfrBo61pTATAuUsmjz+cgaBRUGRtUbFQoA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.72) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from AM6P195CA0096.EURP195.PROD.OUTLOOK.COM (2603:10a6:209:86::37) by VI1PR10MB2669.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:e7::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.21; Mon, 9 May 2022 10:16:33 +0000 Received: from VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com (2603:10a6:209:86:cafe::7e) by AM6P195CA0096.outlook.office365.com (2603:10a6:209:86::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18 via Frontend Transport; Mon, 9 May 2022 10:16:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.72) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.72 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.72; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.72) by VE1EUR01FT105.mail.protection.outlook.com (10.152.3.156) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:33 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SMA.ad011.siemens.net (194.138.21.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 5/7] scripts: add isar-sstate Date: Mon, 9 May 2022 12:16:02 +0200 Message-ID: <20220509101604.3249558-6-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 43be577a-cbf4-4a13-b373-08da31a4fe19 X-MS-TrafficTypeDiagnostic: VI1PR10MB2669:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: grdTINr9qPfDmmtmNVGN/aV0AgtaP/df5O4Pxi9IUZRFfro6Pjqk4jRaClBvFd+8YTxT7vN+z5qdrf6PERgxejaZXL7Tl4lt2bqUNo4Sf5DB73xpTiy1d1iytpi0QcbK7jcrVELD8wTYtfgCsqPyzKr+WDfqxmX63esHAzVreMTS8WLNgTBlqxjmS4WkgKATy+0ujvVEudd/CD0AtgyVcFEXcEUkIfaF5SVZtO8xl7UaQPj7aBdafqsE8tb6rbzuthlNaeJZaHdsfluJk/ZCIDWaoQ1ehmFfw0JxSIhYHuG9Doz+kdpdZmHz28XnwROMquasLTlTJkTHkrOclsUALiyzbKieJMO65JAfbMWtuQLiD49sKJeG4dB2MI9CJC1jEwrmPW427W51d7i61f4E1ogzHYMDo6XLrOSMQ4u3B/TcXvWFoHwtwacHAurbj9DP/NAOrOlTsnfyjw1zndmdIi1mBAmmz1/wtUyLMjUu6IWX2XVROn8YLRjGOYsO/A+CsV2tHAA3NihNJwujUT0ZlPWUdVxQ6qMo6QZVMjpNZCY6Z5fxKne8odb6Gaty3w7XlduNOTMbd8EAyxNCszu6JE6u9f2Xl4n/Nc0zb0nzScVVYeyr+2+PzC5RBUVCeNM/Fxc28AECBXC0k9R6CcR9bsYZKKPEnMzLmPDrmvkKMcG0wy3aMnuim1lscp2yVHiPwbSosrHSDzb+SXpN1IAcjQ== X-Forefront-Antispam-Report: CIP:194.138.21.72; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(8676002)(966005)(4326008)(70206006)(40460700003)(5660300002)(956004)(70586007)(508600001)(8936002)(86362001)(82960400001)(44832011)(36860700001)(81166007)(30864003)(83380400001)(316002)(2906002)(356005)(16526019)(36756003)(82310400005)(186003)(47076005)(336012)(6916009)(6666004)(2616005)(107886003)(1076003)(26005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:33.3213 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 43be577a-cbf4-4a13-b373-08da31a4fe19 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.72]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR10MB2669 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=jtpoRWYv; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe08::601 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX This adds a maintenance helper script to work with remote/shared sstate caches. Signed-off-by: Adriaan Schmidt --- scripts/isar-sstate | 794 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 794 insertions(+) create mode 100755 scripts/isar-sstate diff --git a/scripts/isar-sstate b/scripts/isar-sstate new file mode 100755 index 00000000..8b541cf4 --- /dev/null +++ b/scripts/isar-sstate @@ -0,0 +1,794 @@ +#!/usr/bin/env python3 +""" +This software is part of Isar +Copyright (c) Siemens AG, 2022 + +# isar-sstate: Helper for management of shared sstate caches + +Isar uses the sstate cache feature of bitbake to cache the output of certain +build tasks, potentially speeding up builds significantly. This script is +meant to help managing shared sstate caches, speeding up builds using cache +artifacts created elsewhere. There are two main ways of accessing a shared +sstate cache: + - Point `SSTATE_DIR` to a persistent location that is used by multiple + builds. bitbake will read artifacts from there, and also immediately + store generated cache artifacts in this location. This speeds up local + builds, and if `SSTATE_DIR` is located on a shared filesystem, it can + also benefit others. + - Point `SSTATE_DIR` to a local directory (e.g., simply use the default + value `${TOPDIR}/sstate-cache`), and additionally set `SSTATE_MIRRORS` + to a remote sstate cache. bitbake will use artifacts from both locations, + but will write newly created artifacts only to the local folder + `SSTATE_DIR`. To share them, you need to explicitly upload them to + the shared location, which is what isar-sstate is for. + +isar-sstate implements four commands (upload, clean, info, analyze), +and supports three remote backends (filesystem, http/webdav, AWS S3). + +## Commands + +### upload + +The `upload` command pushes the contents of a local sstate cache to the +remote location, uploading all files that don't already exist on the remote. + +### clean + +The `clean` command deletes old artifacts from the remote cache. It takes two +arguments, `--max-age` and `--max-sig-age`, each of which must be a number, +followed by one of `w`, `d`, `h`, `m`, or `s` (for weeks, days, hours, minutes, +seconds, respectively). + +`--max-age` specifies up to which age artifacts should be kept in the cache. +Anything older will be removed. Note that this only applies to the `.tgz` files +containing the actual cached items, not the `.siginfo` files containing the +cache metadata (signatures and hashes). +To permit analysis of caching details using the `analyze` command, the siginfo +files can be kept longer, as indicated by `--max-sig-age`. If not set explicitly, +this defaults to `max_age`, and any explicitly given value can't be smaller +than `max_age`. + +### info + +The `info` command scans the remote cache and displays some basic statistics. +The argument `--verbose` increases the amount of information displayed. + +### analyze + +The `analyze` command iterates over all artifacts in the local sstate cache, +and compares them to the contents of the remote cache. If an item is not +present in the remote cache, the signature of the local item is compared +to all potential matches in the remote cache, identified by matching +architecture, recipe (`PN`), and task. This analysis has the same output +format as `bitbake-diffsigs`. + +## Backends + +### Filesystem backend + +This uses a filesystem location as the remote cache. In case you can access +your remote cache this way, you could also have bitbake write to the cache +directly, by setting `SSTATE_DIR`. However, using `isar-sstate` gives +you a uniform interface, and lets you use the same code/CI scripts across +heterogeneous setups. Also, it gives you the `analyze` command. + +### http backend + +A http server with webdav extension can be used as remote cache. +Apache can easily be configured to function as a remote sstate cache, e.g.: +``` + + Alias /sstate/ /path/to/sstate/location/ + + Dav on + Options Indexes + Require all granted + + +``` +In addition you need to load Apache's dav module: +``` +a2enmod dav +``` + +To use the http backend, you need to install the Python webdavclient library. +On Debian you would: +``` +apt-get install python3-webdavclient +``` + +### S3 backend + +An AWS S3 bucket can be used as remote cache. You need to ensure that AWS +credentials are present (e.g., in your AWS config file or as environment +variables). + +To use the S3 backend you need to install the Python botocore library. +On Debian you would: +``` +apt-get install python3-botocore +``` +""" + +import argparse +from collections import namedtuple +import datetime +import os +import re +import shutil +import sys +from tempfile import NamedTemporaryFile +import time + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), '..', 'bitbake', 'lib')) +from bb.siggen import compare_sigfiles + +# runtime detection of supported targets +webdav_supported = True +try: + import webdav3.client + import webdav3.exceptions +except ModuleNotFoundError: + webdav_supported = False + +s3_supported = True +try: + import botocore.exceptions + import botocore.session +except ModuleNotFoundError: + s3_supported = False + +SstateCacheEntry = namedtuple( + 'SstateCacheEntry', 'hash path arch pn task suffix islink age size'.split()) + +# The filename of sstate items is defined in Isar: +# SSTATE_PKGSPEC = "sstate:${PN}:${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}:" +# "${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:" + +# This regex extracts relevant fields: +SstateRegex = re.compile(r'sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' + r'(?P[^:]*):[^:]*:(?P[0-9a-f]*)_' + r'(?P[^\.]*)\.(?P.*)') + + +class SstateTargetBase(object): + def __init__(self, path, cached=False): + """Constructor + + :param path: URI of the remote (without leading 'protocol://') + """ + self.use_cache = False + if cached: + self.enable_cache() + + def __del__(self): + if self.use_cache: + self.cleanup_cache() + + def __repr__(self): + """Format remote for printing + + :returns: URI string, including 'protocol://' + """ + pass + + def exists(self, path=''): + """Check if a remote path exists + + :param path: path (file or directory) to check + :returns: True if path exists, False otherwise + """ + pass + + def create(self): + """Try to create the remote + + :returns: True if remote could be created, False otherwise + """ + pass + + def mkdir(self, path): + """Create a directory on the remote + + :param path: path to create + :returns: True on success, False on failure + """ + pass + + def upload(self, path, filename): + """Uploads a local file to the remote + + :param path: remote path to upload to + :param filename: local file to upload + """ + pass + + def delete(self, path): + """Delete remote file and remove potential empty directories + + :param path: remote file to delete + """ + pass + + def list_all(self): + """List all sstate files in the remote + + :returns: list of SstateCacheEntry objects + """ + pass + + def download(self, path): + """Prepare to temporarily access a remote file for reading + + This is meant to provide access to siginfo files during analysis. Files + must not be modified, and should be released using release() once they + are no longer used. + + :param path: remote path + :returns: local path to file + """ + pass + + def release(self, download_path): + """Release a temporary file + + :param download_path: local file + """ + pass + + def enable_cache(self): + """Enable caching of downloads + + This is a separate function, so you can decide after creation + if you want to enable caching. + """ + self.use_cache = True + self.cache = {} + self.real_download = self.download + self.real_release = self.release + self.download = self.download_cached + self.release = self.release_cached + + def download_cached(self, path): + """Download using cache + + This function replaces download() when using the cache. + DO NOT OVERRIDE. + """ + if path in self.cache: + return self.cache[path] + data = self.real_download(path) + self.cache[path] = data + return data + + def release_cached(self, download_path): + """Release when using cache + + This function replaces release() when using the cache. + DO NOT OVERRIDE. + """ + pass + + def cleanup_cache(self): + """Clean up all cached downloads. + + Called by destructor. + """ + for k, v in list(self.cache.items()): + self.real_release(v) + del(self.cache[k]) + + +class SstateFileTarget(SstateTargetBase): + def __init__(self, path, **kwargs): + super().__init__(path, **kwargs) + if path.startswith('file://'): + path = path[len('file://'):] + self.path = path + self.basepath = os.path.abspath(path) + + def __repr__(self): + return f"file://{self.path}" + + def exists(self, path=''): + return os.path.exists(os.path.join(self.basepath, path)) + + def create(self): + return self.mkdir('') + + def mkdir(self, path): + try: + os.makedirs(os.path.join(self.basepath, path), exist_ok=True) + except OSError: + return False + return True + + def upload(self, path, filename): + shutil.copy(filename, os.path.join(self.basepath, path)) + + def delete(self, path): + try: + os.remove(os.path.join(self.basepath, path)) + except FileNotFoundError: + pass + dirs = path.split('/')[:-1] + for d in [dirs[:i] for i in range(len(dirs), 0, -1)]: + try: + os.rmdir(os.path.join(self.basepath, '/'.join(d))) + except FileNotFoundError: + pass + except OSError: # directory is not empty + break + + def list_all(self): + all_files = [] + now = time.time() + for subdir, dirs, files in os.walk(self.basepath): + reldir = subdir[(len(self.basepath)+1):] + for f in files: + m = SstateRegex.match(f) + if m is not None: + islink = os.path.islink(os.path.join(subdir, f)) + age = int(now - os.path.getmtime(os.path.join(subdir, f))) + all_files.append(SstateCacheEntry( + path=os.path.join(reldir, f), + size=os.path.getsize(os.path.join(subdir, f)), + islink=islink, + age=age, + **(m.groupdict()))) + return all_files + + def download(self, path): + # we don't actually download, but instead just pass the local path + if not self.exists(path): + return None + return os.path.join(self.basepath, path) + + def release(self, download_path): + # as we didn't download, there is nothing to clean up + pass + + +class SstateDavTarget(SstateTargetBase): + def __init__(self, url, **kwargs): + if not webdav_supported: + print("ERROR: No webdav support. Please install the webdav3 Python module.") + print("INFO: on Debian: 'apt-get install python3-webdavclient'") + sys.exit(1) + super().__init__(url, **kwargs) + m = re.match('^([^:]+://[^/]+)/(.*)', url) + if not m: + print(f"Cannot parse target path: {url}") + sys.exit(1) + self.host = m.group(1) + self.basepath = m.group(2) + if not self.basepath.endswith('/'): + self.basepath += '/' + self.dav = webdav3.client.Client({'webdav_hostname': self.host}) + self.tmpfiles = [] + + def __repr__(self): + return f"{self.host}/{self.basepath}" + + def exists(self, path=''): + return self.dav.check(self.basepath + path) + + def create(self): + return self.mkdir('') + + def mkdir(self, path): + dirs = (self.basepath + path).split('/') + + for i in range(len(dirs)): + d = '/'.join(dirs[:(i+1)]) + '/' + if not self.dav.check(d): + if not self.dav.mkdir(d): + return False + return True + + def upload(self, path, filename): + return self.dav.upload_sync(remote_path=self.basepath + path, local_path=filename) + + def delete(self, path): + self.dav.clean(self.basepath + path) + dirs = path.split('/')[1:-1] + for d in [dirs[:i] for i in range(len(dirs), 0, -1)]: + items = self.dav.list(self.basepath + '/'.join(d), get_info=True) + if len(items) > 0: + # collection is not empty + break + self.dav.clean(self.basepath + '/'.join(d)) + + def list_all(self): + now = time.time() + + def recurse_dir(path): + files = [] + for item in self.dav.list(path, get_info=True): + if item['isdir'] and not item['path'] == path: + files.extend(recurse_dir(item['path'])) + elif not item['isdir']: + m = SstateRegex.match(item['path'][len(path):]) + if m is not None: + modified = time.mktime( + datetime.datetime.strptime( + item['created'], + '%Y-%m-%dT%H:%M:%SZ').timetuple()) + age = int(now - modified) + files.append(SstateCacheEntry( + path=item['path'][len(self.basepath):], + size=int(item['size']), + islink=False, + age=age, + **(m.groupdict()))) + return files + return recurse_dir(self.basepath) + + def download(self, path): + # download to a temporary file + tmp = NamedTemporaryFile(prefix='isar-sstate-', delete=False) + tmp.close() + try: + self.dav.download_sync(remote_path=self.basepath + path, local_path=tmp.name) + except webdav3.exceptions.RemoteResourceNotFound: + return None + self.tmpfiles.append(tmp.name) + return tmp.name + + def release(self, download_path): + # remove the temporary download + if download_path is not None and download_path in self.tmpfiles: + os.remove(download_path) + self.tmpfiles = [f for f in self.tmpfiles if not f == download_path] + + +class SstateS3Target(SstateTargetBase): + def __init__(self, path, **kwargs): + if not s3_supported: + print("ERROR: No S3 support. Please install the botocore Python module.") + print("INFO: on Debian: 'apt-get install python3-botocore'") + sys.exit(1) + super().__init__(path, **kwargs) + session = botocore.session.get_session() + self.s3 = session.create_client('s3') + if path.startswith('s3://'): + path = path[len('s3://'):] + m = re.match('^([^/]+)(?:/(.+)?)?$', path) + self.bucket = m.group(1) + if m.group(2): + self.basepath = m.group(2) + if not self.basepath.endswith('/'): + self.basepath += '/' + else: + self.basepath = '' + self.tmpfiles = [] + + def __repr__(self): + return f"s3://{self.bucket}/{self.basepath}" + + def exists(self, path=''): + if path == '': + # check if the bucket exists + try: + self.s3.head_bucket(Bucket=self.bucket) + except botocore.exceptions.ClientError as e: + print(e) + print(e.response['Error']['Message']) + return False + return True + try: + self.s3.head_object(Bucket=self.bucket, Key=self.basepath + path) + except botocore.exceptions.ClientError as e: + if e.response['ResponseMetadata']['HTTPStatusCode'] != 404: + print(e) + print(e.response['Error']['Message']) + return False + return True + + def create(self): + return self.exists() + + def mkdir(self, path): + # in S3, folders are implicit and don't need to be created + return True + + def upload(self, path, filename): + try: + self.s3.put_object(Body=open(filename, 'rb'), Bucket=self.bucket, Key=self.basepath + path) + except botocore.exceptions.ClientError as e: + print(e) + print(e.response['Error']['Message']) + + def delete(self, path): + try: + self.s3.delete_object(Bucket=self.bucket, Key=self.basepath + path) + except botocore.exceptions.ClientError as e: + print(e) + print(e.response['Error']['Message']) + + def list_all(self): + now = time.time() + + def recurse_dir(path): + files = [] + try: + result = self.s3.list_objects(Bucket=self.bucket, Prefix=path, Delimiter='/') + except botocore.exceptions.ClientError as e: + print(e) + print(e.response['Error']['Message']) + return [] + for f in result.get('Contents', []): + m = SstateRegex.match(f['Key'][len(path):]) + if m is not None: + modified = time.mktime(f['LastModified'].timetuple()) + age = int(now - modified) + files.append(SstateCacheEntry( + path=f['Key'][len(self.basepath):], + size=f['Size'], + islink=False, + age=age, + **(m.groupdict()))) + for p in result.get('CommonPrefixes', []): + files.extend(recurse_dir(p['Prefix'])) + return files + return recurse_dir(self.basepath) + + def download(self, path): + # download to a temporary file + tmp = NamedTemporaryFile(prefix='isar-sstate-', delete=False) + try: + result = self.s3.get_object(Bucket=self.bucket, Key=self.basepath + path) + except botocore.exceptions.ClientError: + return None + tmp.write(result['Body'].read()) + tmp.close() + self.tmpfiles.append(tmp.name) + return tmp.name + + def release(self, download_path): + # remove the temporary download + if download_path is not None and download_path in self.tmpfiles: + os.remove(download_path) + self.tmpfiles = [f for f in self.tmpfiles if not f == download_path] + + +def arguments(): + parser = argparse.ArgumentParser() + parser.add_argument( + 'command', type=str, metavar='command', + choices='info upload clean analyze'.split(), + help="command to execute (info, upload, clean, analyze)") + parser.add_argument( + 'source', type=str, nargs='?', + help="local sstate dir (for uploads or analysis)") + parser.add_argument( + 'target', type=str, + help="remote sstate location (a file://, http://, or s3:// URI)") + parser.add_argument( + '-v', '--verbose', default=False, action='store_true') + parser.add_argument( + '--max-age', type=str, default='1d', + help="clean: remove tgz files older than MAX_AGE (a number followed by w|d|h|m|s)") + parser.add_argument( + '--max-sig-age', type=str, default=None, + help="clean: remove siginfo files older than MAX_SIG_AGE (defaults to MAX_AGE)") + + args = parser.parse_args() + if args.command in 'upload analyze'.split() and args.source is None: + print(f"ERROR: '{args.command}' needs a source and target") + sys.exit(1) + elif args.command in 'info clean'.split() and args.source is not None: + print(f"ERROR: '{args.command}' must not have a source (only a target)") + sys.exit(1) + return args + + +def sstate_upload(source, target, verbose, **kwargs): + if not os.path.isdir(source): + print(f"WARNING: source {source} does not exist. Not uploading.") + return 0 + + if not target.exists() and not target.create(): + print(f"ERROR: target {target} does not exist and could not be created.") + return -1 + + print(f"INFO: uploading {source} to {target}") + os.chdir(source) + upload, exists = [], [] + for subdir, dirs, files in os.walk('.'): + target_dirs = subdir.split('/')[1:] + for f in files: + file_path = (('/'.join(target_dirs) + '/') if len(target_dirs) > 0 else '') + f + if target.exists(file_path): + if verbose: + print(f"[EXISTS] {file_path}") + exists.append(file_path) + else: + upload.append((file_path, target_dirs)) + upload_gb = (sum([os.path.getsize(f[0]) for f in upload]) / 1024.0 / 1024.0 / 1024.0) + print(f"INFO: uploading {len(upload)} files ({upload_gb:.02f} GB)") + print(f"INFO: {len(exists)} files already present on target") + for file_path, target_dirs in upload: + if verbose: + print(f"[UPLOAD] {file_path}") + target.mkdir('/'.join(target_dirs)) + target.upload(file_path, file_path) + return 0 + + +def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): + def convert_to_seconds(x): + seconds_per_unit = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400, 'w': 604800} + m = re.match(r'^(\d+)(w|d|h|m|s)?', x) + if m is None: + print(f"ERROR: cannot parse MAX_AGE '{max_age}', needs to be a number followed by w|d|h|m|s") + sys.exit(-1) + unit = m.group(2) + if unit is None: + print("WARNING: MAX_AGE without unit, assuming 'days'") + unit = 'd' + return int(m.group(1)) * seconds_per_unit[unit] + + max_age_seconds = convert_to_seconds(max_age) + if max_sig_age is None: + max_sig_age = max_age + max_sig_age_seconds = max(max_age_seconds, convert_to_seconds(max_sig_age)) + + if not target.exists(): + print(f"INFO: cannot access target {target}. Nothing to clean.") + return 0 + + print(f"INFO: scanning {target}") + all_files = target.list_all() + links = [f for f in all_files if f.islink] + if links: + print(f"NOTE: we have links: {links}") + tgz_files = [f for f in all_files if f.suffix == 'tgz'] + siginfo_files = [f for f in all_files if f.suffix == 'tgz.siginfo'] + del_tgz_files = [f for f in tgz_files if f.age >= max_age_seconds] + del_tgz_hashes = [f.hash for f in del_tgz_files] + del_siginfo_files = [f for f in siginfo_files if + f.age >= max_sig_age_seconds or f.hash in del_tgz_hashes] + print(f"INFO: found {len(tgz_files)} tgz files, {len(del_tgz_files)} of which are older than {max_age}") + print(f"INFO: found {len(siginfo_files)} siginfo files, {len(del_siginfo_files)} of which " + f"correspond to old tgz files or are older than {max_sig_age}") + + for f in del_tgz_files + del_siginfo_files: + if verbose: + print(f"[DELETE] {f.path}") + target.delete(f.path) + freed_gb = sum([x.size for x in del_tgz_files + del_siginfo_files]) / 1024.0 / 1024.0 / 1024.0 + print(f"INFO: freed {freed_gb:.02f} GB") + return 0 + + +def sstate_info(target, verbose, **kwargs): + if not target.exists(): + print(f"INFO: cannot access target {target}. No info to show.") + return 0 + + print(f"INFO: scanning {target}") + all_files = target.list_all() + size_gb = sum([x.size for x in all_files]) / 1024.0 / 1024.0 / 1024.0 + print(f"INFO: found {len(all_files)} files ({size_gb:0.2f} GB)") + + if not verbose: + return 0 + + archs = list(set([f.arch for f in all_files])) + print(f"INFO: found the following archs: {archs}") + + key_task = {'deb': 'dpkg_build', + 'rootfs': 'rootfs_install', + 'bootstrap': 'bootstrap'} + recipes = {k: [] for k in key_task.keys()} + others = [] + for pn in set([f.pn for f in all_files]): + tasks = set([f.task for f in all_files if f.pn == pn]) + ks = [k for k, v in key_task.items() if v in tasks] + if len(ks) == 1: + recipes[ks[0]].append(pn) + elif len(ks) == 0: + others.append(pn) + else: + print(f"WARNING: {pn} could be any of {ks}") + for k, entries in recipes.items(): + print(f"Cache hits for {k}:") + for pn in entries: + hits = [f for f in all_files if f.pn == pn and f.task == key_task[k] and f.suffix == 'tgz'] + print(f" - {pn}: {len(hits)} hits") + print("Other cache hits:") + for pn in others: + print(f" - {pn}") + return 0 + + +def sstate_analyze(source, target, **kwargs): + if not os.path.isdir(source): + print(f"ERROR: source {source} does not exist. Nothing to analyze.") + return -1 + if not target.exists(): + print(f"ERROR: target {target} does not exist. Nothing to analyze.") + return -1 + + source = SstateFileTarget(source) + target.enable_cache() + local_sigs = {s.hash: s for s in source.list_all() if s.suffix.endswith('.siginfo')} + remote_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + + key_tasks = 'dpkg_build rootfs_install bootstrap'.split() + + check = [k for k, v in local_sigs.items() if v.task in key_tasks] + for local_hash in check: + s = local_sigs[local_hash] + print(f"\033[1;33m==== checking local item {s.arch}:{s.pn}:{s.task} ({s.hash[:8]}) ====\033[0m") + if local_hash in remote_sigs: + print(" -> found hit in remote cache") + continue + remote_matches = [k for k, v in remote_sigs.items() if s.arch == v.arch and s.pn == v.pn and s.task == v.task] + if len(remote_matches) == 0: + print(" -> found no hit, and no potential remote matches") + else: + print(f" -> found no hit, but {len(remote_matches)} potential remote matches") + for r in remote_matches: + t = remote_sigs[r] + print(f"\033[0;33m**** comparing to {r[:8]} ****\033[0m") + + def recursecb(key, remote_hash, local_hash): + recout = [] + if remote_hash in remote_sigs.keys(): + remote_file = target.download(remote_sigs[remote_hash].path) + elif remote_hash in local_sigs.keys(): + recout.append(f"found remote hash in local signatures ({key})!?! (please implement that case!)") + return recout + else: + recout.append(f"could not find remote signature {remote_hash[:8]} for job {key}") + return recout + if local_hash in local_sigs.keys(): + local_file = source.download(local_sigs[local_hash].path) + elif local_hash in remote_sigs.keys(): + local_file = target.download(remote_sigs[local_hash].path) + else: + recout.append(f"could not find local signature {local_hash[:8]} for job {key}") + return recout + if local_file is None or remote_file is None: + out = "Aborting analysis because siginfo files disappered unexpectedly" + else: + out = compare_sigfiles(remote_file, local_file, recursecb, color=True) + if local_hash in local_sigs.keys(): + source.release(local_file) + else: + target.release(local_file) + target.release(remote_file) + for change in out: + recout.extend([' ' + line for line in change.splitlines()]) + return recout + + local_file = source.download(s.path) + remote_file = target.download(t.path) + out = compare_sigfiles(remote_file, local_file, recursecb, color=True) + source.release(local_file) + target.release(remote_file) + # shorten hashes from 64 to 8 characters for better readability + out = [re.sub(r'([0-9a-f]{8})[0-9a-f]{56}', r'\1', line) for line in out] + print('\n'.join(out)) + + +def main(): + args = arguments() + + if args.target.startswith('http://'): + target = SstateDavTarget(args.target) + elif args.target.startswith('s3://'): + target = SstateS3Target(args.target) + elif args.target.startswith('file://'): + target = SstateFileTarget(args.target) + else: # no protocol given, assume file:// + target = SstateFileTarget(args.target) + + args.target = target + return globals()[f'sstate_{args.command}'](**vars(args)) + + +if __name__ == '__main__': + sys.exit(main()) From patchwork Mon May 9 02:16:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1779 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:43 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f59.google.com (mail-lf1-f59.google.com [209.85.167.59]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGgaD003595 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:42 +0200 Received: by mail-lf1-f59.google.com with SMTP id bp17-20020a056512159100b00472631eb445sf5621176lfb.13 for ; Mon, 09 May 2022 03:16:42 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091397; cv=pass; d=google.com; s=arc-20160816; b=rCb7i4w77OMGZaKU8YNJT5YPSSHUz65dnYzn6FfKiDwZcfp1dJW/UNl3bgQdmg2LsX WO/a5zxcwL1ivjcSPCJ8rtdQMPsvj0gzZWXDHHInwI8HcTExaX1NGETPq6Z26C1VxCxb BR5G27yA/4qLs2V71+pYFFeBEAgLNfekXZodPYjp5hLqP7cTA6eE8lQUukFLZMCsgOBA FV7B8lpFMMJOHlYj/7C+l2rZLZs4D7hepNmjz9jV6UvMVKCqnH5wJJwPIGx1KbFinTw6 3g7RMGGj/2svavPKU407eWXv5TaR+PkC8TQkt8+EvEIysGavAg6DNkJhTQXa9sZBNh9C QeHw== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=sV8H9nElDQxF43eZygNWSpR9LJtBaJjO6p8K687TwBU=; b=RRMWRsE0MlyhOmNHmGVXvQBlcfo1fOdmgYR/+4o800/mwmbIqQhGBvkbZGfurHUQ+K aICNfiL5DNeVC8g3j+6LTZFZ8Q8jPmxwadhpuG+79cPrNdU71dN9zR4/5dXwwSubKiCs AeTf+CL28JiUkSHjlifNofIPbHmDuyMFZHoglVumVwR6PXh6vvrsHGYBTkP2x16y5XbJ cB36SKxslXF1MDxvKvGA1aMdQF20OpHeddJT5mZPjL9Rx4/0d3zRnWsZZAwof17edSHT +YqVAKfWsdITyaY9giroht4F70NxgTgK9MOLnEAeRG5rYkjPOjt3q0GC3PmuExrNRFNq uKjA== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=w5YSqd7h; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::609 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=sV8H9nElDQxF43eZygNWSpR9LJtBaJjO6p8K687TwBU=; b=ln1flFr1HyjkGDv79Xsp7GGlK0DZYuxYUV6VAUN6qPwQDh3cT0d4kJn/zzoUTofliW wzdpb8N0hoJ85LArNlP3ya+DIcraUz1pOmtIWtHOKdSdNq5J6kOIevpsIKKk56wUMW5m 2c7L1BiC2/I7wFwDEIPUO1eZGdDLoaMvp6ckc750aDHhxZFpSdYK9agwqbrib78XXXPH Q13xRASn3cwpbKNlCeggQWdlMDT4XUAQScF5D5O6rdbCI+CUzNt/2XcJLCyEmZ8FP+hG DtHSNcv+Vek9w3ttGI7HjX5tt7XU9p3jyewKrh0iYU+jUTas4+5tgM1r2ZpfOSNBqRTF 4cnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=sV8H9nElDQxF43eZygNWSpR9LJtBaJjO6p8K687TwBU=; b=EhYLdn6EUPTqSTi96NYVK41wEV5zMWoFWA8YVm7oPKiSGAFbD6qltEJStNgob6e0I4 9ov/XbDehaTtckyWXFHUzyNL8Xu81/qvfADfy4BNebTn6ByJKnGJjbpX5zZzot42VmDp j1FUQ/yOhrw9SjwutrQb79z1uJWw7SVBTDqxj/5rF/o5IHxGqf52nyQnx/tKHLY4ACLD xq3D0IOj7cJTQSEQU23E/xlNE53eKxNz9IJr8azPhKcrqx+KZr50YxNP3jIOC5KAEpFQ tMwHskqMo3aqfU3gBKykX09HwTE+2gVxYuVqYFxUetlx9wNmskEkgYrBX1mimGl7+Dy2 IZPQ== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM530YYUpkcqrdC9mCGj+3Vxfr72f4ENNDQNXDJ5EGpvw73f5oiewC T9w6QyBuHC9h9KLob8d7heE= X-Google-Smtp-Source: ABdhPJwNuUiYyVaNDjvP+5Xb3jSEtPGS+2hI/7ZcOJuFcSJ4dvT6v1PRKfBWWUlI3vzgnIhgEyReNg== X-Received: by 2002:a2e:8404:0:b0:250:cde7:e9e3 with SMTP id z4-20020a2e8404000000b00250cde7e9e3mr1556984ljg.289.1652091397058; Mon, 09 May 2022 03:16:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:b1c:b0:472:38f0:bc75 with SMTP id w28-20020a0565120b1c00b0047238f0bc75ls1798457lfu.0.gmail; Mon, 09 May 2022 03:16:36 -0700 (PDT) X-Received: by 2002:ac2:596f:0:b0:472:5c36:17f5 with SMTP id h15-20020ac2596f000000b004725c3617f5mr11862035lfp.142.1652091396070; Mon, 09 May 2022 03:16:36 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091396; cv=pass; d=google.com; s=arc-20160816; b=K+BqlmkFBcRk6cLoniq6f30JlVP2HEKmKnU9/D8urnrw5AnVwbx3FTS2U9huJytcEe monq/ssih7oMqAiqzfEZi8S6OJICDmd+bMfG6/KCxa6sEOhxfZt4yyqfOrNpCuDJDgLs m92w2svywVFyYi8OvNxBkmpED5lMFuA5g5NAX3DPp5KFPbEkO7IETgW6VqVcQBYK8cnh PqvvxIC80AISQiP+Bu6aFeXlROHszYzSEy+yvuDVgu8cljUmZxS4/zvtM/+bKJ1MTCBw xmSNwV9Gr+dbO5N5maHALj82vJi+UKRpC27Z1oijBBV2JVUflAr55RyX8XqtysZ5roQ+ YGRA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=xH794yLJ3URSQMMdAIjsC7aYSKXBnf4W12jdkCLlSXE=; b=tLZRvb+S8R7fgL+w9SDwqGsjK06jVf0oOIFrQISdBW0kGxB53ti9H0Xq/VZtjD/hWU +1JxYr8e8mhnE3tt2k47+/Zbr+6F6nAeeQmMRa0uvepTjyqK9SY0nIhoddPT6pC4H/+X RVaAPWvhb/2b1JBkbJ1j4CbBBfYsaqVQjl4zMIhiJcrgiyVCys5IMALhk1IyOAOX6xR8 WYYDO3jAMxhvbF1vr3VkHtz03jGTXBZdp7FkmWOSu9nTxlYLC/RRRNvV1b6DnBM7o+TO /dkY9UZ1eFhWZTy8RbmrGlPAPKIr/z6XlDAHvJeJ8vCVo82Ix538E9ROA+FRs1F58GTd 40GQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=w5YSqd7h; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::609 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on0609.outbound.protection.outlook.com. [2a01:111:f400:fe0c::609]) by gmr-mx.google.com with ESMTPS id e18-20020a2e8ed2000000b0024f0dcb32f8si465112ljl.5.2022.05.09.03.16.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:35 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::609 as permitted sender) client-ip=2a01:111:f400:fe0c::609; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Egj455azM1dBT8dsKtb0W0cs0exJEqOjjLhSr/y2DjL/HC8kL4v3r3xTA/Ja2h3eu8k5bGkzsmReIF+Lvu1BQ0oVTYMjenN2iCcHP6KIIcDF+OnhpUhv0a4isx94TEDfppIAvOmXRvlvfutOj3bgsiRvnFxBNhiXt3Y+/lVhE4rcn+TpJy+kCowPLzqTnPrBvyfJHcoNTw7rzN/Xra5z9/f0p67fPKjOwRS9nkyRuRUVqzCXLDgwlzHKbsaoCsVy4Gx7CU7cFhG53dQflAjult/4y+HMU5iy+xk/7XHeI64RXaa9BWbTc9rpLcqw4+QRPSt0yIFVj3DDEAE8Iz626w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xH794yLJ3URSQMMdAIjsC7aYSKXBnf4W12jdkCLlSXE=; b=QzjlPl2Ajg4Ll52Niseixr6BAvELdxGlLAOCMhZXSyMDgNTGmgBJXoiFochLoGPL9MpRGiH4ht3fgQteT0NleoF1XiSnM10sjFwfKMU5Wnn9pcQqXAbAF8r/l2+onxJmkb38/Zqptib8apWKeXx10xI71evno/kLhRHu9XeUmz4QPPbyfbA6yhsJPdrUMMNj+zF8Bijazl2v2HA9nMXcOVceGA9QnX2QWbLzL5YXaa3JX55FgJ7kTzRrK5mo0/jXk/6Uj2svkwuGqxfx+coA1YRjwm9/qnLeDquw2+QDRubdNY43CS4dodFEOhhlTgz/JdbquKuEN2RyrOhP4FuUow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.72) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from AM6P195CA0098.EURP195.PROD.OUTLOOK.COM (2603:10a6:209:86::39) by AS4PR10MB5695.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:4f2::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.22; Mon, 9 May 2022 10:16:34 +0000 Received: from VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com (2603:10a6:209:86:cafe::8a) by AM6P195CA0098.outlook.office365.com (2603:10a6:209:86::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.21 via Frontend Transport; Mon, 9 May 2022 10:16:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.72) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.72 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.72; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.72) by VE1EUR01FT105.mail.protection.outlook.com (10.152.3.156) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:34 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SMA.ad011.siemens.net (194.138.21.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:33 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Felix Moessbauer Subject: [PATCH v2 6/7] isar-sstate: add tool to check for caching issues Date: Mon, 9 May 2022 12:16:03 +0200 Message-ID: <20220509101604.3249558-7-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b43a4054-6333-4f7a-6321-08da31a4feb9 X-MS-TrafficTypeDiagnostic: AS4PR10MB5695:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KjPGAMZoHBxZDGxiPtU4yX3KEFHc2rp5IoYvZ36M+W8nJmN58QyHCTcRuq1gmXorvtROG/w00ZdCsWcomuc3CyTFXRoETIh9MXmwqW1dG6KdRuiGpGXpUk6LzEEXKZSDz3VLhzPQErLL4Zx52IrlwVHSILUrPfNnW9Uv0c3ItqogOb2fwBvGEwDVyIKvKP+xB5InPpEbufp8PA/RTa4xbLNRB0x//RUbgSlIsYm2DAuH9yBVa91PGDuZpnxfixtUeX+V+pfIzxGrtMkYOKkWxxppAlUHtxkShl1QOTO5mavfkJZkZx6ogMUYAdZV9i1jd8r4XYLR8OvO7vHhvLVOoaxR3NOlA3n7gbMpsU3CE9RzZX+XahVfk3vPknPl9sakgYaaO9cyvv4sXIRCHmvnU1CP7NfNz58eVV4soRSn5X9f8xGlFIXVskcj7xdE5wA+FiaT0f2dFIhPCYeqxjeZB0EySElzSRBlZdF3B8EqQzoWcRhsEO5Wuj91+2Uvo2/980EZwOFvzEUOI7WJFYfj9m+RnKQ4+x15hTKTZXiOM50cxIs4GF644eynRfSqF6IVBdhM09Ho5MYlU/hFDu16pxwZ9MtNHFC/5FdsCWpG35RzcoEZJVyxFYUC43ykjouhssJmbSj91FQTU6BYW8t7vWeeFb1D6k36wzdof+XQ9VvuiLFb4WmUka6punXsYDhH6x7669BWZ7CD3V6MAD3C3g== X-Forefront-Antispam-Report: CIP:194.138.21.72; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(5660300002)(82310400005)(83380400001)(508600001)(81166007)(6916009)(36756003)(2906002)(26005)(70206006)(47076005)(16526019)(336012)(4326008)(40460700003)(44832011)(70586007)(86362001)(1076003)(316002)(6666004)(82960400001)(8676002)(356005)(36860700001)(956004)(2616005)(8936002)(107886003)(186003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:34.3840 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b43a4054-6333-4f7a-6321-08da31a4feb9 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.72]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR10MB5695 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=w5YSqd7h; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::609 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX From: Felix Moessbauer This patch adds the 'lint' command to the isar-sstate script that helps in finding cachability issues. Signed-off-by: Felix Moessbauer --- scripts/isar-sstate | 73 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 2 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 8b541cf4..8ea85edc 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -62,6 +62,11 @@ to all potential matches in the remote cache, identified by matching architecture, recipe (`PN`), and task. This analysis has the same output format as `bitbake-diffsigs`. +### lint + +The `lint` command searches form common flaws that reduce the +cachability of a layer. + ## Backends ### Filesystem backend @@ -119,6 +124,7 @@ import shutil import sys from tempfile import NamedTemporaryFile import time +import pickle sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), '..', 'bitbake', 'lib')) from bb.siggen import compare_sigfiles @@ -556,8 +562,8 @@ def arguments(): parser = argparse.ArgumentParser() parser.add_argument( 'command', type=str, metavar='command', - choices='info upload clean analyze'.split(), - help="command to execute (info, upload, clean, analyze)") + choices='info upload clean analyze lint'.split(), + help="command to execute (info, upload, clean, analyze, lint)") parser.add_argument( 'source', type=str, nargs='?', help="local sstate dir (for uploads or analysis)") @@ -572,6 +578,15 @@ def arguments(): parser.add_argument( '--max-sig-age', type=str, default=None, help="clean: remove siginfo files older than MAX_SIG_AGE (defaults to MAX_AGE)") + parser.add_argument( + '--sources-dir', type=str, default='/work/', + help="lint: absolute path to sources folder (e.g. layerbase)") + parser.add_argument( + '--build-dir', type=str, default='/build/tmp/', + help="lint: absolute path to build folder") + parser.add_argument( + '--exit-code', type=int, default=None, + help="lint: return this instead of number of found issues") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -774,6 +789,60 @@ def sstate_analyze(source, target, **kwargs): print('\n'.join(out)) +def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, **kwargs): + ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() + if not target.exists(): + print(f"ERROR: target {target} does not exist. Nothing to analyze.") + return -1 + + cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + + hits_srcdir = 0 + hits_builddir = 0 + hits_other = 0 + for sig in cache_sigs.values(): + sig_file = target.download(sig.path) + with open(sig_file, 'rb') as f: + sigdata_raw = pickle.Unpickler(f) + sigdata = sigdata_raw.load() + + pn_issues = [] + for name, val in sigdata['varvals'].items(): + if not name[0].isupper(): + continue + if sigdata['basewhitelist'] and name in sigdata['basewhitelist'] or \ + sigdata['taskwhitelist'] and name in sigdata['taskwhitelist'] or \ + name in ADDITIONAL_IGNORED_VARNAMES: + continue + if not val or not val[0] == '/': + continue + task = sigdata['task'] + if val.startswith(build_dir): + pn_issues.append(f'\033[0;31m-> path in build-dir: {name} = "{val}"\033[0m') + hits_builddir += 1 + elif val.startswith(sources_dir): + pn_issues.append(f'\033[0;31m-> path in sources-dir: {name} = "{val}"\033[0m') + hits_srcdir += 1 + else: + hits_other += 1 + if verbose: + pn_issues.append(f'\033[0;34m-> other absolute path: {name} = "{val}"\033[0m') + if len(pn_issues) > 0: + print(f"\033[1;33m==== issues found in {sig.arch}:{sig.pn}:{sig.task} ({sig.hash[:8]}) ====\033[0m") + print('\n'.join(pn_issues)) + target.release(sig_file) + + sum_hits = hits_srcdir + hits_builddir + if sum_hits == 0: + print(f'no cachability issues found (scanned {len(cache_sigs)} signatures)') + else: + print(f'warning: found cachability issues (scanned {len(cache_sigs)} signatures)') + print(f'-> absolute paths: sources-dir {hits_srcdir}, build-dir {hits_builddir}, other {hits_other}') + if exit_code is not None: + return exit_code + return hits_srcdir + hits_builddir + + def main(): args = arguments() From patchwork Mon May 9 02:16:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Schmidt, Adriaan" X-Patchwork-Id: 1778 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3) with LMTPA; Mon, 09 May 2022 12:16:43 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f62.google.com (mail-wr1-f62.google.com [209.85.221.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id 249AGgEb003599 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 May 2022 12:16:42 +0200 Received: by mail-wr1-f62.google.com with SMTP id p18-20020adf9592000000b00207bc12decbsf5588887wrp.21 for ; Mon, 09 May 2022 03:16:42 -0700 (PDT) ARC-Seal: i=3; a=rsa-sha256; t=1652091397; cv=pass; d=google.com; s=arc-20160816; b=q42ewzcHh6ic5ZGVDiUP4S759NcqXe69FxH6/6ihFlXVj7jPejLY/FlhS3/xkJDl76 rDqpXH+Vop5b6J4IaiHe/wDRwG7rAzQcVTGPMdoz2z+vRQL2M3NpUxshcCeAC+fcazRS 2I/P82UDIQJPQNw8QeViRLxMXdWAo/pjNwD+ujQ/wydKaCJiPnvGtSDWQadIybz/LnUJ hIUqTC/hEWoWgX7e+1ocIKpWfY1voWGDuVbHxc2CcIumYW9sN+rxD09Dhwz2tNi8nFXX 2nnTi8NLBwLV2uLhjYSYBf8VzUgiA+3+16dycoejxib6mn0uHyRE9wCxUrNnvDACs7JI mL2g== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=tFErhcLvBkjYOkm9AXOOv3oI2AZDZTJO1ChPCfOJTn4=; b=NUq8m7fXI6mr/j0tJ0Qrc/XGad7liqJueP/Eec8h9Hir5UvsrOfltXjUXgjwa8+Pkv rrvbXB0eXOuzBXnwS149Z7+uS2KjxDlYrlGwSEkqw/Y/SmVBuI3AkmvBcHSIno1DGZCD KnMAum9Yid6+vIu0mPjwvdT9l8ZQYsi5j4TPSAx8nTavnksQvKZtXfAJEu4s237Bglnr a8dKF8vjkDEiBhtzXRSVx3wRxQ6CeGoEMKn2KrUVhkJ/t/jiLe9iyBm5FGSkns291PDC cvnKNLRNtQpgHyKVG58DRYN2YdBw/kr9zx3TL7nOjqp3wvjEXHJnDM3Pqo4r38ALgHFJ KqSQ== ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=Fw9kab9N; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe06::60e as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=tFErhcLvBkjYOkm9AXOOv3oI2AZDZTJO1ChPCfOJTn4=; b=EV88IwYqGzRBHgXjh2DZXfuroV2FB2kW842RA/DTT5UyJ/pmmi2uFEvicXcTlr8rlU uHD28g5DlPaX25gix3ZVCvHZeIbJZzITRkcWQzzeinN78LzCpFEY/sYWsggZfUNMzYJX 24HHmBTqvNR7I4szQ9k/PEIx9RwZriqxlCmex5FFUzvk6STpPDlV992aaCjw1vrDt0oY RymP9bLD4bS/AJsu9/sXsfUcrHHRh9T8D21FH7JhOPxJP4WbKRt2WF7TsOnqXPGmX8aA ZwrJX3EIHAq1XuQ688KoRp9xL8Vy32UDfGNrGEj+14rc2TFKqvM4Xi04tbwBvoRWPQam G2Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=tFErhcLvBkjYOkm9AXOOv3oI2AZDZTJO1ChPCfOJTn4=; b=e4waU1kY9tSOQBtuXvbyw0RKFZUS7917lTE7pVp0nEGbC4Fe2+qLfAvrf11/STe299 xTnuUxO7y6OBqqRNJT5u6VCZV2T9yrau+/vpjHzjuxrTJ5aDkBiO71sGOqWlcNE0Lnq4 peyZ+8UgIAeqSLWMJTGHU1eX637sR3XD6npdnYl3zQ7dsl+Zb8WQ2B0vuD0sF8xkZudD kZ7PnmWoINwbDBU6O9mjgvPr5fKkjYiwsgLK/MzseMSH8PxbOsaA2VooW/PeDqUzyhB/ N9UCXuQ9kkArf0prRbRwf2USLzLUAKKixcKzzKP+DEyLjZua2EtoLP41rRdW0ehmontD d3Dg== Sender: isar-users@googlegroups.com X-Gm-Message-State: AOAM531p6JSPQFd+2WMnX+EB/3SHleP+8LeIQDsBW7XWoOH+ggTmjXBg OcD6VEQsbRI1snGUy6O47wo= X-Google-Smtp-Source: ABdhPJym1LgZmEDa8uJmPqZtXg/gwG2Nr4QeUKrdF5pLuVWnRoXwWRPGARSLGJRrELolhnRehJ9Dpg== X-Received: by 2002:a05:600c:4f13:b0:394:8978:7707 with SMTP id l19-20020a05600c4f1300b0039489787707mr7159458wmq.34.1652091397490; Mon, 09 May 2022 03:16:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6000:1688:b0:20c:67b9:e68b with SMTP id y8-20020a056000168800b0020c67b9e68bls1543975wrd.3.gmail; Mon, 09 May 2022 03:16:36 -0700 (PDT) X-Received: by 2002:adf:cd12:0:b0:20c:ba84:287c with SMTP id w18-20020adfcd12000000b0020cba84287cmr7482122wrm.490.1652091396523; Mon, 09 May 2022 03:16:36 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091396; cv=pass; d=google.com; s=arc-20160816; b=Q/qg6qSl4bQ7q0ynlu1qkabumfcZUq6RW+bZYO13q4fLQgVWSx6CudCf1VelYHH0h6 8ul3p5k/RQzTQJNETVQ+FgxLlDolOiOG3lTelWCr691+Oj1bR8R0oKabSi642m3sfj1P 7LTk+wvDSER/YiUUHMMfheiaFs/a3j1/cFKZZoz1YI4Yfdla/6ABiCcpasLuMdt4KS0x F28XyM6d7AJsuTLf3fCo68Zkz0bTIovhrDHfBjk0MLbfgfIWyh6V7W6nyd34nFa+jUZg AU2MNM7EZfOsDRhAO/EUYHV9fgjVBL9FGfWtHen4y7fMRPlh/Q/mKP3B2nrzoEvyBi5B Ps8Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=TLWPrnFd94QE9PjiI3lBiSPdmRugGVocCH6HXMmbugA=; b=MM13qePbNaMveoewhfHV+G7jxTRsgTvOLCPvF51oziUV98RxERMNvlwLBm3m3pCrzL RmiLh5Sw90pJyW7SrE7NHd7iH4rn5vgqk/xC7IR2Q5RlUFyC9dH9GvTasQITcRbBFsXj hZCYAbtuKLpBlpuCGlp6DUH6uRS8ESeAaQYLSuJRyaGqjkYdrinsuMKOaCzdanhMWlyE S4BsXOD9jRqEyIBFS5JcZsKCxQnUjkzBl3QtJI4A3nmRNeW8UtP5ADZR/c628ENHQiG0 zzVHqoGDQ8EzsHgBEYxNAoek7dFm0DVPJdHzFIPyZQTGzv0SxoHOSJe9odxKX9hmjZpu ED1Q== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=Fw9kab9N; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe06::60e as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-ve1eur02on060e.outbound.protection.outlook.com. [2a01:111:f400:fe06::60e]) by gmr-mx.google.com with ESMTPS id k3-20020a1ca103000000b00393faebeaa1si363022wme.4.2022.05.09.03.16.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:36 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe06::60e as permitted sender) client-ip=2a01:111:f400:fe06::60e; ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=emtkbNc64EAHLIzH8d2Qvi508p0eaJ8Z1smKDTjbn0uDyJbbuLZzoA+Gb4qbYKtix9bx/WzTGFjqNZ+Z5lxiZUgSuF69tS/YxFFsfgwUc4U3aHB9+xoks5WTKN+tb5qJ9OYPxcQWP6o2MQbqe1bwBnt9norEVUMuVL/P278J5jsIdZ372KWIZD6qxuT7grOds07fMkhQMXzqZf/0O5sgKEbIRCFQT+Y/htfOOnu+MtYAU0aOOwCREx41hEFkQPE9bvnX3zVj3X5rGMsGDkBVCqnwyorotBwFV1tmNV1otmQ+Bug2ut6zFBKofRcnamvemRxuYd7a34nVltcsBwCVSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TLWPrnFd94QE9PjiI3lBiSPdmRugGVocCH6HXMmbugA=; b=gAyq6WshY1lTGjLyfIHl3uhhPZXgw778FiLbIYgYO+jNIv4Uhvelsu562dCIJ9qCG2uKrpOaVAMKvxEKSVI+rZX5r4Zmld86h67XOfPA3GNUbED3yeLSjBG3Dx7qT+nwU5IzlIJ/piqrCiWkruz7UcKVRcedYWnGuQWS9vLk7v1ELzRV1Hb8DXonVFhDa1q2ef4zMRKANFd4cJ8OzE8DHl1vqLuwTdVn1w1Hja8k+Rzq3cec486SBMqxlJngaACGxqMr/Hf6bl25Ovtxw+pXkOLLNYPFdmdZDdvWu87QwW+olTh+c5eCXEgcYNUYlQo8KlND0R0R9QfAO76UTlcTbw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.72) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none Received: from AM6P195CA0099.EURP195.PROD.OUTLOOK.COM (2603:10a6:209:86::40) by AS1PR10MB5554.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:475::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23; Mon, 9 May 2022 10:16:35 +0000 Received: from VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com (2603:10a6:209:86:cafe::ab) by AM6P195CA0099.outlook.office365.com (2603:10a6:209:86::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18 via Frontend Transport; Mon, 9 May 2022 10:16:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.72) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.72 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.72; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.72) by VE1EUR01FT105.mail.protection.outlook.com (10.152.3.156) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:34 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SMA.ad011.siemens.net (194.138.21.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:33 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 7/7] testsuite: add cachability analysis to sstate test Date: Mon, 9 May 2022 12:16:04 +0200 Message-ID: <20220509101604.3249558-8-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 383e511d-0826-4105-4b19-08da31a4ff03 X-MS-TrafficTypeDiagnostic: AS1PR10MB5554:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kYv2paJ3ASKHW5t8WtBqRs+mJqicOlMWP5hD2qD9/jw+/+x6DpE5E+pB6F2EhjEsCr3kpahAjan0nKsVnIAe820WHd5WdaZUsfMZr3pbYwnCJeoAH70o6o92MqQ5DUZwxoS+a/xDVhFuPOzcx9Z2xAqedCEcBrCLju+rzaDTL11VZkTOYRY6xQPDI6m5XrjBt5tn5CSzFN7ZbkVh4U1AFtGYj6pun5NB/+4rii1hoBT8mJQegPy0ehDQHXbm628VBer0ecoxONEFVlEM7uCmpleFJ2UlFjxqBnLpTsrw1nNIXzgICNnDsOX6d4mLnjWjG3f1DSKUPkhfRuwSZOhr3CQfTQbjTVM00GYOqs8pq/fZa1uhzKypDe6OkXUFXDGI9mzvqMIiHiA6hcefluICmizOb7zFbh03oXjXg/9NfTBNoYuRzMURCRb7ERAPbK5//U2F1hguyDrkI+UjFDOKFTTmc16H3SpAIg5ahzK6ZJNDhiDHU++YSVlfVcoA6CmazF0qN4AV8DE7YmYhoZZX190RmxnebjJU5KLJ2r217vOcjmaWYmpILBShI5MpvmW5qmsoD0d3TqqG+5xFeHnmCH4Njo8Rlc/Y3Gpq/jVIxB8In447jd3L8iOfmLj7dmowOou3rkVRIkaQEPVJ8PEciSOOGjZMynxA+Ic3IxYaCpaJNXpjboDrBtEzpmy0nhZoVYYWMEZoNQVsrE9tqAZ/gw== X-Forefront-Antispam-Report: CIP:194.138.21.72; CTRY:DE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:hybrid.siemens.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(83380400001)(16526019)(186003)(6666004)(36860700001)(1076003)(26005)(2616005)(956004)(2906002)(44832011)(107886003)(36756003)(40460700003)(336012)(47076005)(82310400005)(8936002)(508600001)(5660300002)(86362001)(82960400001)(70206006)(6916009)(70586007)(8676002)(81166007)(4326008)(316002)(356005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:34.8524 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 383e511d-0826-4105-4b19-08da31a4ff03 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; Ip=[194.138.21.72]; Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS1PR10MB5554 X-Original-Sender: adriaan.schmidt@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=Fw9kab9N; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe06::60e as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-getmail-retrieved-from-mailbox: INBOX Call `isar-sstate lint` on the populated sstate cache to ensure that no undesired absolute paths make it into the sstate signatures. Signed-off-by: Adriaan Schmidt --- testsuite/cibase.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/testsuite/cibase.py b/testsuite/cibase.py index a1494ce6..2ffb8191 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -6,7 +6,7 @@ import re import tempfile import time -from cibuilder import CIBuilder +from cibuilder import CIBuilder, isar_root from avocado.utils import process class CIBaseTest(CIBuilder): @@ -103,6 +103,12 @@ class CIBaseTest(CIBuilder): # Populate cache self.bitbake(image_target, **kwargs) + # Check signature files for cachability issues like absolute paths in signatures + result = process.run(f'{isar_root}/scripts/isar-sstate lint {self.build_dir}/sstate-cache ' + f'--build-dir {self.build_dir} --sources-dir {isar_root}') + if result.exit_status > 0: + self.fail("Detected cachability issues") + # Save contents of image deploy dir expected_files = set(glob.glob(f'{self.build_dir}/tmp/deploy/images/*/*'))