From patchwork Mon Nov 9 09:18:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Clara Kowalsky X-Patchwork-Id: 5596 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 09 Nov 2026 10:19:47 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-qt1-f190.google.com (mail-qt1-f190.google.com [209.85.160.190]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 6A99Jj1E026155 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 Nov 2026 10:19:46 +0100 Received: by mail-qt1-f190.google.com with SMTP id d75a77b69052e-5363b06bcb2sf15600751cf.2 for ; Mon, 09 Nov 2026 01:19:46 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1794215980; cv=pass; d=google.com; s=arc-20260327; b=PQa6/tsIwBz14vgBJ+xhEKzx+BIRq8HLnYvK63/VW7IORZ/EmwwdCJSb+kZdRzFIWB fPmS8OOBq2oLhnsyQjZaBQqqXO7e1HcpqY+ifgbcnH6lwa80C6pY9LTU5+7GW+NwOO7f 4KCdyR9a2MqkwlmaVSFNMMqDv96PNYUYGnB4lu6eDES46aNrEcXHOOItNE0J0nwxBjIZ qn1kX76TE7DMBRfw2GpYJhCP+X+QwS54F9AC4GhSsRJorFt5CQtN2QTy1b/U9vi5UHJ1 RHLfj9vrRRzGa0AnC0yZXuOCiSMJKsqWlBvxmkyvgjNYovnzFY64X91J6cpYMaV7nJJu mqpw== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:mime-version :content-transfer-encoding:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=go0O3LhBPODqIwOuFwJmfZyhfxmswrdel7+1l87nL8k=; fh=T/14US3MXEI7wiGo4OQbI5b+xPrHdyivY60/hBV4vMk=; b=phXgnRRhZcc64XfPEGwR9gSjYxog/5kQ7GHOPT9uomczxkLhzPcjr5jQOnOtG1/CG4 mKqN/GnKracU48rAGhZn3macIAX1ihTGqpvjKwV3D3bkWjSygatW1i7GaVvnEc+d/FZ+ 9uZgopCUFV8NNw/JOuapR+ZU19vdsXK5UkXXO1GLazPENq9Gqb3Vcv+1KIlmGM3Wn/en 7xeZHiSKNCAXPsYLH/h5shBanbt1skTKWpURoTX8ibuZoVkP6BcK02/jQn+NzQBZpOL3 LH8BqU70gqT60mKLNNNS397IBnEW8T0ND/u/FyYAglLRFvreOeUK+AgVYzIofvLKMSWM /ixg==; darn=isar-build.org ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FUN8WYfb; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20260707; t=1794215979; x=1794820779; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:mime-version :content-transfer-encoding:content-type:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=go0O3LhBPODqIwOuFwJmfZyhfxmswrdel7+1l87nL8k=; b=YBg2LfpHLgRNisA7AE0h6P6Bdyx8w1coFPQmMP4uIWRJ65Yedx205+EIHobvVfGJ1U UR1P/zaHqHDiAt2wfHhAsFO6/VkS19AZPqwPIMEgfcEXSwiKRU8yo7ijrzjhzG6ZxIMg hCDD1vXSrqH+T7SF8ysXDFs8xVCmDUO2Vy3mkcsWX32xGbYJtJRVfrY1AwtisdWL/2Hp k230Q8/p6b4vIlDjzSLDqF3iQBm1JJ0v9Tu9OCgzoxEvraiDmHLAieUqS4QO7lItAems XkDE5dsOCmY+vlGXQsl5YjBRn6JBzOYHC29d/k5wUQHoJbj1GODQkddInMRYiMI6leF8 HpjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794215979; x=1794820779; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:mime-version :content-transfer-encoding:content-type:references:in-reply-to :message-id:date:subject:cc:to:from:x-beenthere:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to:content-type; bh=go0O3LhBPODqIwOuFwJmfZyhfxmswrdel7+1l87nL8k=; b=efsVbyDMGcJsGIUBapIZ+U9/2maPTuBsaDzdsImuIhnAt5t7AhhItDW/f/rSTfIICy pCd6ESL3vlsEuUoqi1aCNymwx3M7yzvl8kaMr6FCtbQuZFH3TAHISy8aCnPMuWhf1D1r /1DDy55ISZTmwtYyjy+qvpEHjcQXxbLl3d64sj3bCugl9SSBDeb0CvWTRIdpYSvtRlmQ SmA8+R7LnZUe+gsBqktlMOCyzKLtXOoeZj4G3n77VnmfeEVVdGf5T6xz1se99GnyeE+c DymZ0QZFQmrgCufObipbhf09JLPqq4QgR+IEPd6GX9OQwHBowtTlVMZgV9XMJg2f861v kovw== X-Forwarded-Encrypted: i=3; AL+hErEcpAS5djS9kPEju0xoOhHgJNswaOc90qbdpwlgwHQy3bPF9BoOTDpxYgOvfN3Xjl8mrasLMVI=@isar-build.org X-Gm-Message-State: APuU4Z8m2caxWntWc1qmz579DP4C701TCOVSmFfrwjEsCNYrHiqRUAES Nva5KpsLin2p+ZWbtcnyXXpbda+CnzPuug/mw35RxuqOxsojZ3AQdHzp X-Received: by 2002:a05:622a:5588:b0:533:7a4c:d8e9 with SMTP id d75a77b69052e-552a4cc7e62mr45622461cf.37.1794215978520; Mon, 09 Nov 2026 01:19:38 -0800 (PST) X-BeenThere: isar-users@googlegroups.com; h="ATskLdf1UXYb2149+mDFsZEEcbWygd0WR0kUXpd6zVbvw5KeZg==" Received: by 2002:a05:6214:2589:b0:914:1182:d90e with SMTP id 6a1803df08f44-91eccb3713bls35188576d6.1.-pod-prod-03-us; Mon, 09 Nov 2026 01:19:37 -0800 (PST) X-Received: by 2002:a05:6102:512c:b0:7d2:d45e:5c9d with SMTP id ada2fe7eead31-7d2db0e27d2mr663159137.13.1794215976808; Mon, 09 Nov 2026 01:19:36 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1794215976; cv=pass; d=google.com; s=arc-20260327; b=Hr74bfo++q4FBZFso7tt3eeEXwkNdO743MRNRA10lTW/hBrCpk1FdtQuvwJRdUhesr tUFVAj2VbB8DBGyQ+cUtlHRjI8noapo9zcq+RptcS4anVeUPwGMwPwi0iz5Q60FS1/H5 j3ZtW8ZTdFJwvOD1ITAXyK+WgVne5TwUiueD4NaRTzwdBCPRuAe3NmKHrHM9UdO33RkY eKyjK9LvQpcqc3idFHPUwIRVhStWqjoqJE3++85fMkZC6qasaGoOjzta8iFlicLTlPR5 Dp7D/k6Qpqolxrt4Q3YMDH5tSYjfns9HKis4J/jWHLgarxZZI8aEFZgIaqbQOZYHs8/5 86pw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ODJ0ZYuPAilOAyIgo4ZWu/UQ6yKRLrzrSIq4t8GFbKo=; fh=AqT8zD6jUs4YP+XkSwmnbRYu3wrat9Vyg/xvm1MGQCM=; b=eShR2ASGH4HuSN2ibUak9yrKngawqd/5xo2UuUTPGk/0T+ykRT8J9xc029Ka5HDujD NgPPUHBgLeJPNOfKfOGgtdeZ3NUfHLCF+rIBOEi0cmxizLGAp4Xzj3iNY23XtFlXBe9L H/oylsNoH/lueqXPfhj/4/BdLOMZOr1P/ZRR/IErYieWcJA1qUjnZCpZ+LMHCDxK8+ef zpGF547iurj5gpX9/GLxR5/3pg+nwwg/FQrMxyU36F1V5ChR58+Z25Ihz/LusM3DUT/o 11uDsKiP35RLu3ECce4ffugJD+7fhzni59ASuvXiF8F8krrx5xcS+r4IzB1D7AipJjZz F8BA==; dara=google.com ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FUN8WYfb; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazlp170110003.outbound.protection.outlook.com. [2a01:111:f403:c201::3]) by gmr-mx.google.com with ESMTPS id a1e0cc1a2514c-99c0292009csi92776241.1.2026.11.09.01.19.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Nov 2026 01:19:36 -0800 (PST) Received-SPF: pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) client-ip=2a01:111:f403:c201::3; ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=f7822B1i620WFGhx0qfkCLklQeb5e/X5Ojs5OOG6oIZZKtAZgcB0CfP8H73rg+jbziq8ldi/dMi9tuiEPL3mffn2AXBEaVG/tPACJDFpx15ABxmbQIgmBXk4LeY1Qu8KLdRS3lT4WW7QYvKerQJj/sm+7iigBxZd04jO4fzd/nLESxmflyxH/JAOsWqyUOsqAs6ilEaEJIFKmepC/GSEdxgMkfRR5mVdkRU2RupJVXS69wIqkrne+Uew+KJ5cbDYCBfs4ipQEuPxHIteNWrQh2OAVCl9oOjRaeYo1/lW0DVvb4eJd9UhihImJ+UqfoiNV2nYt1lY/Pp1n/3CuhjZ1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ODJ0ZYuPAilOAyIgo4ZWu/UQ6yKRLrzrSIq4t8GFbKo=; b=p1Vo+eVSms3F7y79N/kUrFUt/dRO1wkl+wzezqc3IWaDQq6EalMt7NoQnwQn2DMwJNMDbdYSXPvqvvFEmydKTr/MMJP/cOQvHENnvFuvSWTyrhao3iP2h9IsnqkJM+Ph44kXe8hWak5A+eQ35pQtnxjd9813Zeapouu/UT52SGn9qhmLRDlaCLCAp4EmMB1bbdFaMrccJAv82oHt1Te32d0jFMFuI9K6dA9+yyFZURmkkZKrVE35lXI1NsxIWPHMLg3yA86ZJfffEZR5ehCMhZGjNBDyFBBtdeuWFt4ocJ1GtUjSylp41tqWXOYQOqK/RF+QdrfzyZPkKpctwYpLmA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none Received: from AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:740::12) by DU0PR10MB7554.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:426::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.615.7; Mon, 9 Nov 2026 09:18:54 +0000 Received: from AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM ([fe80::527e:14a7:bd3e:86be]) by AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM ([fe80::527e:14a7:bd3e:86be%6]) with mapi id 15.21.0615.007; Mon, 9 Nov 2026 09:18:54 +0000 X-Patchwork-Original-From: "'Clara Kowalsky' via isar-users" From: Clara Kowalsky To: isar-users@googlegroups.com Cc: Clara Kowalsky Subject: [PATCH 8/8] bitbake: update to v2.20.0 Date: Mon, 9 Nov 2026 10:18:01 +0100 Message-ID: <20261109091802.202521-9-clara.kowalsky@siemens.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261109091802.202521-1-clara.kowalsky@siemens.com> References: <20261109091802.202521-1-clara.kowalsky@siemens.com> X-ClientProxiedBy: FR4P281CA0130.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:b9::18) To AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:740::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM0PR10MB9645:EE_|DU0PR10MB7554:EE_ X-MS-Office365-Filtering-Correlation-Id: c29c303d-1dd7-4f46-27fc-08df3e427695 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|69100299015|366016|376014|3109299003|7049299003|23010399003|1800799024|39142699007|31052699007|260925021911599003|260925022911599003|260925021311599003|261009223327099003|5023799004|11063799006|56012099006|261021133027099003|261021123027099003|261021135927099003|261021130227099003|261021121427099003|261022113327099003|261022115727099003|261022075727099003|261022105027099003|261013131602099003|261009221227099003|261009220027099003|261009221727099003|261009223027099003|261026130827099003|261026140227099003|261026135527099003|261026121527099003|261028144727099003|261102130727099003|261102124527099003|261102125427099003|261102123227099003|10067099003|12006099003|8126099003|21046099003|10086099003|10136099003|25016099003|21026099003|55112099003|20052099010|18092099006|18002099003|22082099003|17002099007|16056099006|10063799003|9063799003|29003799003|6133799003|13003099007|4133799003|3023799007|15866825006|191535003; X-Microsoft-Antispam-Message-Info: HHc2Av906Gi+WoQ/FiXFEulCGjfyH1YudPhhmyB9kELRnKM0SNjh/rRTfxRv9tRND78xp/nDC6PBUTqvY4v0ap9As0g8rXUCQnKfcVb8DOEnmD/maNFjkR2P0g/fvhCqAUiHqTeSMO0aSG6ScXysarF/qxy+/Y18dQ9q9nGvT55rnCP5hy4bRbzZB5jU3acb7yE0Tj1755gPdoU+eFdHz1G5psO4mOunJUK2J1Z8Jct1LgYRCs5VoIZ0vnk4GkO8+Q7682Ieia7+0Lt6UqopVSr0k4XksbC7yK836U+mCl9h6tGNupDv4/PUlNB1eH/TcPNfcpTnr9UF6lU4CNVVjmqmjGQXeN4CkjtJhpSpXl9AOcgMWuYugJEjfzza6bEtw7iMV0WIEIPiP6hQDLC2T9wtQ/uRDBFJ/ca4kTUby2hqOmFdzAcjdSPRkSGkVkILhQJK7apDhIcFBFRYRy0hFlLlPOr/KMmVG57PPxiRew5lOgrYP8pDq242YwTMV3NEgu66+671kHUZvi4BPbN8/YHyvcIyyhXbKnRufA1gv7B5uvmfCOfeyYmdatUVZedrqUQjYRKyL9enRQERiFdoDuwPG6axmkboo671C8o8/qn1TP1uCIK9CNFvSQ+AIhhNPE/PEW1O2HddRPyfJYSOeVOyuI9RpwfoE0jfG0ZRlKb9C9TnY9VXGruOBl1EUhJV X-Forefront-Antispam-Report: =?utf-8?q?CIP=3A255=2E255=2E255=2E255=3BCTRY=3A?= =?utf-8?q?=3BLANG=3Aen=3BSCL=3A1=3BSRV=3A=3BIPV=3ANLI=3BSFV=3ANSPM=3BH=3AAM?= =?utf-8?q?0PR10MB9645=2EEURPRD10=2EPROD=2EOUTLOOK=2ECOM=3BPTR=3A=3BCAT=3ANO?= =?utf-8?q?NE=3BSFS=3A=2813230040=29=2869100299015=29=28366016=29=28376014?= =?utf-8?q?=29=283109299003=29=287049299003=29=2823010399003=29=281800799024?= =?utf-8?q?=29=2839142699007=29=2831052699007=29=28260925021911599003=29=282?= =?utf-8?q?60925022911599003=29=28260925021311599003=29=28261009223327099003?= =?utf-8?q?=29=285023799004=29=2811063799006=29=2856012099006=29=28261021133?= =?utf-8?q?027099003=29=28261021123027099003=29=28261021135927099003=29=2826?= =?utf-8?q?1021130227099003=29=28261021121427099003=29=28261022113327099003?= =?utf-8?q?=29=28261022115727099003=29=28261022075727099003=29=2826102210502?= =?utf-8?q?7099003=29=28261013131602099003=29=28261009221227099003=29=282610?= =?utf-8?q?09220027099003=29=28261009221727099003=29=28261009223027099003=29?= =?utf-8?q?=28261026130827099003=29=28261026140227099003=29=2826102613552709?= =?utf-8?q?9003=29=28261026121527099003=29=28261028144727099003=29=282611021?= =?utf-8?q?30727099003=29=28261102124527099003=29=28261102125427099003=29=28?= =?utf-8?q?261102123227099003=29=2810067099003=29=2812006099003=29=288126099?= =?utf-8?q?003=29=2821046099003=29=2810086099003=29=2810136099003=29=2825016?= =?utf-8?q?099003=29=2821026099003=29=2855112099003=29=2820052099010=29=2818?= =?utf-8?q?092099006=29=2818002099003=29=2822082099003=29=2817002099007=29?= =?utf-8?q?=2816056099006=29=2810063799003=29=289063799003=29=2829003799003?= =?utf-8?q?=29=286133799003=29=2813003099007=29=284133799003=29=283023799007?= =?utf-8?q?=29=2815866825006=29=28191535003=29=3BDIR=3AOUT=3BSFP=3A1101=3B?= X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?kIIVudagSLPfjhDDvFWdMkN8RTpx?= =?utf-8?q?cQjgH1EoIF57wL7zYTbVgSiAUC1fncgide07sbnHazMyMyLfTh2TLcqEKx9TXsj2z?= =?utf-8?q?7Y4dNyZ5pQ6PidLeXbLWMDpnfLBOe38WwZR4qyW7IUnG2HyDwvrTW3U5FaZWeZUgW?= =?utf-8?q?5+wlrUEpau9MCiOleWBvWPcfGP95z2xrvejo93dlIm8LsmTui5R1eLJI6JCJPtCP7?= =?utf-8?q?Peh5ZzIYO0oFkkrKnTjQs9FQGk0SCUKznpGL974+xElSU2GGeIZBsPLCEXc6eNvKy?= =?utf-8?q?VNHrIhE+z95jYDBk//aW6ffV3+5OngGpePYTURJk7mGE0UxVSpaf1EYNNgWnMI9Hi?= =?utf-8?q?s73k7AfkYk8+WzU877A7l180+ELOV4QgS6gUX6YjbQ02aeF4kxR4Y3tx7qEl3fiOw?= =?utf-8?q?FmqaS6nDZ75Yy5jSnBLi5/FqK5zfcSig414g/EqPbizEMn5l6yPjkii+oEtoo8FGQ?= =?utf-8?q?4PBvFNA3TpSKbbxuKuzjHamB7bWF1alL8Lpl/0Apl9mj2mAenZX0UWTnPqaEIGB/2?= =?utf-8?q?KMcWafOopvFSqtVwos0ll6IXbUKSghfuLFYSJ6fDVisjVLJo2FLfuFhWY5x8QkB4l?= =?utf-8?q?Dkdk4aspGjUoFWkSzzIFBbFX4ryCFK6nlPIMqZYEU4yhjzs1jRL4x26vPLqDCVKAy?= =?utf-8?q?OmIAlgSVRkMrU1Z8lVCOewozd2MpuDW7uQ9RnnDo/jiL6rzEOKebqWvel2Lx5yJ3u?= =?utf-8?q?qwyYadgDb2YcUPKc8HH5fH7xmfLf5Cfj4CaxoLJBtNI22eh5QHY4rLanktXatrgMP?= =?utf-8?q?fmrkCeuvISp4eoUJU5ZSlwLkNAIAvapuFzxqUNKkWTleVDU2UurKqMx2EtMRCzxfb?= =?utf-8?q?mgcUZCKhuheezcdMJM4inibYDhunWC8H3GmFn9zf+5gNlLGHtQSYM+WGIJ+QeZBAf?= =?utf-8?q?YQm2AQvRkvbjQbrkrYXlGDEBX1zW0bsfWkTI2H9prRUpxkQHq5Yr/VT+zFPconcjG?= =?utf-8?q?84RX1+TUunW1G8TEG++/MUxl4G5j1IUUgVk7YeYAA63GYYjCUINUFGPEmd4XmxpgG?= =?utf-8?q?SmpNmRbRS+XX9wss2ChtnlfeZklIoVYY+YV7cGyGbC7Zc6TerRPY+4wzYwmMR7Wn+?= =?utf-8?q?3KBiNJP/olTL8Pb8zxmgiuTcXNqTVIN3VpJQjFTVVW06B+77wRy1UhvF2WnEcr47j?= =?utf-8?q?prXC2BQqqdakeUCt8bTsY76Pbm+2c+NsbcBUevsmhIXbgE4G7RzaniEihurtWNO8B?= =?utf-8?q?aOGsbhGJyOQIi7yConfDzR9LZco0nCulzOtqru4FF1cvyw/V5ovNx4XcZy5Bua1aO?= =?utf-8?q?r6/n7zdJ0Pkauj83ihRdCXjWW12zdo1BzYHUfSfOzIBUZtCx5Rqj20voLEJ6CmFpw?= =?utf-8?q?PNAxegMJJTtNaw2YHEh4OL+P5VdqR36jAuFZMwptbTSfIHT8CFJFAKfcFVnws1KoK?= =?utf-8?q?HrEwwlKKSuoL48XE5K/XjQ3f2xD8YShlBsmmgoL5q4PXplA4txvNzxBBaIKwie4eZ?= =?utf-8?q?dSCtAzjluuBewfUuh5avTEfKkVJ52WgbOVynGssW/xUNAhGcaYMFbhBnZznc4KEGH?= =?utf-8?q?DPKFxcOXkic1pNvTythi8DaUp44bBjstDLwH0yUFyZIl2qyovkJpFlBE6FPIAqQyN?= =?utf-8?q?29pe8XXWnYN4nDwxpp/GKUGBRdPsye5gfx4abKIuynrRaM8Nur+4yyMWmNLv2tkG7?= =?utf-8?q?ho8QZ8UckCkgokFBalQRBFSAFBeojb3AtsYkb3vX07gfdprr67IME=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: c29c303d-1dd7-4f46-27fc-08df3e427695 X-MS-Exchange-CrossTenant-AuthSource: AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 09:18:53.9751 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /ggjromjMhGgeLAivxiEXCtHfJ+Z+cfg8uprrU9X15zH8l0HEQsWxZN7/3q5aegEkvkAaZdpMAS8Nh2s1MwxRU3AdB214WSHWfyQDydInf4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR10MB7554 X-Original-Sender: clara.kowalsky@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FUN8WYfb; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Clara Kowalsky Reply-To: Clara Kowalsky Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= Update BitBake from v2.8.0 to v2.20.0. This is upstream commit 634f8dede6b9d71b5a6691499fd05a8678b46eae, tagged yocto-6.1. Drop Isar's lowered Python version check and retain upstream's Python 3.9 minimum. WIC now runs with the build host's Python, so imaging Debian buster no longer requires this workaround. In gitignore, set /build* so Git does not ignore BitBake's new bs4/builder directory. Signed-off-by: Clara Kowalsky --- .gitignore | 2 +- bitbake/.b4-config | 4 + bitbake/.gitignore | 2 + bitbake/AGENTS.md | 55 + bitbake/README | 2 +- bitbake/TODO | 65 +- bitbake/bin/bitbake | 4 +- bitbake/bin/bitbake-config-build | 1 + bitbake/bin/bitbake-diffsigs | 16 +- bitbake/bin/bitbake-getvar | 21 +- bitbake/bin/bitbake-hashclient | 115 +- bitbake/bin/bitbake-hashserv | 17 +- bitbake/bin/bitbake-layers | 46 +- bitbake/bin/bitbake-prserv | 34 +- bitbake/bin/bitbake-selftest | 13 +- bitbake/bin/bitbake-server | 14 +- bitbake/bin/bitbake-setup | 1481 ++++++ bitbake/bin/bitbake-worker | 42 +- bitbake/bin/git-make-shallow | 175 - bitbake/bin/toaster | 23 +- bitbake/bin/toaster-eventreplay | 7 + bitbake/conf/bitbake.conf | 2 +- bitbake/contrib/b4-wrapper-bitbake.py | 40 + bitbake/contrib/hashserv/Dockerfile | 5 +- bitbake/contrib/prserv/Dockerfile | 3 +- bitbake/contrib/vim/LICENSE.txt | 18 - bitbake/contrib/vim/ftdetect/bitbake.vim | 24 - bitbake/contrib/vim/ftplugin/bitbake.vim | 13 - bitbake/contrib/vim/indent/bitbake.vim | 343 -- bitbake/contrib/vim/plugin/newbb.vim | 88 - bitbake/contrib/vim/plugin/newbbappend.vim | 46 - bitbake/contrib/vim/syntax/bitbake.vim | 131 - .../oe-nodistro-blacksail.conf.json | 48 + .../oe-nodistro-master.conf.json | 48 + .../oe-nodistro-whinlatter.conf.json | 56 + .../oe-nodistro-wrynose.conf.json | 48 + .../configurations/poky-blacksail.conf.json | 74 + .../configurations/poky-master.conf.json | 74 + .../configurations/poky-whinlatter.conf.json | 80 + .../configurations/poky-wrynose.conf.json | 74 + bitbake/doc/.gitignore | 3 + bitbake/doc/Makefile | 2 +- bitbake/doc/README | 2 +- .../bitbake-user-manual-environment-setup.rst | 1332 +++++ .../bitbake-user-manual-execution.rst | 35 +- .../bitbake-user-manual-fetching.rst | 305 +- .../bitbake-user-manual-intro.rst | 267 +- .../bitbake-user-manual-library-functions.rst | 59 + .../bitbake-user-manual-metadata.rst | 462 +- .../bitbake-user-manual-ref-variables.rst | 464 +- bitbake/doc/conf.py | 41 +- bitbake/doc/index.rst | 2 + bitbake/doc/releases.rst | 174 - bitbake/doc/setversions.py | 374 ++ bitbake/doc/sphinx-static/switchers.js | 233 - bitbake/doc/sphinx-static/switchers.js.in | 249 + bitbake/doc/sphinx-static/theme_overrides.css | 17 +- bitbake/lib/bb/COW.py | 10 +- bitbake/lib/bb/__init__.py | 158 +- .../_vendor/beautifulsoup4.LICENSE} | 11 +- bitbake/lib/bb/_vendor/bs4/__init__.py | 1177 +++++ bitbake/lib/bb/_vendor/bs4/_deprecation.py | 80 + bitbake/lib/bb/_vendor/bs4/_typing.py | 205 + bitbake/lib/bb/_vendor/bs4/_warnings.py | 98 + .../lib/bb/_vendor/bs4/builder/__init__.py | 848 ++++ .../lib/bb/_vendor/bs4/builder/_html5lib.py | 611 +++ .../lib/bb/_vendor/bs4/builder/_htmlparser.py | 517 ++ bitbake/lib/bb/_vendor/bs4/builder/_lxml.py | 501 ++ bitbake/lib/bb/_vendor/bs4/css.py | 330 ++ bitbake/lib/bb/_vendor/bs4/dammit.py | 1525 ++++++ bitbake/lib/bb/_vendor/bs4/diagnose.py | 268 + bitbake/lib/bb/_vendor/bs4/element.py | 3395 +++++++++++++ bitbake/lib/bb/_vendor/bs4/exceptions.py | 28 + bitbake/lib/bb/_vendor/bs4/filter.py | 764 +++ bitbake/lib/bb/_vendor/bs4/formatter.py | 276 ++ bitbake/lib/bb/_vendor/bs4/py.typed | 0 bitbake/lib/bb/_vendor/ply.pyi | 1 + bitbake/lib/{ => bb/_vendor}/ply/__init__.py | 1 + bitbake/lib/bb/_vendor/ply/cpp.py | 918 ++++ bitbake/lib/bb/_vendor/ply/ctokens.py | 133 + bitbake/lib/bb/_vendor/ply/lex.py | 901 ++++ bitbake/lib/{ => bb/_vendor}/ply/yacc.py | 2238 +++++---- bitbake/lib/bb/_vendor/ply/ygen.py | 74 + bitbake/lib/bb/_vendor/progressbar.pyi | 1 + .../{ => bb/_vendor}/progressbar/LICENSE.txt | 0 .../{ => bb/_vendor}/progressbar/__init__.py | 4 +- .../{ => bb/_vendor}/progressbar/compat.py | 0 .../_vendor}/progressbar/progressbar.py | 37 +- .../{ => bb/_vendor}/progressbar/widgets.py | 14 +- bitbake/lib/bb/_vendor/simplediff.pyi | 1 + .../lib/{ => bb/_vendor}/simplediff/LICENSE | 0 .../{ => bb/_vendor}/simplediff/__init__.py | 2 +- bitbake/lib/bb/_vendor/tomli/LICENSE | 21 + bitbake/lib/bb/_vendor/tomli/__init__.py | 8 + bitbake/lib/bb/_vendor/tomli/_parser.py | 793 +++ bitbake/lib/bb/_vendor/tomli/_re.py | 119 + bitbake/lib/bb/_vendor/tomli/_types.py | 10 + bitbake/lib/bb/_vendor/tomli/py.typed | 1 + .../lib/bb/_vendor/typing_extensions.LICENSE | 279 ++ bitbake/lib/bb/_vendor/typing_extensions.py | 4317 +++++++++++++++++ bitbake/lib/bb/_vendor/typing_extensions.pyi | 1 + bitbake/lib/bb/acl.py | 2 - bitbake/lib/bb/asyncrpc/__init__.py | 3 +- bitbake/lib/bb/asyncrpc/client.py | 162 +- bitbake/lib/bb/asyncrpc/serv.py | 58 +- bitbake/lib/bb/asyncrpc/taskgroup.py | 52 + bitbake/lib/bb/build.py | 29 +- bitbake/lib/bb/cache.py | 37 +- bitbake/lib/bb/checksum.py | 25 + bitbake/lib/bb/codeparser.py | 102 +- bitbake/lib/bb/command.py | 50 +- bitbake/lib/bb/compress/lz4.py | 4 +- bitbake/lib/bb/cooker.py | 319 +- bitbake/lib/bb/cookerdata.py | 40 +- bitbake/lib/bb/daemonize.py | 3 + bitbake/lib/bb/data.py | 19 +- bitbake/lib/bb/data_smart.py | 157 +- bitbake/lib/bb/event.py | 29 +- bitbake/lib/bb/{fetch2 => fetch}/README | 0 bitbake/lib/bb/{fetch2 => fetch}/__init__.py | 526 +- bitbake/lib/bb/{fetch2 => fetch}/az.py | 23 +- bitbake/lib/bb/{fetch2 => fetch}/clearcase.py | 63 +- bitbake/lib/bb/fetch/crate.py | 198 + bitbake/lib/bb/{fetch2 => fetch}/gcp.py | 45 +- bitbake/lib/bb/{fetch2 => fetch}/git.py | 674 +-- bitbake/lib/bb/{fetch2 => fetch}/gitannex.py | 22 +- bitbake/lib/bb/{fetch2 => fetch}/gitsm.py | 144 +- bitbake/lib/bb/fetch/gomod.py | 270 ++ bitbake/lib/bb/{fetch2 => fetch}/hg.py | 65 +- bitbake/lib/bb/{fetch2 => fetch}/local.py | 15 +- bitbake/lib/bb/{fetch2 => fetch}/npm.py | 169 +- bitbake/lib/bb/{fetch2 => fetch}/npmsw.py | 120 +- bitbake/lib/bb/{fetch2 => fetch}/perforce.py | 46 +- bitbake/lib/bb/{fetch2 => fetch}/repo.py | 26 +- bitbake/lib/bb/{fetch2 => fetch}/s3.py | 21 +- bitbake/lib/bb/{fetch2 => fetch}/sftp.py | 23 +- bitbake/lib/bb/{fetch2 => fetch}/ssh.py | 33 +- bitbake/lib/bb/{fetch2 => fetch}/svn.py | 88 +- bitbake/lib/bb/{fetch2 => fetch}/wget.py | 316 +- bitbake/lib/bb/fetch2.py | 19 + bitbake/lib/bb/fetch2/bzr.py | 128 - bitbake/lib/bb/fetch2/crate.py | 141 - bitbake/lib/bb/fetch2/cvs.py | 157 - bitbake/lib/bb/fetch2/osc.py | 165 - bitbake/lib/bb/filter.py | 142 + bitbake/lib/bb/main.py | 8 +- bitbake/lib/bb/parse/__init__.py | 49 +- bitbake/lib/bb/parse/ast.py | 201 +- bitbake/lib/bb/parse/parse_py/BBHandler.py | 53 +- bitbake/lib/bb/parse/parse_py/ConfHandler.py | 24 +- bitbake/lib/bb/persist_data.py | 271 -- bitbake/lib/bb/pysh/pyshlex.py | 2 +- bitbake/lib/bb/pysh/pyshyacc.py | 2 +- bitbake/lib/bb/runqueue.py | 487 +- bitbake/lib/bb/server/process.py | 86 +- bitbake/lib/bb/server/xmlrpcserver.py | 11 +- bitbake/lib/bb/siggen.py | 102 +- bitbake/lib/bb/taskdata.py | 8 +- bitbake/lib/bb/tests/codeparser.py | 40 + bitbake/lib/bb/tests/compression.py | 4 +- bitbake/lib/bb/tests/cooker.py | 342 +- bitbake/lib/bb/tests/data.py | 207 +- .../software/miniupnp/download.php | 3528 ++++++++++++++ bitbake/lib/bb/tests/fetch.py | 2583 +++++++--- bitbake/lib/bb/tests/fetch_import.py | 86 + bitbake/lib/bb/tests/filter.py | 88 + .../bb/tests/parse-tests/classes/base.bbclass | 5 + .../classes/recipe-file-class.bbclass | 2 + .../bb/tests/parse-tests/conf/bitbake.conf | 15 + bitbake/lib/bb/tests/parse.py | 413 +- bitbake/lib/bb/tests/persist_data.py | 129 - .../tests/runqueue-tests/classes/base.bbclass | 4 +- .../lib/bb/tests/runqueue-tests/recipes/g1.bb | 2 + .../lib/bb/tests/runqueue-tests/recipes/h1.bb | 0 bitbake/lib/bb/tests/runqueue.py | 20 +- bitbake/lib/bb/tests/setup.py | 1089 +++++ bitbake/lib/bb/tests/siggen.py | 47 + bitbake/lib/bb/tests/support/httpserver.py | 4 +- bitbake/lib/bb/tests/utils.py | 19 + bitbake/lib/bb/tinfoil.py | 227 +- bitbake/lib/bb/ui/buildinfohelper.py | 5 +- bitbake/lib/bb/ui/knotty.py | 203 +- bitbake/lib/bb/ui/ncurses.py | 2 +- bitbake/lib/bb/ui/taskexp_ncurses.py | 3 + bitbake/lib/bb/ui/teamcity.py | 14 +- bitbake/lib/bb/ui/uihelper.py | 12 +- bitbake/lib/bb/utils.py | 867 +++- bitbake/lib/bblayers/action.py | 4 +- bitbake/lib/bblayers/layerindex.py | 12 +- bitbake/lib/bblayers/query.py | 37 +- bitbake/lib/bs4/AUTHORS.txt | 43 - bitbake/lib/bs4/NEWS.txt | 1066 ---- bitbake/lib/bs4/__init__.py | 468 -- bitbake/lib/bs4/builder/__init__.py | 324 -- bitbake/lib/bs4/builder/_html5lib.py | 338 -- bitbake/lib/bs4/builder/_htmlparser.py | 262 - bitbake/lib/bs4/builder/_lxml.py | 248 - bitbake/lib/bs4/dammit.py | 832 ---- bitbake/lib/bs4/diagnose.py | 216 - bitbake/lib/bs4/element.py | 1724 ------- bitbake/lib/bs4/testing.py | 686 --- bitbake/lib/bs4/tests/__init__.py | 1 - .../lib/bs4/tests/test_builder_registry.py | 147 - bitbake/lib/bs4/tests/test_docs.py | 32 - bitbake/lib/bs4/tests/test_html5lib.py | 98 - bitbake/lib/bs4/tests/test_htmlparser.py | 31 - bitbake/lib/bs4/tests/test_lxml.py | 70 - bitbake/lib/bs4/tests/test_soup.py | 479 -- bitbake/lib/bs4/tests/test_tree.py | 2004 -------- bitbake/lib/codegen.py | 564 --- bitbake/lib/hashserv/__init__.py | 15 +- bitbake/lib/hashserv/client.py | 423 +- bitbake/lib/hashserv/server.py | 257 +- bitbake/lib/hashserv/sqlite.py | 17 + bitbake/lib/hashserv/tests.py | 589 ++- bitbake/lib/layerindexlib/__init__.py | 6 +- bitbake/lib/layerindexlib/cooker.py | 16 +- bitbake/lib/layerindexlib/restapi.py | 2 +- bitbake/lib/ply/lex.py | 1054 ---- bitbake/lib/prserv/__init__.py | 97 +- bitbake/lib/prserv/client.py | 15 +- bitbake/lib/prserv/db.py | 452 +- bitbake/lib/prserv/serv.py | 140 +- bitbake/lib/prserv/tests.py | 388 ++ bitbake/lib/pyinotify.py | 2326 --------- bitbake/lib/toaster/bldcollector/admin.py | 8 +- bitbake/lib/toaster/bldcontrol/admin.py | 2 +- .../bldcontrol/localhostbecontroller.py | 175 +- .../toaster/orm/fixtures/check_fixtures.py | 76 + .../lib/toaster/orm/fixtures/gen_fixtures.py | 137 +- bitbake/lib/toaster/orm/fixtures/oe-core.xml | 72 +- bitbake/lib/toaster/orm/fixtures/poky.xml | 136 +- .../orm/management/commands/lsupdates.py | 29 +- bitbake/lib/toaster/orm/models.py | 3 +- .../tests/browser/selenium_helpers_base.py | 75 +- .../tests/browser/test_all_builds_page.py | 7 +- .../tests/browser/test_all_projects_page.py | 12 +- .../tests/browser/test_builddashboard_page.py | 2 +- .../tests/browser/test_landing_page.py | 14 +- .../tests/browser/test_layerdetails_page.py | 65 +- .../browser/test_new_custom_image_page.py | 2 +- .../tests/browser/test_new_project_page.py | 12 +- bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- .../tests/functional/functional_helpers.py | 88 + .../functional/test_create_new_project.py | 121 +- .../tests/functional/test_functional_basic.py | 200 +- .../tests/functional/test_project_config.py | 99 +- .../tests/functional/test_project_page.py | 417 +- .../test_project_page_tab_config.py | 313 +- bitbake/lib/toaster/tests/functional/utils.py | 21 +- .../tests/toaster-tests-requirements.txt | 4 +- bitbake/lib/toaster/tests/views/test_views.py | 2 +- .../toastergui/templates/layerdetails.html | 8 +- .../toastergui/templatetags/projecttags.py | 1 - bitbake/lib/toaster/toastergui/views.py | 18 +- bitbake/lib/toaster/toastermain/settings.py | 5 +- bitbake/packaging-pypi/bitbake-setup/BUILD.md | 67 + bitbake/packaging-pypi/bitbake-setup/LICENSE | 23 + .../packaging-pypi/bitbake-setup/README.md | 42 + .../bitbake-setup/package-bitbake-setup.py | 79 + .../bitbake-setup/pyproject.toml | 92 + bitbake/pyproject.toml | 16 + .../setup-schema/bitbake-setup.schema.json | 211 + bitbake/setup-schema/layers.schema.json | 107 + bitbake/toaster-requirements.txt | 2 +- bitbake/vendor.txt | 6 + bitbake/vendor/licenses/ply/LICENSE | 32 + ...0010-lib-bs4-Avoid-soupsieve-warning.patch | 40 + .../patches/bs4-remove-double-imports.patch | 23 + ...mplement-basic-task-progress-support.patch | 125 + ...ial-pass-of-SPDX-license-headers-to-.patch | 79 + ...bar-accept-value-over-initial-maxval.patch | 54 + ...self._fd_console-to-use-for-self._ha.patch | 53 + ...ke-bars-show-correctly-with-maxval-0.patch | 31 + ...ty-Allow-mixing-log-messages-and-pro.patch | 31 + 275 files changed, 41536 insertions(+), 21231 deletions(-) create mode 100644 bitbake/.b4-config create mode 100644 bitbake/AGENTS.md create mode 120000 bitbake/bin/bitbake-config-build create mode 100755 bitbake/bin/bitbake-setup delete mode 100755 bitbake/bin/git-make-shallow create mode 100755 bitbake/contrib/b4-wrapper-bitbake.py delete mode 100644 bitbake/contrib/vim/LICENSE.txt delete mode 100644 bitbake/contrib/vim/ftdetect/bitbake.vim delete mode 100644 bitbake/contrib/vim/ftplugin/bitbake.vim delete mode 100644 bitbake/contrib/vim/indent/bitbake.vim delete mode 100644 bitbake/contrib/vim/plugin/newbb.vim delete mode 100644 bitbake/contrib/vim/plugin/newbbappend.vim delete mode 100644 bitbake/contrib/vim/syntax/bitbake.vim create mode 100644 bitbake/default-registry/configurations/oe-nodistro-blacksail.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-master.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json create mode 100644 bitbake/default-registry/configurations/poky-blacksail.conf.json create mode 100644 bitbake/default-registry/configurations/poky-master.conf.json create mode 100644 bitbake/default-registry/configurations/poky-whinlatter.conf.json create mode 100644 bitbake/default-registry/configurations/poky-wrynose.conf.json create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst delete mode 100644 bitbake/doc/releases.rst create mode 100755 bitbake/doc/setversions.py delete mode 100644 bitbake/doc/sphinx-static/switchers.js create mode 100644 bitbake/doc/sphinx-static/switchers.js.in rename bitbake/lib/{bs4/COPYING.txt => bb/_vendor/beautifulsoup4.LICENSE} (80%) create mode 100644 bitbake/lib/bb/_vendor/bs4/__init__.py create mode 100644 bitbake/lib/bb/_vendor/bs4/_deprecation.py create mode 100644 bitbake/lib/bb/_vendor/bs4/_typing.py create mode 100644 bitbake/lib/bb/_vendor/bs4/_warnings.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/__init__.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_html5lib.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_htmlparser.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_lxml.py create mode 100644 bitbake/lib/bb/_vendor/bs4/css.py create mode 100644 bitbake/lib/bb/_vendor/bs4/dammit.py create mode 100644 bitbake/lib/bb/_vendor/bs4/diagnose.py create mode 100644 bitbake/lib/bb/_vendor/bs4/element.py create mode 100644 bitbake/lib/bb/_vendor/bs4/exceptions.py create mode 100644 bitbake/lib/bb/_vendor/bs4/filter.py create mode 100644 bitbake/lib/bb/_vendor/bs4/formatter.py create mode 100644 bitbake/lib/bb/_vendor/bs4/py.typed create mode 100644 bitbake/lib/bb/_vendor/ply.pyi rename bitbake/lib/{ => bb/_vendor}/ply/__init__.py (80%) create mode 100644 bitbake/lib/bb/_vendor/ply/cpp.py create mode 100644 bitbake/lib/bb/_vendor/ply/ctokens.py create mode 100644 bitbake/lib/bb/_vendor/ply/lex.py rename bitbake/lib/{ => bb/_vendor}/ply/yacc.py (60%) create mode 100644 bitbake/lib/bb/_vendor/ply/ygen.py create mode 100644 bitbake/lib/bb/_vendor/progressbar.pyi rename bitbake/lib/{ => bb/_vendor}/progressbar/LICENSE.txt (100%) rename bitbake/lib/{ => bb/_vendor}/progressbar/__init__.py (96%) rename bitbake/lib/{ => bb/_vendor}/progressbar/compat.py (100%) rename bitbake/lib/{ => bb/_vendor}/progressbar/progressbar.py (88%) rename bitbake/lib/{ => bb/_vendor}/progressbar/widgets.py (97%) create mode 100644 bitbake/lib/bb/_vendor/simplediff.pyi rename bitbake/lib/{ => bb/_vendor}/simplediff/LICENSE (100%) rename bitbake/lib/{ => bb/_vendor}/simplediff/__init__.py (99%) create mode 100644 bitbake/lib/bb/_vendor/tomli/LICENSE create mode 100644 bitbake/lib/bb/_vendor/tomli/__init__.py create mode 100644 bitbake/lib/bb/_vendor/tomli/_parser.py create mode 100644 bitbake/lib/bb/_vendor/tomli/_re.py create mode 100644 bitbake/lib/bb/_vendor/tomli/_types.py create mode 100644 bitbake/lib/bb/_vendor/tomli/py.typed create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.LICENSE create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.py create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.pyi create mode 100644 bitbake/lib/bb/asyncrpc/taskgroup.py rename bitbake/lib/bb/{fetch2 => fetch}/README (100%) rename bitbake/lib/bb/{fetch2 => fetch}/__init__.py (84%) rename bitbake/lib/bb/{fetch2 => fetch}/az.py (78%) rename bitbake/lib/bb/{fetch2 => fetch}/clearcase.py (84%) create mode 100644 bitbake/lib/bb/fetch/crate.py rename bitbake/lib/bb/{fetch2 => fetch}/gcp.py (62%) rename bitbake/lib/bb/{fetch2 => fetch}/git.py (52%) rename bitbake/lib/bb/{fetch2 => fetch}/gitannex.py (65%) rename bitbake/lib/bb/{fetch2 => fetch}/gitsm.py (63%) create mode 100644 bitbake/lib/bb/fetch/gomod.py rename bitbake/lib/bb/{fetch2 => fetch}/hg.py (72%) rename bitbake/lib/bb/{fetch2 => fetch}/local.py (83%) rename bitbake/lib/bb/{fetch2 => fetch}/npm.py (62%) rename bitbake/lib/bb/{fetch2 => fetch}/npmsw.py (70%) rename bitbake/lib/bb/{fetch2 => fetch}/perforce.py (87%) rename bitbake/lib/bb/{fetch2 => fetch}/repo.py (68%) rename bitbake/lib/bb/{fetch2 => fetch}/s3.py (87%) rename bitbake/lib/bb/{fetch2 => fetch}/sftp.py (85%) rename bitbake/lib/bb/{fetch2 => fetch}/ssh.py (85%) rename bitbake/lib/bb/{fetch2 => fetch}/svn.py (65%) rename bitbake/lib/bb/{fetch2 => fetch}/wget.py (68%) create mode 100644 bitbake/lib/bb/fetch2.py delete mode 100644 bitbake/lib/bb/fetch2/bzr.py delete mode 100644 bitbake/lib/bb/fetch2/crate.py delete mode 100644 bitbake/lib/bb/fetch2/cvs.py delete mode 100644 bitbake/lib/bb/fetch2/osc.py create mode 100644 bitbake/lib/bb/filter.py delete mode 100644 bitbake/lib/bb/persist_data.py create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php create mode 100644 bitbake/lib/bb/tests/fetch_import.py create mode 100644 bitbake/lib/bb/tests/filter.py create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/base.bbclass create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass create mode 100644 bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf delete mode 100644 bitbake/lib/bb/tests/persist_data.py create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb create mode 100644 bitbake/lib/bb/tests/setup.py delete mode 100644 bitbake/lib/bs4/AUTHORS.txt delete mode 100644 bitbake/lib/bs4/NEWS.txt delete mode 100644 bitbake/lib/bs4/__init__.py delete mode 100644 bitbake/lib/bs4/builder/__init__.py delete mode 100644 bitbake/lib/bs4/builder/_html5lib.py delete mode 100644 bitbake/lib/bs4/builder/_htmlparser.py delete mode 100644 bitbake/lib/bs4/builder/_lxml.py delete mode 100644 bitbake/lib/bs4/dammit.py delete mode 100644 bitbake/lib/bs4/diagnose.py delete mode 100644 bitbake/lib/bs4/element.py delete mode 100644 bitbake/lib/bs4/testing.py delete mode 100644 bitbake/lib/bs4/tests/__init__.py delete mode 100644 bitbake/lib/bs4/tests/test_builder_registry.py delete mode 100644 bitbake/lib/bs4/tests/test_docs.py delete mode 100644 bitbake/lib/bs4/tests/test_html5lib.py delete mode 100644 bitbake/lib/bs4/tests/test_htmlparser.py delete mode 100644 bitbake/lib/bs4/tests/test_lxml.py delete mode 100644 bitbake/lib/bs4/tests/test_soup.py delete mode 100644 bitbake/lib/bs4/tests/test_tree.py delete mode 100644 bitbake/lib/codegen.py delete mode 100644 bitbake/lib/ply/lex.py create mode 100644 bitbake/lib/prserv/tests.py delete mode 100644 bitbake/lib/pyinotify.py create mode 100755 bitbake/lib/toaster/orm/fixtures/check_fixtures.py create mode 100644 bitbake/packaging-pypi/bitbake-setup/BUILD.md create mode 100644 bitbake/packaging-pypi/bitbake-setup/LICENSE create mode 100644 bitbake/packaging-pypi/bitbake-setup/README.md create mode 100755 bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py create mode 100644 bitbake/packaging-pypi/bitbake-setup/pyproject.toml create mode 100644 bitbake/pyproject.toml create mode 100644 bitbake/setup-schema/bitbake-setup.schema.json create mode 100644 bitbake/setup-schema/layers.schema.json create mode 100644 bitbake/vendor.txt create mode 100644 bitbake/vendor/licenses/ply/LICENSE create mode 100644 bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch create mode 100644 bitbake/vendor/patches/bs4-remove-double-imports.patch create mode 100644 bitbake/vendor/patches/progressbar-0001-lib-implement-basic-task-progress-support.patch create mode 100644 bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pass-of-SPDX-license-headers-to-.patch create mode 100644 bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-accept-value-over-initial-maxval.patch create mode 100644 bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._fd_console-to-use-for-self._ha.patch create mode 100644 bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-show-correctly-with-maxval-0.patch create mode 100644 bitbake/vendor/patches/progressbar-0006-progressbar-knotty-Allow-mixing-log-messages-and-pro.patch diff --git a/.gitignore b/.gitignore index b5f6149e..f397ff51 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,4 @@ __pycache__ .config.yaml* .kas_shell_history .isar_test_shell_history -build* +/build* diff --git a/bitbake/.b4-config b/bitbake/.b4-config new file mode 100644 index 00000000..047f0b94 --- /dev/null +++ b/bitbake/.b4-config @@ -0,0 +1,4 @@ +[b4] + send-series-to = bitbake-devel@lists.openembedded.org + send-auto-cc-cmd = ./contrib/b4-wrapper-bitbake.py send-auto-cc-cmd + prep-pre-flight-checks = disable-needs-checking diff --git a/bitbake/.gitignore b/bitbake/.gitignore index fbf9eb20..23cb9688 100644 --- a/bitbake/.gitignore +++ b/bitbake/.gitignore @@ -3,6 +3,7 @@ *~ pyshtables.py /doc/manual/html/ +**/bitbake-builds/ /build/ /bin/bitbakec *.swp @@ -16,3 +17,4 @@ lib/toaster/contrib/tts/backlog.txt lib/toaster/contrib/tts/log/* lib/toaster/contrib/tts/.cache/* lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log +packaging-workspace/ diff --git a/bitbake/AGENTS.md b/bitbake/AGENTS.md new file mode 100644 index 00000000..833eca7f --- /dev/null +++ b/bitbake/AGENTS.md @@ -0,0 +1,55 @@ +# AGENTS.md — BitBake + +Guidance for AI agents working in the `bitbake` repository. + +## Making changes + +- No web PR workflow. The project uses **patches on mailing lists** + (bitbake-devel@lists.openembedded.org), tracked via patchwork. Don't assume + GitHub PR conventions apply. +- One Git commit per logical change (e.g., one commit per recipe added/upgraded). +- Commit message format: + - Summary line prefixed with the recipe name or short file path, + e.g. `openssl: upgrade 3.2.0 -> 3.2.1` or `classes/rootfs-postcommands: fix typo`. + Check `git log --oneline ` for the prefix convention already in use. + - Body explains what/why/how, and how it was tested. + - Prefer explaining *why* a change was made over restating what's + already visible in the diff, unless the diff is long/hard to follow + or the reason for the change isn't known — in those cases, summarize + what changed instead. + - Be succinct: use as few sentences and lines as possible while still + covering what/why/how. Cut restatements, filler, and any detail a + reviewer wouldn't need (e.g. exact test-run counts, step-by-step + narration). One short paragraph is usually enough; only add a second + if a distinct point (e.g. testing) needs it. + - The most relevant information about the change should be presented first. + - Reference bugs as `[YOCTO #1234]` when applicable. + - Use `Reported-by`, `Suggested-by`, `Tested-by`, `Reviewed-by`, `Cc` tags + where relevant. + +### AI-generated code + +- **Mandatory**: any AI-generated code/commit must add an `AI-Generated:` + tag line in the commit message, placed *before* `Signed-off-by` (if present), + naming the tool used, e.g.: + + ``` + component: Add the ability to ... + + AI-Generated: Uses GitHub Copilot + ``` + +- Also add a code comment or other in-patch indication that the code is + AI-generated (recommended, not just the commit tag). +- **Mandatory** Do not automatically add a `Signed-off-by`. This tag must + always be written by the human contributor, never by the AI tool — the + DCO/Developer's Statement of Origin still applies in full. +- Contributor is responsible for confirming the AI tool's terms don't + conflict with the project's open source license/IP policy, and that + any third-party copyrighted material in the AI output is properly + licensed before contributing it. See the Linux Foundation generative AI + guidance: https://www.linuxfoundation.org/legal/generative-ai. +- `Co-authored-by` should be omitted in favor of the `AI-generated` tag. + +When in doubt about a convention, grep `git log --oneline ` in the +target repo for precedent before inventing a new style. diff --git a/bitbake/README b/bitbake/README index e9f4c858..39e4a978 100644 --- a/bitbake/README +++ b/bitbake/README @@ -13,7 +13,7 @@ Bitbake plain documentation can be found under the doc directory or its integrat html version at the Yocto Project website: https://docs.yoctoproject.org -Bitbake requires Python version 3.8 or newer. +Bitbake requires Python version 3.9 or newer. Contributing ------------ diff --git a/bitbake/TODO b/bitbake/TODO index 54d091fb..4e0bd9a7 100644 --- a/bitbake/TODO +++ b/bitbake/TODO @@ -1,36 +1,31 @@ -- Reimplement the interactive mode as a proper ui -- Continue dropping fatal/SystemExit/sys.exit usage in favor of raising - appropriate exceptions -- Continue pylint / pyflakes / pychecker / pep8 fixups + +- Interfaces + - Add an interactive mode to knotty (similar to the interactive mode we dropped) + - Fix up or drop the ncurses interface? Currently too limited to be useful. + - Concerns about whether teamcity interface works. No testing. + +- Improve exception handling: + - dropping fatal/SystemExit/sys.exit usage in favor of raising + appropriate exceptions + - Audit bb.fatal usage - these should all be able to be replaced with + exceptions + +- Continue pylint / pyflakes / pychecker / pep8 fixups - focus on real issues + or performance issues, not cosmetic ones. + - Drop os.system usage in favor of direct subprocess usage or a subprocess wrapper -- Kill the execution of 'tee' for the task log file in build.py -- Fix up the exception handling - - - Kill exec_task's catch of FuncFailed, instead catch it in the other - callers of exec_task/exec_func - - What exactly is the purpose of the "EventException"? I can see using an - exception like that, *perhaps*, to abstract away exceptions raised by - event handlers, but it has no place in bb.build.exec_task - BUG: if you chmod 000 local.conf, it silently doesn't parse it, when it should really fail, so the user can fix the problem. -- Audit bb.fatal usage - these should all be able to be replaced with - exceptions -- Figure out how to handle the ncurses UI. Should some of our logging - formatting stuff be made common to all of bitbake, or perhaps all UIs via - the UIHelper? Long term, high impact: - - Change override application to actually *move* it over -- so the original - override specific version of the variable goes away, rather than sticking - around as a duplicate. - - Change the behavior when a variable is referenced and is unset. Today, it - evaluates to ${FOO} and then shell has a chance to expand it, but this is - far from ideal. We had considered evaluating it to the empty string, but - that has other potential problems. Frans Meulenbroeks has proposed just + - Consider changing the behavior when a variable is referenced and is unset + Today, it evaluates to ${FOO} and then shell has a chance to expand it, but + this is far from ideal. We had considered evaluating it to the empty string, + but that has other potential problems. Frans Meulenbroeks has proposed just erroring when this occurs, as we can always define default values for the variables in bitbake.conf. This seems reasonable. My only concern with that is the case where you want to reference a shell variable with odd @@ -38,25 +33,3 @@ Long term, high impact: expansion rather than normal $. To handle that case, we'd really need a way to escape / disable bitbake variable expansion, \${} perhaps. -Uncertain: - - - Leverage the python 2.6 multiprocessing module - - - Worker processes for bb.cooker - - Server / UI processes - - - Create a bitbake configuration class which is utilized by the library, not - just bin/bitbake. This class should be responsible for extracting - configuration parameters from the metadata for bitbake internal use, as well - as pulling specific items like BBDEBUG, and importing settings from an - optparse options object. - - - Python version bits - - - Utilize the new string formatting where appropriate - - Do we need to take into account the bytes literals changes? - - Do we have any file-like objects that would benefit from using the "io" - module? - - Do we want to leverage the abstract base classes in collections? - - Aside: Set methods now accept multiple iterables - diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake index f494eaa1..71bc3791 100755 --- a/bitbake/bin/bitbake +++ b/bitbake/bin/bitbake @@ -27,11 +27,11 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException bb.utils.check_system_locale() -__version__ = "2.8.0" +__version__ = "2.20.0" if __name__ == "__main__": if __version__ != bb.__version__: - sys.exit("Bitbake core version and program version mismatch!") + sys.exit("Bitbake core library version and program version mismatch!") try: sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), cookerdata.CookerConfiguration())) diff --git a/bitbake/bin/bitbake-config-build b/bitbake/bin/bitbake-config-build new file mode 120000 index 00000000..11e6df80 --- /dev/null +++ b/bitbake/bin/bitbake-config-build @@ -0,0 +1 @@ +bitbake-layers \ No newline at end of file diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs index 8202c786..99660d5c 100755 --- a/bitbake/bin/bitbake-diffsigs +++ b/bitbake/bin/bitbake-diffsigs @@ -22,6 +22,13 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), ' import bb.tinfoil import bb.siggen import bb.msg +from bb import __version__ as libbb_version + +__version__ = "2.20.0" + +if __version__ != libbb_version: + sys.exit("Bitbake core library version and program version mismatch!") + myname = os.path.basename(sys.argv[0]) logger = bb.msg.logger_create(myname) @@ -72,16 +79,17 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): elif sig2 not in sigfiles: logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2)) sys.exit(1) + + latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']] else: sigfiles = find_siginfo(bbhandler, pn, taskname) latestsigs = sorted(sigfiles.keys(), key=lambda h: sigfiles[h]['time'])[-2:] if not latestsigs: logger.error('No sigdata files found matching %s %s' % (pn, taskname)) sys.exit(1) - sig1 = latestsigs[0] - sig2 = latestsigs[1] - - latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']] + latestfiles = [sigfiles[latestsigs[0]]['path']] + if len(latestsigs) > 1: + latestfiles.append(sigfiles[latestsigs[1]]['path']) return latestfiles diff --git a/bitbake/bin/bitbake-getvar b/bitbake/bin/bitbake-getvar index 8901f99a..7f4d3745 100755 --- a/bitbake/bin/bitbake-getvar +++ b/bitbake/bin/bitbake-getvar @@ -10,15 +10,23 @@ import io import os import sys import warnings +import logging warnings.simplefilter("default") bindir = os.path.dirname(__file__) topdir = os.path.dirname(bindir) sys.path[0:0] = [os.path.join(topdir, 'lib')] +import bb.providers import bb.tinfoil +from bb import __version__ as libbb_version + +__version__ = "2.20.0" if __name__ == "__main__": + if __version__ != libbb_version: + sys.exit("Bitbake core library version and program version mismatch!") + parser = argparse.ArgumentParser(description="Bitbake Query Variable") parser.add_argument("variable", help="variable name to query") parser.add_argument("-r", "--recipe", help="Recipe name to query", default=None, required=False) @@ -37,13 +45,22 @@ if __name__ == "__main__": sys.exit("--flag only makes sense with --value") quiet = args.quiet or args.value + if quiet: + logger = logging.getLogger("BitBake") + logger.setLevel(logging.WARNING) + with bb.tinfoil.Tinfoil(tracking=True, setup_logging=not quiet) as tinfoil: if args.recipe: tinfoil.prepare(quiet=3 if quiet else 2) - d = tinfoil.parse_recipe(args.recipe) + try: + d = tinfoil.parse_recipe(args.recipe) + except bb.providers.NoProvider as e: + sys.exit(str(e)) else: tinfoil.prepare(quiet=2, config_only=True) - d = tinfoil.config_data + # Expand keys and run anonymous functions to get identical result to + # "bitbake -e" + d = tinfoil.finalizeData() value = None if args.flag: diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient index 610787ed..ce5fd173 100755 --- a/bitbake/bin/bitbake-hashclient +++ b/bitbake/bin/bitbake-hashclient @@ -16,8 +16,12 @@ import time import warnings import netrc import json +import statistics +import textwrap warnings.simplefilter("default") +__version__ = "2.20.0" + try: import tqdm ProgressBar = tqdm.tqdm @@ -39,6 +43,12 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import hashserv import bb.asyncrpc +from bb import __version__ as libbb_version + +if __version__ != libbb_version: + sys.exit("Bitbake core library version and program version mismatch!") +if __version__ != hashserv.__version__: + sys.exit("Bitbake hashserv library version and program version mismatch!") DEFAULT_ADDRESS = 'unix://./hashserve.sock' METHOD = 'stress.test.method' @@ -81,6 +91,7 @@ def main(): nonlocal found_hashes nonlocal missed_hashes nonlocal max_time + nonlocal times with hashserv.create_client(args.address) as client: for i in range(args.requests): @@ -98,29 +109,41 @@ def main(): else: missed_hashes += 1 - max_time = max(elapsed, max_time) + times.append(elapsed) pbar.update() max_time = 0 found_hashes = 0 missed_hashes = 0 lock = threading.Lock() - total_requests = args.clients * args.requests + times = [] start_time = time.perf_counter() - with ProgressBar(total=total_requests) as pbar: + with ProgressBar(total=args.clients * args.requests) as pbar: threads = [threading.Thread(target=thread_main, args=(pbar, lock), daemon=False) for _ in range(args.clients)] for t in threads: t.start() for t in threads: t.join() + total_elapsed = time.perf_counter() - start_time - elapsed = time.perf_counter() - start_time with lock: - print("%d requests in %.1fs. %.1f requests per second" % (total_requests, elapsed, total_requests / elapsed)) - print("Average request time %.8fs" % (elapsed / total_requests)) - print("Max request time was %.8fs" % max_time) - print("Found %d hashes, missed %d" % (found_hashes, missed_hashes)) + mean = statistics.mean(times) + median = statistics.median(times) + stddev = statistics.pstdev(times) + + print(f"Number of clients: {args.clients}") + print(f"Requests per client: {args.requests}") + print(f"Number of requests: {len(times)}") + print(f"Total elapsed time: {total_elapsed:.3f}s") + print(f"Total request rate: {len(times)/total_elapsed:.3f} req/s") + print(f"Average request time: {mean:.3f}s") + print(f"Median request time: {median:.3f}s") + print(f"Request time std dev: {stddev:.3f}s") + print(f"Maximum request time: {max(times):.3f}s") + print(f"Minimum request time: {min(times):.3f}s") + print(f"Hashes found: {found_hashes}") + print(f"Hashes missed: {missed_hashes}") if args.report: with ProgressBar(total=args.requests) as pbar: @@ -212,6 +235,27 @@ def main(): print("New hashes marked: %d" % result["count"]) return 0 + def handle_gc_mark_stream(args, client): + stdin = (l.strip() for l in sys.stdin) + marked_hashes = 0 + + try: + result = client.gc_mark_batch(args.mark, stdin) + marked_hashes = result["count"] + except ConnectionError: + logger.warning( + "Server doesn't seem to support `gc-mark-stream`. Sending " + "hashes sequentially using `gc-mark` API." + ) + for line in stdin: + pairs = line.split() + condition = dict(zip(pairs[::2], pairs[1::2])) + result = client.gc_mark(args.mark, condition) + marked_hashes += result["count"] + + print("New hashes marked: %d" % marked_hashes) + return 0 + def handle_gc_sweep(args, client): result = client.gc_sweep(args.mark) print("Removed %d rows" % result["count"]) @@ -225,7 +269,45 @@ def main(): print("true" if result else "false") return 0 - parser = argparse.ArgumentParser(description='Hash Equivalence Client') + def handle_ping(args, client): + times = [] + for i in range(1, args.count + 1): + if not args.quiet: + print(f"Ping {i} of {args.count}... ", end="") + start_time = time.perf_counter() + client.ping() + elapsed = time.perf_counter() - start_time + times.append(elapsed) + if not args.quiet: + print(f"{elapsed:.3f}s") + + mean = statistics.mean(times) + median = statistics.median(times) + std_dev = statistics.pstdev(times) + + if not args.quiet: + print("------------------------") + print(f"Number of pings: {len(times)}") + print(f"Average round trip time: {mean:.3f}s") + print(f"Median round trip time: {median:.3f}s") + print(f"Round trip time std dev: {std_dev:.3f}s") + print(f"Min time is: {min(times):.3f}s") + print(f"Max time is: {max(times):.3f}s") + return 0 + + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description='Hash Equivalence Client', + epilog=textwrap.dedent( + """ + Possible ADDRESS options are: + unix://PATH Connect to UNIX domain socket at PATH + ws://HOST[:PORT] Connect to websocket at HOST:PORT (default port is 80) + wss://HOST[:PORT] Connect to secure websocket at HOST:PORT (default port is 443) + HOST:PORT Connect to TCP server at HOST:PORT + """ + ), + ) parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")') parser.add_argument('--log', default='WARNING', help='Set logging level') parser.add_argument('--login', '-l', metavar="USERNAME", help="Authenticate as USERNAME") @@ -313,6 +395,16 @@ def main(): help="Keep entries in table where KEY == VALUE") gc_mark_parser.set_defaults(func=handle_gc_mark) + gc_mark_parser_stream = subparsers.add_parser( + 'gc-mark-stream', + help=( + "Mark multiple hashes to be retained for garbage collection. Input should be provided via stdin, " + "with each line formatted as key-value pairs separated by spaces, for example 'column1 foo column2 bar'." + ) + ) + gc_mark_parser_stream.add_argument("mark", help="Mark for this garbage collection operation") + gc_mark_parser_stream.set_defaults(func=handle_gc_mark_stream) + gc_sweep_parser = subparsers.add_parser('gc-sweep', help="Perform garbage collection and delete any entries that are not marked") gc_sweep_parser.add_argument("mark", help="Mark for this garbage collection operation") gc_sweep_parser.set_defaults(func=handle_gc_sweep) @@ -322,6 +414,11 @@ def main(): unihash_exists_parser.add_argument("unihash", help="Unihash to check") unihash_exists_parser.set_defaults(func=handle_unihash_exists) + ping_parser = subparsers.add_parser('ping', help="Ping server") + ping_parser.add_argument("-n", "--count", type=int, help="Number of pings. Default is %(default)s", default=10) + ping_parser.add_argument("-q", "--quiet", action="store_true", help="Don't print each ping; only print results") + ping_parser.set_defaults(func=handle_ping) + args = parser.parse_args() logger = logging.getLogger('hashserv') diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv index 4bfb7abf..3e68b6a2 100755 --- a/bitbake/bin/bitbake-hashserv +++ b/bitbake/bin/bitbake-hashserv @@ -12,6 +12,8 @@ import argparse import sqlite3 import warnings +__version__ = "2.20.0" + warnings.simplefilter("default") sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib")) @@ -19,14 +21,15 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib import hashserv from hashserv.server import DEFAULT_ANON_PERMS -VERSION = "1.0.0" +if __version__ != hashserv.__version__: + sys.exit("Bitbake library hashserv version and program version mismatch!") DEFAULT_BIND = "unix://./hashserve.sock" def main(): parser = argparse.ArgumentParser( - description="Hash Equivalence Reference Server. Version=%s" % VERSION, + description="Hash Equivalence Reference Server. Version=%s" % __version__, formatter_class=argparse.RawTextHelpFormatter, epilog=""" The bind address may take one of the following formats: @@ -125,6 +128,11 @@ The following permissions are supported by the server: default=os.environ.get("HASHSERVER_ADMIN_PASSWORD", None), help="Create default admin user with password ADMIN_PASSWORD ($HASHSERVER_ADMIN_PASSWORD)", ) + parser.add_argument( + "--reuseport", + action="store_true", + help="Enable SO_REUSEPORT, allowing multiple servers to bind to the same port for load balancing", + ) args = parser.parse_args() @@ -132,7 +140,9 @@ The following permissions are supported by the server: level = getattr(logging, args.log.upper(), None) if not isinstance(level, int): - raise ValueError("Invalid log level: %s (Try ERROR/WARNING/INFO/DEBUG)" % args.log) + raise ValueError( + "Invalid log level: %s (Try ERROR/WARNING/INFO/DEBUG)" % args.log + ) logger.setLevel(level) console = logging.StreamHandler() @@ -155,6 +165,7 @@ The following permissions are supported by the server: anon_perms=anon_perms, admin_username=args.admin_user, admin_password=args.admin_password, + reuseport=args.reuseport, ) server.serve_forever() return 0 diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers index d4b1d1aa..c1eafb22 100755 --- a/bitbake/bin/bitbake-layers +++ b/bitbake/bin/bitbake-layers @@ -17,25 +17,38 @@ import argparse import warnings warnings.simplefilter("default") +__version__ = "2.20.0" + bindir = os.path.dirname(__file__) +toolname = os.path.basename(__file__).split(".")[0] topdir = os.path.dirname(bindir) sys.path[0:0] = [os.path.join(topdir, 'lib')] import bb.tinfoil +import bb.cooker import bb.msg +from bb import __version__ as libbb_version + +if __version__ != libbb_version: + sys.exit("Bitbake library version and program version mismatch!") -logger = bb.msg.logger_create('bitbake-layers', sys.stdout) +logger = bb.msg.logger_create(toolname, sys.stdout) def main(): parser = argparse.ArgumentParser( - description="BitBake layers utility", + description={"bitbake-layers":"BitBake layers utility", "bitbake-config-build":"BitBake build configuration utility"}[toolname], epilog="Use %(prog)s --help to get help on a specific command", add_help=False) parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') - parser.add_argument('-F', '--force', help='Force add without recipe parse verification', action='store_true') + if toolname == "bitbake-layers": + parser.add_argument('-F', '--force', help='Forced execution: can be specified multiple times. -F will force add without recipe parse verification and -FF will additionally force the run without layer parsing.', action='count') parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') + # Set the default for --force here rather than above so that it is always + # defined even if the option is not available. + parser.set_defaults(force=0) + global_args, unparsed_args = parser.parse_known_args() # Help is added here rather than via add_help=True, as we don't want it to @@ -57,18 +70,27 @@ def main(): level=logger.getEffectiveLevel()) plugins = [] - tinfoil = bb.tinfoil.Tinfoil(tracking=True) - tinfoil.logger.setLevel(logger.getEffectiveLevel()) - try: - tinfoil.prepare(True) - for path in ([topdir] + - tinfoil.config_data.getVar('BBPATH').split(':')): - pluginpath = os.path.join(path, 'lib', 'bblayers') + with bb.tinfoil.Tinfoil(tracking=True) as tinfoil: + tinfoil.logger.setLevel(logger.getEffectiveLevel()) + + if global_args.force > 1: + bbpaths = [] + else: + if toolname == "bitbake-config-build": + extra_features = [bb.cooker.CookerFeatures.SKIP_FRAGMENTS] + else: + extra_features = None + tinfoil.prepare(True, extra_features=extra_features) + bbpaths = tinfoil.config_data.getVar('BBPATH').split(':') + + for path in ([topdir] + bbpaths): + pluginbasepath = {"bitbake-layers":'bblayers', 'bitbake-config-build':'bbconfigbuild'}[toolname] + pluginpath = os.path.join(path, 'lib', pluginbasepath) bb.utils.load_plugins(logger, plugins, pluginpath) registered = False for plugin in plugins: - if hasattr(plugin, 'tinfoil_init'): + if hasattr(plugin, 'tinfoil_init') and global_args.force <= 1: plugin.tinfoil_init(tinfoil) if hasattr(plugin, 'register_commands'): registered = True @@ -86,8 +108,6 @@ def main(): tinfoil.config_data.enableTracking() return args.func(args) - finally: - tinfoil.shutdown() if __name__ == "__main__": diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv index ad0a0694..8ba246c0 100755 --- a/bitbake/bin/bitbake-prserv +++ b/bitbake/bin/bitbake-prserv @@ -11,19 +11,30 @@ import argparse import warnings warnings.simplefilter("default") +__version__ = "2.20.0" + sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib")) import prserv import prserv.serv - -VERSION = "1.1.0" + +if __name__ == "__main__": + if __version__ != prserv.__version__: + sys.exit("Bitbake prserv library version and program version mismatch!") PRHOST_DEFAULT="0.0.0.0" PRPORT_DEFAULT=8585 +def init_logger(logfile, loglevel): + numeric_level = getattr(logging, loglevel.upper(), None) + if not isinstance(numeric_level, int): + raise ValueError("Invalid log level: %s" % loglevel) + FORMAT = "%(asctime)-15s %(message)s" + logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT) + def main(): parser = argparse.ArgumentParser( - description="BitBake PR Server. Version=%s" % VERSION, + description="BitBake PR Server. Version=%s" % __version__, formatter_class=argparse.RawTextHelpFormatter) parser.add_argument( @@ -70,12 +81,25 @@ def main(): action="store_true", help="open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=os.environ.get("PRSERV_UPSTREAM", None), + help="Upstream PR service (host:port)", + ) args = parser.parse_args() - prserv.init_logger(os.path.abspath(args.log), args.loglevel) + init_logger(os.path.abspath(args.log), args.loglevel) if args.start: - ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + ret=prserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=prserv.serv.stop_daemon(args.host, args.port) else: diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest index f25f23b1..855abb1f 100755 --- a/bitbake/bin/bitbake-selftest +++ b/bitbake/bin/bitbake-selftest @@ -11,14 +11,20 @@ import warnings warnings.simplefilter("default") sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +__version__ = "2.20.0" + import unittest try: import bb import hashserv + import prserv import layerindexlib except RuntimeError as exc: sys.exit(str(exc)) +if __version__ != bb.__version__: + sys.exit("Bitbake core library version and program version mismatch!") + tests = ["bb.tests.codeparser", "bb.tests.color", "bb.tests.cooker", @@ -26,13 +32,16 @@ tests = ["bb.tests.codeparser", "bb.tests.data", "bb.tests.event", "bb.tests.fetch", + "bb.tests.fetch_import", "bb.tests.parse", - "bb.tests.persist_data", "bb.tests.runqueue", + "bb.tests.setup", "bb.tests.siggen", "bb.tests.utils", "bb.tests.compression", + "bb.tests.filter", "hashserv.tests", + "prserv.tests", "layerindexlib.tests.layerindexobj", "layerindexlib.tests.restapi", "layerindexlib.tests.cooker"] @@ -64,6 +73,8 @@ ENV_HELP = """\ Environment variables: BB_SKIP_NETTESTS set to 'yes' in order to skip tests using network connection + BB_SKIP_PYPI_TESTS set to 'no' to run PyPI packaging tests + (default: yes/skip) BB_TMPDIR_NOCLEAN set to 'yes' to preserve test tmp directories """ diff --git a/bitbake/bin/bitbake-server b/bitbake/bin/bitbake-server index 454a3919..6157fb33 100755 --- a/bitbake/bin/bitbake-server +++ b/bitbake/bin/bitbake-server @@ -12,6 +12,8 @@ warnings.simplefilter("default") import logging sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) +__version__ = "2.20.0" + import bb bb.utils.check_system_locale() @@ -23,13 +25,17 @@ if len(sys.argv) != 11 or not sys.argv[1].startswith("decafbad"): import bb.server.process +if __name__ == "__main__": + if __version__ != bb.__version__: + sys.exit("Bitbake core library version and program version mismatch!") + lockfd = int(sys.argv[2]) readypipeinfd = int(sys.argv[3]) logfile = sys.argv[4] lockname = sys.argv[5] sockname = sys.argv[6] timeout = float(sys.argv[7]) -profile = bool(int(sys.argv[8])) +profile = sys.argv[8] xmlrpcinterface = (sys.argv[9], int(sys.argv[10])) if xmlrpcinterface[0] == "None": xmlrpcinterface = (None, xmlrpcinterface[1]) @@ -38,9 +44,9 @@ if xmlrpcinterface[0] == "None": with open('/dev/null', 'r') as si: os.dup2(si.fileno(), sys.stdin.fileno()) -so = open(logfile, 'a+') -os.dup2(so.fileno(), sys.stdout.fileno()) -os.dup2(so.fileno(), sys.stderr.fileno()) +with open(logfile, 'a+') as so: + os.dup2(so.fileno(), sys.stdout.fileno()) + os.dup2(so.fileno(), sys.stderr.fileno()) # Have stdout and stderr be the same so log output matches chronologically # and there aren't two seperate buffers diff --git a/bitbake/bin/bitbake-setup b/bitbake/bin/bitbake-setup new file mode 100755 index 00000000..e50117ea --- /dev/null +++ b/bitbake/bin/bitbake-setup @@ -0,0 +1,1481 @@ +#!/usr/bin/env python3 + +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import argparse +import configparser +import copy +import datetime +import functools +import glob +import json +import logging +import os +import shutil +import signal +import string +import subprocess +import sys +import textwrap +import time + +__version__ = "2.20.0" + +bindir = os.path.abspath(os.path.dirname(__file__)) +sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')] + +import bb.msg # noqa: E402 +import bb.fetch # noqa: E402 +import bb.process # noqa: E402 +from bb import __version__ as libbb_version # noqa: E402 + +if __name__ == "__main__": + if __version__ != libbb_version: + sys.exit("Bitbake core library version and program version mismatch!") + +logger = bb.msg.logger_create('bitbake-setup', sys.stdout) + +# These settings can only be set in the global context +GLOBAL_ONLY_SETTINGS = ( + "top-dir-prefix", + "top-dir-name", +) + +def color_enabled() -> bool: + """ + Our logger has a BBLogFormatter formatter which holds whether color is + enabled or not. Return this value. + """ + return logger.handlers[0].formatter.color_enabled + +def get_diff_color_param() -> str: + return "--color=always" if color_enabled() else "--color=never" + +def print_configs(prompt: str, choices: list[str], descriptions: list[str] = []): + """ + Helper function to print a list of choices and align the output. + Each option name is made bold to stand out, unless color is not enabled in + our logger. + """ + if not prompt.endswith(':'): + prompt += ":" + logger.plain(prompt) + + if not descriptions: + descriptions = ["" for _ in choices] + + # maximum size of all choices, for alignment + cmax = max([len(c) for c in choices]) + 1 + + for n, c in enumerate(choices): + msg = f"{n + 1}. " + if color_enabled(): + # make it bold + msg += "\033[1m" + msg += f"{c:<{cmax}}" + if color_enabled(): + msg += "\033[0m" + msg += f" {descriptions[n]}" + logger.plain(msg) + +# If bitbake is from a release tarball or somewhere like pypi where +# updates may not be straightforward, prefer to use the git repo as the +# default registry +def get_default_registry(): + internal_registry = os.path.normpath(os.path.dirname(__file__) + "/../default-registry") + git_registry = "git://git.openembedded.org/bitbake;protocol=https;branch=master;rev=master" + if os.path.exists(os.path.dirname(__file__) + "/../.git"): + return internal_registry + else: + return git_registry + +def cache_dir(top_dir): + return os.path.join(top_dir, '.bitbake-setup-cache') + +def init_bb_cache(top_dir, settings, args): + dldir = settings["default"]["dl-dir"] + bb_cachedir = os.path.join(cache_dir(top_dir), 'bitbake-cache') + os.makedirs(bb_cachedir, exist_ok=True) + + d = bb.data.init() + d.setVar("DL_DIR", dldir) + d.setVar("BB_CACHEDIR", bb_cachedir) + d.setVar("__BBSRCREV_SEEN", "1") + if args.no_network: + d.setVar("BB_SRCREV_POLICY", "cache") + bb.fetch.fetcher_init(d) + return d + +def save_bb_cache(): + bb.fetch.fetcher_parse_save() + bb.fetch.fetcher_parse_done() + +def get_config_name(config): + suffix = '.conf.json' + config_file = os.path.basename(config) + if config_file.endswith(suffix): + return config_file[:-len(suffix)] + else: + raise Exception("Config file {} does not end with {}, please rename the file.".format(config, suffix)) + +def write_upstream_config(config_dir, config_data): + with open(os.path.join(config_dir, "config-upstream.json"),'w') as s: + json.dump(config_data, s, sort_keys=True, indent=4) + +def write_sources_fixed_revisions(config_dir, layer_dir, config_data): + json_path = os.path.join(config_dir, "sources-fixed-revisions.json") + json_link = os.path.join(layer_dir, "sources-fixed-revisions.json") + sources = {} + sources['sources'] = config_data + with open(os.path.join(config_dir, "sources-fixed-revisions.json"),'w') as s: + json.dump(sources, s, sort_keys=True, indent=4) + if not os.path.lexists(json_link): + os.symlink(os.path.relpath(json_path ,layer_dir), json_link) + +def commit_config(config_dir): + bb.process.run(['git', '-C', config_dir, 'add', '.']) + bb.process.run(['git', '-C', config_dir, 'commit', '--allow-empty', '--no-verify', '-a', '-m', 'Configuration at ' + time.asctime()]) + +def _write_layer_list(dest, repodirs): + layers = [] + for r in repodirs: + for root, _, files in os.walk(os.path.join(dest,r)): + if os.path.basename(root) == 'conf' and 'layer.conf' in files: + layers.append(os.path.relpath(os.path.dirname(root), dest)) + layers_f = os.path.join(dest, ".oe-layers.json") + with open(layers_f, 'w') as f: + json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4) + +def add_unique_timestamp_to_path(path): + timestamp = time.strftime("%Y%m%d%H%M%S") + path_unique = "{}.{}".format(path, timestamp) + if os.path.exists(path_unique): + import itertools + for i in itertools.count(start=1): + path_unique = "{}.{}.{}".format(path, timestamp, i) + if not os.path.exists(path_unique): + break + return path_unique + +def _get_remotes(r_remote): + remotes = [] + + if 'remotes' not in r_remote and 'uri' not in r_remote: + raise Exception("Expected key(s): 'remotes', 'uri'") + + if 'remotes' in r_remote: + for remote in r_remote['remotes']: + remotes.append(r_remote['remotes'][remote]['uri']) + + if 'uri' in r_remote: + remotes.append(r_remote['uri']) + + return remotes + +def checkout_layers(layers, confdir, layerdir, d, rebase_conflicts_strategy='abort'): + def _checkout_git_remote(r_remote, repodir, layers_fixed_revisions): + rev = r_remote['rev'] + branch = r_remote.get('branch', None) + + remotes = _get_remotes(r_remote) + + for remote in remotes: + prot,host,path,user,pswd,params = bb.fetch.decodeurl(remote) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + logger.plain(" {}".format(r_name)) + if branch: + src_uri = f"{fetchuri};protocol={prot};rev={rev};branch={branch};destsuffix={repodir}" + else: + src_uri = f"{fetchuri};protocol={prot};rev={rev};nobranch=1;destsuffix={repodir}" + fetcher = bb.fetch.Fetch([src_uri], d) + repodir_path = os.path.join(layerdir, repodir) + try: + do_fetch(fetcher, layerdir) + except (bb.fetch.LocalModificationsError, bb.fetch.RebaseError) as e: + if rebase_conflicts_strategy != 'backup': + e.msg += ("\nUse 'bitbake-setup update --rebase-conflicts-strategy=backup'" + " to automatically back up the directory and re-clone from upstream," + " or use 'bitbake-setup init -L %s /path/to/local/checkout'" + " to work with a local checkout instead." % r_name) + raise + backup_path = add_unique_timestamp_to_path(repodir_path + '-backup') + logger.warning( + "%s\n" + "Renaming %s to %s to preserve your work, then re-cloning from upstream.", + e, repodir_path, backup_path) + os.rename(repodir_path, backup_path) + fetcher.unpack(layerdir) + urldata = fetcher.ud[src_uri] + revision = urldata.revision + layers_fixed_revisions[r_name]['git-remote']['rev'] = revision + + def _symlink_local(src, dst): + logger.plain("Making a symbolic link {} pointing to {}".format(dst, src)) + os.symlink(src, dst) + + layers_fixed_revisions = copy.deepcopy(layers) + repodirs = [] + oesetupbuild = None + logger.plain("Fetching layer/tool repositories into {}".format(layerdir)) + for r_name in layers: + r_data = layers[r_name] + repodir = r_data.get("path", r_name) + repodirs.append(repodir) + + r_remote = r_data.get('git-remote') + r_local = r_data.get('local') + if r_remote and r_local: + raise Exception("Source {} contains both git-remote and local properties.".format(r_name)) + + repodir_path = os.path.join(layerdir, repodir) + if os.path.lexists(repodir_path): + if os.path.islink(repodir_path): + os.remove(repodir_path) + elif r_local: + backup_path = add_unique_timestamp_to_path(repodir_path + '-backup') + logger.warning("""Source {} in {} contains local modifications. Renaming to {} to preserve them. +For local development work it is recommended to clone the needed layers separately and re-initialize using -L option: +bitbake-setup init -L {} /path/to/repo/checkout""".format( + r_name, repodir_path, backup_path, r_name)) + os.rename(repodir_path, backup_path) + + if r_remote: + _checkout_git_remote(r_remote, repodir, layers_fixed_revisions) + if r_local: + _symlink_local(os.path.expanduser(r_local["path"]), repodir_path) + + if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build') + oeinitbuildenvdir = os.path.join(layerdir, repodir) + + logger.plain(" ") + _write_layer_list(layerdir, repodirs) + + if oesetupbuild: + links = {'setup-build': oesetupbuild, 'oe-scripts': os.path.dirname(oesetupbuild), 'oe-init-build-env-dir': oeinitbuildenvdir} + for link,item in links.items(): + symlink = os.path.join(layerdir, link) + if os.path.lexists(symlink): + os.remove(symlink) + os.symlink(os.path.relpath(item,layerdir),symlink) + + return layers_fixed_revisions + +def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode=False): + def _setup_build_conf(layers, filerelative_layers, build_conf_dir): + os.makedirs(build_conf_dir) + layers_s = [] + + for layer in layers: + layer = os.path.join(layerdir, layer) + layers_s.append(" {} \\".format(layer)) + + for layer in filerelative_layers: + if thisdir: + layer = os.path.join(thisdir, layer) + else: + raise Exception("Configuration is using bb-layers-file-relative to specify " \ + "a layer path relative to itself. This can be done only " \ + "when the configuration is specified by its path on local " \ + "disk, not when it's in a registry or is fetched over http.") + layers_s.append(" {} \\".format(layer)) + + layers_s = "\n".join(layers_s) + bblayers_conf = """BBLAYERS ?= " \\ +{} + " +""".format(layers_s) + with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as f: + f.write(bblayers_conf) + + local_conf = """# +# This file is intended for local configuration tweaks. +# +# If you would like to publish and share changes made to this file, +# it is recommended to put them into a distro config, or to create +# layer fragments from changes made here. +# +""" + with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f: + f.write(local_conf) + + with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') as f: + f.write("") + + with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') as f: + f.write(bitbake_config["description"] + "\n") + + with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f: + notes = bitbake_config.get("notes") + if isinstance(notes, list): + notes = "\n".join(notes) + f.write(notes + "\n" if notes else "") + + def _make_init_build_env(builddir, oeinitbuildenvdir): + builddir = os.path.realpath(builddir) + cmd = 'cd {}\nset {}\n. ./oe-init-build-env\nset --\n'.format(oeinitbuildenvdir, builddir) + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + + with open(initbuild_in_builddir, 'w') as f: + f.write("# init-build-env wrapper created by bitbake-setup\n") + f.write(cmd + '\n') + + def _prepend_passthrough_to_init_build_env(builddir): + env = bitbake_config.get("bb-env-passthrough-additions") + if not env: + return + + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir) as f: + content = f.read() + + joined = " \\\n".join(env) + env = "export BB_ENV_PASSTHROUGH_ADDITIONS=\" \\\n" + env += "${BB_ENV_PASSTHROUGH_ADDITIONS} \\\n" + env += joined + env += '"' + + with open(initbuild_in_builddir, 'w') as f: + f.write("# environment passthrough added by bitbake-setup\n") + f.write(env + '\n') + f.write('\n') + f.write(content) + + bitbake_builddir = os.path.join(setupdir, "build") + logger.plain("Setting up bitbake configuration in\n {}\n".format(bitbake_builddir)) + + template = bitbake_config.get("oe-template") + layers = bitbake_config.get("bb-layers") + if not template and not layers: + logger.error("Bitbake configuration does not contain a reference to an OpenEmbedded build template via 'oe-template' or a list of layers via 'bb-layers'; please use oe-setup-build, oe-init-build-env or another mechanism manually to complete the setup.") + return + oesetupbuild = os.path.join(layerdir, 'setup-build') + if template and not os.path.exists(oesetupbuild): + raise Exception("Cannot complete setting up a bitbake build directory from OpenEmbedded template '{}' as oe-setup-build was not found in any layers; please use oe-init-build-env manually.".format(template)) + + bitbake_confdir = os.path.join(bitbake_builddir, 'conf') + backup_bitbake_confdir = add_unique_timestamp_to_path(os.path.join(bitbake_builddir, 'conf-backup')) + upstream_bitbake_confdir = add_unique_timestamp_to_path(os.path.join(bitbake_builddir, 'conf-upstream')) + + if os.path.exists(bitbake_confdir): + os.rename(bitbake_confdir, backup_bitbake_confdir) + + if layers: + filerelative_layers = bitbake_config.get("bb-layers-file-relative") or [] + _setup_build_conf(layers, filerelative_layers, bitbake_confdir) + + if template: + bb.process.run([oesetupbuild, "setup", "-c", template, "-b", bitbake_builddir, "--no-shell"]) + else: + oeinitbuildenvdir = os.path.join(layerdir, 'oe-init-build-env-dir') + if not os.path.exists(os.path.join(oeinitbuildenvdir, "oe-init-build-env")): + logger.error("Could not find oe-init-build-env in any of the layers; please use another mechanism to initialize the bitbake environment") + return + _make_init_build_env(bitbake_builddir, os.path.realpath(oeinitbuildenvdir)) + + _prepend_passthrough_to_init_build_env(bitbake_builddir) + + siteconf_symlink = os.path.join(bitbake_confdir, "site.conf") + siteconf = os.path.normpath(os.path.join(setupdir, '..', "site.conf")) + if os.path.lexists(siteconf_symlink): + os.remove(siteconf_symlink) + os.symlink(os.path.relpath(siteconf, bitbake_confdir), siteconf_symlink) + + + init_script = os.path.join(bitbake_builddir, "init-build-env") + workspace_file = os.path.join(setupdir, "bitbake.code-workspace") + shell = "bash" + # oe-fragments entries can be strings or dicts with 'name'/'description' keys + oe_fragments = [f["name"] if isinstance(f, dict) else f for f in bitbake_config.get("oe-fragments", [])] + fragments = oe_fragments + sorted(bitbake_config.get("oe-fragment-choices",{}).values()) + if fragments: + bb.process.run("{} -c '. {} && bitbake-config-build enable-fragment {}'".format(shell, init_script, " ".join(fragments))) + + if os.path.exists(backup_bitbake_confdir): + conf_diff = get_diff(backup_bitbake_confdir, bitbake_confdir) + if not conf_diff: + logger.plain('New bitbake configuration from upstream is the same as the current one, no need to update it.') + shutil.rmtree(bitbake_confdir) + os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) + return + + logger.plain('Upstream bitbake configuration changes were found:') + logger.plain(conf_diff) + + if update_bb_conf == 'prompt': + y_or_n = input('Apply these changes to the current configuration? (y/N): ') + if y_or_n != 'y': + update_bb_conf = 'no' + + if update_bb_conf == 'no': + logger.plain('Ignoring upstream bitbake configuration changes') + logger.plain(f'Leaving the upstream configuration in {upstream_bitbake_confdir}') + os.rename(bitbake_confdir, upstream_bitbake_confdir) + os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) + return + + logger.plain('Applying upstream bitbake configuration changes') + logger.plain(f'Leaving the previous configuration in {backup_bitbake_confdir}') + + fragment_note = "Run 'bitbake-config-build enable-fragment ' to enable additional fragments or replace built-in ones (e.g. machine/ or distro/ to change MACHINE or DISTRO)." + + readme_extra = "" + if init_vscode: + readme_extra = "\nTo edit the code in VSCode, open the workspace: code {}\n".format(workspace_file) + + readme = """{}\n\nAdditional information is in {} and {}\n +Source the environment using '. {}' to run builds from the command line.\n +{}\n +The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf +{}""".format( + bitbake_config["description"], + os.path.join(bitbake_builddir,'conf/conf-summary.txt'), + os.path.join(bitbake_builddir,'conf/conf-notes.txt'), + init_script, + fragment_note, + bitbake_builddir, + readme_extra + ) + readme_file = os.path.join(bitbake_builddir, "README") + with open(readme_file, 'w') as f: + f.write(readme) + + logger.plain("This bitbake configuration provides:\n {}\n".format(bitbake_config["description"])) + logger.plain("Usage instructions and additional information are in\n {}\n".format(readme_file)) + logger.plain("To run builds, source the environment using\n . {}\n".format(init_script)) + logger.plain("{}\n".format(fragment_note)) + logger.plain("The bitbake configuration files (local.conf, bblayers.conf and more) can be found in\n {}/conf\n".format(bitbake_builddir)) + if init_vscode: + logger.plain("To edit the code in VSCode, open the workspace:\n code {}\n".format(workspace_file)) + + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) + +def get_registry_config(registry_path, id): + for root, _, files in os.walk(registry_path): + for file in files: + if file.endswith('.conf.json') and id == get_config_name(file): + return os.path.join(root, file) + raise Exception("Unable to find {} in available configurations; use 'list' sub-command to see what is available".format(id)) + +def merge_overrides_into_sources(sources, overrides): + layers = copy.deepcopy(sources) + for k,v in overrides.items(): + if k in layers: + layers[k] = v + return layers + +def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt", init_vscode=False, rebase_conflicts_strategy='abort'): + layer_config = merge_overrides_into_sources(config["data"]["sources"], config["source-overrides"]["sources"]) + sources_fixed_revisions = checkout_layers(layer_config, confdir, layerdir, d, rebase_conflicts_strategy=rebase_conflicts_strategy) + bitbake_config = config["bitbake-config"] + thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None + setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode) + write_sources_fixed_revisions(confdir, layerdir, sources_fixed_revisions) + commit_config(confdir) + +def int_input(allowed_values, prompt=''): + n = None + while n is None: + try: + n = int(input(prompt)) + except ValueError: + prompt = 'Not a valid number, please try again: ' + continue + if n not in allowed_values: + prompt = 'Number {} not one of {}, please try again: '.format(n, allowed_values) + n = None + return n + +def flatten_bitbake_configs(configs): + def merge_configs(c1,c2): + c_merged = {} + for k,v in c2.items(): + if k not in c1.keys(): + c_merged[k] = v + for k,v in c1.items(): + if k not in c2.keys(): + c_merged[k] = v + else: + c_merged[k] = c1[k] + c2[k] + del c_merged['configurations'] + return c_merged + + flattened_configs = [] + for c in configs: + if 'configurations' not in c: + flattened_configs.append(c) + else: + for sub_c in flatten_bitbake_configs(c['configurations']): + flattened_configs.append(merge_configs(c, sub_c)) + return flattened_configs + +def choose_bitbake_config(configs, parameters, non_interactive): + flattened_configs = flatten_bitbake_configs(configs) + configs_dict = {i["name"]:i for i in flattened_configs} + + if parameters: + config_id = parameters[0] + if config_id not in configs_dict: + raise Exception("Bitbake configuration {} not found; replace with one of {}".format(config_id, configs_dict)) + return configs_dict[config_id] + + enumerated_configs = list(enumerate(flattened_configs, 1)) + if len(enumerated_configs) == 1: + only_config = flattened_configs[0] + logger.plain("\nSelecting the only available bitbake configuration {}".format(only_config["name"])) + return only_config + + if non_interactive: + raise Exception("Unable to choose from bitbake configurations in non-interactive mode: {}".format(configs_dict)) + + logger.plain("") + print_configs("Available bitbake configurations", + [c["name"] for c in flattened_configs], + [c["description"] for c in flattened_configs]) + config_n = int_input([i[0] for i in enumerated_configs], + "\nPlease select one of the above bitbake configurations by its number: ") - 1 + return flattened_configs[config_n] + +def choose_config(configs, non_interactive): + not_expired_configs = [k for k in sorted(configs.keys()) if not has_expired(configs[k].get("expires", None))] + if len(not_expired_configs) == 1: + only_config = not_expired_configs[0] + logger.plain("\nSelecting the only available configuration {}\n".format(only_config)) + return only_config + + if non_interactive: + raise Exception("Unable to choose from configurations in non-interactive mode: {}".format(not_expired_configs)) + + descs = [] + for c in not_expired_configs: + d = configs[c]["description"] + expiry_date = configs[c].get("expires", None) + if expiry_date: + d += f" (supported until {expiry_date})" + descs.append(d) + + logger.plain("") + print_configs("Available Configuration Templates", + [c for c in not_expired_configs], + descs) + config_n = int_input([i[0] for i in list(enumerate(not_expired_configs, 1))], + "\nPlease select one of the above configurations by its number: ") - 1 + return not_expired_configs[config_n] + +def choose_fragments(possibilities, parameters, non_interactive, skip_selection): + choices = {} + for k,v in possibilities.items(): + if skip_selection and k in skip_selection: + logger.info("Skipping a selection of {}, as requested on command line. The resulting bitbake configuration may require further manual adjustments.".format(k)) + continue + + # options can be a list of strings or a list of dicts + options = v["options"] + if len(options) > 0 and isinstance(v["options"][0], str): + options = [{"name": o, "description": ""} for o in v["options"]] + + choice = [o["name"] for o in options if o["name"] in parameters] + if len(choice) > 1: + raise Exception("Options specified on command line do not allow a single selection " + f"from possibilities {[o['name'] for o in options]}, please " + f"remove one or more from {parameters}") + if len(choice) == 1: + choices[k] = choice[0] + continue + + if non_interactive: + raise Exception(f"Unable to choose from options in non-interactive mode: {[o['name'] for o in options]}") + + logger.plain("") + print_configs(v["description"], + [o['name'] for o in options], + [o['description'] for o in options]) + options_enumerated = list(enumerate(options, 1)) + option_n = int_input([i[0] for i in options_enumerated], + "\nPlease select one of the above options by its number: ") - 1 + choices[k] = options_enumerated[option_n][1]["name"] + return choices + +def obtain_config(top_dir, registry, args, source_overrides, d): + if args.config: + config_id = args.config[0] + config_parameters = args.config[1:] + if os.path.exists(config_id): + config_id = os.path.abspath(config_id) + logger.info("Reading configuration from local file\n {}".format(config_id)) + upstream_config = {'type':'local', + 'path':config_id, + 'name':get_config_name(config_id), + 'data':json.load(open(config_id)) + } + elif config_id.startswith("http://") or config_id.startswith("https://"): + logger.info("Reading configuration from network URI\n {}".format(config_id)) + import urllib.request + try: + with urllib.request.urlopen(config_id) as f: + json_data = json.load(f) + upstream_config = {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json_data} + except json.JSONDecodeError as e: + raise Exception ("Invalid JSON from {}. Are you pointing to an HTML page? {}".format(config_id, e)) from e + else: + logger.info("Looking up config {} in configuration registry".format(config_id)) + registry_path = update_registry(registry, cache_dir(top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + if config_id not in registry_configs: + raise Exception("Config {} not found in configuration registry, re-run 'init' without parameters to choose from available configurations.".format(config_id)) + upstream_config = {'type':'registry','registry':registry,'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + expiry_date = upstream_config['data'].get("expires", None) + if has_expired(expiry_date): + logger.warning("This configuration is no longer supported after {}. Please consider changing to a supported configuration.".format(expiry_date)) + else: + registry_path = update_registry(registry, cache_dir(top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + config_id = choose_config(registry_configs, args.non_interactive) + config_parameters = [] + upstream_config = {'type':'registry','registry':registry,'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + + upstream_config['bitbake-config'] = choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], config_parameters, args.non_interactive) + upstream_config['bitbake-config']['oe-fragment-choices'] = choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), config_parameters[1:], args.non_interactive, args.skip_selection) + upstream_config['non-interactive-cmdline-options'] = [config_id, upstream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values()) + upstream_config['source-overrides'] = source_overrides + upstream_config['skip-selection'] = args.skip_selection + return upstream_config + +def obtain_overrides(args): + overrides = {'sources':{}} + if args.source_overrides: + overrides = json.load(open(args.source_overrides)) + overrides_dir = os.path.dirname(os.path.abspath(args.source_overrides)) + for s,v in overrides['sources'].items(): + local = v.get('local') + if local: + path = os.path.expanduser(local['path']) + if not os.path.isabs(path): + overrides['sources'][s]['local']['path'] = os.path.join(overrides_dir, path) + + for local_name, local_path in args.use_local_source: + overrides['sources'][local_name] = {'local':{'path':os.path.abspath(os.path.expanduser(local_path))}} + + return overrides + +def configure_vscode(setupdir, layerdir, builddir, init_script): + """ + Configure the VSCode environment by creating or updating a workspace file. + + Create or update a bitbake.code-workspace file with folders for the layers and build/conf. + Managed folders are regenerated; user-added folders are kept. Settings are merged, with + managed keys (bitbake.*, python extra paths) always overwritten. + """ + logger.debug("configure_vscode: setupdir={}, layerdir={}, builddir={}, init_script={}".format( + setupdir, layerdir, builddir, init_script)) + + # Get git repository directories + git_repos = [] + if os.path.exists(layerdir): + for entry in os.listdir(layerdir): + entry_path = os.path.join(layerdir, entry) + if os.path.isdir(entry_path) and not os.path.islink(entry_path): + # Check if it's a git repository + if os.path.exists(os.path.join(entry_path, '.git')): + git_repos.append(entry) + logger.debug("configure_vscode: found {} git repos: {}".format(len(git_repos), git_repos)) + + conf_path = os.path.relpath(os.path.join(builddir, "conf"), setupdir) + repo_paths = [os.path.relpath(os.path.join(layerdir, repo), setupdir) for repo in git_repos] + logger.debug("configure_vscode: conf_path={}, repo_paths={}".format(conf_path, repo_paths)) + + # Load existing workspace + workspace_file = os.path.join(setupdir, "bitbake.code-workspace") + workspace = { + "extensions": { + "recommendations": [ + "yocto-project.yocto-bitbake" + ] + } + } + if os.path.exists(workspace_file): + logger.debug("configure_vscode: loading existing workspace file: {}".format(workspace_file)) + try: + with open(workspace_file, 'r') as f: + workspace = json.load(f) + logger.debug("configure_vscode: loaded workspace with {} folders, {} settings".format( + len(workspace.get("folders", [])), len(workspace.get("settings", {})))) + except (json.JSONDecodeError, OSError) as e: + logger.error( + "Unable to read existing workspace file {}: {}. Skipping update.".format( + workspace_file, str(e) + ) + ) + return + else: + logger.debug("configure_vscode: creating new workspace file: {}".format(workspace_file)) + + # Update folders + existing_folders = workspace.get("folders", []) + new_folders = [{"name": "conf", "path": conf_path}] + for rp in repo_paths: + repo_name = os.path.basename(rp) + new_folders.append({"name": repo_name, "path": rp}) + # Keep any user-added folders that are not managed + managed_paths = {f["path"] for f in new_folders} + for f in existing_folders: + if f["path"] not in managed_paths: + new_folders.append(f) + logger.debug("configure_vscode: keeping user-added folder: {}".format(f["path"])) + workspace["folders"] = new_folders + logger.debug("configure_vscode: updated workspace with {} folders".format(len(new_folders))) + + # Build Python extra paths for each layer - only check top level of each repo + extra_paths = [] + subdirs_to_check = ['lib', 'scripts'] + for repo in git_repos: + repo_path_abs = os.path.join(layerdir, repo) + for subdir in subdirs_to_check: + sub_path = os.path.join(repo_path_abs, subdir) + if os.path.isdir(sub_path): + extra_paths.append(sub_path) + + # Update settings + existing_settings = workspace.get("settings", {}) + new_settings = { + "bitbake.disableConfigModification": True, + "bitbake.pathToBitbakeFolder": os.path.join(layerdir, "bitbake"), + "bitbake.pathToBuildFolder": builddir, + "bitbake.pathToEnvScript": init_script, + "bitbake.workingDirectory": builddir, + "files.associations": { + "*.conf": "bitbake", + "*.inc": "bitbake" + }, + "files.exclude": { + "**/.git/**": True + }, + "search.exclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "files.watcherExclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "python.analysis.exclude": [ + "**/.git/**", + "**/logs/**" + ], + "python.autoComplete.extraPaths": extra_paths, + "python.analysis.extraPaths": extra_paths + } + + # Merge settings: add missing, always update bitbake paths and python extra paths + for key, value in new_settings.items(): + if key not in existing_settings: + existing_settings[key] = value + elif key.startswith("bitbake.") or key in [ + "python.autoComplete.extraPaths", + "python.analysis.extraPaths", + ]: + # Always replace - these are managed/machine-generated settings + existing_settings[key] = value + elif key in [ + "files.associations", + "files.exclude", + "search.exclude", + "files.watcherExclude", + "python.analysis.exclude", + ]: + # For dicts and lists, merge new values in without removing user additions + if isinstance(value, dict): + if not isinstance(existing_settings[key], dict): + existing_settings[key] = {} + for k, v in value.items(): + if k not in existing_settings[key]: + existing_settings[key][k] = v + elif isinstance(value, list): + if not isinstance(existing_settings[key], list): + existing_settings[key] = [] + for item in value: + if item not in existing_settings[key]: + existing_settings[key].append(item) + + workspace["settings"] = existing_settings + logger.debug("configure_vscode: merged settings, total {} keys".format(len(existing_settings))) + + with open(workspace_file, 'w') as f: + json.dump(workspace, f, indent=4) + logger.debug("configure_vscode: wrote workspace file: {}".format(workspace_file)) + +def init_config(top_dir, settings, args): + create_siteconf(top_dir, args.non_interactive, settings) + + d = init_bb_cache(top_dir, settings, args) + + def handle_task_progress(event, d): + rate = event.rate if event.rate else '' + progress = event.progress if event.progress > 0 else 0 + logger.handlers[0].terminator = '\r' + logger.plain("{}% {} ".format(progress, rate)) + logger.handlers[0].terminator = '\n' + + source_overrides = obtain_overrides(args) + upstream_config = obtain_config(top_dir, settings["default"]["registry"], args, source_overrides, d) + logger.info("Run 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options']))) + + if args.setup_dir_name: + setup_dir_name = args.setup_dir_name + else: + setup_dir_name = "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_")) + if 'setup-dir-name' in upstream_config['bitbake-config']: + mapping = { + k: v.partition("/")[2].replace(" ", "-").replace("/", "_") + for k, v in upstream_config['bitbake-config']['oe-fragment-choices'].items() + } + config_setup_dir_name = string.Template(upstream_config['bitbake-config']['setup-dir-name']).substitute(mapping) + config_setup_dir = os.path.join(top_dir, config_setup_dir_name) + if os.path.exists(config_setup_dir): + logger.info("Setup directory {} (as suggested by configuration) already exists, using the full name instead.\n".format(config_setup_dir)) + elif settings['default']['use-full-setup-dir-name'] != 'no': + logger.info("Using the full setup directory name instead of {} suggested by configuration, as set in the settings.\n".format(config_setup_dir)) + else: + setup_dir_name = config_setup_dir_name + + if not args.non_interactive: + n = input(f"Enter setup directory name [{setup_dir_name}]: ") + if n: + setup_dir_name = n + + setupdir = os.path.join(os.path.abspath(top_dir), setup_dir_name) + if os.path.exists(os.path.join(setupdir, "layers")): + logger.info(f"Setup already initialized in:\n {setupdir}\nUse 'bitbake-setup status' to check if it needs to be updated, or 'bitbake-setup update' to perform the update.\nIf you would like to start over and re-initialize in this directory, remove it, and run 'bitbake-setup init' again.") + return + + logger.plain("Initializing a setup directory in\n {}".format(setupdir)) + if not args.non_interactive: + y_or_n = input('Continue? (y/N): ') + if y_or_n != 'y': + exit() + logger.plain("") + + os.makedirs(setupdir, exist_ok=True) + + confdir = os.path.join(setupdir, "config") + layerdir = os.path.join(setupdir, "layers") + + os.makedirs(confdir) + os.makedirs(layerdir) + + bb.process.run(["git", "-C", confdir, "init", "-b", "main"]) + # Make sure commiting doesn't fail if no default git user is configured on the machine + bb.process.run(["git", "-C", confdir, "config", "user.name", "bitbake-setup"]) + bb.process.run(["git", "-C", confdir, "config", "user.email", "bitbake-setup@not.set"]) + bb.process.run(["git", "-C", confdir, "commit", "--no-verify", "--allow-empty", "-m", "Initial commit"]) + + bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d) + + write_upstream_config(confdir, upstream_config) + update_build(upstream_config, confdir, setupdir, layerdir, d, update_bb_conf="yes", init_vscode=args.init_vscode) + + bb.event.remove("bb.build.TaskProgress", None) + +def get_diff(file1, file2): + try: + bb.process.run(["diff", get_diff_color_param(), "-uNr", file1, file2]) + except bb.process.ExecutionError as e: + if e.exitcode == 1: + return e.stdout + else: + raise e + return None + +def are_layers_changed(layers, layerdir, d): + def _is_git_remote_changed(r_remote, repodir): + from bb.fetch.git import git_hash_re + + rev = r_remote['rev'] + branch = r_remote.get('branch', None) + + rev_parse_result = bb.process.run(['git', '-C', os.path.join(layerdir, repodir), "rev-parse", "HEAD"]) + local_revision = rev_parse_result[0].strip() + if git_hash_re.match(rev): + if rev != local_revision: + logger.info('Layer repository checked out into {} is at revision {} but should be at {}'.format(os.path.join(layerdir, repodir),local_revision, rev)) + return True + return False + + remotes = _get_remotes(r_remote) + changed = False + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remote) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + if branch: + fetcher = bb.fetch.FetchData("{};protocol={};rev={};branch={};destsuffix={}".format(fetchuri,type,rev,branch,repodir), d) + else: + fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d) + upstream_revision = fetcher.method.latest_revision(fetcher, d, 'default') + if upstream_revision != local_revision: + changed = True + logger.info('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remote, os.path.join(layerdir, repodir), rev, local_revision, upstream_revision)) + return changed + + changed = False + for r_name in layers: + r_data = layers[r_name] + repodir = r_data.get("path", r_name) + + git_remote = r_data.get('git-remote') + if git_remote: + changed = changed | _is_git_remote_changed(git_remote, repodir) + + return changed + +def build_status(top_dir, settings, args, d, update=False): + setupdir = os.path.abspath(args.setup_dir) + + confdir = os.path.join(setupdir, "config") + layerdir = os.path.join(setupdir, "layers") + + current_upstream_config = json.load(open(os.path.join(confdir, "config-upstream.json"))) + + args.config = current_upstream_config['non-interactive-cmdline-options'] + args.non_interactive = True + args.skip_selection = current_upstream_config['skip-selection'] + source_overrides = current_upstream_config["source-overrides"] + registry = current_upstream_config.get("registry") + new_upstream_config = obtain_config(top_dir, registry, args, source_overrides, d) + + write_upstream_config(confdir, new_upstream_config) + config_diff = bb.process.run(["git", "-C" , confdir, "diff", get_diff_color_param()])[0] + + if config_diff: + logger.plain('\nConfiguration in {} has changed:\n{}'.format(setupdir, config_diff)) + if update: + update_build(new_upstream_config, confdir, setupdir, layerdir, d, + update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy) + else: + bb.process.run(["git", "-C", confdir, "restore", "config-upstream.json"]) + return + + layer_config = merge_overrides_into_sources(current_upstream_config["data"]["sources"], current_upstream_config["source-overrides"]["sources"]) + if are_layers_changed(layer_config, layerdir, d): + if update: + update_build(current_upstream_config, confdir, setupdir, layerdir, + d, update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy) + return + + logger.plain("\nConfiguration in {} has not changed.".format(setupdir)) + +def build_update(top_dir, settings, args, d): + build_status(top_dir, settings, args, d, update=True) + +def do_fetch(fetcher, dir): + # git fetcher simply dumps git output to stdout; in bitbake context that is redirected to temp/log.do_fetch + # and we need to set up smth similar here + fetchlogdir = os.path.join(dir, 'logs') + os.makedirs(fetchlogdir, exist_ok=True) + fetchlog = os.path.join(fetchlogdir, 'fetch_log.{}'.format(datetime.datetime.now().strftime("%Y%m%d%H%M%S"))) + with open(fetchlog, 'a') as f: + oldstdout = sys.stdout + sys.stdout = f + try: + fetcher.download() + fetcher.unpack_update(dir) + except: + if os.path.exists(fetchlog) and os.path.getsize(fetchlog) > 0: + logger.error("Check {} for additional information that may help diagnoze the issue.".format(fetchlog)) + raise + finally: + sys.stdout = oldstdout + +def update_registry(registry, cachedir, d): + registrydir = 'configurations' + if registry.startswith("."): + full_registrydir = os.path.join(os.getcwd(), registry, registrydir) + elif registry.startswith("/"): + full_registrydir = os.path.join(registry, registrydir) + else: + full_registrydir = os.path.join(cachedir, registrydir) + logger.info("Fetching configuration registry\n {}\ninto\n {}".format(registry, full_registrydir)) + fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d) + do_fetch(fetcher, cachedir) + return full_registrydir + +def has_expired(expiry_date): + if expiry_date: + return datetime.datetime.now() > datetime.datetime.fromisoformat(expiry_date) + return False + +def list_registry(registry_path, with_expired): + json_data = {} + + for root, _, files in os.walk(registry_path): + for file in files: + if file.endswith('.conf.json'): + config_name = get_config_name(file) + config_data = json.load(open(os.path.join(root, file))) + config_desc = config_data["description"] + expiry_date = config_data.get("expires", None) + if expiry_date: + if with_expired or not has_expired(expiry_date): + json_data[config_name] = {"description": config_desc, "expires": expiry_date} + else: + json_data[config_name] = {"description": config_desc} + return json_data + +def list_configs(settings, args): + import tempfile + top_dir = tempfile.mkdtemp(prefix="bitbake-setup-list-") + settings['default']['dl-dir'] = os.path.join(top_dir, '.bitbake-setup-downloads') + d = init_bb_cache(top_dir, settings, args) + registry_path = update_registry(settings["default"]["registry"], cache_dir(top_dir), d) + json_data = list_registry(registry_path, args.with_expired) + shutil.rmtree(top_dir) + + logger.plain("Available configurations:") + for config_name, config_data in sorted(json_data.items()): + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + if args.with_expired or not has_expired(expiry_date): + logger.plain("{}\t{} (supported until {})".format(config_name, config_desc, expiry_date)) + else: + logger.plain("{}\t{}".format(config_name, config_desc)) + logger.plain("\nRun 'init' with one of the above configuration identifiers to set up a build.") + + if args.write_json: + with open(args.write_json, 'w') as f: + json.dump(json_data, f, sort_keys=True, indent=4) + logger.plain("Available configurations written into {}".format(args.write_json)) + +def install_buildtools(top_dir, settings, args, d): + buildtools_install_dir = os.path.join(args.setup_dir, 'buildtools') + if os.path.exists(buildtools_install_dir): + if not args.force: + logger.plain("Buildtools are already installed in {}.".format(buildtools_install_dir)) + env_scripts = glob.glob(os.path.join(buildtools_install_dir, 'environment-setup-*')) + if env_scripts: + logger.plain("If you wish to use them, you need to source the environment setup script e.g.") + for s in env_scripts: + logger.plain("$ . {}".format(s)) + logger.plain("You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation.") + return + shutil.rmtree(buildtools_install_dir) + + install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools') + + buildtools_config = {} + upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json') + try: + with open(upstream_config_path) as f: + upstream_config = json.load(f) + buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {}) + except FileNotFoundError: + pass + except json.JSONDecodeError as e: + logger.error("Failed to parse %s: %s" % (upstream_config_path, e)) + sys.exit(1) + + url = args.url if args.url is not None else buildtools_config.get('url') + sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum') + + buildtools_download_dir = add_unique_timestamp_to_path( + os.path.join(args.setup_dir, 'buildtools-downloads/buildtools')) + + if args.default or (not url and not sha256): + # No config or --default: fall back to install-buildtools defaults + logger.plain("Using install-buildtools script defaults") + subprocess.check_call([install_buildtools_script, + '-d', buildtools_install_dir, + '--downloads-directory', buildtools_download_dir]) + return + + if not url: + logger.error("No buildtools URL configured. " + "Set 'url' in the install-buildtools config section or pass --url.") + sys.exit(1) + + if not sha256: + logger.error("No sha256sum configured for buildtools installer. " + "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n" + "You can obtain it with: sha256sum ") + sys.exit(1) + + # Download using bb.fetch with sha256 validation + src_uri = "{};sha256sum={}".format(url, sha256) + + logger.plain("Fetching buildtools installer using bitbake fetcher") + logger.plain(" {}".format(url)) + + fetcher = bb.fetch.Fetch([src_uri], d) + try: + fetcher.download() + except bb.fetch.ChecksumError as e: + logger.error("Checksum mismatch for buildtools installer. " + "Verify the URL and downloaded file.\n%s" % str(e)) + sys.exit(1) + except bb.fetch.FetchError as e: + logger.error("Failed to download buildtools installer: %s" % str(e)) + sys.exit(1) + + # Unpack into the standard buildtools download location + os.makedirs(buildtools_download_dir, exist_ok=True) + fetcher.unpack(buildtools_download_dir) + + filename = os.path.basename(url.rstrip('/')) + matches = glob.glob(os.path.join(buildtools_download_dir, '**', filename), recursive=True) + if not matches: + logger.error("Unpacked file not found: %s" % filename) + sys.exit(1) + local_path = matches[0] + + logger.plain("Buildtools installer will be installed into {}".format( + buildtools_install_dir)) + + cmd = [ + install_buildtools_script, + '--local-file', local_path, + '-d', buildtools_install_dir, + ] + + try: + subprocess.check_call(cmd) + except subprocess.CalledProcessError as e: + logger.error("install-buildtools failed with exit code %d" % e.returncode) + sys.exit(1) + +def create_siteconf(top_dir, non_interactive, settings): + siteconfpath = os.path.join(top_dir, 'site.conf') + if os.path.exists(siteconfpath): + logger.info('A site.conf file already exists. Please remove it if you would like to replace it with a default one') + else: + logger.plain(f'{top_dir} looks like a new top directory. If you would like to use a different directory, answer "n" below and either:') + logger.plain('\t1) Change the default bitbake-setup settings:') + logger.plain('\t\tbitbake-setup settings set default top-dir-prefix ') + logger.plain('\t\tbitbake-setup settings set default top-dir-name ') + logger.plain('\t2) Pass one or more options on the command line to change the top level directory in that invocation only:') + logger.plain('\t\tbitbake-setup --setting default top-dir-prefix ...') + logger.plain('\t\tbitbake-setup --setting default top-dir-name ...') + logger.plain('') + + logger.plain('A common site.conf file will be created, please check it is correct before running builds\n {}\n'.format(siteconfpath)) + if not non_interactive: + y_or_n = input('Proceed? (y/N): ') + if y_or_n != 'y': + exit() + + os.makedirs(top_dir, exist_ok=True) + with open(siteconfpath, 'w') as siteconffile: + sstate_settings = textwrap.dedent( + """ + # + # Where to place shared-state files + # + # BitBake has the capability to accelerate builds based on previously built output. + # This is done using "shared state" files which can be thought of as cache objects + # and this option determines where those files are placed. + # + # You can wipe out TMPDIR leaving this directory intact and the build would regenerate + # from these files if no changes were made to the configuration. If changes were made + # to the configuration, only shared state files where the state was still valid would + # be used (done using checksums). + SSTATE_DIR ?= "{sstate_dir}" + # + # Hash Equivalence database location + # + # Hash equivalence improves reuse of sstate by detecting when a given sstate + # artifact can be reused as equivalent, even if the current task hash doesn't + # match the one that generated the artifact. This variable controls where the + # Hash Equivalence database ("hashserv.db") is stored and can be shared between + # concurrent builds. + BB_HASHSERVE_DB_DIR ?= "${{SSTATE_DIR}}" + """.format(sstate_dir=os.path.join(top_dir, ".sstate-cache")) + ) + siteconffile.write( + textwrap.dedent( + """\ + # This file is intended for build host-specific bitbake settings + + # Where to place downloads + # + # During a first build the system will download many different source code + # tarballs from various upstream projects. This can take a while, particularly + # if your network connection is slow. These are all stored in DL_DIR. When + # wiping and rebuilding you can preserve this directory to speed up this part of + # subsequent builds. This directory is safe to share between multiple builds on + # the same machine too. + DL_DIR ?= "{dl_dir}" + """.format( + dl_dir=settings["default"]["dl-dir"], + ) + ) + (sstate_settings if settings["default"]["common-sstate"] == 'yes' else "") + ) + + +def topdir_settings_path(top_dir): + return os.path.join(top_dir, 'settings.conf') + +def global_settings_path(args): + return os.path.abspath(args.global_settings) if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'settings.conf') + +def load_settings(settings_path): + settings = configparser.ConfigParser() + if os.path.exists(settings_path): + logger.info('Loading settings from {}'.format(settings_path)) + settings.read_file(open(settings_path)) + + for section in settings.sections(): + for key, value in settings[section].items(): + settings[section][key] = value.strip('\"\'') + + return settings + +def change_setting(top_dir, args): + if vars(args)['global']: + settings_path = global_settings_path(args) + elif args.setting in GLOBAL_ONLY_SETTINGS: + logger.info(f"{args.setting} can only be set in the global config; '--global' is implied") + settings_path = global_settings_path(args) + else: + settings_path = topdir_settings_path(top_dir) + settings = load_settings(settings_path) + + if args.subcommand == 'set': + if args.section not in settings.keys(): + settings[args.section] = {} + settings[args.section][args.setting] = args.value + logger.plain(f"From section '{args.section}' the setting '{args.setting}' was changed to '{args.value}'") + if args.subcommand == 'unset': + if args.section in settings.keys() and args.setting in settings[args.section].keys(): + del settings[args.section][args.setting] + logger.plain(f"From section '{args.section}' the setting '{args.setting}' has been removed") + + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + logger.info(f"Settings written to {settings_path}") + +def list_settings(all_settings): + for section, section_settings in all_settings.items(): + for key, value in section_settings.items(): + logger.plain("{} {} {}".format(section, key, value)) + +def settings_func(top_dir, all_settings, args): + if args.subcommand == 'list': + list_settings(all_settings) + elif args.subcommand == 'set' or args.subcommand == 'unset': + change_setting(top_dir, args) + +def get_setup_dir_via_bbpath(): + bbpath = os.environ.get('BBPATH') + if bbpath: + bitbake_dir = os.path.normpath(bbpath.split(':')[0]) + if os.path.exists(os.path.join(bitbake_dir,'init-build-env')): + setup_dir = os.path.dirname(bitbake_dir) + return setup_dir + return None + +def get_top_dir(args, settings): + setup_dir_via_bbpath = get_setup_dir_via_bbpath() + if setup_dir_via_bbpath: + top_dir = os.path.dirname(setup_dir_via_bbpath) + if os.path.exists(cache_dir(top_dir)): + return top_dir + + if hasattr(args, 'setup_dir'): + top_dir = os.path.dirname(os.path.normpath(args.setup_dir)) + return top_dir + + top_dir_prefix = settings['default']['top-dir-prefix'] + top_dir_name = settings['default']['top-dir-name'] + return os.path.join(top_dir_prefix, top_dir_name) + +def merge_settings(builtin_settings, global_settings, topdir_settings, cmdline_settings): + all_settings = builtin_settings + + for s in (global_settings, topdir_settings): + for section, section_settings in s.items(): + for setting, value in section_settings.items(): + if section not in all_settings.keys(): + all_settings[section] = {} + all_settings[section][setting] = value + + for (section, setting, value) in cmdline_settings: + if section not in all_settings.keys(): + all_settings[section] = {} + all_settings[section][setting] = value + + return all_settings + +def sigint_handler(sig, frame, func, top_dir): + logger.plain('\nShutting down...') + if isinstance(top_dir, str) and os.path.exists(top_dir): + if func in [init_config, build_update]: + logger.warning(f'{top_dir} may contain an incomplete setup!') + elif func == install_buildtools: + logger.warning(f'{top_dir} may contain an incomplete buildtools installation!') + exit() + +def main(): + def add_setup_dir_arg(parser): + setup_dir = get_setup_dir_via_bbpath() + if setup_dir: + parser.add_argument('--setup-dir', default=setup_dir, help="Path to the setup, default is %(default)s via BBPATH") + else: + parser.add_argument('--setup-dir', required=True, help="Path to the setup") + + parser = argparse.ArgumentParser( + description="BitBake setup utility. Run with 'init' argument to get started.", + epilog="Use %(prog)s --help to get help on a specific command" + ) + parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') + parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') + parser.add_argument('--version', action='version', version='%(prog)s ' + __version__) + parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') + parser.add_argument('--no-network', action='store_true', help='Do not check whether configuration repositories and layer repositories have been updated; use only the local cache.') + parser.add_argument('--global-settings', action='store', metavar='PATH', help='Path to the global settings file.') + parser.add_argument('--setting', default=[], action='append', dest='cmdline_settings', + nargs=3, metavar=('SECTION', 'SETTING', 'VALUE'), + help='Modify a setting (for this bitbake-setup invocation only), for example "--setting default top-dir-prefix /path/to/top/dir".') + + subparsers = parser.add_subparsers() + + parser_list = subparsers.add_parser('list', help='List available configurations') + parser_list.add_argument('--with-expired', action='store_true', help='List also configurations that are no longer supported due to reaching their end-of-life dates.') + parser_list.add_argument('--write-json', action='store', help='Write available configurations into a json file so they can be programmatically processed.') + parser_list.set_defaults(func=list_configs) + + parser_init = subparsers.add_parser('init', help='Select a configuration and initialize a setup from it') + parser_init.add_argument('config', nargs='*', help="path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available choices if command line doesn't completely specify them.") + parser_init.add_argument('--non-interactive', action='store_true', help='Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.') + parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.') + parser_init.add_argument('--setup-dir-name', action='store', help='A custom setup directory name under the top directory.') + parser_init.add_argument('--skip-selection', action='append', help='Do not select and set an option/fragment from available choices; the resulting bitbake configuration may be incomplete.') + parser_init.add_argument('-L', '--use-local-source', default=[], action='append', nargs=2, metavar=('SOURCE_NAME', 'PATH'), + help='Symlink local source into a build, instead of getting it as prescribed by a configuration (useful for local development).') + parser_init.add_argument('--init-vscode', action=argparse.BooleanOptionalAction, default=bool(shutil.which('code')), + help='Generate VSCode workspace configuration (default: %(default)s)') + parser_init.set_defaults(func=init_config) + + parser_status = subparsers.add_parser('status', help='Check if the setup needs to be synchronized with configuration') + add_setup_dir_arg(parser_status) + parser_status.set_defaults(func=build_status) + + parser_update = subparsers.add_parser('update', help='Update a setup to be in sync with configuration') + add_setup_dir_arg(parser_update) + parser_update.add_argument('--update-bb-conf', choices=['prompt', 'yes', 'no'], default='prompt', help='Update bitbake configuration files (bblayers.conf, local.conf) (default: prompt)') + parser_update.add_argument('--rebase-conflicts-strategy', choices=['abort', 'backup'], default='abort', + help="What to do when a layer repository has local modifications that prevent " + "an in-place update: 'abort' (default) aborts with an error message; " + "'backup' renames the directory to a timestamped backup and re-clones from upstream.") + parser_update.set_defaults(func=build_update) + + parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine') + add_setup_dir_arg(parser_install_buildtools) + parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.') + parser_install_buildtools.add_argument('--default', action='store_true', help='Use install-buildtools script defaults, ignoring any config.') + parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.') + parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.') + parser_install_buildtools.set_defaults(func=install_buildtools) + + parser_settings_arg_global = argparse.ArgumentParser(add_help=False) + parser_settings_arg_global.add_argument('--global', action='store_true', help="Modify the setting in a global settings file, rather than one specific to a top directory") + + parser_settings = subparsers.add_parser('settings', + help='List current settings, or set or unset a setting in a settings file (e.g. the default prefix and name of the top directory, the location of configuration registry, downloads directory and other settings specific to a top directory)') + parser_settings.set_defaults(func=settings_func) + + subparser_settings = parser_settings.add_subparsers(dest="subcommand", required=True, help="The action to perform on the settings file") + + subparser_settings.add_parser('list', help="List all settings with their values") + + parser_settings_set = subparser_settings.add_parser('set', parents=[parser_settings_arg_global], + help="In a Section, set a setting to a certain value") + parser_settings_set.add_argument("section", metavar="
", help="Section in a settings file, typically 'default'") + parser_settings_set.add_argument("setting", metavar="", help="Name of a setting") + parser_settings_set.add_argument("value", metavar="", help="The setting value") + + parser_settings_unset = subparser_settings.add_parser('unset', parents=[parser_settings_arg_global], + help="Unset a setting, e.g. 'bitbake-setup settings unset default registry' would revert to the registry setting in a global settings file") + parser_settings_unset.add_argument("section", metavar="
", help="Section in a settings file, typically 'default'") + parser_settings_unset.add_argument("setting", metavar="", help="The setting to remove") + + args = parser.parse_args() + + if args.debug: + logger.setLevel(logging.DEBUG) + elif args.quiet: + logger.setLevel(logging.ERROR) + + # Need to re-run logger_create with color argument + # (will be the same logger since it has the same name) + bb.msg.logger_create('bitbake-setup', output=sys.stdout, + color=args.color, + level=logger.getEffectiveLevel()) + + if 'func' in args: + if hasattr(args, 'setup_dir'): + if not os.path.exists(os.path.join(args.setup_dir,'build', 'init-build-env')): + logger.error("Not a valid setup directory: build/init-build-env does not exist in {}".format(args.setup_dir)) + return + + if not hasattr(args, 'non_interactive'): + args.non_interactive = True + + builtin_settings = {} + builtin_settings['default'] = { + 'top-dir-prefix':os.getcwd(), + 'top-dir-name':'bitbake-builds', + 'registry':get_default_registry(), + 'use-full-setup-dir-name':'no', + 'common-sstate':'yes', + } + + global_settings = load_settings(global_settings_path(args)) + top_dir = get_top_dir(args, merge_settings(builtin_settings, global_settings, {}, args.cmdline_settings)) + + # register handler now to pass top_dir + _handler = functools.partial(sigint_handler, + func=args.func, + top_dir=os.path.abspath(top_dir)) + signal.signal(signal.SIGINT, _handler) + + # This cannot be set with the rest of the builtin settings as top_dir needs to be determined first + builtin_settings['default']['dl-dir'] = os.path.join(top_dir, '.bitbake-setup-downloads') + + topdir_settings = load_settings(topdir_settings_path(top_dir)) + all_settings = merge_settings(builtin_settings, global_settings, topdir_settings, args.cmdline_settings) + + if args.func == settings_func: + settings_func(top_dir, all_settings, args) + return + if args.func == list_configs: + list_configs(all_settings, args) + return + + logger.info('Bitbake-setup is using {} as top directory.'.format(top_dir)) + + try: + if args.func == init_config: + init_config(top_dir, all_settings, args) + else: + d = init_bb_cache(top_dir, all_settings, args) + args.func(top_dir, all_settings, args, d) + + save_bb_cache() + except (SystemExit, KeyboardInterrupt): + raise + except Exception as e: + if args.debug: + raise + logger.error(str(e)) + sys.exit(1) + else: + parser.print_help() + +if __name__ == '__main__': + main() diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index e8073f2a..61484a5c 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -9,8 +9,9 @@ import os import sys import warnings warnings.simplefilter("default") +warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*use.of.fork.*may.lead.to.deadlocks.in.the.child.*") sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) -from bb import fetch2 +from bb import fetch import logging import bb import select @@ -21,9 +22,21 @@ import traceback import queue import shlex import subprocess -from multiprocessing import Lock +import fcntl from threading import Thread +__version__ = "2.20.0" + +if __name__ == "__main__": + if __version__ != bb.__version__: + sys.exit("Bitbake core library version and program version mismatch!") + +Lock = bb.multiprocessing.Lock + +# Remove when we have a minimum of python 3.10 +if not hasattr(fcntl, 'F_SETPIPE_SZ'): + fcntl.F_SETPIPE_SZ = 1031 + bb.utils.check_system_locale() # Users shouldn't be running this code directly @@ -44,7 +57,6 @@ if sys.argv[1].startswith("decafbadbad"): # updates to log files for use with tail try: if sys.stdout.name == '': - import fcntl fl = fcntl.fcntl(sys.stdout.fileno(), fcntl.F_GETFL) fl |= os.O_SYNC fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, fl) @@ -56,6 +68,12 @@ logger = logging.getLogger("BitBake") worker_pipe = sys.stdout.fileno() bb.utils.nonblockingfd(worker_pipe) +# Try to make the pipe buffers larger as it is much more efficient. If we can't +# e.g. out of buffer space (/proc/sys/fs/pipe-user-pages-soft) then just pass over. +try: + fcntl.fcntl(worker_pipe, fcntl.F_SETPIPE_SZ, 512 * 1024) +except: + pass # Need to guard against multiprocessing being used in child processes # and multiple processes trying to write to the parent at the same time worker_pipe_lock = None @@ -105,7 +123,7 @@ def worker_flush(worker_queue): if not worker_queue.empty(): worker_queue_int.extend(worker_queue.get()) written = os.write(worker_pipe, worker_queue_int) - worker_queue_int = worker_queue_int[written:] + del worker_queue_int[0:written] except (IOError, OSError) as e: if e.errno != errno.EAGAIN and e.errno != errno.EPIPE: raise @@ -171,11 +189,8 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask): elif workerdata["umask"]: umask = workerdata["umask"] if umask: - # umask might come in as a number or text string.. - try: - umask = int(umask, 8) - except TypeError: - pass + # Convert to a python numeric value as it could be a string + umask = bb.utils.to_filemode(umask) dry_run = cfg.dry_run or runtask['dry_run'] @@ -278,9 +293,12 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask): bb.utils.disable_network(uid, gid) else: logger.debug("Skipping disable network for %s since %s is not a local uid." % (taskname, uid)) + bb.utils.landlock_restrict_network() - # exported_vars() returns a generator which *cannot* be passed to os.environ.update() - # successfully. We also need to unset anything from the environment which shouldn't be there + # exported_vars() expands each value, so it must be called before + # empty_environment() below: an expansion that shells out needs the + # environment bitbake was started with. We also need to unset + # anything from the environment which shouldn't be there exports = bb.data.exported_vars(the_data) bb.utils.empty_environment() @@ -357,7 +375,7 @@ class runQueueWorkerPipe(): def read(self): start = len(self.queue) try: - self.queue.extend(self.input.read(102400) or b"") + self.queue.extend(self.input.read(512*1024) or b"") except (OSError, IOError) as e: if e.errno != errno.EAGAIN: raise diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-shallow deleted file mode 100755 index 9de557c1..00000000 --- a/bitbake/bin/git-make-shallow +++ /dev/null @@ -1,175 +0,0 @@ -#!/usr/bin/env python3 -# -# Copyright BitBake Contributors -# -# SPDX-License-Identifier: GPL-2.0-only -# - -"""git-make-shallow: make the current git repository shallow - -Remove the history of the specified revisions, then optionally filter the -available refs to those specified. -""" - -import argparse -import collections -import errno -import itertools -import os -import subprocess -import sys -import warnings -warnings.simplefilter("default") - -version = 1.0 - - -git_cmd = ['git', '-c', 'safe.bareRepository=all'] - -def main(): - if sys.version_info < (3, 4, 0): - sys.exit('Python 3.4 or greater is required') - - git_dir = check_output(git_cmd + ['rev-parse', '--git-dir']).rstrip() - shallow_file = os.path.join(git_dir, 'shallow') - if os.path.exists(shallow_file): - try: - check_output(git_cmd + ['fetch', '--unshallow']) - except subprocess.CalledProcessError: - try: - os.unlink(shallow_file) - except OSError as exc: - if exc.errno != errno.ENOENT: - raise - - args = process_args() - revs = check_output(git_cmd + ['rev-list'] + args.revisions).splitlines() - - make_shallow(shallow_file, args.revisions, args.refs) - - ref_revs = check_output(git_cmd + ['rev-list'] + args.refs).splitlines() - remaining_history = set(revs) & set(ref_revs) - for rev in remaining_history: - if check_output(git_cmd + ['rev-parse', '{}^@'.format(rev)]): - sys.exit('Error: %s was not made shallow' % rev) - - filter_refs(args.refs) - - if args.shrink: - shrink_repo(git_dir) - subprocess.check_call(git_cmd + ['fsck', '--unreachable']) - - -def process_args(): - # TODO: add argument to automatically keep local-only refs, since they - # can't be easily restored with a git fetch. - parser = argparse.ArgumentParser(description='Remove the history of the specified revisions, then optionally filter the available refs to those specified.') - parser.add_argument('--ref', '-r', metavar='REF', action='append', dest='refs', help='remove all but the specified refs (cumulative)') - parser.add_argument('--shrink', '-s', action='store_true', help='shrink the git repository by repacking and pruning') - parser.add_argument('revisions', metavar='REVISION', nargs='+', help='a git revision/commit') - if len(sys.argv) < 2: - parser.print_help() - sys.exit(2) - - args = parser.parse_args() - - if args.refs: - args.refs = check_output(git_cmd + ['rev-parse', '--symbolic-full-name'] + args.refs).splitlines() - else: - args.refs = get_all_refs(lambda r, t, tt: t == 'commit' or tt == 'commit') - - args.refs = list(filter(lambda r: not r.endswith('/HEAD'), args.refs)) - args.revisions = check_output(git_cmd + ['rev-parse'] + ['%s^{}' % i for i in args.revisions]).splitlines() - return args - - -def check_output(cmd, input=None): - return subprocess.check_output(cmd, universal_newlines=True, input=input) - - -def make_shallow(shallow_file, revisions, refs): - """Remove the history of the specified revisions.""" - for rev in follow_history_intersections(revisions, refs): - print("Processing %s" % rev) - with open(shallow_file, 'a') as f: - f.write(rev + '\n') - - -def get_all_refs(ref_filter=None): - """Return all the existing refs in this repository, optionally filtering the refs.""" - ref_output = check_output(git_cmd + ['for-each-ref', '--format=%(refname)\t%(objecttype)\t%(*objecttype)']) - ref_split = [tuple(iter_extend(l.rsplit('\t'), 3)) for l in ref_output.splitlines()] - if ref_filter: - ref_split = (e for e in ref_split if ref_filter(*e)) - refs = [r[0] for r in ref_split] - return refs - - -def iter_extend(iterable, length, obj=None): - """Ensure that iterable is the specified length by extending with obj.""" - return itertools.islice(itertools.chain(iterable, itertools.repeat(obj)), length) - - -def filter_refs(refs): - """Remove all but the specified refs from the git repository.""" - all_refs = get_all_refs() - to_remove = set(all_refs) - set(refs) - if to_remove: - check_output(['xargs', '-0', '-n', '1'] + git_cmd + ['update-ref', '-d', '--no-deref'], - input=''.join(l + '\0' for l in to_remove)) - - -def follow_history_intersections(revisions, refs): - """Determine all the points where the history of the specified revisions intersects the specified refs.""" - queue = collections.deque(revisions) - seen = set() - - for rev in iter_except(queue.popleft, IndexError): - if rev in seen: - continue - - parents = check_output(git_cmd + ['rev-parse', '%s^@' % rev]).splitlines() - - yield rev - seen.add(rev) - - if not parents: - continue - - check_refs = check_output(git_cmd + ['merge-base', '--independent'] + sorted(refs)).splitlines() - for parent in parents: - for ref in check_refs: - print("Checking %s vs %s" % (parent, ref)) - try: - merge_base = check_output(git_cmd + ['merge-base', parent, ref]).rstrip() - except subprocess.CalledProcessError: - continue - else: - queue.append(merge_base) - - -def iter_except(func, exception, start=None): - """Yield a function repeatedly until it raises an exception.""" - try: - if start is not None: - yield start() - while True: - yield func() - except exception: - pass - - -def shrink_repo(git_dir): - """Shrink the newly shallow repository, removing the unreachable objects.""" - subprocess.check_call(git_cmd + ['reflog', 'expire', '--expire-unreachable=now', '--all']) - subprocess.check_call(git_cmd + ['repack', '-ad']) - try: - os.unlink(os.path.join(git_dir, 'objects', 'info', 'alternates')) - except OSError as exc: - if exc.errno != errno.ENOENT: - raise - subprocess.check_call(git_cmd + ['prune', '--expire', 'now']) - - -if __name__ == '__main__': - main() diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster index f002c8c1..16a64bab 100755 --- a/bitbake/bin/toaster +++ b/bitbake/bin/toaster @@ -164,14 +164,20 @@ fi # in the local layers that currently make using an arbitrary # toasterconf.json difficult. -. $OE_ROOT/.templateconf -if [ -n "$TEMPLATECONF" ]; then - if [ ! -d "$TEMPLATECONF" ]; then - # Allow TEMPLATECONF=meta-xyz/conf as a shortcut - if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then - TEMPLATECONF="$OE_ROOT/$TEMPLATECONF" +# BBSETUP no longer supports .templateconf +if [ -f $OE_ROOT/.templateconf ] ; then + . $OE_ROOT/.templateconf + if [ -n "$TEMPLATECONF" ]; then + if [ ! -d "$TEMPLATECONF" ]; then + # Allow TEMPLATECONF=meta-xyz/conf as a shortcut + if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then + TEMPLATECONF="$OE_ROOT/$TEMPLATECONF" + fi fi fi +else + # Force poky + TEMPLATECONF="layers/meta-yocto/meta-poky/conf/templates/default" fi unset OE_ROOT @@ -230,7 +236,7 @@ for param in $*; do manage_cmd="$manage_cmd $param" else echo "$HELP" - exit 1 + return 1 fi ;; @@ -255,7 +261,8 @@ fi # 2) the build dir (in build) # 3) the sqlite db if that is being used. # 4) pid's we need to clean up on exit/shutdown -export TOASTER_DIR=$TOASTERDIR +# By default we move this all into the Toaster's parent build directory +export TOASTER_DIR=$BUILDDIR export BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS TOASTER_DIR" # Determine the action. If specified by arguments, fine, if not, toggle it diff --git a/bitbake/bin/toaster-eventreplay b/bitbake/bin/toaster-eventreplay index 74a31932..742c2b78 100755 --- a/bitbake/bin/toaster-eventreplay +++ b/bitbake/bin/toaster-eventreplay @@ -31,6 +31,13 @@ sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'lib')) import bb.cooker from bb.ui import toasterui from bb.ui import eventreplay +from bb import __version__ as libbb_version + +__version__ = "2.20.0" + +if __name__ == "__main__": + if __version__ != libbb_version: + sys.exit("Bitbake core library version and program version mismatch!") def main(argv): with open(argv[-1]) as eventfile: diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf index f5a5a333..a8cd5c07 100644 --- a/bitbake/conf/bitbake.conf +++ b/bitbake/conf/bitbake.conf @@ -43,4 +43,4 @@ T = "${WORKDIR}/temp" TARGET_ARCH = "${BUILD_ARCH}" TMPDIR = "${TOPDIR}/tmp" WORKDIR = "${TMPDIR}/work/${PF}" -GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}" +GITPKGV = "${@bb.fetch.get_srcrev(d, 'gitpkgv_revision')}" diff --git a/bitbake/contrib/b4-wrapper-bitbake.py b/bitbake/contrib/b4-wrapper-bitbake.py new file mode 100755 index 00000000..87dff2c3 --- /dev/null +++ b/bitbake/contrib/b4-wrapper-bitbake.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# +# This script is to be called by b4: +# - through b4.send-auto-cc-cmd with "send-auto-cc-cmd" as first argument, +# +# When send-auto-cc-cmd is passed: +# +# This returns the list of Cc recipients for a patch. +# +# This script takes as stdin a patch. + +import subprocess +import sys + +cmd = sys.argv[1] +if cmd != "send-auto-cc-cmd": + sys.exit(-1) + +patch = sys.stdin.read() + +if subprocess.call(["which", "lsdiff"], stdout=subprocess.DEVNULL) != 0: + print("lsdiff missing from host, please install patchutils") + sys.exit(-1) + +files = subprocess.check_output(["lsdiff", "--strip-match=1", "--strip=1", "--include=doc/*"], + input=patch, text=True) +if len(files): + print("docs@lists.yoctoproject.org") +else: +# Handle patches made with --no-prefix + files = subprocess.check_output(["lsdiff", "--include=doc/*"], + input=patch, text=True) + if len(files): + print("docs@lists.yoctoproject.org") + +sys.exit(0) diff --git a/bitbake/contrib/hashserv/Dockerfile b/bitbake/contrib/hashserv/Dockerfile index 74b4a3be..5c8b8dd1 100644 --- a/bitbake/contrib/hashserv/Dockerfile +++ b/bitbake/contrib/hashserv/Dockerfile @@ -9,14 +9,13 @@ # docker build -f contrib/hashserv/Dockerfile . # -FROM alpine:3.13.1 +FROM alpine:3.23 -RUN apk add --no-cache python3 +RUN apk add --no-cache python3 py3-websockets libgcc COPY bin/bitbake-hashserv /opt/bbhashserv/bin/ COPY lib/hashserv /opt/bbhashserv/lib/hashserv/ COPY lib/bb /opt/bbhashserv/lib/bb/ -COPY lib/codegen.py /opt/bbhashserv/lib/codegen.py COPY lib/ply /opt/bbhashserv/lib/ply/ COPY lib/bs4 /opt/bbhashserv/lib/bs4/ diff --git a/bitbake/contrib/prserv/Dockerfile b/bitbake/contrib/prserv/Dockerfile index 9585fe3f..694b9b9f 100644 --- a/bitbake/contrib/prserv/Dockerfile +++ b/bitbake/contrib/prserv/Dockerfile @@ -28,14 +28,13 @@ # prserv # -FROM alpine:3.14.4 +FROM alpine:3.23 RUN apk add --no-cache python3 COPY bin/bitbake-prserv /opt/bbprserv/bin/ COPY lib/prserv /opt/bbprserv/lib/prserv/ COPY lib/bb /opt/bbprserv/lib/bb/ -COPY lib/codegen.py /opt/bbprserv/lib/codegen.py COPY lib/ply /opt/bbprserv/lib/ply/ COPY lib/bs4 /opt/bbprserv/lib/bs4/ diff --git a/bitbake/contrib/vim/LICENSE.txt b/bitbake/contrib/vim/LICENSE.txt deleted file mode 100644 index c7d91502..00000000 --- a/bitbake/contrib/vim/LICENSE.txt +++ /dev/null @@ -1,18 +0,0 @@ -The MIT License (MIT) - -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in -the Software without restriction, including without limitation the rights to -use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software is furnished to do so, -subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS -FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR -COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER -IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/bitbake/contrib/vim/ftdetect/bitbake.vim b/bitbake/contrib/vim/ftdetect/bitbake.vim deleted file mode 100644 index 09fc4dc7..00000000 --- a/bitbake/contrib/vim/ftdetect/bitbake.vim +++ /dev/null @@ -1,24 +0,0 @@ -" Vim filetype detection file -" Language: BitBake -" Author: Ricardo Salveti -" Copyright: Copyright (C) 2008 Ricardo Salveti -" Licence: You may redistribute this under the same terms as Vim itself -" -" This sets up the syntax highlighting for BitBake files, like .bb, .bbclass and .inc - -if &compatible || version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif - -" .bb, .bbappend and .bbclass -au BufNewFile,BufRead *.{bb,bbappend,bbclass} set filetype=bitbake - -" .inc -au BufNewFile,BufRead *.inc set filetype=bitbake - -" .conf -au BufNewFile,BufRead *.conf - \ if (match(expand("%:p:h"), "conf") > 0) | - \ set filetype=bitbake | - \ endif - diff --git a/bitbake/contrib/vim/ftplugin/bitbake.vim b/bitbake/contrib/vim/ftplugin/bitbake.vim deleted file mode 100644 index 9e8d3e13..00000000 --- a/bitbake/contrib/vim/ftplugin/bitbake.vim +++ /dev/null @@ -1,13 +0,0 @@ -" Only do this when not done yet for this buffer -if exists("b:did_ftplugin") - finish -endif - -" Don't load another plugin for this buffer -let b:did_ftplugin = 1 - -let b:undo_ftplugin = "setl cms< sts< sw< et< sua<" - -setlocal commentstring=#\ %s -setlocal softtabstop=4 shiftwidth=4 expandtab -setlocal suffixesadd+=.bb,.bbclass diff --git a/bitbake/contrib/vim/indent/bitbake.vim b/bitbake/contrib/vim/indent/bitbake.vim deleted file mode 100644 index 7ee9d699..00000000 --- a/bitbake/contrib/vim/indent/bitbake.vim +++ /dev/null @@ -1,343 +0,0 @@ -" Vim indent file -" Language: BitBake -" Copyright: Copyright (C) 2019 Agilent Technologies, Inc. -" Maintainer: Chris Laplante -" License: You may redistribute this under the same terms as Vim itself - - -if exists("b:did_indent") - finish -endif - -if exists("*BitbakeIndent") - finish -endif - -runtime! indent/sh.vim -unlet b:did_indent - -setlocal indentexpr=BitbakeIndent(v:lnum) -setlocal autoindent nolisp - -function s:is_bb_python_func_def(lnum) - let stack = synstack(a:lnum, 1) - if len(stack) == 0 - return 0 - endif - - let top = synIDattr(stack[0], "name") - echo top - - return synIDattr(stack[0], "name") == "bbPyFuncDef" -endfunction - -"""" begin modified from indent/python.vim, upstream commit 7a9bd7c1e0ce1baf5a02daf36eeae3638aa315c7 -"""" This copied code is licensed the same as Vim itself. -setlocal indentkeys+=<:>,=elif,=except - -let s:keepcpo= &cpo -set cpo&vim - -let s:maxoff = 50 " maximum number of lines to look backwards for () - -function! GetBBPythonIndent(lnum) - - " If this line is explicitly joined: If the previous line was also joined, - " line it up with that one, otherwise add two 'shiftwidth' - if getline(a:lnum - 1) =~ '\\$' - if a:lnum > 1 && getline(a:lnum - 2) =~ '\\$' - return indent(a:lnum - 1) - endif - return indent(a:lnum - 1) + (exists("g:pyindent_continue") ? eval(g:pyindent_continue) : (shiftwidth() * 2)) - endif - - " If the start of the line is in a string don't change the indent. - if has('syntax_items') - \ && synIDattr(synID(a:lnum, 1, 1), "name") =~ "String$" - return -1 - endif - - " Search backwards for the previous non-empty line. - let plnum = prevnonblank(v:lnum - 1) - - if plnum == 0 - " This is the first non-empty line, use zero indent. - return 0 - endif - - call cursor(plnum, 1) - - " Identing inside parentheses can be very slow, regardless of the searchpair() - " timeout, so let the user disable this feature if he doesn't need it - let disable_parentheses_indenting = get(g:, "pyindent_disable_parentheses_indenting", 0) - - if disable_parentheses_indenting == 1 - let plindent = indent(plnum) - let plnumstart = plnum - else - " searchpair() can be slow sometimes, limit the time to 150 msec or what is - " put in g:pyindent_searchpair_timeout - let searchpair_stopline = 0 - let searchpair_timeout = get(g:, 'pyindent_searchpair_timeout', 150) - - " If the previous line is inside parenthesis, use the indent of the starting - " line. - " Trick: use the non-existing "dummy" variable to break out of the loop when - " going too far back. - let parlnum = searchpair('(\|{\|\[', '', ')\|}\|\]', 'nbW', - \ "line('.') < " . (plnum - s:maxoff) . " ? dummy :" - \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')" - \ . " =~ '\\(Comment\\|Todo\\|String\\)$'", - \ searchpair_stopline, searchpair_timeout) - if parlnum > 0 - " We may have found the opening brace of a BitBake Python task, e.g. 'python do_task {' - " If so, ignore it here - it will be handled later. - if s:is_bb_python_func_def(parlnum) - let parlnum = 0 - let plindent = indent(plnum) - let plnumstart = plnum - else - let plindent = indent(parlnum) - let plnumstart = parlnum - endif - else - let plindent = indent(plnum) - let plnumstart = plnum - endif - - " When inside parenthesis: If at the first line below the parenthesis add - " two 'shiftwidth', otherwise same as previous line. - " i = (a - " + b - " + c) - call cursor(a:lnum, 1) - let p = searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW', - \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :" - \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')" - \ . " =~ '\\(Comment\\|Todo\\|String\\)$'", - \ searchpair_stopline, searchpair_timeout) - if p > 0 - if s:is_bb_python_func_def(p) - " Handle first non-empty line inside a BB Python task - if p == plnum - return shiftwidth() - endif - - " Handle the user actually trying to close a BitBake Python task - let line = getline(a:lnum) - if line =~ '^\s*}' - return -2 - endif - - " Otherwise ignore the brace - let p = 0 - else - if p == plnum - " When the start is inside parenthesis, only indent one 'shiftwidth'. - let pp = searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW', - \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :" - \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')" - \ . " =~ '\\(Comment\\|Todo\\|String\\)$'", - \ searchpair_stopline, searchpair_timeout) - if pp > 0 - return indent(plnum) + (exists("g:pyindent_nested_paren") ? eval(g:pyindent_nested_paren) : shiftwidth()) - endif - return indent(plnum) + (exists("g:pyindent_open_paren") ? eval(g:pyindent_open_paren) : (shiftwidth() * 2)) - endif - if plnumstart == p - return indent(plnum) - endif - return plindent - endif - endif - - endif - - - " Get the line and remove a trailing comment. - " Use syntax highlighting attributes when possible. - let pline = getline(plnum) - let pline_len = strlen(pline) - if has('syntax_items') - " If the last character in the line is a comment, do a binary search for - " the start of the comment. synID() is slow, a linear search would take - " too long on a long line. - if synIDattr(synID(plnum, pline_len, 1), "name") =~ "\\(Comment\\|Todo\\)$" - let min = 1 - let max = pline_len - while min < max - let col = (min + max) / 2 - if synIDattr(synID(plnum, col, 1), "name") =~ "\\(Comment\\|Todo\\)$" - let max = col - else - let min = col + 1 - endif - endwhile - let pline = strpart(pline, 0, min - 1) - endif - else - let col = 0 - while col < pline_len - if pline[col] == '#' - let pline = strpart(pline, 0, col) - break - endif - let col = col + 1 - endwhile - endif - - " If the previous line ended with a colon, indent this line - if pline =~ ':\s*$' - return plindent + shiftwidth() - endif - - " If the previous line was a stop-execution statement... - " TODO: utilize this logic to deindent when ending a bbPyDefRegion - if getline(plnum) =~ '^\s*\(break\|continue\|raise\|return\|pass\|bb\.fatal\)\>' - " See if the user has already dedented - if indent(a:lnum) > indent(plnum) - shiftwidth() - " If not, recommend one dedent - return indent(plnum) - shiftwidth() - endif - " Otherwise, trust the user - return -1 - endif - - " If the current line begins with a keyword that lines up with "try" - if getline(a:lnum) =~ '^\s*\(except\|finally\)\>' - let lnum = a:lnum - 1 - while lnum >= 1 - if getline(lnum) =~ '^\s*\(try\|except\)\>' - let ind = indent(lnum) - if ind >= indent(a:lnum) - return -1 " indent is already less than this - endif - return ind " line up with previous try or except - endif - let lnum = lnum - 1 - endwhile - return -1 " no matching "try"! - endif - - " If the current line begins with a header keyword, dedent - if getline(a:lnum) =~ '^\s*\(elif\|else\)\>' - - " Unless the previous line was a one-liner - if getline(plnumstart) =~ '^\s*\(for\|if\|try\)\>' - return plindent - endif - - " Or the user has already dedented - if indent(a:lnum) <= plindent - shiftwidth() - return -1 - endif - - return plindent - shiftwidth() - endif - - " When after a () construct we probably want to go back to the start line. - " a = (b - " + c) - " here - if parlnum > 0 - return plindent - endif - - return -1 - -endfunction - -let &cpo = s:keepcpo -unlet s:keepcpo - -""" end of stuff from indent/python.vim - - -let b:did_indent = 1 -setlocal indentkeys+=0\" - - -function! BitbakeIndent(lnum) - if !has('syntax_items') - return -1 - endif - - let stack = synstack(a:lnum, 1) - if len(stack) == 0 - return -1 - endif - - let name = synIDattr(stack[0], "name") - - " TODO: support different styles of indentation for assignments. For now, - " we only support like this: - " VAR = " \ - " value1 \ - " value2 \ - " " - " - " i.e. each value indented by shiftwidth(), with the final quote " completely unindented. - if name == "bbVarValue" - " Quote handling is tricky. kernel.bbclass has this line for instance: - " EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" " HOSTCPP="${BUILD_CPP}"" - " Instead of trying to handle crazy cases like that, just assume that a - " double-quote on a line by itself (following an assignment) means the - " user is closing the assignment, and de-dent. - if getline(a:lnum) =~ '^\s*"$' - return 0 - endif - - let prevstack = synstack(a:lnum - 1, 1) - if len(prevstack) == 0 - return -1 - endif - - let prevname = synIDattr(prevstack[0], "name") - - " Only indent if there was actually a continuation character on - " the previous line, to avoid misleading indentation. - let prevlinelastchar = synIDattr(synID(a:lnum - 1, col([a:lnum - 1, "$"]) - 1, 1), "name") - let prev_continued = prevlinelastchar == "bbContinue" - - " Did the previous line introduce an assignment? - if index(["bbVarDef", "bbVarFlagDef"], prevname) != -1 - if prev_continued - return shiftwidth() - endif - endif - - if !prev_continued - return 0 - endif - - " Autoindent can take it from here - return -1 - endif - - if index(["bbPyDefRegion", "bbPyFuncRegion"], name) != -1 - let ret = GetBBPythonIndent(a:lnum) - " Should normally always be indented by at least one shiftwidth; but allow - " return of -1 (defer to autoindent) or -2 (force indent to 0) - if ret == 0 - return shiftwidth() - elseif ret == -2 - return 0 - endif - return ret - endif - - " TODO: GetShIndent doesn't detect tasks prepended with 'fakeroot' - " Need to submit a patch upstream to Vim to provide an extension point. - " Unlike the Python indenter, the Sh indenter is way too large to copy and - " modify here. - if name == "bbShFuncRegion" - return GetShIndent() - endif - - " TODO: - " + heuristics for de-denting out of a bbPyDefRegion? e.g. when the user - " types an obvious BB keyword like addhandler or addtask, or starts - " writing a shell task. Maybe too hard to implement... - - return -1 -endfunction diff --git a/bitbake/contrib/vim/plugin/newbb.vim b/bitbake/contrib/vim/plugin/newbb.vim deleted file mode 100644 index 3a420273..00000000 --- a/bitbake/contrib/vim/plugin/newbb.vim +++ /dev/null @@ -1,88 +0,0 @@ -" Vim plugin file -" Purpose: Create a template for new bb files -" Author: Ricardo Salveti -" Copyright: Copyright (C) 2008 Ricardo Salveti -" -" This file is licensed under the MIT license, see COPYING.MIT in -" this source distribution for the terms. -" -" Based on the gentoo-syntax package -" -" Will try to use git to find the user name and email - -if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif - -fun! GetUserName() - let l:user_name = system("git config --get user.name") - if v:shell_error - return "Unknown User" - else - return substitute(l:user_name, "\n", "", "") -endfun - -fun! GetUserEmail() - let l:user_email = system("git config --get user.email") - if v:shell_error - return "unknown@user.org" - else - return substitute(l:user_email, "\n", "", "") -endfun - -fun! BBHeader() - let l:current_year = strftime("%Y") - let l:user_name = GetUserName() - let l:user_email = GetUserEmail() - 0 put ='# Copyright (C) ' . l:current_year . - \ ' ' . l:user_name . ' <' . l:user_email . '>' - put ='# Released under the MIT license (see COPYING.MIT for the terms)' - $ -endfun - -fun! NewBBTemplate() - if line2byte(line('$') + 1) != -1 - return - endif - - let l:paste = &paste - set nopaste - - " Get the header - call BBHeader() - - " New the bb template - put ='SUMMARY = \"\"' - put ='HOMEPAGE = \"\"' - put ='LICENSE = \"\"' - put ='SECTION = \"\"' - put ='DEPENDS = \"\"' - put ='' - put ='SRC_URI = \"\"' - - " Go to the first place to edit - 0 - /^SUMMARY =/ - exec "normal 2f\"" - - if paste == 1 - set paste - endif -endfun - -if !exists("g:bb_create_on_empty") - let g:bb_create_on_empty = 1 -endif - -" disable in case of vimdiff -if v:progname =~ "vimdiff" - let g:bb_create_on_empty = 0 -endif - -augroup NewBB - au BufNewFile,BufReadPost *.bb - \ if g:bb_create_on_empty | - \ call NewBBTemplate() | - \ endif -augroup END - diff --git a/bitbake/contrib/vim/plugin/newbbappend.vim b/bitbake/contrib/vim/plugin/newbbappend.vim deleted file mode 100644 index 3f65f79c..00000000 --- a/bitbake/contrib/vim/plugin/newbbappend.vim +++ /dev/null @@ -1,46 +0,0 @@ -" Vim plugin file -" Purpose: Create a template for new bbappend file -" Author: Joshua Watt -" Copyright: Copyright (C) 2017 Joshua Watt -" -" This file is licensed under the MIT license, see COPYING.MIT in -" this source distribution for the terms. -" - -if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif - -fun! NewBBAppendTemplate() - if line2byte(line('$') + 1) != -1 - return - endif - - let l:paste = &paste - set nopaste - - " New bbappend template - 0 put ='FILESEXTRAPATHS:prepend := \"${THISDIR}/${PN}:\"' - 2 - - if paste == 1 - set paste - endif -endfun - -if !exists("g:bb_create_on_empty") - let g:bb_create_on_empty = 1 -endif - -" disable in case of vimdiff -if v:progname =~ "vimdiff" - let g:bb_create_on_empty = 0 -endif - -augroup NewBBAppend - au BufNewFile,BufReadPost *.bbappend - \ if g:bb_create_on_empty | - \ call NewBBAppendTemplate() | - \ endif -augroup END - diff --git a/bitbake/contrib/vim/syntax/bitbake.vim b/bitbake/contrib/vim/syntax/bitbake.vim deleted file mode 100644 index 8f39b8f9..00000000 --- a/bitbake/contrib/vim/syntax/bitbake.vim +++ /dev/null @@ -1,131 +0,0 @@ -" Vim syntax file -" Language: BitBake bb/bbclasses/inc -" Author: Chris Larson -" Ricardo Salveti -" Copyright: Copyright (C) 2004 Chris Larson -" Copyright (C) 2008 Ricardo Salveti -" -" This file is licensed under the MIT license, see COPYING.MIT in -" this source distribution for the terms. -" -" Syntax highlighting for bb, bbclasses and inc files. -" -" It's an entirely new type, just has specific syntax in shell and python code - -if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif -if exists("b:current_syntax") - finish -endif - -syn include @python syntax/python.vim -if exists("b:current_syntax") - unlet b:current_syntax -endif - -" BitBake syntax - -" Matching case -syn case match - -" Indicates the error when nothing is matched -syn match bbUnmatched "." - -" Comments -syn cluster bbCommentGroup contains=bbTodo,@Spell -syn keyword bbTodo COMBAK FIXME TODO XXX contained -syn match bbComment "#.*$" contains=@bbCommentGroup - -" String helpers -syn match bbQuote +['"]+ contained -syn match bbDelimiter "[(){}=]" contained -syn match bbArrayBrackets "[\[\]]" contained - -" BitBake strings -syn match bbContinue "\\$" -syn region bbString matchgroup=bbQuote start=+"+ skip=+\\$+ end=+"+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell -syn region bbString matchgroup=bbQuote start=+'+ skip=+\\$+ end=+'+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell - -" Vars definition -syn match bbExport "^export" nextgroup=bbIdentifier skipwhite -syn keyword bbExportFlag export contained nextgroup=bbIdentifier skipwhite -syn match bbIdentifier "[a-zA-Z0-9\-_\.\/\+]\+" display contained -syn match bbVarDeref "${[a-zA-Z0-9\-_:\.\/\+]\+}" contained -syn match bbVarEq "\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)" contained nextgroup=bbVarValue -syn match bbVarDef "^\(export\s*\)\?\([a-zA-Z0-9\-_\.\/\+][${}a-zA-Z0-9\-_:\.\/\+]*\)\s*\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)\@=" contains=bbExportFlag,bbIdentifier,bbOverrideOperator,bbVarDeref nextgroup=bbVarEq -syn match bbVarValue ".*$" contained contains=bbString,bbVarDeref,bbVarPyValue -syn region bbVarPyValue start=+${@+ skip=+\\$+ end=+}+ contained contains=@python - -" Vars metadata flags -syn match bbVarFlagDef "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\.+]\+\]\)\@=" contains=bbIdentifier nextgroup=bbVarFlagFlag -syn region bbVarFlagFlag matchgroup=bbArrayBrackets start="\[" end="\]\s*\(:=\|=\|.=\|=.|+=\|=+\|?=\)\@=" contained contains=bbIdentifier nextgroup=bbVarEq - -" Includes and requires -syn keyword bbInclude inherit include require contained -syn match bbIncludeRest ".*$" contained contains=bbString,bbVarDeref,bbVarPyValue -syn match bbIncludeLine "^\(inherit\|include\|require\)\s\+" contains=bbInclude nextgroup=bbIncludeRest - -" Add taks and similar -syn keyword bbStatement addtask deltask addhandler after before EXPORT_FUNCTIONS contained -syn match bbStatementRest /[^\\]*$/ skipwhite contained contains=bbStatement,bbVarDeref,bbVarPyValue -syn region bbStatementRestCont start=/.*\\$/ end=/^[^\\]*$/ contained contains=bbStatement,bbVarDeref,bbVarPyValue,bbContinue keepend -syn match bbStatementLine "^\(addtask\|deltask\|addhandler\|after\|before\|EXPORT_FUNCTIONS\)\s\+" contains=bbStatement nextgroup=bbStatementRest,bbStatementRestCont - -" OE Important Functions -syn keyword bbOEFunctions do_fetch do_unpack do_patch do_configure do_compile do_stage do_install do_package contained - -" Generic Functions -syn match bbFunction "\h[0-9A-Za-z_\-\.]*" display contained contains=bbOEFunctions - -syn keyword bbOverrideOperator append prepend remove contained - -" BitBake shell metadata -syn include @shell syntax/sh.vim -if exists("b:current_syntax") - unlet b:current_syntax -endif -syn keyword bbShFakeRootFlag fakeroot contained -syn match bbShFuncDef "^\(fakeroot\s*\)\?\([\.0-9A-Za-z_:${}\-\.]\+\)\(python\)\@`__: + + .. code-block:: shell + + $ python3 -m venv --clear ./bitbake-setup-venv + $ . ./bitbake-setup-venv/bin/activate + $ pip install bitbake-setup + + .. tip:: + + To update ``bitbake-setup`` when already installed, you can run: + + .. code-block:: console + + $ pip install --upgrade bitbake-setup + +#. To start using ``bitbake-setup``, run: + + .. code-block:: shell + + $ bitbake-setup init + + This command will ask you to choose which configurations to use available as + part of the default BitBake :term:`Configuration Templates `. + + .. note:: + + These default configurations are located in ``./bitbake/default-registry/``. + See the :ref:`ref-bbsetup-section-config-reference` section to learn more + about ``bitbake-setup`` input configuration files. + +#. Here is an example of files and directories structure created by ``bitbake-setup init``:: + + ./bitbake-builds/ + ├── site.conf + └── poky-master-poky-distro_poky-machine_qemux86-64/ +    ├── build/ +    ├── config/ +    └── layers/ + + With: + + - ``./bitbake-builds``: the :term:`Top Directory`, where ``bitbake-setup`` + configures everything. This directory can be configured with the + :ref:`ref-bbsetup-setting-top-dir-prefix` and + :ref:`ref-bbsetup-setting-top-dir-name` settings. + + - ``poky-master-poky-distro_poky-machine_qemux86-64``: a :term:`Setup` + directory, which holds a :term:`Setup`: a result of the choices made + during the ``bitbake-setup init`` execution. + + The name of the directory will vary depending on the choices. + + - ``config/``: holds the :term:`Configuration Instance`, which embeds the + :term:`Configuration Template` (first choice of the :ref:`ref-bbsetup-command-init` command) + and the choices made during the initialization. + + - ``build/``: the :term:`BitBake Build` directory, where BitBake stores + its own configuration and outputs for the builds. + + - ``layers/``: the directory where :ref:`layers + ` and other + repositories managed by ``bitbake-setup`` are stored and updated. + + - ``site.conf``: a BitBake configuration file that contains site specific + configurations for your build environment. When it is created, it contains + some variable definitions that are based on your current :term:`settings`. + Comments in this file will help you understand what these variables + correspond to. + + These configurations are shared between the :term:`setups `. + +#. Source the ``init-build-env`` file present in the :term:`BitBake Build` + directory: + + .. code-block:: shell + + $ source ./poky-master-poky-distro_poky-machine_qemux86-64/build/init-build-env + + This command will prepare your current shell with the BitBake environment. + +#. You can then start running ``bitbake`` in the current shell. For more information + on how to use ``bitbake``, see the :doc:`/bitbake-user-manual/bitbake-user-manual-execution` + section of this manual. + +.. note:: + + The ``bitbake-setup`` tool is developed and maintained in the BitBake + repository and can also be used directly from there: + + .. code-block:: console + + $ git clone https://git.openembedded.org/bitbake ./bitbake + $ ./bitbake/bin/bitbake-setup ... + +Terminology +=========== + +The ``bitbake-setup`` tool revolves around some common terms we define in this +section. + +``bitbake-setup`` works with a specific hierarchy of files and directories, that +can be represented as follows:: + + Top Directory + ├── site.conf + ├── Setup 1 + │   ├── build/ + │ ├── config/ + │   └── layers/ + ├── Setup 2 + │   ├── build/ + │ ├── config/ + │   └── layers/ + ... + +The "Top Directory" and "Setup" directories are defined as follows: + +.. glossary:: + :term:`Top Directory` + The top directory is the working directory of ``bitbake-setup``, where its + outputs end-up (unless otherwise configured by :term:`settings` such as + :ref:`ref-bbsetup-setting-dl-dir`). + + The location of this directory can be changed with the + :ref:`ref-bbsetup-setting-top-dir-prefix` and + :ref:`ref-bbsetup-setting-top-dir-name` settings. + + The top directory contains one or more :term:`Setup` directories, each of + them containing a :term:`Setup`, and a :term:`Site Configuration File` + (named ``site.conf``). + + :term:`Setup` + A Setup is the result of the :ref:`ref-bbsetup-command-init` + command, which creates a :term:`Setup` directory. It is constructed from a + :term:`Configuration Template` and choices made during the ``init`` command. + + It contains at least: + + - A :term:`BitBake Build` (``build/`` directory). + - A :term:`Configuration Instance` (``config/`` directory). + - Sources such as :ref:`layers + ` or other + repositories managed by ``bitbake-setup`` (``layers/`` directory). + + :term:`Site Configuration File` + This file named ``site.conf`` is a unique file located in the :term:`Top + Directory`, and holds top-level BitBake configuration statements shared + between the :term:`Setups `. + + When it is created, it contains some variable definitions that are based + on your current :term:`settings`. Comments in this file will help you + understand what these variables correspond to. + +The following components are involved to create the content of these directories: + +.. glossary:: + :term:`BitBake Build` + A BitBake Build is a sub-tree inside a :term:`Setup` that BitBake itself + operates on. The files in the ``conf/`` directory of a :term:`BitBake + Build` constitute the :ref:`BitBake configuration + `. + + :term:`Configuration Template` + A Configuration Template is a file in JSON format containing a template to + create a :term:`Setup`. These files are used during the :ref:`ref-bbsetup-command-init` + command as a starting point to configure the :term:`Setup`. When the + command runs, the user may be prompted with choices to further specify the + :term:`Setup` to create. + + It is also possible to specify the choices on the command line for a + completely non-interactive initialization. + + :term:`Configuration Template` files are stored in :term:`registries + `, and can be listed with the :ref:`ref-bbsetup-command-list` + command. + + :term:`Configuration Template` files must end with the ``.conf.json`` + suffix for ``bitbake-setup`` to locate them. + + .. note:: + + The default :term:`Configuration Templates ` are + located in the BitBake repository in a local registry. the + ``default-registry/`` directory. This can be modified with the + :ref:`ref-bbsetup-setting-registry` setting. + + :ref:`ref-bbsetup-command-status` will tell if a :term:`Setup` + is in sync with the :term:`Configuration Template` it was constructed from + (typically: layer updates). + + :ref:`ref-bbsetup-command-update` will bring a :term:`Setup` + in sync with its :term:`Configuration Template`. + + :term:`Configuration Instance` + The :term:`Configuration Instance` is stored in the ``config/`` directory + in a :term:`Setup`. It embeds the :term:`Configuration Template` and the + choices made during the initialization. + + It is also a Git repository, that contains a history of the configuration + instance and updates made to it via :ref:`ref-bbsetup-command-update`. + + :term:`Registry` + A configuration registry is a place where one or more :term:`Configuration + Templates ` are stored. + + The directory structure of the registry can be any: ``bitbake-setup`` + recursively find files ending with ``.conf.json`` and consider it a + :term:`Configuration Template`. + + The registry location is configured through the + :ref:`ref-bbsetup-setting-registry` setting. This location can be the URL to + a Git repository, a local directory, or any URI supported by the BitBake + fetcher (see the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` + section for more information on fetchers). + + :term:`Settings` + Settings are operational parameters that are global to all builds under a + :term:`Top Directory`, stored in a ``settings.conf`` file. For example, + this could be the location of the configuration registry, or where the + BitBake fetcher should store the downloads. + + There are also global settings, common to all top directories that are + stored in ``~/.config/bitbake-setup/settings.conf``. + + See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings` + section to see the supported settings and where they can be stored. + + :term:`Source Override` + A source override is a JSON file that can be used to modify revisions and + origins of layers or other sources that need to be checked out into a + :term:`Setup` (in the ``layers/`` directory). It can be useful for example + when the master branches need to be changed to master-next for the purpose + of testing, or to set up a CI pipeline that tests code in a pull request + coming from a developer's repository and branch. + + Such a file is specified with a command-line option to + :ref:`ref-bbsetup-command-init`. + + See the :ref:`ref-bbsetup-source-overrides` section for more information on + the format of these files. + +The ``bitbake-setup`` command +============================= + +The ``bitbake-setup`` program has general options and sub-commands. These can be +obtained using ``bitbake-setup --help``. + +The general options, common to all commands, are: + +- ``-h`` or ``--help``: Show the help message and exit. +- ``-d`` or ``--debug``: Enable debug outputs. +- ``-q`` or ``--quiet``: Print only errors. +- ``--color``: Colorize output (where COLOR is auto, always, never). +- ``--no-network``: Do not check whether configuration repositories and layer + repositories have been updated; use only the local cache. +- ``--global-settings``: Path to the global settings file. +- ``--setting``: Modify a setting (for this bitbake-setup invocation only). + For example ``--setting default top-dir-prefix /path/to/top-dir``. + +.. _ref-bbsetup-command-init: + +``bitbake-setup init`` +---------------------- + +The ``bitbake-setup init`` sub-command helps initializing a :term:`Setup`. + +This command can be run without any arguments to prompt the user with +configuration options to choose from. These configuration options are taken from +the input :term:`Configuration Template` files in the :term:`registry`. + +.. note:: + + The registry location can be set with the :ref:`ref-bbsetup-setting-registry` + setting and the :ref:`ref-bbsetup-command-settings` command. + +Otherwise, the first argument to :ref:`ref-bbsetup-command-init` can be: + +- A Configuration Template ID in the registry. +- A path to a Configuration Template file on a local disk. +- An HTTP URI to the Configuration Template file. + +The choices made during the bare ``bitbake-setup init`` command can also be +passed directly on the command-line, for example:: + + bitbake-setup init poky distro/poky-tiny ... + +``bitbake-setup`` will stop and ask to make a choice if the above command does +not contain all of the required configurations to complete the sequence of +choices. + +In addition, the command can take the following arguments: + +- ``--non-interactive``: can be used to create :term:`Setups ` + without interactions from the user. The command will fail if not all the + required choices are provided in the command. + +- ``--source-overrides``: can be used to pass one or more + :ref:`source override `. See the + :ref:`ref-bbsetup-source-overrides` section. + +- ``--setup-dir-name``: can be used to configure the name of the :term:`Setup` + directory. + +- ``--skip-selection``: can be used to skip some of the choices + (which may result in an incomplete :term:`Setup`!) + +- ``-L`` or ``--use-local-source``: instead of getting a source as prescribed in + a configuration, symlink it into a :term:`Setup` from a path on local disk. This + is useful for local development where that particular source directory is managed + separately, and bitbake-setup will include it in a build but will not otherwise + touch or modify it. This option can be specified multiple times to specify multiple + local sources. + + The option can be seen as a command line shortcut to providing an override file + with a ``local`` source in it. See the :ref:`ref-bbsetup-source-overrides` section + for more information on source overrides. + +- ``--init-vscode`` / ``--no-init-vscode``: generate (or skip generating) a + ``bitbake.code-workspace`` file in the :term:`Setup` directory. The workspace + file configures the `Yocto Project BitBake + `_ + VS Code extension with paths to the build directory and the init script, and + lists the layer directories as workspace folders. Any user-added folders or + settings in an existing workspace file are preserved across updates. + The default is ``true`` when ``code`` (the VS Code binary) is found on + ``PATH``, and ``false`` otherwise. + +``bitbake-setup init`` Examples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- This example prompts the user to choose a :term:`Configuration Template` from + a custom registry: + + .. code-block:: shell + + $ bitbake-setup \ + --setting default registry 'git://example.com/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' \ + init + +- This example takes a :term:`Configuration Template` from a remote location + (here, one of the default configuration in BitBake): + + .. code-block:: shell + + $ bitbake-setup init https://git.openembedded.org/bitbake/plain/default-registry/configurations/oe-nodistro-master.conf.json + +- This example initializes a :term:`Setup` with: + + - A custom :ref:`ref-bbsetup-setting-top-dir-prefix` and :ref:`ref-bbsetup-setting-top-dir-name` + - A :term:`source override`, and additionally overriding openembedded-core with a locally managed directory. + - A standalone :term:`Configuration Template` file. + - Choices passed on the command-line, applied non-interactively. + + .. code-block:: shell + + $ bitbake-setup \ + --setting default top-dir-prefix /work/bitbake-setup \ + --setting default top-dir-name custom-project \ + init \ + --non-interactive \ + --source-overrides develop-branch.json \ + -L openembedded-core ~/development/openembedded-core-gadget \ + ./gadget_master.conf.json \ + gadget distro/gadget machine/gadget + +.. _ref-bbsetup-command-list: + +``bitbake-setup list`` +---------------------- + +The ``bitbake-setup list`` sub-command lists the available :term:`Configuration +Templates ` in the current :term:`registry`. + +In addition, the command can take the following arguments: + +- ``--with-expired``: list the expired configuration (e.g. older Yocto releases + that have reached their End-Of-Life dates). + +- ``--write-json``: write the configurations into a JSON file so they can be + programmatically processed. + +.. _ref-bbsetup-command-status: + +``bitbake-setup status`` +------------------------ + +The ``bitbake-setup status`` sub-command shows the status of a +:term:`Setup`. Any differences between the local copy of the :term:`Configuration +Template` and the upstream one are printed on the console. + +If the BitBake environment is sourced and ready to build, the ``bitbake-setup +status`` command (without any arguments) will show the status of the current +:term:`Setup`. + +In addition, the command can take the following arguments: + +- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not + required if the command is invoked from an initialized BitBake environment + that contains :term:`BBPATH`. + +.. _ref-bbsetup-command-update: + +``bitbake-setup update`` +------------------------ + +The ``bitbake-setup update`` sub-command updates a :term:`Setup` to sync with +the latest changes from the :term:`Configuration Template` it was constructed from. +The :ref:`ref-bbsetup-command-status` command can be used to show the current +status of the :term:`Setup` before updating it. + +This command is intended to be run from a shell where the BitBake environment +has been sourced (e.g. after ``source build/init-build-env``), so that +``bitbake-setup`` can automatically identify the current :term:`Setup` without +requiring the ``--setup-dir`` argument. + +When a layer repository already exists in the :term:`Setup` (i.e. it has been +previously checked out by ``bitbake-setup init`` or a prior ``update``), the +update is performed *in place* using the fetcher's +:ref:`unpack_update ` method: the new upstream revision +is fetched into the local download cache and then rebased on top of the +checkout's current HEAD. This means any local commits in the layer directory +are preserved and rebased onto the new upstream revision. If the working tree +contains staged or unstaged changes to tracked files, the update is blocked +until those changes are committed, stashed or discarded. + +.. note:: + + ``bitbake-setup`` performs the rebase to fast-forward local commits onto + the new upstream revision, but intentionally does not go further by using + ``--autostash`` (which would silently stash uncommitted changes before the + rebase and pop them afterwards). Any uncommitted modifications are surfaced + to the user before the update proceeds, so there are no surprises from an + automatic stash/pop cycle. + +In addition, the command can take the following arguments: + +- ``--update-bb-conf``: whether to update the :term:`BitBake Build` + configuration (``local.conf``, ``bblayers.conf``, etc.). This argument can + take up to three values: + + - ``prompt`` (default): ask the user whether to update. + - ``yes``: update the configuration files. + - ``no``: don't update the configuration files. + +- ``--rebase-conflicts-strategy``: what to do when a layer repository has + local modifications or commits that prevent an in-place update. Accepted + values are: + + - ``abort`` (default): stop with an error message describing the problem. + The repository is left in its previous state (the failed rebase is + automatically aborted). The error message includes a hint to re-run with + ``--rebase-conflicts-strategy=backup``. + - ``backup``: rename the conflicting layer directory to a timestamped + ``-backup-`` path (preserving local work), then + re-clone the layer from upstream into a fresh directory. + +- ``--setup-dir``: path to the :term:`Setup` to update. Not required if the + command is invoked from an initialized BitBake environment that contains + :term:`BBPATH`. + +``bitbake-setup update`` Examples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- This example performs a standard update from an initialized BitBake + environment. Local commits in any layer directory are rebased on top of + the new upstream revision and preserved: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-core + checked out into /path/to/bitbake-builds/poky-master/layers/openembedded-core + updated revision master from d383ea3... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + +- This example shows what happens when a layer directory contains staged or + unstaged changes to tracked files. The update is blocked with an error: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + ERROR: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core has uncommitted changes, unable to update: + M meta/recipes-devtools/ccache/ccache_4.13.1.bb + + Commit, stash or discard your changes and re-run the update. + Use 'bitbake-setup update --rebase-conflicts-strategy=backup' + to automatically back up the directory and re-clone from upstream, + or use 'bitbake-setup init -L openembedded-core /path/to/local/checkout' + to work with a local checkout instead. + + Stashing the changes and re-running resolves the issue: + + .. code-block:: shell + + $ git -C layers/openembedded-core stash + $ bitbake-setup update + $ git -C layers/openembedded-core stash pop + +- This example shows what happens when a layer directory contains local + commits that conflict with the incoming upstream changes. The failed rebase + is automatically aborted, and the ``dldir`` remote is left in the repository + for manual resolution: + + .. code-block:: shell + + $ bitbake-setup update + ERROR: Repository at layers/openembedded-core has local commits that could + not be rebased onto the new upstream revision: + ... + Note: the 'dldir' remote points to the local download cache and may be + used to resolve the conflict manually. + Once resolved, re-run the update. + + The conflict can be resolved manually using the ``dldir`` remote that + ``bitbake-setup`` adds to the repository: + + .. code-block:: shell + + $ git -C layers/openembedded-core rebase dldir/master + # fix conflicts in an editor, then stage the resolved files: + $ git -C layers/openembedded-core add meta/recipes-core/base-files/base-files.bb + $ git -C layers/openembedded-core rebase --continue + $ bitbake-setup update + +- When manual conflict resolution is not desired, the + ``--rebase-conflicts-strategy=backup`` option can be used instead. It + preserves the conflicting directory under a timestamped backup path and + re-clones the layer cleanly from upstream: + + .. code-block:: shell + + $ bitbake-setup update --rebase-conflicts-strategy=backup + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-core checked + out into /path/to/bitbake-builds/poky-master/layers/openembedded-core + updated revision master from 2ec283e... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + WARNING: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core + has local commits that could not be rebased onto the new upstream revision: + ... + Note: the 'dldir' remote points to the local download cache and may be used to resolve the conflict manually. + Once resolved, re-run the update. + Renaming /path/to/bitbake-builds/poky-master/layers/openembedded-core to + /path/to/bitbake-builds/poky-master/layers/openembedded-core-backup.20260329160426 + to preserve your work, then re-cloning from upstream. + + The backup directory is a complete git repository. Local commits can be + recovered from it after the update by fetching a branch from the backup + into the fresh clone (git accepts local paths as remote URLs) and then + cherry-picking the desired commits. For example, given a ``my-wip`` branch + with two commits existing in the backup repository and not in the fresh clone, + the following commands can be used to apply these commits on top of the new + upstream revision in the fresh clone: + + .. code-block:: shell + + $ git -C layers/openembedded-core-backup.20260329160426 log --oneline my-wip + a1b2c3d u-boot: fix compilation with newer GCC + 2ec283e base-files: update version + ... + + $ cd layers/openembedded-core + $ git checkout -b my-wip + Switched to a new branch 'my-wip' + $ git fetch ../openembedded-core-backup.20260329160426 my-wip + + $ git cherry-pick 2ec283e + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-core/base-files/base-files.bb + $ git cherry-pick --continue + + $ git cherry-pick a1b2c3d + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-devtools/u-boot/u-boot_2026.04.bb + $ git cherry-pick --continue + + $ cd ../.. + + The sequence above is: + + #. Inspect the backup's branch history to identify the commits to recover. + #. Change into the fresh clone and create a matching branch. + #. Fetch the backup branch so its objects become available locally. Git + accepts filesystem paths as remote URLs, and from inside + ``layers/openembedded-core/``, ``../`` points to ``layers/``, where the + backup directory sits. + #. Cherry-pick the commits in oldest-first order. + + Once all desired commits have been recovered and verified, the backup + directory can be removed: + + .. code-block:: shell + + $ rm -rf layers/openembedded-core-backup.20260329160426 + + If a VSCode workspace is in use, the backup directory will appear as an + additional workspace folder until it is cleaned up. It can be removed from + the workspace via the VS Code UI by right-clicking the folder and selecting + *Remove Folder from Workspace*. + +.. _ref-bbsetup-command-install-buildtools: + +``bitbake-setup install-buildtools`` +------------------------------------ + +The ``bitbake-setup install-buildtools`` sub-command installs and extracts a +buildtools tarball into the specified :term:`Setup`. + +After completion, help is printed to help the user on how to use the extracted +tarball. + +.. note:: + + The purpose of the Buildtools tarball is to provide tools needed to run + BitBake on build machines where such tools cannot be easily obtained from the + host Linux distribution (for example on older distribution versions that do + not contain a recent enough GCC compiler or Python interpreter, or machines + where the user running BitBake cannot easily install additional software into + the system). This command requires that the OpenEmbedded-core layer is + present in the BitBake configuration. + + See https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions + for more information. + +In addition, the command can take the following arguments: + +- ``--force``: force the re-installation of the tarball. + +- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not + required if :term:`BBPATH` is already configured. + +- ``--url``: full URL to the buildtools SDK installer. Overrides the value + from the configuration file. + +- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the + value from the configuration file. + +When ``url`` and ``sha256sum`` are set in the configuration file, +the installer is downloaded via ``bb.fetch`` (cached in :term:`DL_DIR`) and its +checksum is enforced. If no configuration is present, the script falls back +to its built-in defaults:: + + "bitbake-setup": { + "configurations": [ + { + "name": "my-config", + "install-buildtools": { + "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh", + "sha256sum": "af76648b..." + }, + ... + } + ] + } + +.. _ref-bbsetup-command-settings: + +``bitbake-setup settings`` +-------------------------- + +The ``bitbake-setup settings`` sub-command helps modifying the settings of +``bitbake-setup``. This sub-command has sub-commands itself: + +- ``bitbake-setup settings list`` lists the current settings and their value. +- ``bitbake-setup settings set`` sets a setting. +- ``bitbake-setup settings unset`` removes a setting. + +Settings must be set with a section and a value, for example:: + + bitbake-setup settings set default top-dir-name bitbake-builds + +Will set the value of ``top-dir-name`` in the ``default`` section to +"bitbake-builds". + +In addition, the command can take the following arguments: + +- ``--global``: write to the global settings + (``~/.config/bitbake-setup/settings.conf``) instead of the :term:`Top + Directory` settings. + +See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings` +section to see the supported settings. + +.. note:: + + The supported setting listed in the + :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings` + section are only affected when set in the ``default`` section. + +Settings +======== + +The settings allow configuring ``bitbake-setup``. Settings are stored in a file +named ``settings.conf``, in :wikipedia:`INI ` format. + +There are multiple locations for storing settings. Settings in different +locations can override each other, but the final value of a setting is computed +from reading the files in this order: + +#. Global settings file: ``~/.config/bitbake-setup/settings.conf``. + +#. Local settings file, taken from a ``settings.conf`` file in the :term:`Top + Directory`. + +#. Command-line settings, passed with the ``--setting`` argument. + +A valid settings file would for example be: + +.. code-block:: ini + + [default] + top-dir-prefix = /path/to/workspace + top-dir-name = bitbake-builds + registry = /path/to/bitbake/default-registry + dl-dir = /path/to/bitbake-setup-downloads + use-full-setup-dir-name = yes + common-sstate = yes + +Settings and their values can be listed and modified with the ``bitbake-setup +settings`` command. See the :ref:`ref-bbsetup-command-settings` section for +more information. + +Below are the available settings. + +.. _ref-bbsetup-setting-top-dir-prefix: + +``top-dir-prefix`` +------------------ + +The :ref:`ref-bbsetup-setting-top-dir-prefix` setting helps configuring the +leftmost part of the path to the :term:`Top Directory`. + +For example, with: + +.. code-block:: ini + + [default] + top-dir-prefix = /path/to/workspace + +The :term:`top directory` would be ``/path/to/workspace/`` with +the ```` corresponding to the :ref:`ref-bbsetup-setting-top-dir-name` +setting. + +This is most useful to customize on systems where the default location of the +:term:`Top Directory` (``~/bitbake-builds``) is not suitable, and there is a +dedicated directory for builds somewhere else. + +.. _ref-bbsetup-setting-top-dir-name: + +``top-dir-name`` +---------------- + +The :ref:`ref-bbsetup-setting-top-dir-name` setting helps configuring the +rightmost part of the path to the :term:`Top Directory`. + +For example, with: + +.. code-block:: ini + + [default] + top-dir-name = builds + +The :term:`top directory` would be ``/builds`` with +the ```` corresponding to the :ref:`ref-bbsetup-setting-top-dir-prefix` +setting. + +.. _ref-bbsetup-setting-registry: + +``registry`` +------------ + +The :ref:`ref-bbsetup-setting-registry` setting sets the URI location of the +registry. This URI can be any URI supported by the BitBake fetcher. + +A local registry would be configured as follows: + +.. code-block:: ini + + [default] + registry = /path/to/registry + +When using another fetcher, it must be specified in the URI scheme. For example: + +.. code-block:: ini + + [default] + registry = git://example.com/bitbake-setup-configurations;protocol=https;branch=master;rev=master + +This would fetch the remote configurations from a remote Git remote repository, +on the ``master`` branch. + +See the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` section for more +information on BitBake fetchers. + +.. _ref-bbsetup-setting-dl-dir: + +``dl-dir`` +---------- + +The :ref:`ref-bbsetup-setting-dl-dir` setting sets the location of the download +cache that ``bitbake-setup`` will configure for the purpose of downloading +configuration repositories, layers and other sources using BitBake fetchers. +Please see :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` and the +:term:`DL_DIR` variable for more information. + +The location can be set such that it is shared with :term:`DL_DIR` specified by +BitBake builds, so that there is a single directory containing a copy of +everything needed to set up and run a BitBake build offline in a reproducible +manner. + +.. _ref-bbsetup-setting-use-full-setup-dir-name: + +``use-full-setup-dir-name`` +--------------------------- + +The :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting, if set to ``yes`` +will override the suggestions for the :term:`Setup` directory name made by +``setup-dir-name`` entries in :term:`Configuration Template` files. This +will make the directory names longer, but fully specific: they will contain +all selections made during initialization. + +.. _ref-bbsetup-setting-common-sstate: + +``common-sstate`` +----------------- + +When this setting is set to ``yes`` (which is also the default), bitbake-setup will +set up a common sstate directory and common hash equivalency database for all the +:term:`setups ` in a :term:`Top Directory`. This is very beneficial for speeding +up builds as build artefacts will be reused whenever possible between them. + +Set this to ``no`` for advanced use cases, such as placing the sstate directory on a NFS +mount and maintaining a separate hash equivalency server, so that sstate and hash equivalency +data can be shared between several computers. For such use cases the sstate settings need +to be added to a build configuration separately. + +See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for how to share sstate +on the network. + +.. _ref-bbsetup-section-config-reference: + +Configuration Template Files Reference +====================================== + +:term:`Configuration Templates ` are the input files given +to ``bitbake-setup`` to configure :term:`Setups `. + +These files are written in the JSON file format and are stored in a +:term:`Registry`. They can also be standalone files directly passed to the +:ref:`ref-bbsetup-command-init` command: + +.. code-block:: shell + + $ bitbake-setup init /path/to/config.conf.json + +They contain the following sections: + +- ``version`` (**required**): version of the configuration file. + + Example: + + .. code-block:: json + :force: + + { + "version": "1.0" + } + +- ``description`` (**required**): the description of the configuration. + + Example: + + .. code-block:: json + :force: + + { + "description": "OpenEmbedded - 'nodistro' basic configuration" + } + +- ``sources`` (**required**): sources, such as git repositories that should + be provided under ``layers/`` directory of a :term:`Setup`. Although the + list of sources can be very flexible, at least BitBake should be cloned + to generate a :term:`Setup`. + + Example: + + .. code-block:: json + :force: + + { + "sources": { + "bitbake": { + "git-remote": { + "uri": "https://git.openembedded.org/bitbake", + "branch": "master", + "rev": "master" + }, + "path": "bitbake" + }, + "openembedded-core": { + "local": { + "path": "~/path/to/local/openembedded-core" + } + }, + "meta-private": { + "git-remote": { + "remotes": { + "origin": { + "uri": "ssh://git@git.example.com/meta-private" + } + }, + "branch": "master", + "rev": "master" + }, + "path": "meta-private" + } + } + } + + Sources can be specified with the following options: + + - ``path`` (*optional*): where the source is extracted, relative to the + ``layers/`` directory of a :term:`Setup`. If unspecified, the name of the + source is used. + + - ``git-remote`` (*optional*): specifies URI, branch and revision of a git + repository to fetch from. + + ``git-remote`` entries are specified with the following options: + + - ``remotes`` (*optional*): a dictionary of git remote names, each containing a ``uri`` property + with a URI that follows the git URI syntax. See https://git-scm.com/docs/git-clone#_git_urls + and https://git-scm.com/docs/git-remote for more information. + + This structure is useful if you want to provide several remotes. Please + note that each URI will be cloned, but only the the last treated one + will be used in the unpacked source. + + Example: + + .. code-block:: json + :force: + + { + "sources": { + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + }, + "contrib": { + "uri": "https://git.openembedded.org/bitbake-contrib" + } + }, + "branch": "master", + "rev": "master" + }, + "path": "bitbake" + } + } + } + - ``rev`` (**required**): the revision to checkout. Can be the name of the + branch to checkout on the latest revision of the specified ``branch``. + + If the value is the branch name, ``bitbake-setup`` will check out the + latest revision on that branch, and keep it updated when using the + :ref:`ref-bbsetup-command-update` command. + + - ``branch`` (**required**): the Git branch, used to check that the + specified ``rev`` is indeed on that branch. + + - ``uri`` (*optional*): a URI that follows the git URI syntax. You can replace + the ``remotes`` structure if only one URI is provided. Despite this, ``uri`` + and ``remotes`` can still be used together. + + See https://git-scm.com/docs/git-clone#_git_urls for more information. + + - ``local`` (*optional*): specifies a path on local disk that should be symlinked + to under ``layers/``. This is useful for local development, where some layer + or other component used in a build is managed separately, but should still be + available for bitbake-setup driven builds. + + ``local`` entries are specified with the following options: + + - ``path`` (**required**): the path on local disk where the externally + managed source tree is. ``~`` and ``~user`` are expanded to that user's home + directory. Paths in configuration files must be absolute (after possible + ~ expansion), paths in override files can be relative to the directory where + the override file is. + +- ``expires`` (*optional*): Expiration date of the configuration. This date + should be in :wikipedia:`ISO 8601 ` format (``YYYY-MM-DDTHH:MM:SS``). + +- ``bitbake-setup`` (**required**): contains a list of configurations. + + Example: + + .. code-block:: json + + { + "bitbake-setup": { + "configurations": [ + { + "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"], + "bb-env-passthrough-additions": ["DL_DIR","SSTATE_DIR"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : [ + { "name": "machine/qemux86-64", "description": "Machine configuration for running an x86-64 system on QEMU" }, + { "name": "machine/qemuarm64", "description": "Machine configuration for running an ARMv8 system on QEMU" }, + { "name": "machine/qemuriscv64", "description": "Machine configuration for running an RISC-V system on QEMU" }, + { "name": "machine/genericarm64", "description": "Machine configuration for Arm64 SystemReady IR/ES platforms" }, + { "name": "machine/genericx86-64", "description": "Machine configuration for generic x86_64 (64-bit) PCs and servers" } + ] + }, + "distro": { + "description": "Target Distributions", + "options" : [ + { "name": "distro/poky", "description": "Distro configuration for Poky (Yocto Project Reference Distro)" }, + { "name": "distro/poky-altcfg", "description": "Distro configuration for Poky (systemd init manager)" }, + { "name": "distro/poky-tiny", "description": "Distro configuration for Poky (optimized for size)" } + ] + } + }, + "configurations": [ + { + "name": "poky", + "description": "Poky - The Yocto Project testing distribution" + }, + { + "name": "poky-with-sstate", + "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.", + "oe-fragments": ["core/yocto/sstate-mirror-cdn"] + } + ] + } + ] + } + } + + Configurations can be specified with the following options: + + - ``name`` (**required**): the name of this configuration snippet. This is + what is prompted during the :ref:`ref-bbsetup-command-init` command + execution. + + - ``description`` (**required**): the description of this configuration + snippet. This is what is prompted during the + :ref:`ref-bbsetup-command-init` command execution. + + - ``notes`` (*optional*): additional information written to + ``build/conf/conf-notes.txt`` when ``bitbake-setup`` generates the build + configuration from ``bb-layers``. This can be a string, or a list of + strings which will be written one per line. For ``oe-template`` + configurations, this file is provided by the template. + + - ``configurations``: Configurations can recursively contain as many nested + configurations as needed. This will create more choices when running the + :ref:`ref-bbsetup-command-init` command. + + The purpose of such nesting is to be able to scale the configurations, for + example when there is a need to create multiple configurations that share + some parameters (which are specified in their common parent), but differ + between themselves in other parameters. ``bitbake-setup`` will assemble + configuration choices by putting together information from a leaf + configuration and all of its ancestors. + + When the same keyword is present in a nested configuration and in one of + its ancestors, the values are merged with Python ``+`` semantics. For + example, lists are appended and strings are concatenated directly. String + values such as ``description`` should include any needed separators in + the configuration data. + + - ``bb-env-passthrough-additions`` (*optional*): List of environment + variables to include in :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. + + - ``bb-layers`` (*optional*): List of layers to add to the ``bblayers.conf`` + file. Paths in this list are relative to the ``layers/`` directory of a + :term:`Setup`. + + The ``bb-layers`` keyword cannot be used in conjunction with the + ``oe-template`` option, as the ``bblayers.conf`` file comes from the + template itself. + + - ``bb-layers-file-relative`` (*optional*): List of layers that are not + managed by ``bitbake-setup`` but that need to be included as part of the + ``bblayers.conf`` file. Paths in this list are relative to the + configuration file. + + This is useful when (one or more) configuration files and (one or + more) layers are hosted in the same Git repository, which is cloned + and managed independently from bitbake-setup workflows. For example:: + + ├── meta-myproject/ + └── myproject.conf.json + + Then ``myproject.conf.json`` can contain the following to add + ``meta-myproject`` to ``bblayers.conf``:: + + { + ... + "bb-layers-file-relative": [ + "meta-myproject" + ], + ... + } + + The ``bb-layers-file-relative`` keyword cannot be used in conjunction with the + ``oe-template`` keyword, as the ``bblayers.conf`` file comes from the + template itself. + + - ``oe-template`` (*optional*, OpenEmbedded specific): OpenEmbedded template + to use. This cannot be used in conjunction with the + ``bb-layers`` or ``bb-layers-file-relative`` keywords as it + already provides a ready ``bblayers.conf`` file to use. + + See https://docs.yoctoproject.org/dev-manual/custom-template-configuration-directory.html + for more information of OpenEmbedded templates. + + - ``oe-fragments-one-of`` (*optional*, OpenEmbedded specific): the OpenEmbedded + fragments to select as part of the build. + + This can one of two formats: + + - A list of dictionaries with ``name`` and a ``description``, + corresponding to respectively the fragments names and fragment + descriptions (like in the example above). + + - A list of fragment names. For example: + + .. code-block:: json + + "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"] + + This will trigger choices to make during the + :ref:`ref-bbsetup-command-init` command execution. + + See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for + more information of OpenEmbedded configuration fragments. + + - ``oe-fragments`` (*optional*, OpenEmbedded specific): fragments to select + as part of the build. + + See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for + more information of OpenEmbedded configuration fragments. + + - ``setup-dir-name`` (*optional*): a suggestion for the :term:`Setup` + directory name. Bitbake-setup will use it, unless it already exists, or + the :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting is set + to ``yes`` (in those cases, it will fall back to the built-in full name, + containing the full set of choices made during initialization). + + This key can have variable expansions in the same format as python + string.Template strings (which matches the shell variable expansion rules): + https://docs.python.org/3/library/string.html#template-strings-strings . + + Variables that can be expanded are any fragment configuration prompted + by the user (e.g. the keys in "oe-fragments-one-of"):: + + { + ... + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : ["machine/gadget", "machine/gizmo"] + } + }, + "setup-dir-name": "somebuild-$machine" + ... + } + + would expand to ``somebuild-machine_gadget``. + +Configuration Template Examples +------------------------------- + +OpenEmbedded "nodistro" configuration for master branches: + +.. literalinclude:: ../../default-registry/configurations/oe-nodistro-master.conf.json + :language: json + +Poky distribution configuration for master branches: + +.. literalinclude:: ../../default-registry/configurations/poky-master.conf.json + :language: json + +.. _ref-bbsetup-source-overrides: + +Source Overrides +================ + +See the definition of :term:`Source Override` in the Terminology section. + +These files are written in the JSON file format and are optionally passed to the +``--source-overrides`` argument of the :ref:`ref-bbsetup-command-init` command. +The ``--source-overrides`` option can be passed multiple times, in which case the +overrides are applied in the order specified in the command-line. + +Here is an example file that overrides the branch of the BitBake repository to +"master-next", and provides openembedded-core as a symlink to a path on local disk: + +.. code-block:: json + + { + "description": "Source override file", + "sources": { + "bitbake": { + "git-remote": { + "branch": "master-next", + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "rev": "master-next" + } + }, + "openembedded-core": { + "local": { + "path": "~/path/to/local/openembedded-core" + } + } + }, + "version": "1.0" + } + +- The ``version`` parameter contains the version of the used configuration, and + should match the one of the :term:`Configuration Template` file in use. + +- The ``sources`` section contains the same options as the ``sources`` option + of a :term:`Configuration Template` file. See the + :ref:`ref-bbsetup-section-config-reference` section for more information. + +.. _ref-bbsetup-fixed-revisions: + +Fixed source revisions +====================== + +:term:`Configuration Template` can set source revisions in ``rev`` to a tag or a branch. +Bitbake-setup will make sure the actual revision will match the tag or branch when performing +initializations or updates, and will capture the revisions in a :ref:`ref-bbsetup-source-overrides` +file. + +This file is named ``sources-fixed-revisions.json`` and is available in ``layers/`` +under the :term:`Setup` directory (as well as under ``config/`` for backwards compatibility). +It can be used to keep record of what was checked out when using configurations that do not +specify exact revisions. + +Also, as it is an override file, it can be combined with the original configurations +to initialize a setup in a reproducible way that guarantees an exact, never-changing +set of revisions, by using the ``--source-overrides`` option of the +:ref:`ref-bbsetup-command-init` command. + +For example if the original configuration had specified only a master branch +for a source:: + + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "master", + "rev": "master" + } + + +the override will contain the exact revisions:: + + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "master", + "rev": "720df1a53452983c1c832f624490e255cf389204" + } diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst index d58fbb32..1638a8f5 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst @@ -64,11 +64,11 @@ data itself is of various types: together. The ``layer.conf`` files are used to construct key variables such as -:term:`BBPATH` and :term:`BBFILES`. -:term:`BBPATH` is used to search for configuration and class files under the -``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is used -to locate both recipe and recipe append files (``.bb`` and -``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the +:term:`BBPATH` and :term:`BBFILES`. :term:`BBPATH` is used to search for +configuration files under the ``conf`` directory and class files under the +``classes-global``, ``classes-recipe`` and ``classes`` directories. +:term:`BBFILES` is used to locate both recipe and recipe append files (``.bb`` +and ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the user has set the :term:`BBPATH` and :term:`BBFILES` directly in the environment. Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variable @@ -337,21 +337,22 @@ file that BitBake parses, you can change that preference:: .. note:: - It is common for a recipe to provide two versions -- a stable, - numbered (and preferred) version, and a version that is automatically - checked out from a source code repository that is considered more - "bleeding edge" but can be selected only explicitly. + It is not uncommon for a given recipe to provide more than one + version -- a stable (and preferred) version that should be selected + by default, and a newer, development or "bleeding edge" version that + will be selected only explicitly by the developer. - For example, in the OpenEmbedded codebase, there is a standard, - versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there - is also a Git-based version, ``busybox_git.bb``, which explicitly - contains the line :: + For example, at the moment, in the OpenEmbedded codebase, there are + two versions of the ``nginx`` webserver: a stable "1.28.0" version + that would be selected by default, and a newer "1.29.1" development + version that a developer would need to choose explicitly. - DEFAULT_PREFERENCE = "-1" + This can be seen based on this snippet from the ``nginx`` 1.29.1 + recipe file:: - to ensure that the - numbered, stable version is always preferred unless the developer - selects otherwise. + # 1.28.x branch is the current stable branch, the recommended default + # 1.29.x is the current mainline branches containing all new features + DEFAULT_PREFERENCE = "-1" .. _bb-bitbake-dependencies: diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index fb4f0a23..9f00545d 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -11,10 +11,11 @@ with the intricacies of downloading source code and files from remote systems. Fetching source code is one of the cornerstones of building software. As such, this module forms an important part of BitBake. -The current fetch module is called "fetch2" and refers to the fact that -it is the second major version of the API. The original version is -obsolete and has been removed from the codebase. Thus, in all cases, -"fetch" refers to "fetch2" in this manual. +The fetch module was previously called ``fetch2`` to distinguish the second +major version of the API from the original version, which is obsolete and has +been removed from the codebase. The module has since been renamed back to +``fetch``. Existing code using ``bb.fetch2`` remains supported by a +compatibility alias, but new code should use ``bb.fetch``. The Download (Fetch) ==================== @@ -30,7 +31,7 @@ The code to execute the first part of this process, a fetch, looks something like the following:: src_uri = (d.getVar('SRC_URI') or "").split() - fetcher = bb.fetch2.Fetch(src_uri, d) + fetcher = bb.fetch.Fetch(src_uri, d) fetcher.download() This code sets up an instance of the fetch class. The instance uses a @@ -39,10 +40,10 @@ variable and then calls the ``download`` method to download the files. The instantiation of the fetch class is usually followed by:: - rootdir = l.getVar('WORKDIR') + rootdir = l.getVar('UNPACKDIR') fetcher.unpack(rootdir) -This code unpacks the downloaded files to the specified by ``WORKDIR``. +This code unpacks the downloaded files to the specified by ``UNPACKDIR``. .. note:: @@ -51,7 +52,7 @@ This code unpacks the downloaded files to the specified by ``WORKDIR``. examine the OpenEmbedded class file ``base.bbclass`` . -The :term:`SRC_URI` and ``WORKDIR`` variables are not hardcoded into the +The :term:`SRC_URI` and ``UNPACKDIR`` variables are not hardcoded into the fetcher, since those fetcher methods can be (and are) called with different variable names. In OpenEmbedded for example, the shared state (sstate) code uses the fetch module to fetch the sstate files. @@ -84,11 +85,8 @@ fetcher does know how to use HTTP as a transport. Here are some examples that show commonly used mirror definitions:: PREMIRRORS ?= "\ - bzr://.*/.\* http://somemirror.org/sources/ \ - cvs://.*/.\* http://somemirror.org/sources/ \ git://.*/.\* http://somemirror.org/sources/ \ hg://.*/.\* http://somemirror.org/sources/ \ - osc://.*/.\* http://somemirror.org/sources/ \ p4://.*/.\* http://somemirror.org/sources/ \ svn://.*/.\* http://somemirror.org/sources/" @@ -126,7 +124,7 @@ in the previous example, or you can name the URLs. The following syntax shows how you name the URIs:: SRC_URI = "http://example.com/foobar.tar.bz2;name=foo" - SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d + SRC_URI[foo.md5sum] = "4a8e0f237e961fd7785d19d07fdb994d" After a file has been downloaded and has had its checksum checked, a ".done" stamp is placed in :term:`DL_DIR`. @@ -159,19 +157,25 @@ URLs except Git URLs, BitBake uses the common ``unpack`` method. A number of parameters exist that you can specify within the URL to govern the behavior of the unpack stage: -- *unpack:* Controls whether the URL components are unpacked. If set to +- *"unpack":* Controls whether the URL components are unpacked. If set to "1", which is the default, the components are unpacked. If set to "0", the unpack stage leaves the file alone. This parameter is useful when you want an archive to be copied in and not be unpacked. -- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether +- *"dos":* Applies to ``.zip`` and ``.jar`` files and specifies whether to use DOS line ending conversion on text files. -- *striplevel:* Strip specified number of leading components (levels) - from file names on extraction +- *"striplevel":* Strip specified number of leading components (levels) + from file names on extraction. -- *subdir:* Unpacks the specific URL to the specified subdirectory - within the root directory. +- *"subdir":* Unpacks the specific URL to the specified subdirectory + within the specified root directory. This path can be further modified + by fetcher specific parameters. + +- *"name":* Assigns a name to a given component of the :term:`SRC_URI`. + This component is later referenced by this name when specifying its + :term:`SRCREV` or :term:`SRC_URI` checksum, or to correctly place its + revision in the package version string with aid of :term:`SRCREV_FORMAT`. The unpack call automatically decompresses and extracts files with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and @@ -183,6 +187,51 @@ cloning the tree into the final directory. The process is completed using references so that there is only one central copy of the Git metadata needed. +.. _bb-the-unpack-update: + +The Non-Destructive Update (``unpack_update``) +============================================== + +.. note:: + + This is a specialised method intended for tools that manage persistent + layer checkouts on behalf of the user, such as + :ref:`bitbake-setup update `. It is not part of the + normal recipe fetch/unpack flow. + +For Git URLs, an alternative ``unpack_update()`` method is available +that updates an existing checkout *in place* rather than removing and +re-cloning it. This is useful when the target directory may contain +local commits that should be preserved across updates. + +The code to call the non-destructive update looks like the following:: + + rootdir = l.getVar('UNPACKDIR') + fetcher.unpack_update(rootdir) + +``unpack_update()`` performs the following steps: + +1. A ``dldir`` git remote is added (or updated) in the existing + checkout, pointing at the local download cache. This remote may + also be useful for manually resolving conflicts outside the fetcher. +2. The new upstream revision is fetched from the ``dldir`` remote into + the existing repository. +3. Any local commits are rebased on top of the new upstream revision, + preserving local work. + +The method raises an error if: + +- The working tree contains staged or unstaged changes to tracked + files (``LocalModificationsError``). +- Local commits cannot be cleanly rebased onto the new upstream + revision (``RebaseError``). A failed rebase is automatically aborted + before the exception is raised. +- The download cache does not contain a sufficiently recent clone + of the repository, or the checkout is a shallow clone. + +Currently only the Git fetcher supports ``unpack_update()``. All other +fetcher types raise ``RuntimeError`` if it is called. + .. _bb-fetchers: Fetchers @@ -246,91 +295,19 @@ Some example URLs are as follows:: introduce ambiguity when parsing URLs that also contain semi-colons, for example:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" - + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" Such URLs should should be modified by replacing semi-colons with '&' characters:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" - + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" In most cases this should work. Treating semi-colons and '&' in queries identically is recommended by the World Wide Web Consortium (W3C). Note that due to the nature of the URL, you may have to specify the name of the downloaded file as well:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" - - -.. _cvs-fetcher: - -CVS fetcher (``(cvs://``) -------------------------- - -This submodule handles checking out files from the CVS version control -system. You can configure it using a number of different variables: - -- :term:`FETCHCMD_cvs `: The name of the executable to use when running - the ``cvs`` command. This name is usually "cvs". - -- :term:`SRCDATE`: The date to use when fetching the CVS source code. A - special value of "now" causes the checkout to be updated on every - build. - -- :term:`CVSDIR`: Specifies where a temporary - checkout is saved. The location is often ``DL_DIR/cvs``. - -- CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the - ``cvs`` command. - -- CVS_PROXY_PORT: The port number to use as a "proxyport=" - parameter to the ``cvs`` command. - -As well as the standard username and password URL syntax, you can also -configure the fetcher with various URL parameters: - -The supported parameters are as follows: - -- *"method":* The protocol over which to communicate with the CVS - server. By default, this protocol is "pserver". If "method" is set to - "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You - can use "dir" for local directories. - -- *"module":* Specifies the module to check out. You must supply this - parameter. - -- *"tag":* Describes which CVS TAG should be used for the checkout. By - default, the TAG is empty. - -- *"date":* Specifies a date. If no "date" is specified, the - :term:`SRCDATE` of the configuration is used to - checkout a specific date. The special value of "now" causes the - checkout to be updated on every build. - -- *"localdir":* Used to rename the module. Effectively, you are - renaming the output directory to which the module is unpacked. You - are forcing the module into a special directory relative to - :term:`CVSDIR`. - -- *"rsh":* Used in conjunction with the "method" parameter. - -- *"scmdata":* Causes the CVS metadata to be maintained in the tarball - the fetcher creates when set to "keep". The tarball is expanded into - the work directory. By default, the CVS metadata is removed. - -- *"fullpath":* Controls whether the resulting checkout is at the - module level, which is the default, or is at deeper paths. - -- *"norecurse":* Causes the fetcher to only checkout the specified - directory with no recurse into any subdirectories. - -- *"port":* The port to which the CVS server connects. - -Some example URLs are as follows:: - - SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" - SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" .. _svn-fetcher: @@ -398,16 +375,16 @@ This fetcher supports the following parameters: .. note:: - When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs - from the one that is typically passed to ``git clone`` command and provided - by the Git server to fetch from. For example, the URL returned by GitLab - server for ``mesa`` when cloning over SSH is - ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in - :term:`SRC_URI` is the following:: + When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs + from the one that is typically passed to ``git clone`` command and provided + by the Git server to fetch from. For example, the URL returned by GitLab + server for ``mesa`` when cloning over SSH is + ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in + :term:`SRC_URI` is the following:: - SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." + SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." - Note the ``:`` character changed for a ``/`` before the path to the project. + Note the ``:`` character changed for a ``/`` before the path to the project. - *"nocheckout":* Tells the fetcher to not checkout source code when unpacking when set to "1". Set this option for the URL where there is @@ -436,13 +413,15 @@ This fetcher supports the following parameters: "nobranch" is set to "1", this is a mandatory parameter. The number of branch parameters must match the number of name parameters. -- *"rev":* The revision to use for the checkout. The default is - "master". +- *"rev":* The revision to use for the checkout. If :term:`SRCREV` is also set, + this parameter must match its value. -- *"tag":* Specifies a tag to use for the checkout. To correctly - resolve tags, BitBake must access the network. For that reason, tags - are often not used. As far as Git is concerned, the "tag" parameter - behaves effectively the same as the "rev" parameter. +- *"tag":* Specifies a tag to use when fetching. To correctly resolve + tags, BitBake must access the network. If a ``rev`` parameter or + :term:`SRCREV` is also specified, network access is not necessary to resolve + the tag and instead, it is verified that they both resolve to the same commit + SHA at unpack time. The ``tag`` parameter is optional, but strongly + recommended if the checked out revision is a tag. - *"subpath":* Limits the checkout to a specific subpath of the tree. By default, the whole tree is checked out. @@ -451,7 +430,7 @@ This fetcher supports the following parameters: By default, the path is ``git/``. - *"usehead":* Enables local ``git://`` URLs to use the current branch - HEAD as the revision for use with ``AUTOREV``. The "usehead" + HEAD as the revision for use with :term:`AUTOREV`. The "usehead" parameter implies no branch and only works when the transfer protocol is ``file://``. @@ -461,13 +440,6 @@ Here are some example URLs:: SRC_URI = "git://github.com/asciidoc/asciidoc-py;protocol=https;branch=main" SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." -.. note:: - - When using ``git`` as the fetcher of the main source code of your software, - ``S`` should be set accordingly:: - - S = "${WORKDIR}/git" - .. note:: Specifying passwords directly in ``git://`` urls is not supported. @@ -476,7 +448,7 @@ Here are some example URLs:: easy to share metadata without removing passwords. SSH keys, ``~/.netrc`` and ``~/.ssh/config`` files can be used as alternatives. -Using tags with the git fetcher may cause surprising behaviour. Bitbake needs to +Using tags with the git fetcher may cause surprising behaviour. BitBake needs to resolve the tag to a specific revision and to do that, it has to connect to and use the upstream repository. This is because the revision the tags point at can change and we've seen cases of this happening in well known public repositories. This can mean @@ -530,10 +502,10 @@ The fetcher uses the ``rcleartool`` or Following are options for the :term:`SRC_URI` statement: -- *vob*: The name, which must include the prepending "/" character, +- *"vob":* The name, which must include the prepending "/" character, of the ClearCase VOB. This option is required. -- *module*: The module, which must include the prepending "/" +- *"module":* The module, which must include the prepending "/" character, in the selected VOB. .. note:: @@ -545,7 +517,7 @@ Following are options for the :term:`SRC_URI` statement: load /example_vob/example_module -- *proto*: The protocol, which can be either ``http`` or ``https``. +- *"proto":* The protocol, which can be either ``http`` or ``https``. By default, the fetcher creates a configuration specification. If you want this specification written to an area other than the default, use @@ -554,9 +526,9 @@ the specification is written. .. note:: - the SRCREV loses its functionality if you specify this variable. However, - SRCREV is still used to label the archive after a fetch even though it does - not define what is fetched. + the :term:`SRCREV` loses its functionality if you specify this variable. + However, :term:`SRCREV` is still used to label the archive after a fetch even + though it does not define what is fetched. Here are a couple of other behaviors worth mentioning: @@ -598,7 +570,7 @@ and port, username, and password, and fetches the Head Revision:: SRC_URI = "p4://example-depot/main/source/..." SRCREV = "${AUTOREV}" PV = "p4-${SRCPV}" - S = "${WORKDIR}/p4" + S = "${UNPACKDIR}/p4" Here is an example that specifies the server URL and port, username, and password, and fetches a Revision based on a Label:: @@ -607,25 +579,25 @@ password, and fetches a Revision based on a Label:: SRC_URI = "p4://user:passwd@example-depot/main/source/..." SRCREV = "release-1.0" PV = "p4-${SRCPV}" - S = "${WORKDIR}/p4" + S = "${UNPACKDIR}/p4" .. note:: - You should always set S to "${WORKDIR}/p4" in your recipe. + You should always set S to "${UNPACKDIR}/p4" in your recipe. By default, the fetcher strips the depot location from the local file paths. In the above example, the content of ``example-depot/main/source/`` will be placed -in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot +in ``${UNPACKDIR}/p4``. For situations where preserving parts of the remote depot paths locally is desirable, the fetcher supports two parameters: -- *"module":* - The top-level depot location or directory to fetch. The value of this - parameter can also point to a single file within the depot, in which case - the local file path will include the module path. -- *"remotepath":* - When used with the value "``keep``", the fetcher will mirror the full depot - paths locally for the specified location, even in combination with the - ``module`` parameter. +- *"module":* + The top-level depot location or directory to fetch. The value of this + parameter can also point to a single file within the depot, in which case + the local file path will include the module path. +- *"remotepath":* + When used with the value "``keep``", the fetcher will mirror the full depot + paths locally for the specified location, even in combination with the + ``module`` parameter. Here is an example use of the the ``module`` parameter:: @@ -686,9 +658,9 @@ Such functionality is set by the variable: delegate access to resources, if this variable is set, the Az Fetcher will use it when fetching artifacts from the cloud. -You can specify the AZ_SAS variable as shown below:: +You can specify the AZ_SAS variable prefixed with a ? as shown below:: - AZ_SAS = "se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=&sig=" + AZ_SAS = "?se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=&sig=" Here is an example URL:: @@ -745,6 +717,11 @@ The format for the :term:`SRC_URI` setting must be:: SRC_URI = "crate://REGISTRY/NAME/VERSION" +This fetcher supports the following parameters: + +- *"protocol":* The protocol used to fetch the crates. The default is "https". + You can also use "http". + Here is an example URL:: SRC_URI = "crate://crates.io/glob/0.2.11" @@ -754,6 +731,12 @@ Here is an example URL:: NPM Fetcher (``npm://``) ------------------------ +.. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. + This submodule fetches source code from an `NPM `__ Javascript package registry. @@ -793,6 +776,12 @@ to automatically create a recipe from an NPM URL. NPM shrinkwrap Fetcher (``npmsw://``) ------------------------------------- +.. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. + This submodule fetches source code from an `NPM shrinkwrap `__ description file, which lists the dependencies @@ -828,12 +817,8 @@ Other Fetchers Fetch submodules also exist for the following: -- Bazaar (``bzr://``) - - Mercurial (``hg://``) -- OSC (``osc://``) - - S3 (``s3://``) - Secure FTP (``sftp://``) @@ -848,4 +833,42 @@ submodules. However, you might find the code helpful and readable. Auto Revisions ============== -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. +For recipes which need to use the latest revision of their source code, +the way to achieve it is to use :term:`AUTOREV` as the value of the +source code repository's :term:`SRCREV`:: + + SRCREV = "${AUTOREV}" + +.. note:: + + With :term:`AUTOREV`, BitBake will always need to take the additional step of + querying the remote repository to retrieve the latest available revision. + + Also, recipes using it are not part of the parsing-time cache, + and hence are parsed every time. + +Multiple Source Control Repositories +==================================== + +For some recipes it is necessary to make use of more than one +version controlled source code repository. In such case, the recipe +must provide BitBake with information about how it should include +the different SCM revisions in its package version string, instead of its +usual approach with a single :term:`SRCREV`. + +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` +variable. Consider the following example:: + + SRC_URI = " \ + git://git.some.example.com/source-tree.git;name=machine \ + git://git.some.example.com/metadata.git;name=meta \ + " + SRCREV_machine = "3f9db490a81eeb0077be3c5a5aa1388a2372232f" + SRCREV_meta = "1ac1d0ff730fe1dd1371823d562db8126750a98c" + SRCREV_FORMAT ?= "meta_machine" + +The value given to :term:`SRCREV_FORMAT` references names, which were +assigned using the ``name`` parameter in the :term:`SRC_URI` definition, +and which represent the version controlled source code repositories. +In the above example, the :term:`SRC_URI` contained two URLs named +"meta" and "machine". diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst index 35ffb88b..6801323e 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst @@ -198,7 +198,7 @@ Class files, which are denoted by the ``.bbclass`` extension, contain information that is useful to share between metadata files. The BitBake source tree currently comes with one class metadata file called ``base.bbclass``. You can find this file in the ``classes`` directory. -The ``base.bbclass`` class files is special since it is always included +The ``base.bbclass`` class file is special since it is always included automatically for all recipes and classes. This class contains definitions for standard basic tasks such as fetching, unpacking, configuring (empty by default), compiling (runs any Makefile present), @@ -206,6 +206,35 @@ installing (empty by default) and packaging (empty by default). These tasks are often overridden or extended by other classes added during the project development process. +Class Types +~~~~~~~~~~~ + +BitBake supports class files installed in three different directories: + +- ``classes-global/``: these classes must be inherited globally through the + :term:`INHERIT` variable in a :ref:`configuration file + `. These + classes are included for every recipe being built. For example, you would use + the global class named ``myclass`` like so:: + + INHERIT += "myclass" + +- ``classes-recipe/``: these classes must be inherited from a recipe using the + :ref:`inherit ` or + :ref:`inherit_defer ` + directive. They do not support being inherited globally. For example, you + would use the recipe class named ``myclass`` like so:: + + inherit myclass + +- ``classes/``: this final directory is meant for classes that can be used in + the two contexts explained above. In other words, they can be inherited either + globally or in a recipe. + +For details on how BitBake locates class files, see the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Locating Class Files` +section of the BitBake User Manual. + Layers ------ @@ -242,12 +271,9 @@ same root filename. The filenames can differ only in the file type suffix used (e.g. ``formfactor_0.0.bb`` and ``formfactor_0.0.bbappend``). -Information in append files extends or overrides the information in the -underlying, similarly-named recipe files. - When you name an append file, you can use the "``%``" wildcard character -to allow for matching recipe names. For example, suppose you have an -append file named as follows:: +to allow for a less strict recipe name matching. For example, suppose you have +an append file named as follows:: busybox_1.21.%.bbappend @@ -258,12 +284,8 @@ the append file would match the following recipe names:: busybox_1.21.1.bb busybox_1.21.2.bb busybox_1.21.3.bb - -.. note:: - - The use of the " % " character is limited in that it only works directly in - front of the .bbappend portion of the append file's name. You cannot use the - wildcard character in any other location of the name. + busybox_1.21.10.bb + busybox_1.21.11.bb If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the append name would not match. However, if you named the append file @@ -272,6 +294,47 @@ append name would not match. However, if you named the append file In the most general case, you could name the append file something as simple as ``busybox_%.bbappend`` to be entirely version independent. +A recipe named without a version may only be appended by an append file named +exactly the same. That is, a recipe named:: + + mesa.bb + +can only be appended by an append file named:: + + mesa.bbappend + +That is, ``mesa_%.bbappend`` append file will not match a ``mesa.bb`` recipe. + +.. note:: + + The "``%``" character is a marker for BitBake, and everything after that + marker will not be used for matching recipe names. That is, an append file + named:: + + busybox_1.%.11.bbappend + + will match recipes named:: + + busybox_1.0.11.bb + busybox_1.2.bb + busybox_1.3.48.bb + + Note that the "``%``" character is not specific to the version part of the + recipe name. That is, an append file named:: + + linux-yocto%.bbappend + + will match recipes named:: + + linux-yocto.bb + linux-yocto_6.18.bb + linux-yocto-tiny_6.12.bb + linux-yocto-rt_6.16.bb + + While it can be useful to append to similar recipes with slightly different + names, it is highly discouraged to do so as recipes from other layers may be + named similarly enough that the append file would unwittingly match them. + Obtaining BitBake ================= @@ -349,40 +412,84 @@ Usage and syntax Following is the usage and syntax for BitBake:: $ bitbake -h - Usage: bitbake [options] [recipename/target recipe:do_task ...] - - Executes the specified task (default is 'build') for a given set of target recipes (.bb files). - It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which - will provide the layer, BBFILES and other configuration information. + usage: bitbake [-s] [-e] [-g] [-u UI] [--version] [-h] [-f] [-c CMD] + [-C INVALIDATE_STAMP] [--runall RUNALL] [--runonly RUNONLY] + [--no-setscene] [--skip-setscene] [--setscene-only] [-n] [-p] + [-k] [-P] [-S SIGNATURE_HANDLER] [--revisions-changed] + [-b BUILDFILE] [-D] [-l DEBUG_DOMAINS] [-v] [-q] + [-w WRITEEVENTLOG] [-B BIND] [-T SERVER_TIMEOUT] + [--remote-server REMOTE_SERVER] [-m] [--token XMLRPCTOKEN] + [--observe-only] [--status-only] [--server-only] [-r PREFILE] + [-R POSTFILE] [-I EXTRA_ASSUME_PROVIDED] + [recipename/target ...] + + It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH + which will provide the layer, BBFILES and other configuration information. + + General options: + recipename/target Execute the specified task (default is 'build') for + these target recipes (.bb files). + -s, --show-versions Show current and preferred versions of all recipes. + -e, --environment Show the global or per-recipe environment complete + with information about where variables were + set/changed. + -g, --graphviz Save dependency tree information for the specified + targets in the dot syntax. + -u UI, --ui UI The user interface to use (knotty, ncurses, taskexp, + taskexp_ncurses or teamcity - default knotty). + --version Show programs version and exit. + -h, --help Show this help message and exit. - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -b BUILDFILE, --buildfile=BUILDFILE - Execute tasks from a specific .bb recipe directly. - WARNING: Does not handle any dependencies from other - recipes. - -k, --continue Continue as much as possible after an error. While the - target that failed and anything depending on it cannot - be built, as much as possible will be built before - stopping. + Task control options: -f, --force Force the specified targets/task to run (invalidating any existing stamp file). - -c CMD, --cmd=CMD Specify the task to execute. The exact options + -c CMD, --cmd CMD Specify the task to execute. The exact options available depend on the metadata. Some examples might be 'compile' or 'populate_sysroot' or 'listtasks' may give a list of the tasks available. - -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP + -C INVALIDATE_STAMP, --clear-stamp INVALIDATE_STAMP Invalidate the stamp for the specified task such as 'compile' and then run the default task for the specified target(s). - -r PREFILE, --read=PREFILE - Read the specified file before bitbake.conf. - -R POSTFILE, --postread=POSTFILE - Read the specified file after bitbake.conf. - -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also - print bb.note(...) messages to stdout (in addition to - writing them to ${T}/log.do_<task>). + --runall RUNALL Run the specified task for any recipe in the taskgraph + of the specified target (even if it wouldn't otherwise + have run). + --runonly RUNONLY Run only the specified task within the taskgraph of + the specified targets (and any task dependencies those + tasks may have). + --no-setscene Do not run any setscene tasks. sstate will be ignored + and everything needed, built. + --skip-setscene Skip setscene tasks if they would be executed. Tasks + previously restored from sstate will be kept, unlike + --no-setscene. + --setscene-only Only run setscene tasks, don't run any real tasks. + + Execution control options: + -n, --dry-run Don't execute, just go through the motions. + -p, --parse-only Quit after parsing the BB recipes. + -k, --continue Continue as much as possible after an error. While the + target that failed and anything depending on it cannot + be built, as much as possible will be built before + stopping. + -P, --profile Profile the command and save reports. + -S SIGNATURE_HANDLER, --dump-signatures SIGNATURE_HANDLER + Dump out the signature construction information, with + no task execution. The SIGNATURE_HANDLER parameter is + passed to the handler. Two common values are none and + printdiff but the handler may define more/less. none + means only dump the signature, printdiff means + recursively compare the dumped signature with the most + recent one in a local build or sstate cache (can be + used to find out why tasks re-run when that is not + expected) + --revisions-changed Set the exit code depending on whether upstream + floating revisions have changed or not. + -b BUILDFILE, --buildfile BUILDFILE + Execute tasks from a specific .bb recipe directly. + WARNING: Does not handle any dependencies from other + recipes. + + Logging/output control options: -D, --debug Increase the debug level. You can specify this more than once. -D sets the debug level to 1, where only bb.debug(1, ...) messages are printed to stdout; -DD @@ -392,65 +499,47 @@ Following is the usage and syntax for BitBake:: -D only affects output to stdout. All debug messages are written to ${T}/log.do_taskname, regardless of the debug level. + -l DEBUG_DOMAINS, --log-domains DEBUG_DOMAINS + Show debug logging for the specified logging domains. + -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also + print bb.note(...) messages to stdout (in addition to + writing them to ${T}/log.do_). -q, --quiet Output less log message data to the terminal. You can specify this more than once. - -n, --dry-run Don't execute, just go through the motions. - -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER - Dump out the signature construction information, with - no task execution. The SIGNATURE_HANDLER parameter is - passed to the handler. Two common values are none and - printdiff but the handler may define more/less. none - means only dump the signature, printdiff means compare - the dumped signature with the cached one. - -p, --parse-only Quit after parsing the BB recipes. - -s, --show-versions Show current and preferred versions of all recipes. - -e, --environment Show the global or per-recipe environment complete - with information about where variables were - set/changed. - -g, --graphviz Save dependency tree information for the specified - targets in the dot syntax. - -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED - Assume these dependencies don't exist and are already - provided (equivalent to ASSUME_PROVIDED). Useful to - make dependency graphs more appealing - -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS - Show debug logging for the specified logging domains - -P, --profile Profile the command and save reports. - -u UI, --ui=UI The user interface to use (knotty, ncurses, taskexp or - teamcity - default knotty). - --token=XMLRPCTOKEN Specify the connection token to be used when - connecting to a remote server. - --revisions-changed Set the exit code depending on whether upstream - floating revisions have changed or not. - --server-only Run bitbake without a UI, only starting a server - (cooker) process. - -B BIND, --bind=BIND The name/address for the bitbake xmlrpc server to bind + -w WRITEEVENTLOG, --write-log WRITEEVENTLOG + Writes the event log of the build to a bitbake event + json file. Use '' (empty string) to assign the name + automatically. + + Server options: + -B BIND, --bind BIND The name/address for the bitbake xmlrpc server to bind to. - -T SERVER_TIMEOUT, --idle-timeout=SERVER_TIMEOUT + -T SERVER_TIMEOUT, --idle-timeout SERVER_TIMEOUT Set timeout to unload bitbake server due to inactivity, set to -1 means no unload, default: Environment variable BB_SERVER_TIMEOUT. - --no-setscene Do not run any setscene tasks. sstate will be ignored - and everything needed, built. - --skip-setscene Skip setscene tasks if they would be executed. Tasks - previously restored from sstate will be kept, unlike - --no-setscene - --setscene-only Only run setscene tasks, don't run any real tasks. - --remote-server=REMOTE_SERVER + --remote-server REMOTE_SERVER Connect to the specified server. -m, --kill-server Terminate any running bitbake server. + --token XMLRPCTOKEN Specify the connection token to be used when + connecting to a remote server. --observe-only Connect to a server as an observing-only client. --status-only Check the status of the remote bitbake server. - -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG - Writes the event log of the build to a bitbake event - json file. Use '' (empty string) to assign the name - automatically. - --runall=RUNALL Run the specified task for any recipe in the taskgraph - of the specified target (even if it wouldn't otherwise - have run). - --runonly=RUNONLY Run only the specified task within the taskgraph of - the specified targets (and any task dependencies those - tasks may have). + --server-only Run bitbake without a UI, only starting a server + (cooker) process. + + Configuration options: + -r PREFILE, --read PREFILE + Read the specified file before bitbake.conf. + -R POSTFILE, --postread POSTFILE + Read the specified file after bitbake.conf. + -I EXTRA_ASSUME_PROVIDED, --ignore-deps EXTRA_ASSUME_PROVIDED + Assume these dependencies don't exist and are already + provided (equivalent to ASSUME_PROVIDED). Useful to + make dependency graphs more appealing. + +.. + BitBake help output generated with "stty columns 80; bin/bitbake -h" .. _bitbake-examples: @@ -481,7 +570,7 @@ The following command runs the clean task on the ``foo.bb`` recipe file:: .. note:: - The "-b" option explicitly does not handle recipe dependencies. Other + The ``-b`` option explicitly does not handle recipe dependencies. Other than for debugging purposes, it is instead recommended that you use the syntax presented in the next section. @@ -496,7 +585,7 @@ dependencies, both for build-time and runtime. There must be a way for you to express recipe preferences when multiple recipes provide the same functionality, or when there are multiple versions of a recipe. -The ``bitbake`` command, when not using "--buildfile" or "-b" only +The ``bitbake`` command, when not using ``--buildfile`` or ``-b`` only accepts a "PROVIDES". You cannot provide anything else. By default, a recipe file generally "PROVIDES" its "packagename" as shown in the following example:: @@ -504,7 +593,7 @@ following example:: $ bitbake foo This next example "PROVIDES" the -package name and also uses the "-c" option to tell BitBake to just +package name and also uses the ``-c`` option to tell BitBake to just execute the ``do_clean`` task:: $ bitbake -c clean foo diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst new file mode 100644 index 00000000..09e35394 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst @@ -0,0 +1,59 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +================= +Library Functions +================= + +| + +This chapter lists common library functions available under the ``lib/`` +directory in BitBake. + +These functions can be used in recipes or configuration files with +:ref:`inline-Python ` or :ref:`Python +` functions. + +Logging utilities +================= + +Different logging utilities can be used from Python code in recipes or +configuration files. + +The strings passed below can be formatted with ``str.format()``, for example:: + + bb.warn("Houston, we have a %s", "bit of a problem") + +Formatted string can also be used directly:: + + bb.error("%s, we have a %s" % ("Houston", "big problem")) + +Python f-strings may also be used:: + + h = "Houston" + bb.fatal(f"{h}, we have a critical problem") + +.. automodule:: bb + :members: + debug, + error, + erroronce, + fatal, + note, + plain, + verbnote, + warn, + warnonce, + +``bb.utils`` +============ + +.. automodule:: bb.utils + :members: + :exclude-members: + LogCatcher, + PrCtlError, + VersionStringException, + better_compile, + better_exec, diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst index 58975f4c..621aab25 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst @@ -172,6 +172,26 @@ assignment, ``BAR`` expands to the literal string "${FOO}" as long as BAR = "${FOO}" +Immediate variable expansion (:=) +--------------------------------- + +The ":=" operator results in a variable's contents being expanded +immediately, rather than when the variable is actually used:: + + T = "123" + A := "test ${T}" + T = "456" + B := "${T} ${C}" + C = "cval" + C := "${C}append" + +In this example, ``A`` contains "test 123", even though the final value +of :term:`T` is "456". The variable :term:`B` will end up containing "456 +cvalappend". This is because references to undefined variables are +preserved as is during (immediate)expansion. This is in contrast to GNU +Make, where undefined variables expand to nothing. The variable ``C`` +contains "cvalappend" since ``${C}`` immediately expands to "cval". + Setting a default value (?=) ---------------------------- @@ -235,26 +255,6 @@ After parsing we will have:: W = "xy" -Immediate variable expansion (:=) ---------------------------------- - -The ":=" operator results in a variable's contents being expanded -immediately, rather than when the variable is actually used:: - - T = "123" - A := "test ${T}" - T = "456" - B := "${T} ${C}" - C = "cval" - C := "${C}append" - -In this example, ``A`` contains "test 123", even though the final value -of :term:`T` is "456". The variable :term:`B` will end up containing "456 -cvalappend". This is because references to undefined variables are -preserved as is during (immediate)expansion. This is in contrast to GNU -Make, where undefined variables expand to nothing. The variable ``C`` -contains "cvalappend" since ``${C}`` immediately expands to "cval". - .. _appending-and-prepending: Appending (+=) and prepending (=+) With Spaces @@ -608,7 +608,7 @@ variable. DEPENDS = "glibc ncurses" OVERRIDES = "machine:local" - DEPENDS:append:machine = "libmad" + DEPENDS:append:machine = " libmad" In this example, :term:`DEPENDS` becomes "glibc ncurses libmad". @@ -617,8 +617,8 @@ variable. ``KERNEL_FEATURES`` variable based on the architecture:: KERNEL_FEATURES:append = " ${KERNEL_EXTRA_FEATURES}" - KERNEL_FEATURES:append:qemux86=" cfg/sound.scc cfg/paravirt_kvm.scc" - KERNEL_FEATURES:append:qemux86-64=" cfg/sound.scc cfg/paravirt_kvm.scc" + KERNEL_FEATURES:append:qemux86 = " cfg/sound.scc cfg/paravirt_kvm.scc" + KERNEL_FEATURES:append:qemux86-64 = " cfg/sound.scc cfg/paravirt_kvm.scc" - *Setting a Variable for a Single Task:* BitBake supports setting a variable just for the duration of a single task. Here is an example:: @@ -753,23 +753,17 @@ share the task. This section presents the mechanisms BitBake provides to allow you to share functionality between recipes. Specifically, the mechanisms -include ``include``, ``inherit``, :term:`INHERIT`, and ``require`` -directives. - -Locating Include and Class Files --------------------------------- - -BitBake uses the :term:`BBPATH` variable to locate -needed include and class files. Additionally, BitBake searches the -current directory for ``include`` and ``require`` directives. - -.. note:: - - The BBPATH variable is analogous to the environment variable PATH . - -In order for include and class files to be found by BitBake, they need -to be located in a "classes" subdirectory that can be found in -:term:`BBPATH`. +include :ref:`inherit`, +:ref:`inherit_defer`, +:ref:`include`, +:ref:`include_all`, +:ref:`require` and +:ref:`INHERIT` +directives. There is also a higher-level abstraction called +`configuration fragments` that is enabled with the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive`. + +.. _ref-bitbake-user-manual-metadata-inherit: ``inherit`` Directive --------------------- @@ -809,65 +803,131 @@ An advantage with the inherit directive as compared to both the :ref:`include ` and :ref:`require ` directives is that you can inherit class files conditionally. You can accomplish this by using a variable expression after the ``inherit`` -statement. Here is an example:: +statement. - inherit ${VARNAME} +For inheriting classes conditionally, using the :ref:`inherit_defer +` directive is advised as +:ref:`inherit_defer ` is +evaluated at the end of parsing. -If ``VARNAME`` is -going to be set, it needs to be set before the ``inherit`` statement is -parsed. One way to achieve a conditional inherit in this case is to use -overrides:: +.. _ref-bitbake-user-manual-metadata-inherit-defer: - VARIABLE = "" - VARIABLE:someoverride = "myclass" +``inherit_defer`` Directive +--------------------------- -Another method is by using anonymous Python. Here is an example:: +The :ref:`inherit_defer ` +directive works like the :ref:`inherit +` directive, except that it is only +evaluated at the end of parsing. Its usage is recommended when a conditional +expression is used. - python () { - if condition == value: - d.setVar('VARIABLE', 'myclass') - else: - d.setVar('VARIABLE', '') - } +This allows conditional expressions to be evaluated "late", meaning changes to +the variable after the line is parsed will take effect. With the :ref:`inherit +` directive this is not the case. + +Here is an example:: + + inherit_defer ${VARNAME} + +One way to achieve a conditional inherit in this case is to use +overrides:: -Alternatively, you could use an in-line Python expression in the + VARNAME = "" + VARNAME:someoverride = "myclass" + +:ref:`inherit_defer ` +defers the evaluation of ``${VARNAME}`` until the end of +parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, ``${VARNAME}`` +expands to ``myclass``, which is then inherited. + +Alternatively, you could use an inline Python expression in the following form:: - inherit ${@'classname' if condition else ''} - inherit ${@functionname(params)} + inherit_defer ${@'classname' if condition else ''} + +Or:: + + inherit_defer ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)} In all cases, if the expression evaluates to an empty string, the statement does not trigger a syntax error because it becomes a no-op. +See also :term:`BB_DEFER_BBCLASSES` for automatically promoting classes +``inherit`` calls to ``inherit_defer``. + +.. _ref-include-directive: + ``include`` Directive --------------------- -BitBake understands the ``include`` directive. This directive causes -BitBake to parse whatever file you specify, and to insert that file at -that location. The directive is much like its equivalent in Make except -that if the path specified on the include line is a relative path, -BitBake locates the first file it can find within :term:`BBPATH`. +The ``include`` directive causes BitBake to parse a given file, +and to include that file's contents at the location of the +``include`` statement. This directive is similar to its equivalent +in Make, except that if the path specified on the BitBake ``include`` +line is a relative path, BitBake will search for it on the path designated +by :term:`BBPATH` and will include *only the first matching file*. -The include directive is a more generic method of including +The ``include`` directive is a more generic method of including functionality as compared to the :ref:`inherit ` directive, which is restricted to class (i.e. ``.bbclass``) files. The -include directive is applicable for any other kind of shared or +``include`` directive is applicable for any other kind of shared or encapsulated functionality or configuration that does not suit a ``.bbclass`` file. -As an example, suppose you needed a recipe to include some self-test -definitions:: +For example, if you needed a recipe to include some self-test definitions, +you might write:: include test_defs.inc +The ``include`` directive does not produce an error if the specified file +cannot be found. If the included file *must* exist, then you should use +use :ref:`require ` instead, which will generate an error +if the file cannot be found. + +.. note:: + + Note well that the ``include`` directive will include the first matching + file and nothing further (which is almost always the behaviour you want). + If you need to include all matching files, you need to use the + ``include_all`` directive, explained below. + +.. _ref-include-all-directive: + +``include_all`` Directive +------------------------- + +The ``include_all`` directive works like the :ref:`include +` +directive but will include *all* of the files that match the specified path in +the enabled layers (layers part of :term:`BBLAYERS`). + +Note that only :term:`BBPATH` will be searched, the parent directory of the file +with the ``include_all`` directive will not be searched (unlike for the +``include`` directive). + .. note:: - The include directive does not produce an error when the file cannot be - found. Consequently, it is recommended that if the file you are including is - expected to exist, you should use :ref:`require ` instead - of include . Doing so makes sure that an error is produced if the file cannot - be found. + This behaviour is rarely what you want in normal operation, and should + be reserved for only those situations when you explicitly want to parse + and include all matching files found across all layers, as the following + example shows. + +As a realistic example of this directive, imagine that all of your active +layers contain a file ``conf/distro/include/maintainers.inc``, containing +maintainer information for the recipes in that layer, and you wanted to +collect all of the content from all of those files across all of those layers. +You could use the statement:: + + include_all conf/distro/include/maintainers.inc + +In this case, BitBake will iterate through all of the directories in +the colon-separated :term:`BBPATH` (from left to right) and collect the +contents of all matching files, so you end up with the maintainer +information of all of your active layers, not just the first one. + +As the ``include_all`` directive uses the ``include`` directive in the +background, as with ``include``, no error is produced if no files are matched. .. _require-inclusion: @@ -933,6 +993,197 @@ the ``autotools`` and ``pkgconfig`` classes:: INHERIT += "autotools pkgconfig" +``addfragments`` Directive +-------------------------- + +This directive allows fine-tuning local configurations with configuration +snippets contained in layers in a structured, controlled way. Typically it would +go into ``bitbake.conf``, for example:: + + addfragments conf/fragments OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS OE_FRAGMENTS_BUILTIN + +``addfragments`` takes four parameters: + +- path prefix for fragment files inside the layer file tree that bitbake + uses to construct full paths to the fragment files + +- name of variable that holds the list of enabled fragments in an + active build + +- name of variable that contains a list of variable names containing + fragment-specific metadata (such as descriptions) + +- name of variable that contains definitions for built-in fragments + +This allows listing enabled configuration fragments in ``OE_FRAGMENTS`` +variable like this:: + + OE_FRAGMENTS = "core/domain/somefragment core/someotherfragment anotherlayer/anotherdomain/anotherfragment" + +Fragment names listed in this variable must be prefixed by the layer name +where a fragment file is located, defined by :term:`BBFILE_COLLECTIONS` in ``layer.conf``. + +The implementation then expands this list into +:ref:`require ` +directives with full paths to respective layers:: + + require /path/to/core-layer/conf/fragments/domain/somefragment.conf + require /path/to/core-layer/conf/fragments/someotherfragment.conf + require /path/to/another-layer/conf/fragments/anotherdomain/anotherfragment.conf + +The variable containing a list of fragment metadata variables could look like this:: + + OE_FRAGMENTS_METADATA_VARS = "BB_CONF_FRAGMENT_SUMMARY BB_CONF_FRAGMENT_DESCRIPTION" + +The implementation will add a flag containing the fragment name to each of those variables +when parsing fragments, so that the variables are namespaced by fragment name, and do not override +each other when several fragments are enabled. + +The variable containing a built-in fragment definitions could look like this:: + + OE_FRAGMENTS_BUILTIN = "someprefix:SOMEVARIABLE anotherprefix:ANOTHERVARIABLE" + +and then if 'someprefix/somevalue' is added to the variable that holds the list +of enabled fragments:: + + OE_FRAGMENTS = "... someprefix/somevalue" + +bitbake will treat that as direct value assignment in its configuration:: + + SOMEVARIABLE = "somevalue" + +Locating Include Files +---------------------- + +BitBake first searches the current directory for needed include files for +:ref:`include ` and :ref:`require ` +directives. +BitBake also uses the :term:`BBPATH` variable. + +.. note:: + + The :term:`BBPATH` variable is analogous to the environment variable ``PATH``. + +For these two directives, BitBake includes the first file it finds. + +Let's consider the following statement called from a recipe file located in +``/layers/meta-custom2/recipes-example/example/example_0.1.bb``:: + + require myfile.inc + +Where ``myfile.inc`` is located in ``/layers/meta-custom2/recipes-example/example``. + +And let's assume that the value of :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to find +``myfile.inc`` in this order:: + + /layers/meta-custom2/recipes-example/example/myfile.inc + /layers/meta-custom1/myfile.inc + /layers/meta-custom2/myfile.inc + +In this case the first path of the list matches and BitBake includes this file +in ``example_0.1.bb``. + +Another common example would be:: + + require recipes-other/other/otherfile.inc + +Where ``otherfile.inc`` is located in +``/layers/meta-custom1/recipes-other/other/``. + +In this case, the following paths would be searched:: + + /layers/meta-custom2/recipes-example/example/recipes-other/other/otherfile.inc + /layers/meta-custom1/recipes-other/other/otherfile.inc + /layers/meta-custom2/recipes-other/other/otherfile.inc + +This time, the second item of this list would be matched. + +Note that the first path is based on the location of the file with the +``require`` (or ``include``) directive. Imagine there's a +``/layers/meta-custom2/recipes-bbappend/example/example_0.1.bbappend`` with:: + + require myappend.inc + +In this case, the following paths would be searched:: + + /layers/meta-custom2/recipes-bbappend/example/myappend.inc + /layers/meta-custom1/myappend.inc + /layers/meta-custom2/myappend.inc + +.. note:: + + In the above examples, the exact same search order applies for the + :ref:`include ` directive. + +It is also possible to include *all* occurences of a file with the same name +with the :ref:`include_all ` directive. + +Let's consider the following statement called from a recipe file located in +``/layers/meta-custom2/recipes-example/example/exampleall_0.1.bb``:: + + include_all all.inc + +Where multiple ``all.inc`` are in located in ``/layers/meta-custom2`` and +``/layers/meta-custom1``. + +And let's assume that the value of :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to include +all ``all.inc`` in this order:: + + /layers/meta-custom1/all.inc + /layers/meta-custom2/all.inc + +In this case the first ``/layers/meta-custom1/all.inc`` would be included, and +then ``/layers/meta-custom2/all.inc`` if both are found. + +.. note:: + + The same logic as for the :ref:`ref-include-directive` applies, except that + the path relative to the file where the directive is specified is not + searched when using the ``include_all`` directive. + +Locating Class Files +-------------------- + +Like include files, class files are located using the :term:`BBPATH` variable. +The classes can be included in the ``classes-recipe``, ``classes-global`` and +``classes`` directories, as explained in the +:ref:`bitbake-user-manual/bitbake-user-manual-intro:Class types` section of the +BitBake User Manual. Like for the :ref:`include ` and +:ref:`require ` directives, BitBake stops and inherits the +first class that it finds. + +For classes inherited with the :ref:`inherit +` directive, BitBake will try to +locate the class under each ``classes-recipe`` directory for each path in +:term:`BBPATH`, and then do the same for each ``classes`` directory for each +path in :term:`BBPATH`. + +For example, if the value :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2`` then the ``hello`` class +would be searched in this order:: + + /layers/meta-custom1/classes-recipe/hello.bbclass + /layers/meta-custom2/classes-recipe/hello.bbclass + /layers/meta-custom1/classes/hello.bbclass + /layers/meta-custom2/classes/hello.bbclass + +.. note:: + + Note that the order of the list above does not depend on where the class is + inherited from. + +Likewise, for classes inherited with the :term:`INHERIT` variable, the +``classes-global`` directory is searched first, and the ``classes`` directory is +searched second. Taking the above example, this would result in the following +list:: + + /layers/meta-custom1/classes-global/hello.bbclass + /layers/meta-custom2/classes-global/hello.bbclass + /layers/meta-custom1/classes/hello.bbclass + /layers/meta-custom2/classes/hello.bbclass + Functions ========= @@ -972,7 +1223,9 @@ When you create these types of functions in your recipe or class files, you need to follow the shell programming rules. The scripts are executed by ``/bin/sh``, which may not be a bash shell but might be something such as ``dash``. You should not use -Bash-specific script (bashisms). +Bash-specific script (bashisms), such as ``[[`` (use ``[`` instead). The BitBake +shell parser also has further +:ref:`limitations `. Overrides and override-style operators like ``:append`` and ``:prepend`` can also be applied to shell functions. Most commonly, this application @@ -1013,6 +1266,20 @@ Running ``do_foo`` prints the following:: You can use the ``bitbake -e recipename`` command to view the final assembled function after all overrides have been applied. +Limitations +^^^^^^^^^^^ + +Shell functions must be parsable by BitBake, which cannot handle all +syntax supported by POSIX shells. In particular: + +#. Arithmetic expansion is not supported. This means you cannot do: + + .. code-block:: shell + + VAR=$(( i + 1 )) + + See https://bugzilla.yoctoproject.org/show_bug.cgi?id=11314. + BitBake-Style Python Functions ------------------------------ @@ -1303,8 +1570,8 @@ the task and other tasks. Here is an example that shows how to define a task and declare some dependencies:: python do_printdate () { - import time - print time.strftime('%Y%m%d', time.gmtime()) + import datetime + bb.plain('Date: %s' % (datetime.date.today())) } addtask printdate after do_fetch before do_build @@ -1496,22 +1763,22 @@ functionality of the task: directory listed is used as the current working directory for the task. -- ``[file-checksums]``: Controls the file dependencies for a task. The - baseline file list is the set of files associated with - :term:`SRC_URI`. May be used to set additional dependencies on - files not associated with :term:`SRC_URI`. +- ``[file-checksums]``: Controls the file dependencies for a task. The + baseline file list is the set of files associated with + :term:`SRC_URI`. May be used to set additional dependencies on + files not associated with :term:`SRC_URI`. - The value set to the list is a file-boolean pair where the first - value is the file name and the second is whether or not it - physically exists on the filesystem. :: + The value set to the list is a file-boolean pair where the first + value is the file name and the second is whether or not it + physically exists on the filesystem. :: - do_configure[file-checksums] += "${MY_DIRPATH}/my-file.txt:True" + do_configure[file-checksums] += "${MY_DIRPATH}/my-file.txt:True" - It is important to record any paths which the task looked at and - which didn't exist. This means that if these do exist at a later - time, the task can be rerun with the new additional files. The - "exists" True or False value after the path allows this to be - handled. + It is important to record any paths which the task looked at and + which didn't exist. This means that if these do exist at a later + time, the task can be rerun with the new additional files. The + "exists" True or False value after the path allows this to be + handled. - ``[lockfiles]``: Specifies one or more lockfiles to lock while the task executes. Only one task may hold a lockfile, and any task that @@ -1524,6 +1791,9 @@ functionality of the task: shouldn't access the network outside of ``do_fetch`` as it usually undermines fetcher source mirroring, image and licence manifests, software auditing and supply chain security. + Tasks without the network flag will run with PR_SET_NO_NEW_PRIVS, + which prevents executables from gaining privileges through + file capabilities. - ``[noexec]``: When set to "1", marks the task as being empty, with no execution required. You can use the ``[noexec]`` flag to set up @@ -1635,8 +1905,9 @@ and the content of the :term:`FILE` variable:: print("The name of the Event is %s" % getName(e)) print("The file we run for is %s" % d.getVar('FILE')) } - myclass_eventhandler[eventmask] = "bb.event.BuildStarted - bb.event.BuildCompleted" + myclass_eventhandler[eventmask] = "bb.event.BuildStarted \ + bb.event.BuildCompleted \ + " In the previous example, an eventmask has been set so that the handler only sees the "BuildStarted" and @@ -1972,11 +2243,8 @@ access. Here is a list of available operations: Other Functions --------------- -You can find many other functions that can be called from Python by -looking at the source code of the ``bb`` module, which is in -``bitbake/lib/bb``. For example, ``bitbake/lib/bb/utils.py`` includes -the commonly used functions ``bb.utils.contains()`` and -``bb.utils.mkdirhier()``, which come with docstrings. +Other functions are documented in the +:doc:`/bitbake-user-manual/bitbake-user-manual-library-functions` document. Extending Python Library Code ----------------------------- @@ -2016,12 +2284,12 @@ OpenEmbedded metadata-based example. These checksums are stored in :term:`STAMP`. You can examine the checksums using the following BitBake command:: - $ bitbake-dumpsigs + $ bitbake-dumpsig This command returns the signature data in a readable format that allows you to examine the inputs used when the OpenEmbedded build system generates signatures. For example, using -``bitbake-dumpsigs`` allows you to examine the ``do_compile`` task's +``bitbake-dumpsig`` allows you to examine the ``do_compile`` task's "sigdata" for a C application (e.g. ``bash``). Running the command also reveals that the "CC" variable is part of the inputs that are hashed. Any changes to this variable would invalidate the stamp and cause the diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 899e584f..fd75275b 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -36,6 +36,14 @@ overview of their function and contents. when specified allows for the Git binary from the host to be used rather than building ``git-native``. + :term:`AUTOREV` + This is a special variable used during fetching. When :term:`SRCREV` is + set to the value of this variable, the latest revision from the version + controlled source code repository is used. + It should be set as follows:: + + SRCREV = "${AUTOREV}" + :term:`AZ_SAS` Azure Storage Shared Access Signature, when using the :ref:`Azure Storage fetcher ` @@ -108,6 +116,10 @@ overview of their function and contents. Specifies if SSL certificates should be checked when fetching. The default value is ``1`` and certificates are not checked if the value is set to ``0``. + :term:`BB_CMDLINE` + :term:`BB_CMDLINE` is an internal variable and is read-only. It captures + the exact command line used for the current invocation of BitBake. + :term:`BB_HASH_CODEPARSER_VALS` Specifies values for variables to use when populating the codeparser cache. This can be used selectively to set dummy values for variables to avoid @@ -123,21 +135,24 @@ overview of their function and contents. Specifies the path to a log file into which BitBake's user interface writes output during the build. + This variable's value remains persistent during the entire execution of + the BitBake server. Changes to this variable will only take effect + after a server restart, which can be accomplished with the ``bitbake + --kill-server`` command. + + .. note:: + + For more details on how to make the BitBake server persistent, see the + ``BB_SERVER_TIMEOUT`` variable. + :term:`BB_CURRENTTASK` Contains the name of the currently running task. The name does not include the ``do_`` prefix. - :term:`BB_DANGLINGAPPENDS_WARNONLY` - Defines how BitBake handles situations where an append file - (``.bbappend``) has no corresponding recipe file (``.bb``). This - condition often occurs when layers get out of sync (e.g. ``oe-core`` - bumps a recipe version and the old recipe no longer exists and the - other layer has not been updated to the new version of the recipe - yet). - - The default fatal behavior is safest because it is the sane reaction - given something is out of sync. It is important to realize when your - changes are no longer being applied. + :term:`BB_CURRENT_MC` + Contains the name of the current multiconfig a task is being run under. + The name is taken from the multiconfig configuration file (a file + ``mc1.conf`` would make this variable equal to ``mc1``). :term:`BB_DEFAULT_TASK` The default task to use when none is specified (e.g. with the ``-c`` @@ -148,6 +163,25 @@ overview of their function and contents. The default umask to apply to tasks if specified and no task specific umask flag is set. + :term:`BB_DEFER_BBCLASSES` + The classes listed in this variable have their :ref:`inherit + ` calls automatically promoted + to deferred inherits. See :ref:`inherit_defer + ` for more information on + deferred inherits. + + This means that if :term:`BB_DEFER_BBCLASSES` is set as follows:: + + BB_DEFER_BBCLASSES = "foo" + + The following statement:: + + inherit foo + + Will automatically be equal to calling:: + + inherit_defer foo + :term:`BB_DISKMON_DIRS` Monitors disk space and available inodes during the build and allows you to control the build based on these parameters. @@ -262,8 +296,8 @@ overview of their function and contents. Specifies the internal list of variables to allow through from the external environment into BitBake's datastore. If the value of this variable is not specified (which is the default), the following - list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, - :term:`BB_ENV_PASSTHROUGH`, and :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. + list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, and + :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. .. note:: @@ -317,21 +351,42 @@ overview of their function and contents. For example usage, see :term:`BB_GIT_SHALLOW`. + :term:`BB_GIT_DEFAULT_DESTSUFFIX` + The default destination directory where the :ref:`Git fetcher + ` unpacks the source code. If this variable is not set, the + source code is unpacked in a directory named "git". + :term:`BB_GIT_SHALLOW` Setting this variable to "1" enables the support for fetching, using and generating mirror tarballs of `shallow git repositories `_. - The external `git-make-shallow `_ - script is used for shallow mirror tarball creation. + A git shallow fetch is used for shallow mirror tarball creation (i.e. + ``git fetch --depth ...`` or + ``git fetch --shallow-exclude= ...``. When :term:`BB_GIT_SHALLOW` is enabled, bitbake will attempt to fetch a shallow mirror tarball. If the shallow mirror tarball cannot be fetched, it will try to fetch the full mirror tarball and use that. - When a mirror tarball is not available, a full git clone will be performed - regardless of whether this variable is set or not. Support for shallow - clones is not currently implemented as git does not directly support - shallow cloning a particular git commit hash (it only supports cloning - from a tag or branch reference). + This setting causes an initial shallow clone instead of an initial full bare clone. + The amount of data transferred during the initial clone will be significantly reduced. + + However, every time the source revision (referenced in :term:`SRCREV`) + changes, regardless of whether the cache within the download directory + (defined by :term:`DL_DIR`) has been cleaned up or not, + the data transfer may be significantly higher because entirely + new shallow clones are required for each source revision change. + + Over time, numerous shallow clones may cumulatively transfer + the same amount of data as an initial full bare clone. + This is especially the case with very large repositories. + + Existing initial full bare clones, created without this setting, + will still be utilized. + + If the Git error "Server does not allow request for unadvertised object" + occurs, an initial full bare clone is fetched automatically. + This may happen if the Git server does not allow the request + or if the Git client has issues with this functionality. See also :term:`BB_GIT_SHALLOW_DEPTH` and :term:`BB_GENERATE_SHALLOW_TARBALLS`. @@ -359,6 +414,23 @@ overview of their function and contents. For example usage, see :term:`BB_GIT_SHALLOW`. + :term:`BB_GIT_SHALLOW_EXTRA_REFS` + With :term:`BB_GIT_SHALLOW` enabled, by default, all unused refs (branches + and tags) are removed from the repository, as shallow processing scales + with the number of refs it has to process. + + :term:`BB_GIT_SHALLOW_EXTRA_REFS` allows to explicitly specify additional + refs to keep. This is particularly useful for recipes with custom checkout + processes, or whose Git-based versioning requires a tag be available (i.e. + for ``git describe --tags``). The :term:`BB_GIT_SHALLOW_EXTRA_REFS` + variable is a space-separated list of refs, fully specified, and supports + wildcards. + + Example usage:: + + BB_GIT_SHALLOW_EXTRA_REFS = "refs/tags/v1.0" + BB_GIT_SHALLOW_EXTRA_REFS += "refs/heads/*" + :term:`BB_GLOBAL_PYMODULES` Specifies the list of Python modules to place in the global namespace. It is intended that only the core layer should set this and it is meant @@ -414,6 +486,29 @@ overview of their function and contents. equivalences that correspond to Share State caches that are only available on specific clients. + :term:`BB_HASHSERVE_DB_DIR` + When :term:`BB_HASHSERVE` is set to ``auto``, bitbake will use + a private location inside a particular build directory for the sqlite + database file that holds hash equivalency data. + + This variable allows using a different path, which can be shared + between multiple build directories and bitbake instances + that operate on them. This enables using a common ``${SSTATE_DIR}`` + together with common hash equivalency data for local builds, without having to + separately manage a hash equivalency server. + + .. note:: + + This variable cannot be set to a NFS mount and bitbake will error out then. + The reason is that NFS implementations can have file locking issues, which + can cause data loss and corruption when there are multiple writers operating + on a file at the same time as explained in https://sqlite.org/faq.html#q5 + + If you'd like to share hash equivalency data between multiple computers, you + need to set up a hash equivalency server separately and point :term:`BB_HASHSERVE` + to it. See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for + additional information. + :term:`BB_HASHSERVE_UPSTREAM` Specifies an upstream Hash Equivalence server. @@ -424,7 +519,7 @@ overview of their function and contents. Example usage:: - BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687" + BB_HASHSERVE_UPSTREAM = "hashserv.yoctoproject.org:8686" :term:`BB_INVALIDCONF` Used in combination with the ``ConfigParsed`` event to trigger @@ -525,11 +620,32 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represents the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. + + A default value to limit the CPU pressure to be set in ``conf/local.conf`` + could be:: + + BB_PRESSURE_MAX_CPU = "15000" + + Multiple values should be tested on the build host to determine what suits + best, depending on the need for performance versus load average during + the build. + + .. note:: + + You can track the pressure information while a build is running with: + + .. code-block:: console - This threshold can be set in ``conf/local.conf`` as:: + $ tail -F tmp/log/cooker//console-latest.log | grep "Pressure status changed" + NOTE: Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads + ... - BB_PRESSURE_MAX_CPU = "500" + If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_CPU` + should be increased to a reasonable value for limiting the CPU pressure + on the system. Monitor the varying value after ``CPU:`` above to set a + sensible value. :term:`BB_PRESSURE_MAX_IO` Specifies a maximum I/O pressure threshold, above which BitBake's @@ -541,14 +657,38 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represents the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. At this point in time, experiments show that IO pressure tends to be short-lived and regulating just the CPU with :term:`BB_PRESSURE_MAX_CPU` can help to reduce it. - :term:`BB_PRESSURE_MAX_MEMORY` + A default value to limit the IO pressure to be set in ``conf/local.conf`` + could be:: + + BB_PRESSURE_MAX_IO = "15000" + + Multiple values should be tested on the build host to determine what suits + best, depending on the need for performance versus I/O usage during the + build. + + .. note:: + + You can track the pressure information while a build is running with: + + .. code-block:: console + $ tail -F tmp/log/cooker//console-latest.log | grep "Pressure status changed" + NOTE: Pressure status changed to CPU: None, IO: True, Mem: False (CPU: 2236.0/None, IO: 153.6/2.0, Mem: 0.0/2.0) - using 19/64 bitbake threads + ... + + If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_IO` + should be increased to a reasonable value for limiting the I/O pressure + on the system. Monitor the varying value after ``IO:`` above to set a + sensible value. + + :term:`BB_PRESSURE_MAX_MEMORY` Specifies a maximum memory pressure threshold, above which BitBake's scheduler will not start new tasks (providing there is at least one active task). If no value is set, memory pressure is not @@ -558,7 +698,8 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represents the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. @@ -566,6 +707,30 @@ overview of their function and contents. might be useful as a last resort to prevent OOM errors if they are occurring during builds. + A default value to limit the memory pressure to be set in + ``conf/local.conf`` could be:: + + BB_PRESSURE_MAX_MEMORY = "15000" + + Multiple values should be tested on the build host to determine what suits + best, depending on the need for performance versus memory consumption + during the build. + + .. note:: + + You can track the pressure information while a build is running with: + + .. code-block:: console + + $ tail -F tmp/log/cooker//console-latest.log | grep "Pressure status changed" + NOTE: Pressure status changed to CPU: None, IO: False, Mem: True (CPU: 29.5/None, IO: 0.0/2.0, Mem: 2553.3/2.0) - using 17/64 bitbake threads + ... + + If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_MEMORY` + should be increased to a reasonable value for limiting the memory + pressure on the system. Monitor the varying value after ``Mem:`` above + to set a sensible value. + :term:`BB_RUNFMT` Specifies the name of the executable script files (i.e. run files) saved into ``${``\ :term:`T`\ ``}``. By default, the @@ -680,11 +845,11 @@ overview of their function and contents. .. note:: In order for your I/O priority settings to take effect, you need the - Completely Fair Queuing (CFQ) Scheduler selected for the backing block + Budget Fair Queuing (BFQ) Scheduler selected for the backing block device. To select the scheduler, use the following command form where device is the device (e.g. sda, sdb, and so forth):: - $ sudo sh -c "echo cfq > /sys/block/device/queu/scheduler" + $ sudo sh -c "echo bfq > /sys/block/device/queue/scheduler" :term:`BB_TASK_NICE_LEVEL` Allows specific tasks to change their priority (i.e. nice level). @@ -699,6 +864,12 @@ overview of their function and contents. Within an executing task, this variable holds the hash of the task as returned by the currently enabled signature generator. + :term:`BB_USE_HOME_NPMRC` + Controls whether or not BitBake uses the user's .npmrc file within their + home directory within the npm fetcher. This can be used for authentication + of private NPM registries, among other uses. This is turned off by default + and requires the user to explicitly set it to "1" to enable. + :term:`BB_VERBOSE_LOGS` Controls how verbose BitBake is during builds. If set, shell scripts echo commands and shell script output appears on standard out @@ -766,6 +937,10 @@ overview of their function and contents. :term:`BBFILE_PRIORITY` Assigns the priority for recipe files in each layer. + This variable is used in the ``conf/layer.conf`` file and must be + suffixed with a `_` followed by the name of the specific layer (e.g. + ``BBFILE_PRIORITY_emenlow``). Colon as separator is not supported. + This variable is useful in situations where the same recipe appears in more than one layer. Setting this variable allows you to prioritize a layer against other layers that contain the same recipe @@ -780,7 +955,7 @@ overview of their function and contents. higher precedence. For example, the value 6 has a higher precedence than the value 5. If not specified, the :term:`BBFILE_PRIORITY` variable is set based on layer dependencies (see the :term:`LAYERDEPENDS` variable - for more information. The default priority, if unspecified for a + for more information). The default priority, if unspecified for a layer with no dependencies, is the lowest defined priority + 1 (or 1 if no priorities are defined). @@ -884,28 +1059,101 @@ overview of their function and contents. compiler. Consequently, the syntax follows Python's Regular Expression (re) syntax. The expressions are compared against the full paths to the files. For complete syntax information, see Python's - documentation at http://docs.python.org/3/library/re.html. + documentation at https://docs.python.org/3/library/re.html. The following example uses a complete regular expression to tell - BitBake to ignore all recipe and recipe append files in the - ``meta-ti/recipes-misc/`` directory:: + BitBake to ignore all recipe and recipe append files in + ``recipes-bsp`` directory (recursively) of ``meta-ti-bsp``:: - BBMASK = "meta-ti/recipes-misc/" + BBMASK = "${BBFILE_PATTERN_meta-ti-bsp}/recipes-bsp/" If you want to mask out multiple directories or recipes, you can specify multiple regular expression fragments. This next example masks out multiple directories and individual recipes:: - BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/" - BBMASK += "/meta-oe/recipes-support/" - BBMASK += "/meta-foo/.*/openldap" - BBMASK += "opencv.*\.bbappend" - BBMASK += "lzma" + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal/" + BBMASK += "${BBFILE_PATTERN_openembedded-layer}/recipes-support/" + BBMASK += "${BBFILE_PATTERN_openembedded-layer}/.*/openldap" + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/.*/optee.*\.bbappend" + + This masks: + + - everything under the ``recipes-graphics/libgal/`` directory from + ``meta-ti-bsp``, + - everything under the ``recipes-support/`` directory in ``meta-oe``, + - everything under a directory whose name starts with ``openldap``, and + every file with the same naming scheme, in ``meta-oe`` at any directory + depth > 1 (e.g. in ``meta-oe``, ``recipes-foo/openldap-stuff/`` or + ``recipes-bar/baz/openldap_0.1.bb`` but not ``openldap/``), + - every append file whose name starts with ``optee`` in ``meta-ti-bsp`` at + any directory depth > 1 (e.g. ``optee/optee-examples_%.bbappend`` and + ``recipes-security/optee/optee-client_%.bbappend``). .. note:: - When specifying a directory name, use the trailing slash character - to ensure you match just that directory name. + Because these are complete regular expressions, if you want to match a + directory and not a file, you must end the expression with a trailing + slash. That is:: + + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal/" + + Will match anything under ``recipes-graphics/ligbal/`` directory of + ``meta-ti-bsp``. And:: + + BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal" + + Will match in ``meta-ti-bsp`` any file prefixed with ``libgal`` in + ``recipes-graphics/`` and any directory (recursively; and its + recipes and recipe append files regardless how they are named) prefixed + with ``libgal`` in ``recipes-graphics/``. That is, provided your layers + are available at ``/bitbake-builds/poky-master/layers/``, it'll match:: + + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal.bb + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal_%.bbappend + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal-foo/foo.bb + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal-foo/foo/bz.bbappend + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal/bar.bb + + .. note:: + + Because these are complete regular expressions, failing to start the + pattern with a ``^`` sign (which is usually the first character in + :term:`BBFILE_PATTERN`) means it can match *any* portion of a path. + Take the following as an example:: + + BBMASK = "recipes-graphics/libgal/" + + This will match subdirectories and files in any path containing + ``recipes-graphics/libgal/``, meaning (considering your layers are + available at ``/bitbake-builds/poky-master/layers/``):: + + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal/ + /bitbake-builds/poky-master/layers/my-layer/foo-recipes-graphics/libgal/ + + will be both matched. This may be a more relaxed way of matching + directories, recipes and recipe append files from any third party layer + but is generally discouraged as it may be casting too wide of a net. + + .. note:: + + Because these are complete regular expressions, a leading slash does + not mean the path is absolute. It simply forces the directory to be + named exactly that. Take:: + + BBMASK = "recipes-graphics/libgal/" + + If you happen to have a directory ``foo-recipes-graphics/libgal/``, it + will be matched. + + Leading with a slash:: + + BBMASK = "/recipes-graphics/libgal/" + + makes sure that doesn't happen. However, this doesn't prevent matching + a directory ``recipes-graphics/libgal/`` from another layer. + + Again, we highly recommend using :term:`BBFILE_PATTERN` in the + expressions to specify absolute paths specific to one layer. :term:`BBMULTICONFIG` Enables BitBake to perform multiple configuration builds and lists @@ -963,18 +1211,10 @@ overview of their function and contents. A name assigned to the build. The name defaults to a datetime stamp of when the build was started but can be defined by the metadata. - :term:`BZRDIR` - The directory in which files checked out of a Bazaar system are - stored. - :term:`CACHE` Specifies the directory BitBake uses to store a cache of the metadata so it does not need to be parsed every time BitBake is started. - :term:`CVSDIR` - The directory in which files checked out under the CVS system are - stored. - :term:`DEFAULT_PREFERENCE` Specifies a weak bias for recipe selection priority. @@ -1100,9 +1340,17 @@ overview of their function and contents. :term:`INHERIT` Causes the named class or classes to be inherited globally. Anonymous - functions in the class or classes are not executed for the base - configuration and in each individual recipe. The OpenEmbedded build - system ignores changes to :term:`INHERIT` in individual recipes. + functions in the class or classes are executed in two disjoint situations: + + - When only the + :ref:`base configuration ` + is parsed. For example as a result of the following BitBake invocation:: + + $ bitbake -e + + - When recipes are parsed - then for each parsed recipe. + + BitBake ignores changes to :term:`INHERIT` in individual recipes. For more information on :term:`INHERIT`, see the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`inherit\`\` configuration directive`" @@ -1178,6 +1426,58 @@ overview of their function and contents. upstream source, and then locations specified by :term:`MIRRORS` in that order. + This variable has a specific syntax. Each line must contain a regular + expression matching the original source URL and a replacement URL for it. + For example:: + + MIRRORS:prepend = " \ + git://git.openembedded.org/.* git:///mirrors/openembedded/BASENAME \ + " + + In the above example: + + - The leftmost part means that any Git repository (``git://`` fetcher) + whose URL starts with ``git.openembedded.org/`` will be matched. + + - The rightmost part is the replacement URL for the original repository, + which is hosted locally in the ``/mirrors/openembedded`` directory. It + also uses the ``BASENAME`` keyword which is detailed below. + + The rightmost part of each line can contain keywords which are + automatically replaced by BitBake. These keywords can contain either a + sub-part of the original URL or meta-information about it. + + For what follows, let's take the following URL as an example for the + original source (matched by the leftmost part of the line): + ``git://git.openembedded.org/project/repo.git``. + + A replacement for this URL can be created using the following keywords: + + - ``TYPE``: the URL scheme or fetcher type. For the example above: ``git``. + - ``HOST``: the URL host. For the example above: ``git.openembedded.org``. + - ``PATH``: the URL path. For the example above: ``/project/repo.git``. + - ``BASENAME``: the URL basename. For the example above: ``repo.git``. + - ``MIRRORNAME``: a translation of the host and path with common + characters substituted to dots (``.``). For the example above: + ``git.openembedded.org.project.repo.git``. + + The rightmost part of the line can also be a bare URL that uses no special + keywords (usually ``https://`` or ``file://``). For example:: + + MIRRORS:prepend = " \ + git://git.openembedded.org/.* file:///mirrors/openembedded/ \ + " + + In the above example, the fetcher will try to find a mirror tarball in the + ``/mirrors/openembedded`` directory. The name of this tarball should follow + the name of the tarball created by BitBake (for Git repositories, + the :term:`BB_GENERATE_MIRROR_TARBALLS` variable must be set to "1" to + generate them). + + If we take ``git://git.openembedded.org/project/repo.git`` as an example, + the tarball would have to be named + ``git2_git.openembedded.org.project.repo.git.tar.gz``. + :term:`OVERRIDES` A colon-separated list that BitBake uses to control what variables are overridden after BitBake parses recipes and configuration files. @@ -1213,8 +1513,13 @@ overview of their function and contents. :term:`PERSISTENT_DIR` Specifies the directory BitBake uses to store data that should be preserved between builds. In particular, the data stored is the data - that uses BitBake's persistent data API and the data used by the PR - Server and PR Service. + that uses BitBake's persistent data API, the data used by the PR + Server and PR Service, and the default location of the Hash Equivalence + database (when :term:`BB_HASHSERVE` is set to ``auto``). + + This directory should not be shared between different builds. If you need + to share the Hash Equivalence database, you should setup a Hash + Equivalence server instead. :term:`PF` Specifies the recipe or package name and includes all version and @@ -1253,6 +1558,20 @@ overview of their function and contents. PREFERRED_PROVIDER_xxx = "yyy" PREFERRED_PROVIDER_aaa = "bbb" + :term:`PREFERRED_RPROVIDER` + Determines which *recipe* should be given preference when + multiple packages declare runtime-providing (:term:`RPROVIDES`) + the same item. Some examples:: + + PREFERRED_RPROVIDER_initd-functions ?= "initscripts" + PREFERRED_RPROVIDER_virtual-libegl-icd ?= "mesa" + + The former will select as the package built by the `initscripts` + recipe declaring to be runtime-provider for `initd-functions`, + which is in this case `initscripts-functions`:: + + .../initscripts_1.0.bb:RPROVIDES:${PN}-functions = "initd-functions" + :term:`PREFERRED_VERSION` If there are multiple versions of a recipe available, this variable determines which version should be given preference. You must always @@ -1301,6 +1620,10 @@ overview of their function and contents. use ``file://`` URLs to point to local directories or network shares as well. + The syntax of the :term:`PREMIRRORS` variable is the same as + :term:`MIRRORS`. See the definition of the :term:`MIRRORS` for more + details. + :term:`PROVIDES` A list of aliases by which a particular recipe can be known. By default, a recipe's own :term:`PN` is implicitly already in its @@ -1460,14 +1783,8 @@ overview of their function and contents. - ``az://``: Fetches files from an Azure Storage account using HTTPS. - - ``bzr://``: Fetches files from a Bazaar revision control - repository. - - ``ccrc://``: Fetches files from a ClearCase repository. - - ``cvs://``: Fetches files from a CVS revision control - repository. - - ``file://``: Fetches files, which are usually files shipped with the Metadata, from the local machine. The path is relative to the :term:`FILESPATH` @@ -1502,8 +1819,11 @@ overview of their function and contents. - ``npm://``: Fetches JavaScript modules from a registry. - - ``osc://``: Fetches files from an OSC (OpenSUSE Build service) - revision control repository. + .. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. - ``p4://``: Fetches files from a Perforce (``p4``) revision control repository. @@ -1522,7 +1842,8 @@ overview of their function and contents. - ``name``: Specifies a name to be used for association with :term:`SRC_URI` checksums or :term:`SRCREV` when you have more than one - file or git repository specified in :term:`SRC_URI`. For example:: + file or source control repository specified in :term:`SRC_URI`. + For example:: SRC_URI = "git://example.com/foo.git;branch=main;name=first \ git://example.com/bar.git;branch=main;name=second \ @@ -1535,7 +1856,8 @@ overview of their function and contents. - ``subdir``: Places the file (or extracts its contents) into the specified subdirectory. This option is useful for unusual tarballs or other archives that do not have their files already in a - subdirectory within the archive. + subdirectory within the archive. This path can be further modified + by fetcher specific parameters. - ``subpath``: Limits the checkout to a specific subpath of the tree when using the Git fetcher is used. @@ -1550,18 +1872,18 @@ overview of their function and contents. :term:`SRCREV` The revision of the source code used to build the package. This - variable applies only when using Subversion, Git, Mercurial and - Bazaar. If you want to build a fixed revision and you want to avoid + variable applies only when using Subversion, Git and Mercurial. + If you want to build a fixed revision and you want to avoid performing a query on the remote repository every time BitBake parses your recipe, you should specify a :term:`SRCREV` that is a full revision identifier and not just a tag. :term:`SRCREV_FORMAT` - Helps construct valid :term:`SRCREV` values when + Helps construct a valid package version string when multiple source controlled URLs are used in :term:`SRC_URI`. - The system needs help constructing these values under these + The system needs help constructing this value under these circumstances. Each component in the :term:`SRC_URI` is assigned a name and these are referenced in the :term:`SRCREV_FORMAT` variable. Consider an example with URLs named "machine" and "meta". In this case, @@ -1570,6 +1892,12 @@ overview of their function and contents. ``AUTOINC`` placeholder is added and if needed. And, this placeholder is placed at the start of the returned string. + The :term:`SRCREV_FORMAT` can also take the form "_component2". + This assumes that there is a component in the :term:`SRC_URI` that does not + have a name assigned. While this is not considered good practice, it can be + usefull if a ``.bbappend`` file needs to extend the :term:`SRC_URI` with + an additional repository. + :term:`STAMP` Specifies the base path used to create recipe stamp files. The path to an actual stamp file is constructed by evaluating this string and diff --git a/bitbake/doc/conf.py b/bitbake/doc/conf.py index fc2ee081..9a53e55c 100644 --- a/bitbake/doc/conf.py +++ b/bitbake/doc/conf.py @@ -6,18 +6,27 @@ # -- Path setup -------------------------------------------------------------- -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -# -# import os -# import sys -# sys.path.insert(0, os.path.abspath('.')) - -import sys import datetime +import os +import subprocess +import sys + +from pathlib import Path -current_version = "dev" +# Test that we are building from a Git repository +try: + subprocess.run(["git", "rev-parse", "--is-inside-work-tree"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True) +except subprocess.CalledProcessError: + sys.exit("Building bitbake's documentation must be done from its Git repository.\n" + "Clone the repository with the following command:\n" + "git clone https://git.openembedded.org/bitbake ") + +sys.path.insert(0, os.path.abspath('.')) +import setversions + +current_version = setversions.get_current_version() +print(f"Version calculated to be {current_version}") # String used in sidebar version = 'Version: ' + current_version @@ -26,6 +35,12 @@ if current_version == 'dev': # Version seen in documentation_options.js and hence in js switchers code release = current_version +setversions.write_switchers_js("sphinx-static/switchers.js.in", + "sphinx-static/switchers.js", + current_version) + +setversions.write_releases_rst("releases.rst") + # -- Project information ----------------------------------------------------- project = 'Bitbake' @@ -37,6 +52,7 @@ author = 'Richard Purdie, Chris Larson, and Phil Blundell' extlinks = { 'yocto_docs': ('https://docs.yoctoproject.org%s', None), 'oe_lists': ('https://lists.openembedded.org%s', None), + 'wikipedia': ('https://en.wikipedia.org/wiki/%s', None), } # -- General configuration --------------------------------------------------- @@ -47,6 +63,7 @@ extlinks = { extensions = [ 'sphinx.ext.autosectionlabel', 'sphinx.ext.extlinks', + 'sphinx.ext.autodoc', ] autosectionlabel_prefix_document = True @@ -99,3 +116,7 @@ html_last_updated_fmt = '%b %d, %Y' # Remove the trailing 'dot' in section numbers html_secnumber_suffix = " " + +# autoconf needs the modules available to auto-generate documentation from the +# code +sys.path.insert(0, str(Path('..', 'lib').resolve())) diff --git a/bitbake/doc/index.rst b/bitbake/doc/index.rst index ee1660ac..9f2a9067 100644 --- a/bitbake/doc/index.rst +++ b/bitbake/doc/index.rst @@ -12,10 +12,12 @@ BitBake User Manual bitbake-user-manual/bitbake-user-manual-intro bitbake-user-manual/bitbake-user-manual-execution + bitbake-user-manual/bitbake-user-manual-environment-setup bitbake-user-manual/bitbake-user-manual-metadata bitbake-user-manual/bitbake-user-manual-ref-variables-context bitbake-user-manual/bitbake-user-manual-fetching bitbake-user-manual/bitbake-user-manual-ref-variables + bitbake-user-manual/bitbake-user-manual-library-functions bitbake-user-manual/bitbake-user-manual-hello .. toctree:: diff --git a/bitbake/doc/releases.rst b/bitbake/doc/releases.rst deleted file mode 100644 index b38b1c06..00000000 --- a/bitbake/doc/releases.rst +++ /dev/null @@ -1,174 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.5 - -================================= -BitBake Supported Release Manuals -================================= - -******************************* -Release Series 4.2 (mickledore) -******************************* - -- :yocto_docs:`BitBake 2.4 User Manual ` - -****************************** -Release Series 4.0 (kirkstone) -****************************** - -- :yocto_docs:`BitBake 2.0 User Manual ` - -**************************** -Release Series 3.1 (dunfell) -**************************** - -- :yocto_docs:`BitBake 1.46 User Manual ` - -================================ -BitBake Outdated Release Manuals -================================ - -***************************** -Release Series 4.1 (langdale) -***************************** - -- :yocto_docs:`BitBake 2.2 User Manual ` - -****************************** -Release Series 3.4 (honister) -****************************** - -- :yocto_docs:`BitBake 1.52 User Manual ` - -****************************** -Release Series 3.3 (hardknott) -****************************** - -- :yocto_docs:`BitBake 1.50 User Manual ` - -******************************* -Release Series 3.2 (gatesgarth) -******************************* - -- :yocto_docs:`BitBake 1.48 User Manual ` - -******************************************* -Release Series 3.1 (dunfell first versions) -******************************************* - -- :yocto_docs:`3.1 BitBake User Manual ` -- :yocto_docs:`3.1.1 BitBake User Manual ` -- :yocto_docs:`3.1.2 BitBake User Manual ` -- :yocto_docs:`3.1.3 BitBake User Manual ` - -************************* -Release Series 3.0 (zeus) -************************* - -- :yocto_docs:`3.0 BitBake User Manual ` -- :yocto_docs:`3.0.1 BitBake User Manual ` -- :yocto_docs:`3.0.2 BitBake User Manual ` -- :yocto_docs:`3.0.3 BitBake User Manual ` -- :yocto_docs:`3.0.4 BitBake User Manual ` - -**************************** -Release Series 2.7 (warrior) -**************************** - -- :yocto_docs:`2.7 BitBake User Manual ` -- :yocto_docs:`2.7.1 BitBake User Manual ` -- :yocto_docs:`2.7.2 BitBake User Manual ` -- :yocto_docs:`2.7.3 BitBake User Manual ` -- :yocto_docs:`2.7.4 BitBake User Manual ` - -************************* -Release Series 2.6 (thud) -************************* - -- :yocto_docs:`2.6 BitBake User Manual ` -- :yocto_docs:`2.6.1 BitBake User Manual ` -- :yocto_docs:`2.6.2 BitBake User Manual ` -- :yocto_docs:`2.6.3 BitBake User Manual ` -- :yocto_docs:`2.6.4 BitBake User Manual ` - -************************* -Release Series 2.5 (sumo) -************************* - -- :yocto_docs:`2.5 Documentation ` -- :yocto_docs:`2.5.1 Documentation ` -- :yocto_docs:`2.5.2 Documentation ` -- :yocto_docs:`2.5.3 Documentation ` - -************************** -Release Series 2.4 (rocko) -************************** - -- :yocto_docs:`2.4 BitBake User Manual ` -- :yocto_docs:`2.4.1 BitBake User Manual ` -- :yocto_docs:`2.4.2 BitBake User Manual ` -- :yocto_docs:`2.4.3 BitBake User Manual ` -- :yocto_docs:`2.4.4 BitBake User Manual ` - -************************* -Release Series 2.3 (pyro) -************************* - -- :yocto_docs:`2.3 BitBake User Manual ` -- :yocto_docs:`2.3.1 BitBake User Manual ` -- :yocto_docs:`2.3.2 BitBake User Manual ` -- :yocto_docs:`2.3.3 BitBake User Manual ` -- :yocto_docs:`2.3.4 BitBake User Manual ` - -************************** -Release Series 2.2 (morty) -************************** - -- :yocto_docs:`2.2 BitBake User Manual ` -- :yocto_docs:`2.2.1 BitBake User Manual ` -- :yocto_docs:`2.2.2 BitBake User Manual ` -- :yocto_docs:`2.2.3 BitBake User Manual ` - -**************************** -Release Series 2.1 (krogoth) -**************************** - -- :yocto_docs:`2.1 BitBake User Manual ` -- :yocto_docs:`2.1.1 BitBake User Manual ` -- :yocto_docs:`2.1.2 BitBake User Manual ` -- :yocto_docs:`2.1.3 BitBake User Manual ` - -*************************** -Release Series 2.0 (jethro) -*************************** - -- :yocto_docs:`1.9 BitBake User Manual ` -- :yocto_docs:`2.0 BitBake User Manual ` -- :yocto_docs:`2.0.1 BitBake User Manual ` -- :yocto_docs:`2.0.2 BitBake User Manual ` -- :yocto_docs:`2.0.3 BitBake User Manual ` - -************************* -Release Series 1.8 (fido) -************************* - -- :yocto_docs:`1.8 BitBake User Manual ` -- :yocto_docs:`1.8.1 BitBake User Manual ` -- :yocto_docs:`1.8.2 BitBake User Manual ` - -************************** -Release Series 1.7 (dizzy) -************************** - -- :yocto_docs:`1.7 BitBake User Manual ` -- :yocto_docs:`1.7.1 BitBake User Manual ` -- :yocto_docs:`1.7.2 BitBake User Manual ` -- :yocto_docs:`1.7.3 BitBake User Manual ` - -************************** -Release Series 1.6 (daisy) -************************** - -- :yocto_docs:`1.6 BitBake User Manual ` -- :yocto_docs:`1.6.1 BitBake User Manual ` -- :yocto_docs:`1.6.2 BitBake User Manual ` -- :yocto_docs:`1.6.3 BitBake User Manual ` - diff --git a/bitbake/doc/setversions.py b/bitbake/doc/setversions.py new file mode 100755 index 00000000..2df384e9 --- /dev/null +++ b/bitbake/doc/setversions.py @@ -0,0 +1,374 @@ +#!/usr/bin/env python3 +# +# This file defines is used in doc/conf.py to setup the version information for +# the documentation: +# - get_current_version() used in doc/conf.py computes the current version by +# trying to guess the approximate versions we're at using git tags and +# branches from the repository. +# - write_switchers_js() write the switchers.js file used for switching between +# versions of the documentation. +# +# Copyright (c) 2026 Antonin Godard +# +# SPDX-License-Identifier: MIT +# + +import argparse +import itertools +import json +import os +import re +import subprocess +import sys +import textwrap + +from urllib.request import urlopen, URLError + +# NOTE: the following variables contain default values in case we are not able to fetch +# the releases.json file from https://dashboard.yoctoproject.org/releases.json +DEVBRANCH = "2.18" +LTSSERIES = ["2.8", "2.0"] +ACTIVERELEASES = ["2.16"] + LTSSERIES +YOCTO_MAPPING = { + "2.18": "wrynose", + "2.16": "whinlatter", + "2.8": "scarthgap", + "2.0": "kirkstone", +} + +RELEASES_FROM_JSON = {} + +# Use the local releases.json file if found, fetch it from the dashboard otherwise +releases_json_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "releases.json") +try: + with open(releases_json_path, "r") as f: + RELEASES_FROM_JSON = json.load(f) +except FileNotFoundError: + print("Fetching releases.json from https://dashboard.yoctoproject.org/releases.json...", + file=sys.stderr) + try: + with urlopen("https://dashboard.yoctoproject.org/releases.json") as r, \ + open(releases_json_path, "w") as f: + RELEASES_FROM_JSON = json.load(r) + json.dump(RELEASES_FROM_JSON, f) + except URLError: + print("WARNING: tried to fetch https://dashboard.yoctoproject.org/releases.json " + "but failed, using default values for active releases", file=sys.stderr) + pass + +if RELEASES_FROM_JSON: + ACTIVERELEASES = [] + DEVBRANCH = "" + LTSSERIES = [] + YOCTO_MAPPING = {} + + for release in RELEASES_FROM_JSON: + bb_ver = release["bitbake_version"] + if release["status"] == "Active Development": + DEVBRANCH = bb_ver + if "LTS until" in release["status"]: + LTSSERIES.append(bb_ver) + if release["bitbake_version"]: + YOCTO_MAPPING[bb_ver] = release["release_codename"] + + # Find the first non-dev release, which should be displayed as the default + # page on the docs website. + current_branch = "" + for release in RELEASES_FROM_JSON: + if release["status"] != "Active Development": + current_branch = release["bitbake_version"] + break + + if not current_branch: + sys.exit("Unable to find a current release! Exiting...") + + # make the list of releases unique, there can be duplication when the + # current releases is also an LTS + ACTIVERELEASES = list(dict.fromkeys([current_branch] + LTSSERIES)) + +print(f"ACTIVERELEASES calculated to be {ACTIVERELEASES}", file=sys.stderr) +print(f"DEVBRANCH calculated to be {DEVBRANCH}", file=sys.stderr) +print(f"LTSSERIES calculated to be {LTSSERIES}", file=sys.stderr) + +BB_RELEASE_TAG_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+$") + +def main(): + parser = argparse.ArgumentParser( + description="Parse https://dashboard.yoctoproject.org/releases.json to get current releases information" + ) + parser.add_argument("--get-latest-branch", + help="Print current latest branch and exit", + action="store_true", + default=False) + args = parser.parse_args() + + if args.get_latest_branch: + print(ACTIVERELEASES[0]) + sys.exit(0) + +def get_current_version(): + # Test tags exist and inform the user to fetch if not + try: + subprocess.run(["git", "show", f"{LTSSERIES[-1]}.0"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True) + except subprocess.CalledProcessError: + sys.exit("Please run 'git fetch --tags' before building the documentation") + + # Try and figure out what we are + tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout + for t in tags.split(): + if re.match(BB_RELEASE_TAG_RE, t): + return t + + # We're floating on a branch + branch = subprocess.run(["git", "branch", "--show-current"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout.strip() + + if branch == "" or branch not in list(YOCTO_MAPPING.keys()) + ["master", "master-next"]: + # We're not on a known release branch so we have to guess. Compare the + # numbers of commits from each release branch and assume the smallest + # number of commits is the one we're based off + possible_branch = None + branch_count = 0 + for b in itertools.chain(YOCTO_MAPPING.keys(), ["master"]): + result = subprocess.run(["git", "log", "--format=oneline", "HEAD..origin/" + b], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True) + if result.returncode == 0: + count = result.stdout.count('\n') + if not possible_branch or count < branch_count: + print("Branch %s has count %s" % (b, count)) + possible_branch = b + branch_count = count + if possible_branch: + branch = possible_branch + else: + branch = "master" + print("Nearest release branch estimated to be %s" % branch) + + if branch == "master": + return "dev" + + if branch == "master-next": + return "next" + + ourversion = branch + head_commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout.strip() + branch_commit = subprocess.run(["git", "rev-parse", "--short", branch], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True).stdout.strip() + if head_commit != branch_commit: + ourversion += f" ({head_commit})" + + return ourversion + +def write_switchers_js(js_in, js_out, current_version): + with open(js_in, "r") as r, open(js_out, "w") as w: + lines = r.readlines() + for line in lines: + if "VERSIONS_PLACEHOLDER" in line: + if current_version != "dev": + w.write(" 'dev': 'Unstable (dev)',\n") + for series in ACTIVERELEASES: + w.write(f" '{series}': '{series} ({YOCTO_MAPPING[series]})',\n") + else: + w.write(line) + print("switchers.js generated from switchers.js.in") + +def _release_section(series_version: str, codename: str, bitbake_version: str) -> str: + """ + Helper function to generate a release section, as: + + ******************** + Release Series xxxxx + ******************** + + - + """ + section_length = len(series_version) + len(codename) + 18 + return textwrap.dedent( + f"""\ + {'*' * section_length} + Release Series {series_version} ({codename}) + {'*' * section_length} + + - :yocto_docs:`BitBake {bitbake_version} User Manual ` + + """) + +def write_releases_rst(releases_rst_out: str): + """ + Generates the releases.rst file automatically, based on what is found + in the releases.json file. + """ + with open(releases_rst_out, "w") as f: + f.write(textwrap.dedent("""\ + .. SPDX-License-Identifier: CC-BY-2.5 + + ================================= + BitBake Supported Release Manuals + ================================= + + """)) + + for release in RELEASES_FROM_JSON: + if release["status"] == "Active Development": + continue + + if not release["bitbake_version"]: + continue + + if release["series"] == "current": + f.write(_release_section( + release["series_version"], + release["release_codename"], + release["bitbake_version"])) + + f.write(textwrap.dedent("""\ + ================================ + BitBake Outdated Release Manuals + ================================ + + """)) + + for release in RELEASES_FROM_JSON: + if not release["series"] == "previous": + continue + + if not release["bitbake_version"]: + continue + + f.write(_release_section( + release["series_version"], + release["release_codename"], + release["bitbake_version"])) + + # old legacy links, which cannot be auto-generated + f.write(textwrap.dedent( + """\ + - :yocto_docs:`3.1.2 BitBake User Manual ` + - :yocto_docs:`3.1 BitBake User Manual ` + - :yocto_docs:`3.1.1 BitBake User Manual ` + - :yocto_docs:`3.1.3 BitBake User Manual ` + + ************************* + Release Series 3.0 (Zeus) + ************************* + + - :yocto_docs:`3.0 BitBake User Manual ` + - :yocto_docs:`3.0.1 BitBake User Manual ` + - :yocto_docs:`3.0.2 BitBake User Manual ` + - :yocto_docs:`3.0.3 BitBake User Manual ` + - :yocto_docs:`3.0.4 BitBake User Manual ` + + **************************** + Release Series 2.7 (Warrior) + **************************** + + - :yocto_docs:`2.7 BitBake User Manual ` + - :yocto_docs:`2.7.1 BitBake User Manual ` + - :yocto_docs:`2.7.2 BitBake User Manual ` + - :yocto_docs:`2.7.3 BitBake User Manual ` + - :yocto_docs:`2.7.4 BitBake User Manual ` + + ************************* + Release Series 2.6 (Thud) + ************************* + + - :yocto_docs:`2.6 BitBake User Manual ` + - :yocto_docs:`2.6.1 BitBake User Manual ` + - :yocto_docs:`2.6.2 BitBake User Manual ` + - :yocto_docs:`2.6.3 BitBake User Manual ` + - :yocto_docs:`2.6.4 BitBake User Manual ` + + ************************* + Release Series 2.5 (Sumo) + ************************* + + - :yocto_docs:`2.5 Documentation ` + - :yocto_docs:`2.5.1 Documentation ` + - :yocto_docs:`2.5.2 Documentation ` + - :yocto_docs:`2.5.3 Documentation ` + + ************************** + Release Series 2.4 (Rocko) + ************************** + + - :yocto_docs:`2.4 BitBake User Manual ` + - :yocto_docs:`2.4.1 BitBake User Manual ` + - :yocto_docs:`2.4.2 BitBake User Manual ` + - :yocto_docs:`2.4.3 BitBake User Manual ` + - :yocto_docs:`2.4.4 BitBake User Manual ` + + ************************* + Release Series 2.3 (Pyro) + ************************* + + - :yocto_docs:`2.3 BitBake User Manual ` + - :yocto_docs:`2.3.1 BitBake User Manual ` + - :yocto_docs:`2.3.2 BitBake User Manual ` + - :yocto_docs:`2.3.3 BitBake User Manual ` + - :yocto_docs:`2.3.4 BitBake User Manual ` + + ************************** + Release Series 2.2 (Morty) + ************************** + + - :yocto_docs:`2.2 BitBake User Manual ` + - :yocto_docs:`2.2.1 BitBake User Manual ` + - :yocto_docs:`2.2.2 BitBake User Manual ` + - :yocto_docs:`2.2.3 BitBake User Manual ` + + **************************** + Release Series 2.1 (Krogoth) + **************************** + + - :yocto_docs:`2.1 BitBake User Manual ` + - :yocto_docs:`2.1.1 BitBake User Manual ` + - :yocto_docs:`2.1.2 BitBake User Manual ` + - :yocto_docs:`2.1.3 BitBake User Manual ` + + *************************** + Release Series 2.0 (Jethro) + *************************** + + - :yocto_docs:`1.9 BitBake User Manual ` + - :yocto_docs:`2.0 BitBake User Manual ` + - :yocto_docs:`2.0.1 BitBake User Manual ` + - :yocto_docs:`2.0.2 BitBake User Manual ` + - :yocto_docs:`2.0.3 BitBake User Manual ` + + ************************* + Release Series 1.8 (Fido) + ************************* + + - :yocto_docs:`1.8 BitBake User Manual ` + - :yocto_docs:`1.8.1 BitBake User Manual ` + - :yocto_docs:`1.8.2 BitBake User Manual ` + + ************************** + Release Series 1.7 (Dizzy) + ************************** + + - :yocto_docs:`1.7 BitBake User Manual ` + - :yocto_docs:`1.7.1 BitBake User Manual ` + - :yocto_docs:`1.7.2 BitBake User Manual ` + - :yocto_docs:`1.7.3 BitBake User Manual ` + + ************************** + Release Series 1.6 (Daisy) + ************************** + + - :yocto_docs:`1.6 BitBake User Manual ` + - :yocto_docs:`1.6.1 BitBake User Manual ` + - :yocto_docs:`1.6.2 BitBake User Manual ` + - :yocto_docs:`1.6.3 BitBake User Manual ` + """)) + +if __name__ == "__main__": + main() diff --git a/bitbake/doc/sphinx-static/switchers.js b/bitbake/doc/sphinx-static/switchers.js deleted file mode 100644 index 32113cfa..00000000 --- a/bitbake/doc/sphinx-static/switchers.js +++ /dev/null @@ -1,233 +0,0 @@ -(function() { - 'use strict'; - - var all_versions = { - 'dev': 'dev (3.2)', - '3.1.2': '3.1.2', - '3.0.3': '3.0.3', - '2.7.4': '2.7.4', - }; - - var all_doctypes = { - 'single': 'Individual Webpages', - 'mega': "All-in-one 'Mega' Manual", - }; - - // Simple version comparision - // Return 1 if a > b - // Return -1 if a < b - // Return 0 if a == b - function ver_compare(a, b) { - if (a == "dev") { - return 1; - } - - if (a === b) { - return 0; - } - - var a_components = a.split("."); - var b_components = b.split("."); - - var len = Math.min(a_components.length, b_components.length); - - // loop while the components are equal - for (var i = 0; i < len; i++) { - // A bigger than B - if (parseInt(a_components[i]) > parseInt(b_components[i])) { - return 1; - } - - // B bigger than A - if (parseInt(a_components[i]) < parseInt(b_components[i])) { - return -1; - } - } - - // If one's a prefix of the other, the longer one is greater. - if (a_components.length > b_components.length) { - return 1; - } - - if (a_components.length < b_components.length) { - return -1; - } - - // Otherwise they are the same. - return 0; - } - - function build_version_select(current_series, current_version) { - var buf = [''); - return buf.join(''); - } - - function build_doctype_select(current_doctype) { - var buf = [''); - return buf.join(''); - } - - function navigate_to_first_existing(urls) { - // Navigate to the first existing URL in urls. - var url = urls.shift(); - - // Web browsers won't redirect file:// urls to file urls using ajax but - // its useful for local testing - if (url.startsWith("file://")) { - window.location.href = url; - return; - } - - if (urls.length == 0) { - window.location.href = url; - return; - } - $.ajax({ - url: url, - success: function() { - window.location.href = url; - }, - error: function() { - navigate_to_first_existing(urls); - } - }); - } - - function get_docroot_url() { - var url = window.location.href; - var root = DOCUMENTATION_OPTIONS.URL_ROOT; - - var urlarray = url.split('/'); - // Trim off anything after '/' - urlarray.pop(); - var depth = (root.match(/\.\.\//g) || []).length; - for (var i = 0; i < depth; i++) { - urlarray.pop(); - } - - return urlarray.join('/') + '/'; - } - - function on_version_switch() { - var selected_version = $(this).children('option:selected').attr('value'); - var url = window.location.href; - var current_version = DOCUMENTATION_OPTIONS.VERSION; - var docroot = get_docroot_url() - - var new_versionpath = selected_version + '/'; - if (selected_version == "dev") - new_versionpath = ''; - - // dev versions have no version prefix - if (current_version == "dev") { - var new_url = docroot + new_versionpath + url.replace(docroot, ""); - var fallback_url = docroot + new_versionpath; - } else { - var new_url = url.replace('/' + current_version + '/', '/' + new_versionpath); - var fallback_url = new_url.replace(url.replace(docroot, ""), ""); - } - - console.log(get_docroot_url()) - console.log(url + " to url " + new_url); - console.log(url + " to fallback " + fallback_url); - - if (new_url != url) { - navigate_to_first_existing([ - new_url, - fallback_url, - 'https://www.yoctoproject.org/docs/', - ]); - } - } - - function on_doctype_switch() { - var selected_doctype = $(this).children('option:selected').attr('value'); - var url = window.location.href; - if (selected_doctype == 'mega') { - var docroot = get_docroot_url() - var current_version = DOCUMENTATION_OPTIONS.VERSION; - // Assume manuals before 3.2 are using old docbook mega-manual - if (ver_compare(current_version, "3.2") < 0) { - var new_url = docroot + "mega-manual/mega-manual.html"; - } else { - var new_url = docroot + "singleindex.html"; - } - } else { - var new_url = url.replace("singleindex.html", "index.html") - } - - if (new_url != url) { - navigate_to_first_existing([ - new_url, - 'https://www.yoctoproject.org/docs/', - ]); - } - } - - // Returns the current doctype based upon the url - function doctype_segment_from_url(url) { - if (url.includes("singleindex") || url.includes("mega-manual")) - return "mega"; - return "single"; - } - - $(document).ready(function() { - var release = DOCUMENTATION_OPTIONS.VERSION; - var current_doctype = doctype_segment_from_url(window.location.href); - var current_series = release.substr(0, 3); - var version_select = build_version_select(current_series, release); - - $('.version_switcher_placeholder').html(version_select); - $('.version_switcher_placeholder select').bind('change', on_version_switch); - - var doctype_select = build_doctype_select(current_doctype); - - $('.doctype_switcher_placeholder').html(doctype_select); - $('.doctype_switcher_placeholder select').bind('change', on_doctype_switch); - - if (ver_compare(release, "3.1") < 0) { - $('#outdated-warning').html('Version ' + release + ' of the project is now considered obsolete, please select and use a more recent version'); - $('#outdated-warning').css('padding', '.5em'); - } else if (release != "dev") { - $.each(all_versions, function(version, title) { - var series = version.substr(0, 3); - if (series == current_series && version != release) { - $('#outdated-warning').html('This document is for outdated version ' + release + ', you should select the latest release version in this series, ' + version + '.'); - $('#outdated-warning').css('padding', '.5em'); - } - }); - } - }); -})(); diff --git a/bitbake/doc/sphinx-static/switchers.js.in b/bitbake/doc/sphinx-static/switchers.js.in new file mode 100644 index 00000000..0b209e95 --- /dev/null +++ b/bitbake/doc/sphinx-static/switchers.js.in @@ -0,0 +1,249 @@ +// SPDX-License-Identifier: MIT +(function () { + "use strict"; + + var all_versions = { + VERSIONS_PLACEHOLDER, + }; + + var all_doctypes = { + single: "Individual Webpages", + mega: "All-in-one 'Mega' Manual", + }; + + // Simple version comparision + // Return 1 if a > b + // Return -1 if a < b + // Return 0 if a == b + function ver_compare(a, b) { + if (a == "dev") { + return 1; + } + + if (a === b) { + return 0; + } + + var a_components = a.split("."); + var b_components = b.split("."); + + var len = Math.min(a_components.length, b_components.length); + + // loop while the components are equal + for (var i = 0; i < len; i++) { + // A bigger than B + if (parseInt(a_components[i]) > parseInt(b_components[i])) { + return 1; + } + + // B bigger than A + if (parseInt(a_components[i]) < parseInt(b_components[i])) { + return -1; + } + } + + // If one's a prefix of the other, the longer one is greater. + if (a_components.length > b_components.length) { + return 1; + } + + if (a_components.length < b_components.length) { + return -1; + } + + // Otherwise they are the same. + return 0; + } + + function build_version_select(current_version) { + var buf = [""); + return buf.join(""); + } + + function build_doctype_select(current_doctype) { + var buf = [""); + return buf.join(""); + } + + function navigate_to_first_existing(urls) { + // Navigate to the first existing URL in urls. + var url = urls.shift(); + + // Web browsers won't redirect file:// urls to file urls using ajax but + // its useful for local testing + if (url.startsWith("file://")) { + window.location.href = url; + return; + } + + if (urls.length == 0) { + window.location.href = url; + return; + } + $.ajax({ + url: url, + success: function () { + window.location.href = url; + }, + error: function () { + navigate_to_first_existing(urls); + }, + }); + } + + function get_docroot_url() { + var url = window.location.href; + // Try to get the variable from documentation_options.js + var root = DOCUMENTATION_OPTIONS.URL_ROOT; + if (root == null) { + // In recent versions of Sphinx, URL_ROOT was removed from + // documentation_options.js, so get it like searchtools.js does. + root = document.documentElement.dataset.content_root; + } + + var urlarray = url.split("/"); + // Trim off anything after '/' + urlarray.pop(); + var depth = (root.match(/\.\.\//g) || []).length; + for (var i = 0; i < depth; i++) { + urlarray.pop(); + } + + return urlarray.join("/") + "/"; + } + + function on_version_switch() { + var selected_version = $(this).children("option:selected").attr("value"); + var url = window.location.href; + var current_version = DOCUMENTATION_OPTIONS.VERSION; + var docroot = get_docroot_url(); + + var new_versionpath = selected_version + "/"; + + // latest tag is also the default page (without version information) + if (docroot.endsWith("dev/")) { + var new_url = url.replace("/dev/", "/" + new_versionpath); + var fallback_url = new_url.replace(url.replace(docroot, ""), ""); + } else if (docroot.endsWith(current_version + "/") == false) { + var new_url = docroot + new_versionpath + url.replace(docroot, ""); + var fallback_url = docroot + new_versionpath; + } else { + var new_url = url.replace( + "/" + current_version + "/", + "/" + new_versionpath, + ); + var fallback_url = new_url.replace(url.replace(docroot, ""), ""); + } + + console.log(url + " to url " + new_url); + console.log(url + " to fallback " + fallback_url); + + if (new_url != url) { + navigate_to_first_existing([ + new_url, + fallback_url, + "https://www.yoctoproject.org/bitbake/", + ]); + } + } + + function on_doctype_switch() { + var selected_doctype = $(this).children("option:selected").attr("value"); + var url = window.location.href; + if (selected_doctype == "mega") { + var docroot = get_docroot_url(); + var current_version = DOCUMENTATION_OPTIONS.VERSION; + var new_url = docroot + "singleindex.html"; + } else { + var new_url = url.replace("singleindex.html", "index.html"); + } + + if (new_url != url) { + navigate_to_first_existing([ + new_url, + "https://www.yoctoproject.org/docs/", + ]); + } + } + + // Returns the current doctype based upon the url + function doctype_segment_from_url(url) { + if (url.includes("singleindex") || url.includes("mega-manual")) + return "mega"; + return "single"; + } + + $(document).ready(function () { + var release = DOCUMENTATION_OPTIONS.VERSION; + var current_doctype = doctype_segment_from_url(window.location.href); + var current_series = release.substr(0, 3); + var version_select = build_version_select(release); + + $(".version_switcher_placeholder").html(version_select); + $(".version_switcher_placeholder select").bind("change", on_version_switch); + + var doctype_select = build_doctype_select(current_doctype); + + $(".doctype_switcher_placeholder").html(doctype_select); + $(".doctype_switcher_placeholder select").bind("change", on_doctype_switch); + + // if release = "X.Y ()", remove the "()" so that only X.Y is compared + release = release.split(" ")[0]; + if (!(["dev", "next"].includes(release)) && !(release in all_versions)) { + $("#outdated-warning").html( + "Version " + release + " of the project is now considered obsolete, please select and use a more recent version", + ); + $("#outdated-warning").css("padding", ".5em"); + } + }); +})(); diff --git a/bitbake/doc/sphinx-static/theme_overrides.css b/bitbake/doc/sphinx-static/theme_overrides.css index e362677a..3a303c8e 100644 --- a/bitbake/doc/sphinx-static/theme_overrides.css +++ b/bitbake/doc/sphinx-static/theme_overrides.css @@ -99,14 +99,19 @@ em { [alt='Permalink'] { color: #eee; } [alt='Permalink']:hover { color: black; } +.literal-block { + background: #f8f8f8; +} + @media screen { - /* content column - * - * RTD theme's default is 800px as max width for the content, but we have - * tables with tons of columns, which need the full width of the view-port. - */ + .wy-nav-content { + max-width: 1000px; + background: #fcfcfc; + } - .wy-nav-content{max-width: none; } + .wy-nav-content-wrap { + background: #efefef; + } /* inline literal: drop the borderbox, padding and red color */ code, .rst-content tt, .rst-content code { diff --git a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index 76bc08a3..4af03c54 100644 --- a/bitbake/lib/bb/COW.py +++ b/bitbake/lib/bb/COW.py @@ -36,8 +36,9 @@ class COWDictMeta(COWMeta): __marker__ = tuple() def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, len(cls.__dict__) - 3) + ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"]) + keys = set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, len(keys)) __repr__ = __str__ @@ -161,8 +162,9 @@ class COWDictMeta(COWMeta): class COWSetMeta(COWDictMeta): def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, len(cls.__dict__) - 3) + ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"]) + keys = set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, len(keys)) __repr__ = __str__ diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index 7d9bc147..01c73f1b 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -9,12 +9,11 @@ # SPDX-License-Identifier: GPL-2.0-only # -__version__ = "2.8.0" +__version__ = "2.20.0" import sys -# It was 3.8.0 originally but set to 3.7.3 for Debian Buster -if sys.version_info < (3, 7, 3): - raise RuntimeError("Sorry, python 3.7.3 or later is required for this version of bitbake") +if sys.version_info < (3, 9, 0): + raise RuntimeError("Sorry, python 3.9.0 or later is required for this version of bitbake") if sys.version_info < (3, 10, 0): # With python 3.8 and 3.9, we see errors of "libgcc_s.so.1 must be installed for pthread_cancel to work" @@ -37,6 +36,35 @@ class BBHandledException(Exception): import os import logging +from collections import namedtuple +import multiprocessing as mp + +# Python 3.14 changes the default multiprocessing context from "fork" to +# "forkserver". However, bitbake heavily relies on "fork" behavior to +# efficiently pass data to the child processes. Places that need this should do: +# from bb import multiprocessing +# in place of +# import multiprocessing + +class MultiprocessingContext(object): + """ + Multiprocessing proxy object that uses the "fork" context for a property if + available, otherwise goes to the main multiprocessing module. This allows + it to be a drop-in replacement for the multiprocessing module, but use the + fork context + """ + def __init__(self): + super().__setattr__("_ctx", mp.get_context("fork")) + + def __getattr__(self, name): + if hasattr(self._ctx, name): + return getattr(self._ctx, name) + return getattr(mp, name) + + def __setattr__(self, name, value): + raise AttributeError(f"Unable to set attribute {name}") + +multiprocessing = MultiprocessingContext() class NullHandler(logging.Handler): @@ -62,6 +90,7 @@ class BBLoggerMixin(object): return self.bbdebug(3, *args, **kwargs) def bbdebug(self, level, msg, *args, **kwargs): + import bb.event loglevel = logging.DEBUG - level + 1 if not bb.event.worker_pid: if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]): @@ -104,26 +133,6 @@ class BBLoggerAdapter(logging.LoggerAdapter, BBLoggerMixin): self.setup_bblogger(logger.name) super().__init__(logger, *args, **kwargs) - if sys.version_info < (3, 6): - # These properties were added in Python 3.6. Add them in older versions - # for compatibility - @property - def manager(self): - return self.logger.manager - - @manager.setter - def manager(self, value): - self.logger.manager = value - - @property - def name(self): - return self.logger.name - - def __repr__(self): - logger = self.logger - level = logger.getLevelName(logger.getEffectiveLevel()) - return '<%s %s (%s)>' % (self.__class__.__name__, logger.name, level) - logging.LoggerAdapter = BBLoggerAdapter logger = logging.getLogger("BitBake") @@ -144,14 +153,39 @@ class PrefixLoggerAdapter(logging.LoggerAdapter): # can result in construction of the various loggers. import bb.msg -from bb import fetch2 as fetch -sys.modules['bb.fetch'] = sys.modules['bb.fetch2'] +# There's lots of code (in OE and other layers) that accesses bb.fetch2 without +# first importing it. It used to work because fetch2 was added to the bb module in +# bb/__init__.py, as part of installing the fetch = fetch2 shim. But now that +# the new fetch2 shim is an actual module, this will break. So, provide lazy +# access to bb.fetch2 via __getattr__ so layers keep working. +def __getattr__(name): + if name == "fetch2": + import bb.fetch2 as fetch2 + return fetch2 + + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") # Messaging convenience functions def plain(*args): + """ + Prints a message at "plain" level (higher level than a ``bb.note()``). + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.plain(''.join(args)) def debug(lvl, *args): + """ + Prints a debug message. + + Arguments: + + - ``lvl``: debug level. Higher value increases the debug level + (determined by ``bitbake -D``). + - ``args``: one or more strings to print. + """ if isinstance(lvl, str): mainlogger.warning("Passed invalid debug level '%s' to bb.debug", lvl) args = (lvl,) + args @@ -159,33 +193,81 @@ def debug(lvl, *args): mainlogger.bbdebug(lvl, ''.join(args)) def note(*args): + """ + Prints a message at "note" level. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.info(''.join(args)) -# -# A higher prioity note which will show on the console but isn't a warning -# -# Something is happening the user should be aware of but they probably did -# something to make it happen -# def verbnote(*args): + """ + A higher priority note which will show on the console but isn't a warning. + + Use in contexts when something is happening the user should be aware of but + they probably did something to make it happen. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.verbnote(''.join(args)) # # Warnings - things the user likely needs to pay attention to and fix # def warn(*args): + """ + Prints a warning message. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.warning(''.join(args)) def warnonce(*args): + """ + Prints a warning message like ``bb.warn()``, but only prints the message + once. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.warnonce(''.join(args)) def error(*args, **kwargs): + """ + Prints an error message. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.error(''.join(args), extra=kwargs) def erroronce(*args): + """ + Prints an error message like ``bb.error()``, but only prints the message + once. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.erroronce(''.join(args)) def fatal(*args, **kwargs): + """ + Prints an error message and stops the BitBake execution. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.critical(''.join(args), extra=kwargs) raise BBHandledException() @@ -214,7 +296,6 @@ def deprecated(func, name=None, advice=""): # For compatibility def deprecate_import(current, modulename, fromlist, renames = None): """Import objects from one module into another, wrapping them with a DeprecationWarning""" - import sys module = __import__(modulename, fromlist = fromlist) for position, objname in enumerate(fromlist): @@ -228,3 +309,14 @@ def deprecate_import(current, modulename, fromlist, renames = None): setattr(sys.modules[current], newname, newobj) +TaskData = namedtuple("TaskData", [ + "pn", + "taskname", + "fn", + "deps", + "provides", + "taskhash", + "unihash", + "hashfn", + "taskhash_deps", +]) diff --git a/bitbake/lib/bs4/COPYING.txt b/bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE similarity index 80% rename from bitbake/lib/bs4/COPYING.txt rename to bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE index d668d13f..08e3a9cf 100644 --- a/bitbake/lib/bs4/COPYING.txt +++ b/bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE @@ -1,6 +1,6 @@ Beautiful Soup is made available under the MIT license: - Copyright (c) 2004-2012 Leonard Richardson + Copyright (c) Leonard Richardson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the @@ -20,7 +20,12 @@ Beautiful Soup is made available under the MIT license: BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE - SOFTWARE, DAMMIT. + SOFTWARE. Beautiful Soup incorporates code from the html5lib library, which is -also made available under the MIT license. +also made available under the MIT license. Copyright (c) James Graham +and other contributors + +Beautiful Soup has an optional dependency on the soupsieve library, +which is also made available under the MIT license. Copyright (c) +Isaac Muse diff --git a/bitbake/lib/bb/_vendor/bs4/__init__.py b/bitbake/lib/bb/_vendor/bs4/__init__.py new file mode 100644 index 00000000..58079b9e --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/__init__.py @@ -0,0 +1,1177 @@ +"""Beautiful Soup Elixir and Tonic - "The Screen-Scraper's Friend". + +http://www.crummy.com/software/BeautifulSoup/ + +Beautiful Soup uses a pluggable XML or HTML parser to parse a +(possibly invalid) document into a tree representation. Beautiful Soup +provides methods and Pythonic idioms that make it easy to navigate, +search, and modify the parse tree. + +Beautiful Soup works with Python 3.7 and up. It works better if lxml +and/or html5lib is installed, but they are not required. + +For more than you ever wanted to know about Beautiful Soup, see the +documentation: http://www.crummy.com/software/BeautifulSoup/bs4/doc/ +""" + +__author__ = "Leonard Richardson (leonardr@segfault.org)" +__version__ = "4.15.0" +__copyright__ = "Copyright (c) 2004-2026 Leonard Richardson" +# Use of this source code is governed by the MIT license. +__license__ = "MIT" + +__all__ = [ + "AttributeResemblesVariableWarning", + "BeautifulSoup", + "Comment", + "Declaration", + "ProcessingInstruction", + "ResultSet", + "CSS", + "Script", + "Stylesheet", + "Tag", + "TemplateString", + "ElementFilter", + "UnicodeDammit", + "CData", + "Doctype", + + # Exceptions + "FeatureNotFound", + "ParserRejectedMarkup", + "StopParsing", + + # Warnings + "AttributeResemblesVariableWarning", + "GuessedAtParserWarning", + "MarkupResemblesLocatorWarning", + "UnusualUsageWarning", + "XMLParsedAsHTMLWarning", +] + +from collections import Counter +import io +import sys +import warnings + +# The very first thing we do is give a useful error if someone is +# running this code under Python 2. +if sys.version_info.major < 3: + raise ImportError( + "You are trying to use a Python 3-specific version of Beautiful Soup under Python 2. This will not work. The final version of Beautiful Soup to support Python 2 was 4.9.3." + ) + +from .builder import ( + builder_registry, + TreeBuilder, +) +from .builder._htmlparser import HTMLParserTreeBuilder +from .dammit import UnicodeDammit +from .css import CSS +from ._deprecation import ( + _deprecated, +) +from .element import ( + CData, + Comment, + DEFAULT_OUTPUT_ENCODING, + Declaration, + Doctype, + NavigableString, + PageElement, + ProcessingInstruction, + PYTHON_SPECIFIC_ENCODINGS, + ResultSet, + Script, + Stylesheet, + Tag, + TemplateString, +) +from .formatter import Formatter +from .filter import ( + ElementFilter, + SoupStrainer, +) +from typing import ( + Any, + cast, + Counter as CounterType, + Dict, + Iterator, + List, + Sequence, + Sized, + Optional, + Type, + Union, +) + +from bb._vendor.bs4._typing import ( + _Encoding, + _Encodings, + _IncomingMarkup, + _InsertableElement, + _RawAttributeValue, + _RawAttributeValues, + _RawMarkup, +) + +# Import all warnings and exceptions into the main package. +from bb._vendor.bs4.exceptions import ( + FeatureNotFound, + ParserRejectedMarkup, + StopParsing, +) +from bb._vendor.bs4._warnings import ( + AttributeResemblesVariableWarning, + GuessedAtParserWarning, + MarkupResemblesLocatorWarning, + UnusualUsageWarning, + XMLParsedAsHTMLWarning, +) + + +class BeautifulSoup(Tag): + """A data structure representing a parsed HTML or XML document. + + Most of the methods you'll call on a BeautifulSoup object are inherited from + PageElement or Tag. + + Internally, this class defines the basic interface called by the + tree builders when converting an HTML/XML document into a data + structure. The interface abstracts away the differences between + parsers. To write a new tree builder, you'll need to understand + these methods as a whole. + + These methods will be called by the BeautifulSoup constructor: + * reset() + * feed(markup) + + The tree builder may call these methods from its feed() implementation: + * handle_starttag(name, attrs) # See note about return value + * handle_endtag(name) + * handle_data(data) # Appends to the current data node + * endData(containerClass) # Ends the current data node + + No matter how complicated the underlying parser is, you should be + able to build a tree using 'start tag' events, 'end tag' events, + 'data' events, and "done with data" events. + + If you encounter an empty-element tag (aka a self-closing tag, + like HTML's
tag), call handle_starttag and then + handle_endtag. + """ + + #: Since `BeautifulSoup` subclasses `Tag`, it's possible to treat it as + #: a `Tag` with a `Tag.name`. Hoever, this name makes it clear the + #: `BeautifulSoup` object isn't a real markup tag. + ROOT_TAG_NAME: str = "[document]" + + #: If the end-user gives no indication which tree builder they + #: want, look for one with these features. + DEFAULT_BUILDER_FEATURES: Sequence[str] = ["html", "fast"] + + #: A string containing all ASCII whitespace characters, used in + #: during parsing to detect data chunks that seem 'empty'. + ASCII_SPACES: str = "\x20\x0a\x09\x0c\x0d" + + # FUTURE PYTHON: + element_classes: Dict[Type[PageElement], Type[PageElement]] #: :meta private: + builder: TreeBuilder #: :meta private: + is_xml: bool + known_xml: Optional[bool] + parse_only: Optional[SoupStrainer] #: :meta private: + + # These members are only used while parsing markup. + markup: Optional[_RawMarkup] #: :meta private: + current_data: List[str] #: :meta private: + currentTag: Optional[Tag] #: :meta private: + tagStack: List[Tag] #: :meta private: + open_tag_counter: CounterType[str] #: :meta private: + preserve_whitespace_tag_stack: List[Tag] #: :meta private: + string_container_stack: List[Tag] #: :meta private: + _most_recent_element: Optional[PageElement] #: :meta private: + + #: Beautiful Soup's best guess as to the character encoding of the + #: original document. + original_encoding: Optional[_Encoding] + + #: The character encoding, if any, that was explicitly defined + #: in the original document. This may or may not match + #: `BeautifulSoup.original_encoding`. + declared_html_encoding: Optional[_Encoding] + + #: This is True if the markup that was parsed contains + #: U+FFFD REPLACEMENT_CHARACTER characters which were not present + #: in the original markup. These mark character sequences that + #: could not be represented in Unicode. + contains_replacement_characters: bool + + def __init__( + self, + markup: _IncomingMarkup = "", + features: Optional[Union[str, Sequence[str]]] = None, + builder: Optional[Union[TreeBuilder, Type[TreeBuilder]]] = None, + parse_only: Optional[SoupStrainer] = None, + from_encoding: Optional[_Encoding] = None, + exclude_encodings: Optional[_Encodings] = None, + element_classes: Optional[Dict[Type[PageElement], Type[PageElement]]] = None, + **kwargs: Any, + ): + """Constructor. + + :param markup: A string or a file-like object representing + markup to be parsed. + + :param features: Desirable features of the parser to be + used. This may be the name of a specific parser ("lxml", + "lxml-xml", "html.parser", or "html5lib") or it may be the + type of markup to be used ("html", "html5", "xml"). It's + recommended that you name a specific parser, so that + Beautiful Soup gives you the same results across platforms + and virtual environments. + + :param builder: A TreeBuilder subclass to instantiate (or + instance to use) instead of looking one up based on + `features`. You only need to use this if you've implemented a + custom TreeBuilder. + + :param parse_only: A SoupStrainer. Only parts of the document + matching the SoupStrainer will be considered. This is useful + when parsing part of a document that would otherwise be too + large to fit into memory. + + :param from_encoding: A string indicating the encoding of the + document to be parsed. Pass this in if Beautiful Soup is + guessing wrongly about the document's encoding. + + :param exclude_encodings: A list of strings indicating + encodings known to be wrong. Pass this in if you don't know + the document's encoding but you know Beautiful Soup's guess is + wrong. + + :param element_classes: A dictionary mapping BeautifulSoup + classes like Tag and NavigableString, to other classes you'd + like to be instantiated instead as the parse tree is + built. This is useful for subclassing Tag or NavigableString + to modify default behavior. + + :param kwargs: For backwards compatibility purposes, the + constructor accepts certain keyword arguments used in + Beautiful Soup 3. None of these arguments do anything in + Beautiful Soup 4; they will result in a warning and then be + ignored. + + Apart from this, any keyword arguments passed into the + BeautifulSoup constructor are propagated to the TreeBuilder + constructor. This makes it possible to configure a + TreeBuilder by passing in arguments, not just by saying which + one to use. + """ + if "convertEntities" in kwargs: + del kwargs["convertEntities"] + warnings.warn( + "BS4 does not respect the convertEntities argument to the " + "BeautifulSoup constructor. Entities are always converted " + "to Unicode characters." + ) + + if "markupMassage" in kwargs: + del kwargs["markupMassage"] + warnings.warn( + "BS4 does not respect the markupMassage argument to the " + "BeautifulSoup constructor. The tree builder is responsible " + "for any necessary markup massage." + ) + + if "smartQuotesTo" in kwargs: + del kwargs["smartQuotesTo"] + warnings.warn( + "BS4 does not respect the smartQuotesTo argument to the " + "BeautifulSoup constructor. Smart quotes are always converted " + "to Unicode characters." + ) + + if "selfClosingTags" in kwargs: + del kwargs["selfClosingTags"] + warnings.warn( + "Beautiful Soup 4 does not respect the selfClosingTags argument to the " + "BeautifulSoup constructor. The tree builder is responsible " + "for understanding self-closing tags." + ) + + if "isHTML" in kwargs: + del kwargs["isHTML"] + warnings.warn( + "Beautiful Soup 4 does not respect the isHTML argument to the " + "BeautifulSoup constructor. Suggest you use " + "features='lxml' for HTML and features='lxml-xml' for " + "XML." + ) + + def deprecated_argument(old_name: str, new_name: str) -> Optional[Any]: + if old_name in kwargs: + warnings.warn( + 'The "%s" argument to the BeautifulSoup constructor ' + 'was renamed to "%s" in Beautiful Soup 4.0.0' + % (old_name, new_name), + DeprecationWarning, + stacklevel=3, + ) + return kwargs.pop(old_name) + return None + + parse_only = parse_only or deprecated_argument("parseOnlyThese", "parse_only") + if parse_only is not None: + # Issue a warning if we can tell in advance that + # parse_only will exclude the entire tree. + if parse_only.excludes_everything: + warnings.warn( + f"The given value for parse_only will exclude everything: {parse_only}", + UserWarning, + stacklevel=3, + ) + + from_encoding = from_encoding or deprecated_argument( + "fromEncoding", "from_encoding" + ) + + if from_encoding and isinstance(markup, str): + warnings.warn( + "You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored." + ) + from_encoding = None + + self.element_classes = element_classes or dict() + + # We need this information to track whether or not the builder + # was specified well enough that we can omit the 'you need to + # specify a parser' warning. + original_builder = builder + original_features = features + + builder_class: Optional[Type[TreeBuilder]] = None + if isinstance(builder, type): + # A builder class was passed in; it needs to be instantiated. + builder_class = builder + builder = None + elif builder is None: + if isinstance(features, str): + features = [features] + if features is None or len(features) == 0: + features = self.DEFAULT_BUILDER_FEATURES + possible_builder_class = builder_registry.lookup(*features) + if possible_builder_class is None: + raise FeatureNotFound( + "Couldn't find a tree builder with the features you " + "requested: %s. Do you need to install a parser library?" + % ",".join(features) + ) + builder_class = possible_builder_class + + # At this point either we have a TreeBuilder instance in + # builder, or we have a builder_class that we can instantiate + # with the remaining **kwargs. + if builder is None: + assert builder_class is not None + builder = builder_class(**kwargs) + if ( + not original_builder + and not ( + original_features == builder.NAME + or ( + isinstance(original_features, str) + and original_features in builder.ALTERNATE_NAMES + ) + ) + and markup + ): + # The user did not tell us which TreeBuilder to use, + # and we had to guess. Issue a warning. + if builder.is_xml: + markup_type = "XML" + else: + markup_type = "HTML" + + # This code adapted from warnings.py so that we get the same line + # of code as our warnings.warn() call gets, even if the answer is wrong + # (as it may be in a multithreading situation). + caller = None + try: + caller = sys._getframe(1) + except ValueError: + pass + if caller: + globals = caller.f_globals + line_number = caller.f_lineno + else: + globals = sys.__dict__ + line_number = 1 + filename = globals.get("__file__") + if filename: + fnl = filename.lower() + if fnl.endswith((".pyc", ".pyo")): + filename = filename[:-1] + if filename: + # If there is no filename at all, the user is most likely in a REPL, + # and the warning is not necessary. + values = dict( + filename=filename, + line_number=line_number, + parser=builder.NAME, + markup_type=markup_type, + ) + warnings.warn( + GuessedAtParserWarning.MESSAGE % values, + GuessedAtParserWarning, + stacklevel=2, + ) + else: + if kwargs: + warnings.warn( + "Keyword arguments to the BeautifulSoup constructor will be ignored. These would normally be passed into the TreeBuilder constructor, but a TreeBuilder instance was passed in as `builder`." + ) + + self.builder = builder + self.is_xml = builder.is_xml + self.known_xml = self.is_xml + self._namespaces = dict() + self.parse_only = parse_only + + if hasattr(markup, "read"): # It's a file-type object. + markup = cast(io.IOBase, markup).read() + elif not isinstance(markup, (bytes, str)) and not hasattr(markup, "__len__"): + raise TypeError( + f"Incoming markup is of an invalid type: {markup!r}. Markup must be a string, a bytestring, or an open filehandle." + ) + elif isinstance(markup, Sized) and len(markup) <= 256 and ( + (isinstance(markup, bytes) and b"<" not in markup and b"\n" not in markup) + or (isinstance(markup, str) and "<" not in markup and "\n" not in markup) + ): + # Issue warnings for a couple beginner problems + # involving passing non-markup to Beautiful Soup. + # Beautiful Soup will still parse the input as markup, + # since that is sometimes the intended behavior. + if not self._markup_is_url(markup): + self._markup_resembles_filename(markup) + + # At this point we know markup is a string or bytestring. If + # it was a file-type object, we've read from it. + markup = cast(_RawMarkup, markup) + + rejections = [] + success = False + for ( + self.markup, + self.original_encoding, + self.declared_html_encoding, + self.contains_replacement_characters, + ) in self.builder.prepare_markup( + markup, from_encoding, exclude_encodings=exclude_encodings + ): + self.reset() + self.builder.initialize_soup(self) + try: + self._feed() + success = True + break + except ParserRejectedMarkup as e: + rejections.append(e) + pass + + if not success: + other_exceptions = [str(e) for e in rejections] + raise ParserRejectedMarkup( + "The markup you provided was rejected by the parser. Trying a different parser or a different encoding may help.\n\nOriginal exception(s) from parser:\n " + + "\n ".join(other_exceptions) + ) + + # Clear out the markup and remove the builder's circular + # reference to this object. + self.markup = None + self.builder.soup = None + + def copy_self(self) -> "BeautifulSoup": + """Create a new BeautifulSoup object with the same TreeBuilder, + but not associated with any markup. + + This is the first step of the deepcopy process. + """ + clone = type(self)("", None, self.builder) + + # Keep track of the encoding of the original document, + # since we won't be parsing it again. + clone.original_encoding = self.original_encoding + return clone + + def __getstate__(self) -> Dict[str, Any]: + # Frequently a tree builder can't be pickled. + d = dict(self.__dict__) + if "builder" in d and d["builder"] is not None and not self.builder.picklable: + d["builder"] = type(self.builder) + # Store the contents as a Unicode string. + d["contents"] = [] + d["markup"] = self.decode() + + # If _most_recent_element is present, it's a Tag object left + # over from initial parse. It might not be picklable and we + # don't need it. + if "_most_recent_element" in d: + del d["_most_recent_element"] + return d + + def __setstate__(self, state: Dict[str, Any]) -> None: + # If necessary, restore the TreeBuilder by looking it up. + self.__dict__ = state + if isinstance(self.builder, type): + self.builder = self.builder() + elif not self.builder: + # We don't know which builder was used to build this + # parse tree, so use a default we know is always available. + self.builder = HTMLParserTreeBuilder() + self.builder.soup = self + self.reset() + self._feed() + + @property + def _is_root(self): + """Yes, a BeautifulSoup object is the root of its parse tree. Used by the _root_object internal property.""" + return True + + @classmethod + @_deprecated( + replaced_by="nothing (private method, will be removed)", version="4.13.0" + ) + def _decode_markup(cls, markup: _RawMarkup) -> str: + """Ensure `markup` is Unicode so it's safe to send into warnings.warn. + + warnings.warn had this problem back in 2010 but fortunately + not anymore. This has not been used for a long time; I just + noticed that fact while working on 4.13.0. + """ + if isinstance(markup, bytes): + decoded = markup.decode("utf-8", "replace") + else: + decoded = markup + return decoded + + @classmethod + def _markup_is_url(cls, markup: _RawMarkup) -> bool: + """Error-handling method to raise a warning if incoming markup looks + like a URL. + + :param markup: A string of markup. + :return: Whether or not the markup resembled a URL + closely enough to justify issuing a warning. + """ + problem: bool = False + if isinstance(markup, bytes): + problem = ( + any(markup.startswith(prefix) for prefix in (b"http:", b"https:")) + and b" " not in markup + ) + elif isinstance(markup, str): + problem = ( + any(markup.startswith(prefix) for prefix in ("http:", "https:")) + and " " not in markup + ) + else: + return False + + if not problem: + return False + warnings.warn( + MarkupResemblesLocatorWarning.URL_MESSAGE % dict(what="URL"), + MarkupResemblesLocatorWarning, + stacklevel=3, + ) + return True + + @classmethod + def _markup_resembles_filename(cls, markup: _RawMarkup) -> bool: + """Error-handling method to issue a warning if incoming markup + resembles a filename. + + :param markup: A string of markup. + :return: Whether or not the markup resembled a filename + closely enough to justify issuing a warning. + """ + markup_b: bytes + + # We're only checking ASCII characters, so rather than write + # the same tests twice, convert Unicode to a bytestring and + # operate on the bytestring. + if isinstance(markup, str): + markup_b = markup.encode("utf8") + else: + markup_b = markup + + # Step 1: does it end with a common textual file extension? + filelike = False + lower = markup_b.lower() + extensions = [b".html", b".htm", b".xml", b".xhtml", b".txt"] + if any(lower.endswith(ext) for ext in extensions): + filelike = True + if not filelike: + return False + + # Step 2: it _might_ be a file, but there are a few things + # we can look for that aren't very common in filenames. + + # Characters that have special meaning to Unix shells. (< was + # excluded before this method was called.) + # + # Many of these are also reserved characters that cannot + # appear in Windows filenames. + for byte in markup_b: + if byte in b"?*#&;>$|": + return False + + # Two consecutive forward slashes (as seen in a URL) or two + # consecutive spaces (as seen in fixed-width data). + # + # (Paths to Windows network shares contain consecutive + # backslashes, so checking that doesn't seem as helpful.) + if b"//" in markup_b: + return False + if b" " in markup_b: + return False + + # A colon in any position other than position 1 (e.g. after a + # Windows drive letter). + if markup_b.startswith(b":"): + return False + colon_i = markup_b.rfind(b":") + if colon_i not in (-1, 1): + return False + + # Step 3: If it survived all of those checks, it's similar + # enough to a file to justify issuing a warning. + warnings.warn( + MarkupResemblesLocatorWarning.FILENAME_MESSAGE % dict(what="filename"), + MarkupResemblesLocatorWarning, + stacklevel=3, + ) + return True + + def _feed(self) -> None: + """Internal method that parses previously set markup, creating a large + number of Tag and NavigableString objects. + """ + # Convert the document to Unicode. + self.builder.reset() + + if self.markup is not None: + self.builder.feed(self.markup) + # Close out any unfinished strings and close all the open tags. + self.endData() + while ( + self.currentTag is not None and self.currentTag.name != self.ROOT_TAG_NAME + ): + self.popTag() + + def reset(self) -> None: + """Reset this object to a state as though it had never parsed any + markup. + """ + Tag.__init__(self, self, self.builder, self.ROOT_TAG_NAME) + self.hidden = True + self.builder.reset() + self.current_data = [] + self.currentTag = None + self.tagStack = [] + self.open_tag_counter = Counter() + self.preserve_whitespace_tag_stack = [] + self.string_container_stack = [] + self._most_recent_element = None + self.pushTag(self) + + def new_tag( + self, + name: str, + namespace: Optional[str] = None, + nsprefix: Optional[str] = None, + attrs: Optional[_RawAttributeValues] = None, + sourceline: Optional[int] = None, + sourcepos: Optional[int] = None, + string: Optional[str] = None, + **kwattrs: _RawAttributeValue, + ) -> Tag: + """Create a new Tag associated with this BeautifulSoup object. + + :param name: The name of the new Tag. + :param namespace: The URI of the new Tag's XML namespace, if any. + :param prefix: The prefix for the new Tag's XML namespace, if any. + :param attrs: A dictionary of this Tag's attribute values; can + be used instead of ``kwattrs`` for attributes like 'class' + that are reserved words in Python. + :param sourceline: The line number where this tag was + (purportedly) found in its source document. + :param sourcepos: The character position within ``sourceline`` where this + tag was (purportedly) found. + :param string: String content for the new Tag, if any. + :param kwattrs: Keyword arguments for the new Tag's attribute values. + + """ + attr_container = self.builder.attribute_dict_class(**kwattrs) + if attrs is not None: + attr_container.update(attrs) + tag_class = self.element_classes.get(Tag, Tag) + + # Assume that this is either Tag or a subclass of Tag. If not, + # the user brought type-unsafety upon themselves. + tag_class = cast(Type[Tag], tag_class) + tag = tag_class( + None, + self.builder, + name, + namespace, + nsprefix, + attr_container, + sourceline=sourceline, + sourcepos=sourcepos, + ) + + if string is not None: + tag.string = string + return tag + + def string_container( + self, base_class: Optional[Type[NavigableString]] = None + ) -> Type[NavigableString]: + """Find the class that should be instantiated to hold a given kind of + string. + + This may be a built-in Beautiful Soup class or a custom class passed + in to the BeautifulSoup constructor. + """ + container = base_class or NavigableString + + # The user may want us to use some other class (hopefully a + # custom subclass) instead of the one we'd use normally. + container = cast( + Type[NavigableString], self.element_classes.get(container, container) + ) + + # On top of that, we may be inside a tag that needs a special + # container class. + if self.string_container_stack and container is NavigableString: + container = self.builder.string_containers.get( + self.string_container_stack[-1].name, container + ) + return container + + def new_string( + self, s: str, subclass: Optional[Type[NavigableString]] = None + ) -> NavigableString: + """Create a new `NavigableString` associated with this `BeautifulSoup` + object. + + :param s: The string content of the `NavigableString` + :param subclass: The subclass of `NavigableString`, if any, to + use. If a document is being processed, an appropriate + subclass for the current location in the document will + be determined automatically. + """ + container = self.string_container(subclass) + return container(s) + + def insert_before(self, *args: _InsertableElement) -> List[PageElement]: + """This method is part of the PageElement API, but `BeautifulSoup` doesn't implement + it because there is nothing before or after it in the parse tree. + """ + raise NotImplementedError( + "BeautifulSoup objects don't support insert_before()." + ) + + def insert_after(self, *args: _InsertableElement) -> List[PageElement]: + """This method is part of the PageElement API, but `BeautifulSoup` doesn't implement + it because there is nothing before or after it in the parse tree. + """ + raise NotImplementedError("BeautifulSoup objects don't support insert_after().") + + def popTag(self) -> Optional[Tag]: + """Internal method called by _popToTag when a tag is closed. + + :meta private: + """ + if not self.tagStack: + # Nothing to pop. This shouldn't happen. + return None + tag = self.tagStack.pop() + if tag.name in self.open_tag_counter: + self.open_tag_counter[tag.name] -= 1 + if ( + self.preserve_whitespace_tag_stack + and tag == self.preserve_whitespace_tag_stack[-1] + ): + self.preserve_whitespace_tag_stack.pop() + if self.string_container_stack and tag == self.string_container_stack[-1]: + self.string_container_stack.pop() + # print("Pop", tag.name) + if self.tagStack: + self.currentTag = self.tagStack[-1] + return self.currentTag + + def pushTag(self, tag: Tag) -> None: + """Internal method called by handle_starttag when a tag is opened. + + :meta private: + """ + # print("Push", tag.name) + if self.currentTag is not None: + self.currentTag.contents.append(tag) + self.tagStack.append(tag) + self.currentTag = self.tagStack[-1] + if tag.name != self.ROOT_TAG_NAME: + self.open_tag_counter[tag.name] += 1 + if tag.name in self.builder.preserve_whitespace_tags: + self.preserve_whitespace_tag_stack.append(tag) + if tag.name in self.builder.string_containers: + self.string_container_stack.append(tag) + + def endData(self, containerClass: Optional[Type[NavigableString]] = None) -> None: + """Method called by the TreeBuilder when the end of a data segment + occurs. + + :param containerClass: The class to use when incorporating the + data segment into the parse tree. + + :meta private: + """ + if self.current_data: + current_data = "".join(self.current_data) + # If whitespace is not preserved, and this string contains + # nothing but ASCII spaces, replace it with a single space + # or newline. + if not self.preserve_whitespace_tag_stack: + strippable = True + for i in current_data: + if i not in self.ASCII_SPACES: + strippable = False + break + if strippable: + if "\n" in current_data: + current_data = "\n" + else: + current_data = " " + + # Reset the data collector. + self.current_data = [] + + # Should we add this string to the tree at all? + if ( + self.parse_only + and len(self.tagStack) <= 1 + and (not self.parse_only.allow_string_creation(current_data)) + ): + return + + containerClass = self.string_container(containerClass) + o = containerClass(current_data) + self.object_was_parsed(o) + + def object_was_parsed( + self, + o: PageElement, + parent: Optional[Tag] = None, + most_recent_element: Optional[PageElement] = None, + ) -> None: + """Method called by the TreeBuilder to integrate an object into the + parse tree. + + :meta private: + """ + if parent is None: + parent = self.currentTag + assert parent is not None + previous_element: Optional[PageElement] + if most_recent_element is not None: + previous_element = most_recent_element + else: + previous_element = self._most_recent_element + + next_element = previous_sibling = next_sibling = None + if isinstance(o, Tag): + next_element = o.next_element + next_sibling = o.next_sibling + previous_sibling = o.previous_sibling + if previous_element is None: + previous_element = o.previous_element + + fix = parent.next_element is not None + + o.setup(parent, previous_element, next_element, previous_sibling, next_sibling) + + self._most_recent_element = o + parent.contents.append(o) + + # Check if we are inserting into an already parsed node. + if fix: + self._linkage_fixer(parent) + + def _linkage_fixer(self, el: Tag) -> None: + """Make sure linkage of this fragment is sound.""" + + first = el.contents[0] + child = el.contents[-1] + descendant: PageElement = child + + if child is first and el.parent is not None: + # Parent should be linked to first child + el.next_element = child + # We are no longer linked to whatever this element is + prev_el = child.previous_element + if prev_el is not None and prev_el is not el: + prev_el.next_element = None + # First child should be linked to the parent, and no previous siblings. + child.previous_element = el + child.previous_sibling = None + + # We have no sibling as we've been appended as the last. + child.next_sibling = None + + # This index is a tag, dig deeper for a "last descendant" + if isinstance(child, Tag) and child.contents: + # _last_decendant is typed as returning Optional[PageElement], + # but the value can't be None here, because el is a Tag + # which we know has contents. + descendant = cast(PageElement, child._last_descendant(False)) + + # As the final step, link last descendant. It should be linked + # to the parent's next sibling (if found), else walk up the chain + # and find a parent with a sibling. It should have no next sibling. + descendant.next_element = None + descendant.next_sibling = None + + target: Optional[Tag] = el + while True: + if target is None: + break + elif target.next_sibling is not None: + descendant.next_element = target.next_sibling + target.next_sibling.previous_element = child + break + target = target.parent + + def _popToTag( + self, name: str, nsprefix: Optional[str] = None, inclusivePop: bool = True + ) -> Optional[Tag]: + """Pops the tag stack up to and including the most recent + instance of the given tag. + + If there are no open tags with the given name, nothing will be + popped. + + :param name: Pop up to the most recent tag with this name. + :param nsprefix: The namespace prefix that goes with `name`. + :param inclusivePop: It this is false, pops the tag stack up + to but *not* including the most recent instqance of the + given tag. + + :meta private: + """ + # print("Popping to %s" % name) + if name == self.ROOT_TAG_NAME: + # The BeautifulSoup object itself can never be popped. + return None + + most_recently_popped = None + + stack_size = len(self.tagStack) + for i in range(stack_size - 1, 0, -1): + if not self.open_tag_counter.get(name): + break + t = self.tagStack[i] + if name == t.name and nsprefix == t.prefix: + if inclusivePop: + most_recently_popped = self.popTag() + break + most_recently_popped = self.popTag() + + return most_recently_popped + + def handle_starttag( + self, + name: str, + namespace: Optional[str], + nsprefix: Optional[str], + attrs: _RawAttributeValues, + sourceline: Optional[int] = None, + sourcepos: Optional[int] = None, + namespaces: Optional[Dict[str, str]] = None, + ) -> Optional[Tag]: + """Called by the tree builder when a new tag is encountered. + + :param name: Name of the tag. + :param nsprefix: Namespace prefix for the tag. + :param attrs: A dictionary of attribute values. Note that + attribute values are expected to be simple strings; processing + of multi-valued attributes such as "class" comes later. + :param sourceline: The line number where this tag was found in its + source document. + :param sourcepos: The character position within `sourceline` where this + tag was found. + :param namespaces: A dictionary of all namespace prefix mappings + currently in scope in the document. + + If this method returns None, the tag was rejected by an active + `ElementFilter`. You should proceed as if the tag had not occurred + in the document. For instance, if this was a self-closing tag, + don't call handle_endtag. + + :meta private: + """ + # print("Start tag %s: %s" % (name, attrs)) + self.endData() + + if ( + self.parse_only + and len(self.tagStack) <= 1 + and not self.parse_only.allow_tag_creation(nsprefix, name, attrs) + ): + return None + + tag_class = self.element_classes.get(Tag, Tag) + # Assume that this is either Tag or a subclass of Tag. If not, + # the user brought type-unsafety upon themselves. + tag_class = cast(Type[Tag], tag_class) + tag = tag_class( + self, + self.builder, + name, + namespace, + nsprefix, + attrs, + self.currentTag, + self._most_recent_element, + sourceline=sourceline, + sourcepos=sourcepos, + namespaces=namespaces, + ) + if tag is None: + return tag + if self._most_recent_element is not None: + self._most_recent_element.next_element = tag + self._most_recent_element = tag + self.pushTag(tag) + return tag + + def handle_endtag(self, name: str, nsprefix: Optional[str] = None) -> None: + """Called by the tree builder when an ending tag is encountered. + + :param name: Name of the tag. + :param nsprefix: Namespace prefix for the tag. + + :meta private: + """ + # print("End tag: " + name) + self.endData() + self._popToTag(name, nsprefix) + + def handle_data(self, data: str) -> None: + """Called by the tree builder when a chunk of textual data is + encountered. + + :meta private: + """ + self.current_data.append(data) + + def decode( + self, + indent_level: Optional[int] = None, + eventual_encoding: _Encoding = DEFAULT_OUTPUT_ENCODING, + formatter: Union[Formatter, str] = "minimal", + iterator: Optional[Iterator[PageElement]] = None, + **kwargs: Any, + ) -> str: + """Returns a string representation of the parse tree + as a full HTML or XML document. + + :param indent_level: Each line of the rendering will be + indented this many levels. (The ``formatter`` decides what a + 'level' means, in terms of spaces or other characters + output.) This is used internally in recursive calls while + pretty-printing. + :param eventual_encoding: The encoding of the final document. + If this is None, the document will be a Unicode string. + :param formatter: Either a `Formatter` object, or a string naming one of + the standard formatters. + :param iterator: The iterator to use when navigating over the + parse tree. This is only used by `Tag.decode_contents` and + you probably won't need to use it. + """ + if self.is_xml: + # Print the XML declaration + encoding_part = "" + declared_encoding: Optional[str] = eventual_encoding + if eventual_encoding in PYTHON_SPECIFIC_ENCODINGS: + # This is a special Python encoding; it can't actually + # go into an XML document because it means nothing + # outside of Python. + declared_encoding = None + if declared_encoding is not None: + encoding_part = ' encoding="%s"' % declared_encoding + prefix = '\n' % encoding_part + else: + prefix = "" + + # Prior to 4.13.0, the first argument to this method was a + # bool called pretty_print, which gave the method a different + # signature from its superclass implementation, Tag.decode. + # + # The signatures of the two methods now match, but just in + # case someone is still passing a boolean in as the first + # argument to this method (or a keyword argument with the old + # name), we can handle it and put out a DeprecationWarning. + warning: Optional[str] = None + pretty_print: Optional[bool] = None + if isinstance(indent_level, bool): + if indent_level is True: + indent_level = 0 + elif indent_level is False: + indent_level = None + warning = f"As of 4.13.0, the first argument to BeautifulSoup.decode has been changed from bool to int, to match Tag.decode. Pass in a value of {indent_level} instead." + else: + pretty_print = kwargs.pop("pretty_print", None) + assert not kwargs + if pretty_print is not None: + if pretty_print is True: + indent_level = 0 + elif pretty_print is False: + indent_level = None + warning = f"As of 4.13.0, the pretty_print argument to BeautifulSoup.decode has been removed, to match Tag.decode. Pass in a value of indent_level={indent_level} instead." + + if warning: + warnings.warn(warning, DeprecationWarning, stacklevel=2) + elif indent_level is False or pretty_print is False: + indent_level = None + return prefix + super(BeautifulSoup, self).decode( + indent_level, eventual_encoding, formatter, iterator + ) + + +# Aliases to make it easier to get started quickly, e.g. 'from bs4 import _soup' +_s = BeautifulSoup +_soup = BeautifulSoup + + +class BeautifulStoneSoup(BeautifulSoup): + """Deprecated interface to an XML parser.""" + + def __init__(self, *args: Any, **kwargs: Any): + kwargs["features"] = "xml" + warnings.warn( + "The BeautifulStoneSoup class was deprecated in version 4.0.0. Instead of using " + 'it, pass features="xml" into the BeautifulSoup constructor.', + DeprecationWarning, + stacklevel=2, + ) + super(BeautifulStoneSoup, self).__init__(*args, **kwargs) + + +# If this file is run as a script, act as an HTML pretty-printer. +if __name__ == "__main__": + soup = BeautifulSoup(sys.stdin) + print((soup.prettify())) diff --git a/bitbake/lib/bb/_vendor/bs4/_deprecation.py b/bitbake/lib/bb/_vendor/bs4/_deprecation.py new file mode 100644 index 00000000..a7b5685b --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/_deprecation.py @@ -0,0 +1,80 @@ +"""Helper functions for deprecation. + +This interface is itself unstable and may change without warning. Do +not use these functions yourself, even as a joke. The underscores are +there for a reason. No support will be given. + +In particular, most of this will go away without warning once +Beautiful Soup drops support for Python 3.11, since Python 3.12 +defines a `@typing.deprecated() +decorator. `_ +""" + +import functools +import warnings + +from typing import ( + Any, + Callable, +) + + +def _deprecated_alias(old_name: str, new_name: str, version: str): + """Alias one attribute name to another for backward compatibility + + :meta private: + """ + + @property # type:ignore + def alias(self) -> Any: + ":meta private:" + warnings.warn( + f"Access to deprecated property {old_name}. (Replaced by {new_name}) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=2, + ) + return getattr(self, new_name) + + @alias.setter + def alias(self, value: str) -> None: + ":meta private:" + warnings.warn( + f"Write to deprecated property {old_name}. (Replaced by {new_name}) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=2, + ) + return setattr(self, new_name, value) + + return alias + + +def _deprecated_function_alias( + old_name: str, new_name: str, version: str +) -> Callable[[Any], Any]: + def alias(self, *args: Any, **kwargs: Any) -> Any: + ":meta private:" + warnings.warn( + f"Call to deprecated method {old_name}. (Replaced by {new_name}) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=2, + ) + return getattr(self, new_name)(*args, **kwargs) + + return alias + + +def _deprecated(replaced_by: str, version: str) -> Callable: + def deprecate(func: Callable) -> Callable: + @functools.wraps(func) + def with_warning(*args: Any, **kwargs: Any) -> Any: + ":meta private:" + warnings.warn( + f"Call to deprecated method {func.__name__}. (Replaced by {replaced_by}) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=2, + ) + return func(*args, **kwargs) + + return with_warning + + return deprecate diff --git a/bitbake/lib/bb/_vendor/bs4/_typing.py b/bitbake/lib/bb/_vendor/bs4/_typing.py new file mode 100644 index 00000000..caf2567a --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/_typing.py @@ -0,0 +1,205 @@ +# Custom type aliases used throughout Beautiful Soup to improve readability. + +# Notes on improvements to the type system in newer versions of Python +# that can be used once Beautiful Soup drops support for older +# versions: +# +# * ClassVar can be put on class variables now. +# * In 3.10, x|y is an accepted shorthand for Union[x,y]. +# * In 3.10, TypeAlias gains capabilities that can be used to +# improve the tree matching types (I don't remember what, exactly). +# * In 3.9 it's possible to specialize the re.Match type, +# e.g. re.Match[str]. In 3.8 there's a typing.re namespace for this, +# but it's removed in 3.12, so to support the widest possible set of +# versions I'm not using it. + +from bb._vendor.typing_extensions import ( + runtime_checkable, + Protocol, + TypeAlias, +) +from typing import ( + Any, + Callable, + Dict, + IO, + Iterable, + Mapping, + Optional, + Pattern, + TYPE_CHECKING, + Union, +) + +if TYPE_CHECKING: + from bb._vendor.bs4.element import ( + AttributeValueList, + NamespacedAttribute, + NavigableString, + PageElement, + ResultSet, + Tag, + ) + + +@runtime_checkable +class _RegularExpressionProtocol(Protocol): + """A protocol object which can accept either Python's built-in + `re.Pattern` objects, or the similar ``Regex`` objects defined by the + third-party ``regex`` package. + """ + + def search( + self, string: str, pos: int = ..., endpos: int = ... + ) -> Optional[Any]: ... + + @property + def pattern(self) -> str: ... + + +# Aliases for markup in various stages of processing. +# +#: The rawest form of markup: either a string, bytestring, or an open filehandle. +_IncomingMarkup: TypeAlias = Union[str, bytes, IO[str], IO[bytes]] + +#: Markup that is in memory but has (potentially) yet to be converted +#: to Unicode. +_RawMarkup: TypeAlias = Union[str, bytes] + +# Aliases for character encodings +# + +#: A data encoding. +_Encoding: TypeAlias = str + +#: One or more data encodings. +_Encodings: TypeAlias = Iterable[_Encoding] + +# Aliases for XML namespaces +# + +#: The prefix for an XML namespace. +_NamespacePrefix: TypeAlias = str + +#: The URL of an XML namespace +_NamespaceURL: TypeAlias = str + +#: A mapping of prefixes to namespace URLs. +_NamespaceMapping: TypeAlias = Dict[_NamespacePrefix, _NamespaceURL] + +#: A mapping of namespace URLs to prefixes +_InvertedNamespaceMapping: TypeAlias = Dict[_NamespaceURL, _NamespacePrefix] + +# Aliases for the attribute values associated with HTML/XML tags. +# + +#: The value associated with an HTML or XML attribute. This is the +#: relatively unprocessed value Beautiful Soup expects to come from a +#: `TreeBuilder`. +_RawAttributeValue: TypeAlias = str + +#: A dictionary of names to `_RawAttributeValue` objects. This is how +#: Beautiful Soup expects a `TreeBuilder` to represent a tag's +#: attribute values. +_RawAttributeValues: TypeAlias = ( + "Mapping[Union[str, NamespacedAttribute], _RawAttributeValue]" +) + +#: An attribute value in its final form, as stored in the +# `Tag` class, after it has been processed and (in some cases) +# split into a list of strings. +_AttributeValue: TypeAlias = Union[str, "AttributeValueList"] + +#: A dictionary of names to :py:data:`_AttributeValue` objects. This is what +#: a tag's attributes look like after processing. +_AttributeValues: TypeAlias = Dict[str, _AttributeValue] + +#: The methods that deal with turning :py:data:`_RawAttributeValue` into +#: :py:data:`_AttributeValue` may be called several times, even after the values +#: are already processed (e.g. when cloning a tag), so they need to +#: be able to acommodate both possibilities. +_RawOrProcessedAttributeValues: TypeAlias = Union[_RawAttributeValues, _AttributeValues] + +#: A number of tree manipulation methods can take either a `PageElement` or a +#: normal Python string (which will be converted to a `NavigableString`). +_InsertableElement: TypeAlias = Union["PageElement", str] + +# Aliases to represent the many possibilities for matching bits of a +# parse tree. +# +# This is very complicated because we're applying a formal type system +# to some very DWIM code. The types we end up with will be the types +# of the arguments to the SoupStrainer constructor and (more +# familiarly to Beautiful Soup users) the find* methods. + +#: A function that takes a PageElement and returns a yes-or-no answer. +_PageElementMatchFunction: TypeAlias = Callable[["PageElement"], bool] + +#: A function that takes the raw parsed ingredients of a markup tag +#: and returns a yes-or-no answer. +# Not necessary at the moment. +# _AllowTagCreationFunction:TypeAlias = Callable[[Optional[str], str, Optional[_RawAttributeValues]], bool] + +#: A function that takes the raw parsed ingredients of a markup string node +#: and returns a yes-or-no answer. +# Not necessary at the moment. +# _AllowStringCreationFunction:TypeAlias = Callable[[Optional[str]], bool] + +#: A function that takes a `Tag` and returns a yes-or-no answer. +#: A `TagNameMatchRule` expects this kind of function, if you're +#: going to pass it a function. +_TagMatchFunction: TypeAlias = Callable[["Tag"], bool] + +#: A function that takes a string (or None) and returns a yes-or-no +#: answer. An `AttributeValueMatchRule` expects this kind of function, if +#: you're going to pass it a function. +_NullableStringMatchFunction: TypeAlias = Callable[[Optional[str]], bool] + +#: A function that takes a string and returns a yes-or-no answer. A +# `StringMatchRule` expects this kind of function, if you're going to +# pass it a function. +_StringMatchFunction: TypeAlias = Callable[[str], bool] + +#: Either a tag name, an attribute value or a string can be matched +#: against a string, bytestring, regular expression, or a boolean. +_BaseStrainable: TypeAlias = Union[str, bytes, Pattern[str], bool] + +#: A tag can be matched either with the `_BaseStrainable` options, or +#: using a function that takes the `Tag` as its sole argument. +_BaseStrainableElement: TypeAlias = Union[_BaseStrainable, _TagMatchFunction] + +#: A tag's attribute value can be matched either with the +#: `_BaseStrainable` options, or using a function that takes that +#: value as its sole argument. +_BaseStrainableAttribute: TypeAlias = Union[_BaseStrainable, _NullableStringMatchFunction] + +#: A tag can be matched using either a single criterion or a list of +#: criteria. +_StrainableElement: TypeAlias = Union[ + _BaseStrainableElement, Iterable[_BaseStrainableElement] +] + +#: An attribute value can be matched using either a single criterion +#: or a list of criteria. +_StrainableAttribute: TypeAlias = Union[ + _BaseStrainableAttribute, Iterable[_BaseStrainableAttribute] +] + +#: An string can be matched using the same techniques as +#: an attribute value. +_StrainableString: TypeAlias = _StrainableAttribute + +#: A dictionary may be used to match against multiple attribute vlaues at once. +_StrainableAttributes: TypeAlias = Dict[str, _StrainableAttribute] + +#: Many Beautiful soup methods return a PageElement or an ResultSet of +#: PageElements. A PageElement is either a Tag or a NavigableString. +#: These convenience aliases make it easier for IDE users to see which methods +#: are available on the objects they're dealing with. +_OneElement: TypeAlias = Union["PageElement", "Tag", "NavigableString"] +_AtMostOneElement: TypeAlias = Optional[_OneElement] +_AtMostOneTag: TypeAlias = Optional["Tag"] +_AtMostOneNavigableString: TypeAlias = Optional["NavigableString"] +_QueryResults: TypeAlias = "ResultSet[_OneElement]" +_SomeTags: TypeAlias = "ResultSet[Tag]" +_SomeNavigableStrings: TypeAlias = "ResultSet[NavigableString]" diff --git a/bitbake/lib/bb/_vendor/bs4/_warnings.py b/bitbake/lib/bb/_vendor/bs4/_warnings.py new file mode 100644 index 00000000..89041705 --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/_warnings.py @@ -0,0 +1,98 @@ +"""Define some custom warnings.""" + + +class GuessedAtParserWarning(UserWarning): + """The warning issued when BeautifulSoup has to guess what parser to + use -- probably because no parser was specified in the constructor. + """ + + MESSAGE: str = """No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system ("%(parser)s"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently. + +The code that caused this warning is on line %(line_number)s of the file %(filename)s. To get rid of this warning, pass the additional argument 'features="%(parser)s"' to the BeautifulSoup constructor. +""" + + +class UnusualUsageWarning(UserWarning): + """A superclass for warnings issued when Beautiful Soup sees + something that is typically the result of a mistake in the calling + code, but might be intentional on the part of the user. If it is + in fact intentional, you can filter the individual warning class + to get rid of the warning. If you don't like Beautiful Soup + second-guessing what you are doing, you can filter the + UnusualUsageWarningclass itself and get rid of these entirely. + """ + + +class MarkupResemblesLocatorWarning(UnusualUsageWarning): + """The warning issued when BeautifulSoup is given 'markup' that + actually looks like a resource locator -- a URL or a path to a file + on disk. + """ + + #: :meta private: + GENERIC_MESSAGE: str = """ + +However, if you want to parse some data that happens to look like a %(what)s, then nothing has gone wrong: you are using Beautiful Soup correctly, and this warning is spurious and can be filtered. To make this warning go away, run this code before calling the BeautifulSoup constructor: + + from bb._vendor.bs4 import MarkupResemblesLocatorWarning + import warnings + + warnings.filterwarnings("ignore", category=MarkupResemblesLocatorWarning) + """ + + URL_MESSAGE: str = ( + """The input passed in on this line looks more like a URL than HTML or XML. + +If you meant to use Beautiful Soup to parse the web page found at a certain URL, then something has gone wrong. You should use an Python package like 'requests' to fetch the content behind the URL. Once you have the content as a string, you can feed that string into Beautiful Soup.""" + + GENERIC_MESSAGE + ) + + FILENAME_MESSAGE: str = ( + """The input passed in on this line looks more like a filename than HTML or XML. + +If you meant to use Beautiful Soup to parse the contents of a file on disk, then something has gone wrong. You should open the file first, using code like this: + + filehandle = open(your filename) + +You can then feed the open filehandle into Beautiful Soup instead of using the filename.""" + + GENERIC_MESSAGE + ) + + +class AttributeResemblesVariableWarning(UnusualUsageWarning, SyntaxWarning): + """The warning issued when Beautiful Soup suspects a provided + attribute name may actually be the misspelled name of a Beautiful + Soup variable. Generally speaking, this is only used in cases like + "_class" where it's very unlikely the user would be referencing an + XML attribute with that name. + """ + + MESSAGE: str = """%(original)r is an unusual attribute name and is a common misspelling for %(autocorrect)r. + +If you meant %(autocorrect)r, change your code to use it, and this warning will go away. + +If you really did mean to check the %(original)r attribute, this warning is spurious and can be filtered. To make it go away, run this code before creating your BeautifulSoup object: + + from bb._vendor.bs4 import AttributeResemblesVariableWarning + import warnings + + warnings.filterwarnings("ignore", category=AttributeResemblesVariableWarning) +""" + + +class XMLParsedAsHTMLWarning(UnusualUsageWarning): + """The warning issued when an HTML parser is used to parse + XML that is not (as far as we can tell) XHTML. + """ + + MESSAGE: str = """It looks like you're using an HTML parser to parse an XML document. + +Assuming this really is an XML document, what you're doing might work, but you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the Python package 'lxml' installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor. + +If you want or need to use an HTML parser on this document, you can make this warning go away by filtering it. To do that, run this code before calling the BeautifulSoup constructor: + + from bb._vendor.bs4 import XMLParsedAsHTMLWarning + import warnings + + warnings.filterwarnings("ignore", category=XMLParsedAsHTMLWarning) +""" diff --git a/bitbake/lib/bb/_vendor/bs4/builder/__init__.py b/bitbake/lib/bb/_vendor/bs4/builder/__init__.py new file mode 100644 index 00000000..c9d5786d --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/builder/__init__.py @@ -0,0 +1,848 @@ +from __future__ import annotations + +# Use of this source code is governed by the MIT license. +__license__ = "MIT" + +from collections import defaultdict +import re +from types import ModuleType +from typing import ( + Any, + cast, + Dict, + Iterable, + List, + Optional, + Pattern, + Set, + Tuple, + Type, + TYPE_CHECKING, +) +import warnings +import sys +from bb._vendor.bs4.element import ( + AttributeDict, + AttributeValueList, + CharsetMetaAttributeValue, + ContentMetaAttributeValue, + RubyParenthesisString, + RubyTextString, + Stylesheet, + Script, + TemplateString, + nonwhitespace_re, +) + +# Exceptions were moved to their own module in 4.13. Import here for +# backwards compatibility. +from bb._vendor.bs4.exceptions import ParserRejectedMarkup + +from bb._vendor.bs4._typing import ( + _AttributeValues, + _RawAttributeValue, +) + +from bb._vendor.bs4._warnings import XMLParsedAsHTMLWarning + +if TYPE_CHECKING: + from bb._vendor.bs4 import BeautifulSoup + from bb._vendor.bs4.element import ( + NavigableString, + Tag, + ) + from bb._vendor.bs4._typing import ( + _AttributeValue, + _Encoding, + _Encodings, + _RawOrProcessedAttributeValues, + _RawMarkup, + ) + +__all__ = [ + "HTMLTreeBuilder", + "SAXTreeBuilder", + "TreeBuilder", + "TreeBuilderRegistry", +] + +# Some useful features for a TreeBuilder to have. +FAST = "fast" +PERMISSIVE = "permissive" +STRICT = "strict" +XML = "xml" +HTML = "html" +HTML_5 = "html5" + +__all__ = [ + "TreeBuilderRegistry", + "TreeBuilder", + "HTMLTreeBuilder", + "DetectsXMLParsedAsHTML", + + "ParserRejectedMarkup", # backwards compatibility only as of 4.13.0 +] + +class TreeBuilderRegistry(object): + """A way of looking up TreeBuilder subclasses by their name or by desired + features. + """ + + builders_for_feature: Dict[str, List[Type[TreeBuilder]]] + builders: List[Type[TreeBuilder]] + + def __init__(self) -> None: + self.builders_for_feature = defaultdict(list) + self.builders = [] + + def register(self, treebuilder_class: type[TreeBuilder]) -> None: + """Register a treebuilder based on its advertised features. + + :param treebuilder_class: A subclass of `TreeBuilder`. its + `TreeBuilder.features` attribute should list its features. + """ + for feature in treebuilder_class.features: + self.builders_for_feature[feature].insert(0, treebuilder_class) + self.builders.insert(0, treebuilder_class) + + def lookup(self, *features: str) -> Optional[Type[TreeBuilder]]: + """Look up a TreeBuilder subclass with the desired features. + + :param features: A list of features to look for. If none are + provided, the most recently registered TreeBuilder subclass + will be used. + :return: A TreeBuilder subclass, or None if there's no + registered subclass with all the requested features. + """ + if len(self.builders) == 0: + # There are no builders at all. + return None + + if len(features) == 0: + # They didn't ask for any features. Give them the most + # recently registered builder. + return self.builders[0] + + # Go down the list of features in order, and eliminate any builders + # that don't match every feature. + feature_list = list(features) + feature_list.reverse() + candidates = None + candidate_set = None + while len(feature_list) > 0: + feature = feature_list.pop() + we_have_the_feature = self.builders_for_feature.get(feature, []) + if len(we_have_the_feature) > 0: + if candidates is None: + candidates = we_have_the_feature + candidate_set = set(candidates) + elif candidate_set is not None: + # Eliminate any candidates that don't have this feature. + candidate_set = candidate_set.intersection(set(we_have_the_feature)) + + # The only valid candidates are the ones in candidate_set. + # Go through the original list of candidates and pick the first one + # that's in candidate_set. + if candidate_set is None or candidates is None: + return None + for candidate in candidates: + if candidate in candidate_set: + return candidate + return None + + +#: The `BeautifulSoup` constructor will take a list of features +#: and use it to look up `TreeBuilder` classes in this registry. +builder_registry: TreeBuilderRegistry = TreeBuilderRegistry() + + +class TreeBuilder(object): + """Turn a textual document into a Beautiful Soup object tree. + + This is an abstract superclass which smooths out the behavior of + different parser libraries into a single, unified interface. + + :param multi_valued_attributes: If this is set to None, the + TreeBuilder will not turn any values for attributes like + 'class' into lists. Setting this to a dictionary will + customize this behavior; look at :py:attr:`bs4.builder.HTMLTreeBuilder.DEFAULT_CDATA_LIST_ATTRIBUTES` + for an example. + + Internally, these are called "CDATA list attributes", but that + probably doesn't make sense to an end-user, so the argument name + is ``multi_valued_attributes``. + + :param preserve_whitespace_tags: A set of tags to treat + the way 
 tags are treated in HTML. Tags in this set
+     are immune from pretty-printing; their contents will always be
+     output as-is.
+
+    :param string_containers: A dictionary mapping tag names to
+     the classes that should be instantiated to contain the textual
+     contents of those tags. The default is to use NavigableString
+     for every tag, no matter what the name. You can override the
+     default by changing :py:attr:`DEFAULT_STRING_CONTAINERS`.
+
+    :param store_line_numbers: If the parser keeps track of the line
+     numbers and positions of the original markup, that information
+     will, by default, be stored in each corresponding
+     :py:class:`bs4.element.Tag` object. You can turn this off by
+     passing store_line_numbers=False; then Tag.sourcepos and
+     Tag.sourceline will always be None. If the parser you're using
+     doesn't keep track of this information, then store_line_numbers
+     is irrelevant.
+
+    :param attribute_dict_class: The value of a multi-valued attribute
+      (such as HTML's 'class') willl be stored in an instance of this
+      class.  The default is Beautiful Soup's built-in
+      `AttributeValueList`, which is a normal Python list, and you
+      will probably never need to change it.
+    """
+
+    USE_DEFAULT: Any = object()  #: :meta private:
+
+    def __init__(
+        self,
+        multi_valued_attributes: Dict[str, Set[str]] = USE_DEFAULT,
+        preserve_whitespace_tags: Set[str] = USE_DEFAULT,
+        store_line_numbers: bool = USE_DEFAULT,
+        string_containers: Dict[str, Type[NavigableString]] = USE_DEFAULT,
+        empty_element_tags: Set[str] = USE_DEFAULT,
+        attribute_dict_class: Type[AttributeDict] = AttributeDict,
+        attribute_value_list_class: Type[AttributeValueList] = AttributeValueList,
+    ):
+        self.soup = None
+        if multi_valued_attributes is self.USE_DEFAULT:
+            multi_valued_attributes = self.DEFAULT_CDATA_LIST_ATTRIBUTES
+        self.cdata_list_attributes = multi_valued_attributes
+        if preserve_whitespace_tags is self.USE_DEFAULT:
+            preserve_whitespace_tags = self.DEFAULT_PRESERVE_WHITESPACE_TAGS
+        self.preserve_whitespace_tags = preserve_whitespace_tags
+        if empty_element_tags is self.USE_DEFAULT:
+            self.empty_element_tags = self.DEFAULT_EMPTY_ELEMENT_TAGS
+        else:
+            self.empty_element_tags = empty_element_tags
+        # TODO: store_line_numbers is probably irrelevant now that
+        # the behavior of sourceline and sourcepos has been made consistent
+        # everywhere.
+        if store_line_numbers == self.USE_DEFAULT:
+            store_line_numbers = self.TRACKS_LINE_NUMBERS
+        self.store_line_numbers = store_line_numbers
+        if string_containers == self.USE_DEFAULT:
+            string_containers = self.DEFAULT_STRING_CONTAINERS
+        self.string_containers = string_containers
+        self.attribute_dict_class = attribute_dict_class
+        self.attribute_value_list_class = attribute_value_list_class
+
+    NAME: str = "[Unknown tree builder]"
+    ALTERNATE_NAMES: Iterable[str] = []
+    features: Iterable[str] = []
+
+    is_xml: bool = False
+    picklable: bool = False
+
+    soup: Optional[BeautifulSoup]  #: :meta private:
+
+    #: A tag will be considered an empty-element
+    #: tag when and only when it has no contents.
+    empty_element_tags: Optional[Set[str]] = None  #: :meta private:
+    cdata_list_attributes: Dict[str, Set[str]]  #: :meta private:
+    preserve_whitespace_tags: Set[str]  #: :meta private:
+    string_containers: Dict[str, Type[NavigableString]]  #: :meta private:
+    tracks_line_numbers: bool  #: :meta private:
+
+    #: A value for these tag/attribute combinations is a space- or
+    #: comma-separated list of CDATA, rather than a single CDATA.
+    DEFAULT_CDATA_LIST_ATTRIBUTES: Dict[str, Set[str]] = defaultdict(set)
+
+    #: Whitespace should be preserved inside these tags.
+    DEFAULT_PRESERVE_WHITESPACE_TAGS: Set[str] = set()
+
+    #: The textual contents of tags with these names should be
+    #: instantiated with some class other than `bs4.element.NavigableString`.
+    DEFAULT_STRING_CONTAINERS: Dict[str, Type[bs4.element.NavigableString]] = {} # type:ignore
+
+    #: By default, tags are treated as empty-element tags if they have
+    #: no contents--that is, using XML rules. HTMLTreeBuilder
+    #: defines a different set of DEFAULT_EMPTY_ELEMENT_TAGS based on the
+    #: HTML 4 and HTML5 standards.
+    DEFAULT_EMPTY_ELEMENT_TAGS: Optional[Set[str]] = None
+
+    #: Most parsers don't keep track of line numbers.
+    TRACKS_LINE_NUMBERS: bool = False
+
+    def initialize_soup(self, soup: BeautifulSoup) -> None:
+        """The BeautifulSoup object has been initialized and is now
+        being associated with the TreeBuilder.
+
+        :param soup: A BeautifulSoup object.
+        """
+        self.soup = soup
+
+    def reset(self) -> None:
+        """Do any work necessary to reset the underlying parser
+        for a new document.
+
+        By default, this does nothing.
+        """
+        pass
+
+    def can_be_empty_element(self, tag_name: str) -> bool:
+        """Might a tag with this name be an empty-element tag?
+
+        The final markup may or may not actually present this tag as
+        self-closing.
+
+        For instance: an HTMLBuilder does not consider a 
 tag to be
+        an empty-element tag (it's not in
+        HTMLBuilder.empty_element_tags). This means an empty 
 tag
+        will be presented as "
", not "
" or "
".
+
+        The default implementation has no opinion about which tags are
+        empty-element tags, so a tag will be presented as an
+        empty-element tag if and only if it has no children.
+        "" will become "", and "bar" will
+        be left alone.
+
+        :param tag_name: The name of a markup tag.
+        """
+        if self.empty_element_tags is None:
+            return True
+        return tag_name in self.empty_element_tags
+
+    def feed(self, markup: _RawMarkup) -> None:
+        """Run incoming markup through some parsing process."""
+        raise NotImplementedError()
+
+    def prepare_markup(
+        self,
+        markup: _RawMarkup,
+        user_specified_encoding: Optional[_Encoding] = None,
+        document_declared_encoding: Optional[_Encoding] = None,
+        exclude_encodings: Optional[_Encodings] = None,
+    ) -> Iterable[Tuple[_RawMarkup, Optional[_Encoding], Optional[_Encoding], bool]]:
+        """Run any preliminary steps necessary to make incoming markup
+        acceptable to the parser.
+
+        :param markup: The markup that's about to be parsed.
+        :param user_specified_encoding: The user asked to try this encoding
+           to convert the markup into a Unicode string.
+        :param document_declared_encoding: The markup itself claims to be
+            in this encoding. NOTE: This argument is not used by the
+            calling code and can probably be removed.
+        :param exclude_encodings: The user asked *not* to try any of
+            these encodings.
+
+        :yield: A series of 4-tuples: (markup, encoding, declared encoding,
+            has undergone character replacement)
+
+            Each 4-tuple represents a strategy that the parser can try
+            to convert the document to Unicode and parse it. Each
+            strategy will be tried in turn.
+
+         By default, the only strategy is to parse the markup
+         as-is. See `LXMLTreeBuilderForXML` and
+         `HTMLParserTreeBuilder` for implementations that take into
+         account the quirks of particular parsers.
+
+        :meta private:
+
+        """
+        yield markup, None, None, False
+
+    def test_fragment_to_document(self, fragment: str) -> str:
+        """Wrap an HTML fragment to make it look like a document.
+
+        Different parsers do this differently. For instance, lxml
+        introduces an empty  tag, and html5lib
+        doesn't. Abstracting this away lets us write simple tests
+        which run HTML fragments through the parser and compare the
+        results against other HTML fragments.
+
+        This method should not be used outside of unit tests.
+
+        :param fragment: A fragment of HTML.
+        :return: A full HTML document.
+        :meta private:
+        """
+        return fragment
+
+    def set_up_substitutions(self, tag: Tag) -> bool:
+        """Set up any substitutions that will need to be performed on
+        a `Tag` when it's output as a string.
+
+        By default, this does nothing. See `HTMLTreeBuilder` for a
+        case where this is used.
+
+        :return: Whether or not a substitution was performed.
+        :meta private:
+        """
+        return False
+
+    def _replace_cdata_list_attribute_values(
+        self, tag_name: str, attrs: _RawOrProcessedAttributeValues
+    ) -> _AttributeValues:
+        """When an attribute value is associated with a tag that can
+        have multiple values for that attribute, convert the string
+        value to a list of strings.
+
+        Basically, replaces class="foo bar" with class=["foo", "bar"]
+
+        NOTE: This method modifies its input in place.
+
+        :param tag_name: The name of a tag.
+        :param attrs: A dictionary containing the tag's attributes.
+           Any appropriate attribute values will be modified in place.
+        :return: The modified dictionary that was originally passed in.
+        """
+
+        # First, cast the attrs dict to _AttributeValues. This might
+        # not be accurate yet, but it will be by the time this method
+        # returns.
+        modified_attrs = cast(_AttributeValues, attrs)
+        if not modified_attrs or not self.cdata_list_attributes:
+            # Nothing to do.
+            return modified_attrs
+
+        # There is at least a possibility that we need to modify one of
+        # the attribute values.
+        universal: Set[str] = self.cdata_list_attributes.get("*", set())
+        tag_specific = self.cdata_list_attributes.get(tag_name.lower(), None)
+        for attr in list(modified_attrs.keys()):
+            modified_value: _AttributeValue
+            if attr in universal or (tag_specific and attr in tag_specific):
+                # We have a "class"-type attribute whose string
+                # value is a whitespace-separated list of
+                # values. Split it into a list.
+                original_value: _AttributeValue = modified_attrs[attr]
+                if isinstance(original_value, _RawAttributeValue):
+                    # This is a _RawAttributeValue (a string) that
+                    # needs to be split and converted to a
+                    # AttributeValueList so it can be an
+                    # _AttributeValue.
+                    modified_value = self.attribute_value_list_class(
+                        nonwhitespace_re.findall(original_value)
+                    )
+                else:
+                    # html5lib calls setAttributes twice for the
+                    # same tag when rearranging the parse tree. On
+                    # the second call the attribute value here is
+                    # already a list. This can also happen when a
+                    # Tag object is cloned. If this happens, leave
+                    # the value alone rather than trying to split
+                    # it again.
+                    modified_value = original_value
+                modified_attrs[attr] = modified_value
+        return modified_attrs
+
+
+class SAXTreeBuilder(TreeBuilder):
+    """A Beautiful Soup treebuilder that listens for SAX events.
+
+    This is not currently used for anything, and it will be removed
+    soon. It was a good idea, but it wasn't properly integrated into the
+    rest of Beautiful Soup, so there have been long stretches where it
+    hasn't worked properly.
+    """
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn(
+            "The SAXTreeBuilder class was deprecated in 4.13.0 and will be removed soon thereafter. It is completely untested and probably doesn't work; do not use it.",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        super(SAXTreeBuilder, self).__init__(*args, **kwargs)
+
+    def feed(self, markup: _RawMarkup) -> None:
+        raise NotImplementedError()
+
+    def close(self) -> None:
+        pass
+
+    def startElement(self, name: str, attrs: Dict[str, str]) -> None:
+        attrs = AttributeDict((key[1], value) for key, value in list(attrs.items()))
+        # print("Start %s, %r" % (name, attrs))
+        assert self.soup is not None
+        self.soup.handle_starttag(name, None, None, attrs)
+
+    def endElement(self, name: str) -> None:
+        # print("End %s" % name)
+        assert self.soup is not None
+        self.soup.handle_endtag(name)
+
+    def startElementNS(
+        self, nsTuple: Tuple[str, str], nodeName: str, attrs: Dict[str, str]
+    ) -> None:
+        # Throw away (ns, nodeName) for now.
+        self.startElement(nodeName, attrs)
+
+    def endElementNS(self, nsTuple: Tuple[str, str], nodeName: str) -> None:
+        # Throw away (ns, nodeName) for now.
+        self.endElement(nodeName)
+        # handler.endElementNS((ns, node.nodeName), node.nodeName)
+
+    def startPrefixMapping(self, prefix: str, nodeValue: str) -> None:
+        # Ignore the prefix for now.
+        pass
+
+    def endPrefixMapping(self, prefix: str) -> None:
+        # Ignore the prefix for now.
+        # handler.endPrefixMapping(prefix)
+        pass
+
+    def characters(self, content: str) -> None:
+        assert self.soup is not None
+        self.soup.handle_data(content)
+
+    def startDocument(self) -> None:
+        pass
+
+    def endDocument(self) -> None:
+        pass
+
+
+class HTMLTreeBuilder(TreeBuilder):
+    """This TreeBuilder knows facts about HTML, such as which tags are treated
+    specially by the HTML standard.
+    """
+
+    #: Some HTML tags are defined as having no contents. Beautiful Soup
+    #: treats these specially.
+    DEFAULT_EMPTY_ELEMENT_TAGS: Optional[Set[str]] = set(
+        [
+            # These are from HTML5.
+            "area",
+            "base",
+            "br",
+            "col",
+            "embed",
+            "hr",
+            "img",
+            "input",
+            "keygen",
+            "link",
+            "menuitem",
+            "meta",
+            "param",
+            "source",
+            "track",
+            "wbr",
+            # These are from earlier versions of HTML and are removed in HTML5.
+            "basefont",
+            "bgsound",
+            "command",
+            "frame",
+            "image",
+            "isindex",
+            "nextid",
+            "spacer",
+        ]
+    )
+
+    #: The HTML standard defines these tags as block-level elements. Beautiful
+    #: Soup does not treat these elements differently from other elements,
+    #: but it may do so eventually, and this information is available if
+    #: you need to use it.
+    DEFAULT_BLOCK_ELEMENTS: Set[str] = set(
+        [
+            "address",
+            "article",
+            "aside",
+            "blockquote",
+            "canvas",
+            "dd",
+            "div",
+            "dl",
+            "dt",
+            "fieldset",
+            "figcaption",
+            "figure",
+            "footer",
+            "form",
+            "h1",
+            "h2",
+            "h3",
+            "h4",
+            "h5",
+            "h6",
+            "header",
+            "hr",
+            "li",
+            "main",
+            "nav",
+            "noscript",
+            "ol",
+            "output",
+            "p",
+            "pre",
+            "section",
+            "table",
+            "tfoot",
+            "ul",
+            "video",
+        ]
+    )
+
+    #: These HTML tags need special treatment so they can be
+    #: represented by a string class other than `bs4.element.NavigableString`.
+    #:
+    #: For some of these tags, it's because the HTML standard defines
+    #: an unusual content model for them. I made this list by going
+    #: through the HTML spec
+    #: (https://html.spec.whatwg.org/#metadata-content) and looking for
+    #: "metadata content" elements that can contain strings.
+    #:
+    #: The Ruby tags ( and ) are here despite being normal
+    #: "phrasing content" tags, because the content they contain is
+    #: qualitatively different from other text in the document, and it
+    #: can be useful to be able to distinguish it.
+    #:
+    #: TODO: Arguably 
  foo  
  \tbar\n  \n  
  baz  ")
-        # Everything outside the 
 tag is reformatted, but everything
-        # inside is left alone.
-        self.assertEqual(
-            '
\n foo\n 
  \tbar\n  \n  
\n baz\n
',
-            soup.div.prettify())
-
-    def test_prettify_accepts_formatter(self):
-        soup = BeautifulSoup("foo", 'html.parser')
-        pretty = soup.prettify(formatter = lambda x: x.upper())
-        self.assertTrue("FOO" in pretty)
-
-    def test_prettify_outputs_unicode_by_default(self):
-        soup = self.soup("")
-        self.assertEqual(str, type(soup.prettify()))
-
-    def test_prettify_can_encode_data(self):
-        soup = self.soup("")
-        self.assertEqual(bytes, type(soup.prettify("utf-8")))
-
-    def test_html_entity_substitution_off_by_default(self):
-        markup = "Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!"
-        soup = self.soup(markup)
-        encoded = soup.b.encode("utf-8")
-        self.assertEqual(encoded, markup.encode('utf-8'))
-
-    def test_encoding_substitution(self):
-        # Here's the  tag saying that a document is
-        # encoded in Shift-JIS.
-        meta_tag = ('')
-        soup = self.soup(meta_tag)
-
-        # Parse the document, and the charset apprears unchanged.
-        self.assertEqual(soup.meta['content'], 'text/html; charset=x-sjis')
-
-        # Encode the document into some encoding, and the encoding is
-        # substituted into the meta tag.
-        utf_8 = soup.encode("utf-8")
-        self.assertTrue(b"charset=utf-8" in utf_8)
-
-        euc_jp = soup.encode("euc_jp")
-        self.assertTrue(b"charset=euc_jp" in euc_jp)
-
-        shift_jis = soup.encode("shift-jis")
-        self.assertTrue(b"charset=shift-jis" in shift_jis)
-
-        utf_16_u = soup.encode("utf-16").decode("utf-16")
-        self.assertTrue("charset=utf-16" in utf_16_u)
-
-    def test_encoding_substitution_doesnt_happen_if_tag_is_strained(self):
-        markup = ('
foo
')
-
-        # Beautiful Soup used to try to rewrite the meta tag even if the
-        # meta tag got filtered out by the strainer. This test makes
-        # sure that doesn't happen.
-        strainer = SoupStrainer('pre')
-        soup = self.soup(markup, parse_only=strainer)
-        self.assertEqual(soup.contents[0].name, 'pre')
-
-class TestEncoding(SoupTest):
-    """Test the ability to encode objects into strings."""
-
-    def test_unicode_string_can_be_encoded(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        self.assertEqual(soup.b.string.encode("utf-8"),
-                          "\N{SNOWMAN}".encode("utf-8"))
-
-    def test_tag_containing_unicode_string_can_be_encoded(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        self.assertEqual(
-            soup.b.encode("utf-8"), html.encode("utf-8"))
-
-    def test_encoding_substitutes_unrecognized_characters_by_default(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        self.assertEqual(soup.b.encode("ascii"), b"☃")
-
-    def test_encoding_can_be_made_strict(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        self.assertRaises(
-            UnicodeEncodeError, soup.encode, "ascii", errors="strict")
-
-    def test_decode_contents(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        self.assertEqual("\N{SNOWMAN}", soup.b.decode_contents())
-
-    def test_encode_contents(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        self.assertEqual(
-            "\N{SNOWMAN}".encode("utf8"), soup.b.encode_contents(
-                encoding="utf8"))
-
-    def test_deprecated_renderContents(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        self.assertEqual(
-            "\N{SNOWMAN}".encode("utf8"), soup.b.renderContents())
-
-    def test_repr(self):
-        html = "\N{SNOWMAN}"
-        soup = self.soup(html)
-        if PY3K:
-            self.assertEqual(html, repr(soup))
-        else:
-            self.assertEqual(b'\\u2603', repr(soup))
-
-class TestNavigableStringSubclasses(SoupTest):
-
-    def test_cdata(self):
-        # None of the current builders turn CDATA sections into CData
-        # objects, but you can create them manually.
-        soup = self.soup("")
-        cdata = CData("foo")
-        soup.insert(1, cdata)
-        self.assertEqual(str(soup), "")
-        self.assertEqual(soup.find(text="foo"), "foo")
-        self.assertEqual(soup.contents[0], "foo")
-
-    def test_cdata_is_never_formatted(self):
-        """Text inside a CData object is passed into the formatter.
-
-        But the return value is ignored.
-        """
-
-        self.count = 0
-        def increment(*args):
-            self.count += 1
-            return "BITTER FAILURE"
-
-        soup = self.soup("")
-        cdata = CData("<><><>")
-        soup.insert(1, cdata)
-        self.assertEqual(
-            b"<><>]]>", soup.encode(formatter=increment))
-        self.assertEqual(1, self.count)
-
-    def test_doctype_ends_in_newline(self):
-        # Unlike other NavigableString subclasses, a DOCTYPE always ends
-        # in a newline.
-        doctype = Doctype("foo")
-        soup = self.soup("")
-        soup.insert(1, doctype)
-        self.assertEqual(soup.encode(), b"\n")
-
-    def test_declaration(self):
-        d = Declaration("foo")
-        self.assertEqual("", d.output_ready())
-
-class TestSoupSelector(TreeTest):
-
-    HTML = """
-
-
-
-The title
-
-
-
-Hello there.
-

-

-
An H1

-
Some text

-
Some more text

-
An H2

-
Another

-Bob
-
Another H2

-me
-
-span1a1
-span1a2 test
-
-span2a1
-
-
-
-

-
-

-
-
-
-
-
-
-
-
-
English

-
English UK

-
English US

-
French

-

-
-
-"""
-
-    def setUp(self):
-        self.soup = BeautifulSoup(self.HTML, 'html.parser')
-
-    def assertSelects(self, selector, expected_ids):
-        el_ids = [el['id'] for el in self.soup.select(selector)]
-        el_ids.sort()
-        expected_ids.sort()
-        self.assertEqual(expected_ids, el_ids,
-            "Selector %s, expected [%s], got [%s]" % (
-                selector, ', '.join(expected_ids), ', '.join(el_ids)
-            )
-        )
-
-    assertSelect = assertSelects
-
-    def assertSelectMultiple(self, *tests):
-        for selector, expected_ids in tests:
-            self.assertSelect(selector, expected_ids)
-
-    def test_one_tag_one(self):
-        els = self.soup.select('title')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].name, 'title')
-        self.assertEqual(els[0].contents, ['The title'])
-
-    def test_one_tag_many(self):
-        els = self.soup.select('div')
-        self.assertEqual(len(els), 4)
-        for div in els:
-            self.assertEqual(div.name, 'div')
-
-        el = self.soup.select_one('div')
-        self.assertEqual('main', el['id'])
-
-    def test_select_one_returns_none_if_no_match(self):
-        match = self.soup.select_one('nonexistenttag')
-        self.assertEqual(None, match)
-
-
-    def test_tag_in_tag_one(self):
-        els = self.soup.select('div div')
-        self.assertSelects('div div', ['inner', 'data1'])
-
-    def test_tag_in_tag_many(self):
-        for selector in ('html div', 'html body div', 'body div'):
-            self.assertSelects(selector, ['data1', 'main', 'inner', 'footer'])
-
-    def test_tag_no_match(self):
-        self.assertEqual(len(self.soup.select('del')), 0)
-
-    def test_invalid_tag(self):
-        self.assertRaises(ValueError, self.soup.select, 'tag%t')
-
-    def test_select_dashed_tag_ids(self):
-        self.assertSelects('custom-dashed-tag', ['dash1', 'dash2'])
-
-    def test_select_dashed_by_id(self):
-        dashed = self.soup.select('custom-dashed-tag[id=\"dash2\"]')
-        self.assertEqual(dashed[0].name, 'custom-dashed-tag')
-        self.assertEqual(dashed[0]['id'], 'dash2')
-
-    def test_dashed_tag_text(self):
-        self.assertEqual(self.soup.select('body > custom-dashed-tag')[0].text, 'Hello there.')
-
-    def test_select_dashed_matches_find_all(self):
-        self.assertEqual(self.soup.select('custom-dashed-tag'), self.soup.find_all('custom-dashed-tag'))
-
-    def test_header_tags(self):
-        self.assertSelectMultiple(
-            ('h1', ['header1']),
-            ('h2', ['header2', 'header3']),
-        )
-
-    def test_class_one(self):
-        for selector in ('.onep', 'p.onep', 'html p.onep'):
-            els = self.soup.select(selector)
-            self.assertEqual(len(els), 1)
-            self.assertEqual(els[0].name, 'p')
-            self.assertEqual(els[0]['class'], ['onep'])
-
-    def test_class_mismatched_tag(self):
-        els = self.soup.select('div.onep')
-        self.assertEqual(len(els), 0)
-
-    def test_one_id(self):
-        for selector in ('div#inner', '#inner', 'div div#inner'):
-            self.assertSelects(selector, ['inner'])
-
-    def test_bad_id(self):
-        els = self.soup.select('#doesnotexist')
-        self.assertEqual(len(els), 0)
-
-    def test_items_in_id(self):
-        els = self.soup.select('div#inner p')
-        self.assertEqual(len(els), 3)
-        for el in els:
-            self.assertEqual(el.name, 'p')
-        self.assertEqual(els[1]['class'], ['onep'])
-        self.assertFalse(els[0].has_attr('class'))
-
-    def test_a_bunch_of_emptys(self):
-        for selector in ('div#main del', 'div#main div.oops', 'div div#main'):
-            self.assertEqual(len(self.soup.select(selector)), 0)
-
-    def test_multi_class_support(self):
-        for selector in ('.class1', 'p.class1', '.class2', 'p.class2',
-            '.class3', 'p.class3', 'html p.class2', 'div#inner .class2'):
-            self.assertSelects(selector, ['pmulti'])
-
-    def test_multi_class_selection(self):
-        for selector in ('.class1.class3', '.class3.class2',
-                         '.class1.class2.class3'):
-            self.assertSelects(selector, ['pmulti'])
-
-    def test_child_selector(self):
-        self.assertSelects('.s1 > a', ['s1a1', 's1a2'])
-        self.assertSelects('.s1 > a span', ['s1a2s1'])
-
-    def test_child_selector_id(self):
-        self.assertSelects('.s1 > a#s1a2 span', ['s1a2s1'])
-
-    def test_attribute_equals(self):
-        self.assertSelectMultiple(
-            ('p[class="onep"]', ['p1']),
-            ('p[id="p1"]', ['p1']),
-            ('[class="onep"]', ['p1']),
-            ('[id="p1"]', ['p1']),
-            ('link[rel="stylesheet"]', ['l1']),
-            ('link[type="text/css"]', ['l1']),
-            ('link[href="blah.css"]', ['l1']),
-            ('link[href="no-blah.css"]', []),
-            ('[rel="stylesheet"]', ['l1']),
-            ('[type="text/css"]', ['l1']),
-            ('[href="blah.css"]', ['l1']),
-            ('[href="no-blah.css"]', []),
-            ('p[href="no-blah.css"]', []),
-            ('[href="no-blah.css"]', []),
-        )
-
-    def test_attribute_tilde(self):
-        self.assertSelectMultiple(
-            ('p[class~="class1"]', ['pmulti']),
-            ('p[class~="class2"]', ['pmulti']),
-            ('p[class~="class3"]', ['pmulti']),
-            ('[class~="class1"]', ['pmulti']),
-            ('[class~="class2"]', ['pmulti']),
-            ('[class~="class3"]', ['pmulti']),
-            ('a[rel~="friend"]', ['bob']),
-            ('a[rel~="met"]', ['bob']),
-            ('[rel~="friend"]', ['bob']),
-            ('[rel~="met"]', ['bob']),
-        )
-
-    def test_attribute_startswith(self):
-        self.assertSelectMultiple(
-            ('[rel^="style"]', ['l1']),
-            ('link[rel^="style"]', ['l1']),
-            ('notlink[rel^="notstyle"]', []),
-            ('[rel^="notstyle"]', []),
-            ('link[rel^="notstyle"]', []),
-            ('link[href^="bla"]', ['l1']),
-            ('a[href^="http://"]', ['bob', 'me']),
-            ('[href^="http://"]', ['bob', 'me']),
-            ('[id^="p"]', ['pmulti', 'p1']),
-            ('[id^="m"]', ['me', 'main']),
-            ('div[id^="m"]', ['main']),
-            ('a[id^="m"]', ['me']),
-            ('div[data-tag^="dashed"]', ['data1'])
-        )
-
-    def test_attribute_endswith(self):
-        self.assertSelectMultiple(
-            ('[href$=".css"]', ['l1']),
-            ('link[href$=".css"]', ['l1']),
-            ('link[id$="1"]', ['l1']),
-            ('[id$="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's2a1', 's1a2s1', 'dash1']),
-            ('div[id$="1"]', ['data1']),
-            ('[id$="noending"]', []),
-        )
-
-    def test_attribute_contains(self):
-        self.assertSelectMultiple(
-            # From test_attribute_startswith
-            ('[rel*="style"]', ['l1']),
-            ('link[rel*="style"]', ['l1']),
-            ('notlink[rel*="notstyle"]', []),
-            ('[rel*="notstyle"]', []),
-            ('link[rel*="notstyle"]', []),
-            ('link[href*="bla"]', ['l1']),
-            ('[href*="http://"]', ['bob', 'me']),
-            ('[id*="p"]', ['pmulti', 'p1']),
-            ('div[id*="m"]', ['main']),
-            ('a[id*="m"]', ['me']),
-            # From test_attribute_endswith
-            ('[href*=".css"]', ['l1']),
-            ('link[href*=".css"]', ['l1']),
-            ('link[id*="1"]', ['l1']),
-            ('[id*="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's1a2', 's2a1', 's1a2s1', 'dash1']),
-            ('div[id*="1"]', ['data1']),
-            ('[id*="noending"]', []),
-            # New for this test
-            ('[href*="."]', ['bob', 'me', 'l1']),
-            ('a[href*="."]', ['bob', 'me']),
-            ('link[href*="."]', ['l1']),
-            ('div[id*="n"]', ['main', 'inner']),
-            ('div[id*="nn"]', ['inner']),
-            ('div[data-tag*="edval"]', ['data1'])
-        )
-
-    def test_attribute_exact_or_hypen(self):
-        self.assertSelectMultiple(
-            ('p[lang|="en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']),
-            ('[lang|="en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']),
-            ('p[lang|="fr"]', ['lang-fr']),
-            ('p[lang|="gb"]', []),
-        )
-
-    def test_attribute_exists(self):
-        self.assertSelectMultiple(
-            ('[rel]', ['l1', 'bob', 'me']),
-            ('link[rel]', ['l1']),
-            ('a[rel]', ['bob', 'me']),
-            ('[lang]', ['lang-en', 'lang-en-gb', 'lang-en-us', 'lang-fr']),
-            ('p[class]', ['p1', 'pmulti']),
-            ('[blah]', []),
-            ('p[blah]', []),
-            ('div[data-tag]', ['data1'])
-        )
-
-    def test_unsupported_pseudoclass(self):
-        self.assertRaises(
-            NotImplementedError, self.soup.select, "a:no-such-pseudoclass")
-
-        self.assertRaises(
-            NotImplementedError, self.soup.select, "a:nth-of-type(a)")
-
-
-    def test_nth_of_type(self):
-        # Try to select first paragraph
-        els = self.soup.select('div#inner p:nth-of-type(1)')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].string, 'Some text')
-
-        # Try to select third paragraph
-        els = self.soup.select('div#inner p:nth-of-type(3)')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].string, 'Another')
-
-        # Try to select (non-existent!) fourth paragraph
-        els = self.soup.select('div#inner p:nth-of-type(4)')
-        self.assertEqual(len(els), 0)
-
-        # Pass in an invalid value.
-        self.assertRaises(
-            ValueError, self.soup.select, 'div p:nth-of-type(0)')
-
-    def test_nth_of_type_direct_descendant(self):
-        els = self.soup.select('div#inner > p:nth-of-type(1)')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].string, 'Some text')
-
-    def test_id_child_selector_nth_of_type(self):
-        self.assertSelects('#inner > p:nth-of-type(2)', ['p1'])
-
-    def test_select_on_element(self):
-        # Other tests operate on the tree; this operates on an element
-        # within the tree.
-        inner = self.soup.find("div", id="main")
-        selected = inner.select("div")
-        # The 
 tag was selected. The